Lifelogging caption generation via fourth-person vision in a human–robot symbiotic environment release_fw5ndj6d7fg6dpnx55ugupck64

by Kazuto Nakashima, Yumi Iwashita, Ryo Kurazume

Published in ROBOMECH Journal by Springer Science and Business Media LLC.

2020  

Abstract

<jats:title>Abstract</jats:title> Automatic analysis of our daily lives and activities through a first-person lifelog camera provides us with opportunities to improve our life rhythms or to support our limited visual memories. Notably, to express the visual experiences, the task of generating captions from first-person lifelog images has been actively studied in recent years. First-person images involve scenes approximating what users actually see; therein, the visual cues are not enough to express the user's context since the images are limited by his/her intention. Our challenge is to generate lifelog captions using a meta-perspective called "fourth-person vision". The "fourth-person vision" is a novel concept which complementary exploits the visual information from the first-, second-, and third-person perspectives. First, we assume human–robot symbiotic scenarios that provide a second-person perspective from the camera mounted on the robot and a third-person perspective from the camera fixed in the symbiotic room. To validate our approach in this scenario, we collect perspective-aware lifelog videos and corresponding caption annotations. Subsequently, we propose a multi-perspective image captioning model composed of an image-wise salient region encoder, an attention module that adaptively fuses the salient regions, and a caption decoder that generates scene descriptions. We demonstrate that our proposed model based on the fourth-person concept can greatly improve the captioning performance against single- and double-perspective models.
In application/xml+jats format

Archived Files and Locations

application/pdf  2.6 MB
file_uwzt33h3jngahi4wlntqpxcngi
robomechjournal.springeropen.com (publisher)
web.archive.org (webarchive)
Read Archived PDF
Preserved and Accessible
Type  article-journal
Stage   published
Date   2020-09-24
Language   en ?
Container Metadata
Open Access Publication
In DOAJ
In ISSN ROAD
In Keepers Registry
ISSN-L:  2197-4225
Work Entity
access all versions, variants, and formats of this works (eg, pre-prints)
Catalog Record
Revision: 3621cee0-cd96-4718-b972-b3d7f3bf2a43
API URL: JSON