Context-Aware Emotion Recognition in the Wild Using Spatio-Temporal and Temporal-Pyramid Models release_h5roj3vztvbtharh2chqycsfzi

by Tai Do Nhu, Soo Hyung Kim, Hyung-Jeong Yang, Guee-Sang Lee, Soonja Yeom

Published in Sensors by MDPI AG.

2021   Volume 21, Issue 7, p2344

Abstract

Emotion recognition plays an important role in human–computer interactions. Recent studies have focused on video emotion recognition in the wild and have run into difficulties related to occlusion, illumination, complex behavior over time, and auditory cues. State-of-the-art methods use multiple modalities, such as frame-level, spatiotemporal, and audio approaches. However, such methods have difficulties in exploiting long-term dependencies in temporal information, capturing contextual information, and integrating multi-modal information. In this paper, we introduce a multi-modal flexible system for video-based emotion recognition in the wild. Our system tracks and votes on significant faces corresponding to persons of interest in a video to classify seven basic emotions. The key contribution of this study is that it proposes the use of face feature extraction with context-aware and statistical information for emotion recognition. We also build two model architectures to effectively exploit long-term dependencies in temporal information with a temporal-pyramid model and a spatiotemporal model with "Conv2D+LSTM+3DCNN+Classify" architecture. Finally, we propose the best selection ensemble to improve the accuracy of multi-modal fusion. The best selection ensemble selects the best combination from spatiotemporal and temporal-pyramid models to achieve the best accuracy for classifying the seven basic emotions. In our experiment, we take benchmark measurement on the AFEW dataset with high accuracy.
In application/xml+jats format

Archived Files and Locations

application/pdf  2.3 MB
file_d5bd2ua7qnhntkj6r6l57lf7su
res.mdpi.com (publisher)
web.archive.org (webarchive)
application/pdf  2.3 MB
file_kkkbndjeandztf7ipotnyngeui
mdpi-res.com (web)
web.archive.org (webarchive)
Read Archived PDF
Preserved and Accessible
Type  article-journal
Stage   published
Date   2021-03-27
Language   en ?
DOI  10.3390/s21072344
PubMed  33801739
PMC  PMC8036494
Container Metadata
Open Access Publication
In DOAJ
In ISSN ROAD
In Keepers Registry
ISSN-L:  1424-8220
Work Entity
access all versions, variants, and formats of this works (eg, pre-prints)
Catalog Record
Revision: e838171d-e3c6-400d-b6eb-a751aedb5c82
API URL: JSON