An End-to-End Visual-Audio Attention Network for Emotion Recognition in
User-Generated Videos
release_ckcbrackkzfeditpjdfnoj3wqq
by
Sicheng Zhao, Yunsheng Ma, Yang Gu, Jufeng Yang, Tengfei Xing, Pengfei
Xu, Runbo Hu, Hua Chai, Kurt Keutzer
2020
Abstract
Emotion recognition in user-generated videos plays an important role in
human-centered computing. Existing methods mainly employ traditional two-stage
shallow pipeline, i.e. extracting visual and/or audio features and training
classifiers. In this paper, we propose to recognize video emotions in an
end-to-end manner based on convolutional neural networks (CNNs). Specifically,
we develop a deep Visual-Audio Attention Network (VAANet), a novel architecture
that integrates spatial, channel-wise, and temporal attentions into a visual 3D
CNN and temporal attentions into an audio 2D CNN. Further, we design a special
classification loss, i.e. polarity-consistent cross-entropy loss, based on the
polarity-emotion hierarchy constraint to guide the attention generation.
Extensive experiments conducted on the challenging VideoEmotion-8 and Ekman-6
datasets demonstrate that the proposed VAANet outperforms the state-of-the-art
approaches for video emotion recognition. Our source code is released at:
https://github.com/maysonma/VAANet.
In text/plain
format
Archived Files and Locations
application/pdf 3.9 MB
file_bdhom73r6ffqrkry77q52zarma
|
arxiv.org (repository) web.archive.org (webarchive) |
2003.00832v1
access all versions, variants, and formats of this works (eg, pre-prints)