About Face: Seeing the Talker Improves Spoken Word Recognition but Increases Listening Effort release_agvmhhy3n5fo7opc42mlkv6y2m

by Violet A. Brown, Julia F. Strand

Published in Journal of Cognition by Ubiquity Press, Ltd..

2019   Volume 2, Issue 1, p44

Abstract

It is widely accepted that seeing a talker improves a listener's ability to understand what a talker is saying in background noise (e.g., Erber, 1969; Sumby & Pollack, 1954). The literature is mixed, however, regarding the influence of the visual modality on the listening effort required to recognize speech (e.g., Fraser, Gagné, Alepins, & Dubois, 2010; Sommers & Phelps, 2016). Here, we present data showing that even when the visual modality robustly benefits recognition, processing audiovisual speech can still result in greater cognitive load than processing speech in the auditory modality alone. We show using a dual-task paradigm that the costs associated with audiovisual speech processing are more pronounced in easy listening conditions, in which speech can be recognized at high rates in the auditory modality alone-indeed, effort did not differ between audiovisual and audio-only conditions when the background noise was presented at a more difficult level. Further, we show that though these effects replicate with different stimuli and participants, they do not emerge when effort is assessed with a recall paradigm rather than a dual-task paradigm. Together, these results suggest that the widely cited audiovisual recognition benefit may come at a cost under more favorable listening conditions, and add to the growing body of research suggesting that various measures of effort may not be tapping into the same underlying construct (Strand et al., 2018).
In text/plain format

Archived Files and Locations

application/pdf  2.3 MB
file_3hhqcfehdzf3hfr4aqvx4pfjiy
www.journalofcognition.org (web)
web.archive.org (webarchive)
Read Archived PDF
Preserved and Accessible
Type  article-journal
Stage   published
Date   2019-11-22
Language   en ?
DOI  10.5334/joc.89
PubMed  31807726
PMC  PMC6873894
Journal Metadata
Open Access Publication
In DOAJ
In Keepers Registry
ISSN-L:  2514-4820
Work Entity
access all versions, variants, and formats of this works (eg, pre-prints)
Catalog Record
Revision: a462c94e-4046-4d12-836a-0965eb256617
API URL: JSON