Machine Translation from Signed to Spoken Languages: State of the Art and Challenges
release_o6aatxyuhfaxze4zzupomrfc5u
by
Mathieu De Coster, Dimitar Shterionov, Mieke Van Herreweghe, Joni Dambre
2022
Abstract
Automatic translation from signed to spoken languages is an interdisciplinary
research domain, lying on the intersection of computer vision, machine
translation and linguistics. Nevertheless, research in this domain is performed
mostly by computer scientists in isolation. As the domain is becoming
increasingly popular - the majority of scientific papers on the topic of sign
language translation have been published in the past three years - we provide
an overview of the state of the art as well as some required background in the
different related disciplines. We give a high-level introduction to sign
language linguistics and machine translation to illustrate the requirements of
automatic sign language translation. We present a systematic literature review
to illustrate the state of the art in the domain and then, harking back to the
requirements, lay out several challenges for future research. We find that
significant advances have been made on the shoulders of spoken language machine
translation research. However, current approaches are often not linguistically
motivated or are not adapted to the different input modality of sign languages.
We explore challenges related to the representation of sign language data, the
collection of datasets, the need for interdisciplinary research and
requirements for moving beyond research, towards applications. Based on our
findings, we advocate for interdisciplinary research and to base future
research on linguistic analysis of sign languages. Furthermore, the inclusion
of deaf and hearing end users of sign language translation applications in use
case identification, data collection and evaluation is of the utmost importance
in the creation of useful sign language translation models. We recommend
iterative, human-in-the-loop, design and development of sign language
translation models.
In text/plain
format
Archived Content
There are no accessible files associated with this release. You could check other releases for this work for an accessible version.
Know of a fulltext copy of on the public web? Submit a URL and we will archive it
2202.03086v2
access all versions, variants, and formats of this works (eg, pre-prints)