Exploring automatic text-to-sign translation in a healthcare setting

L Esselink, F Roelofsen, J Dotlacil, S Mende-Gillings, M de Meulder, N Sijm, A Smeijers

Research output: Contribution to journalArticleAcademicpeer-review

Abstract

Communication between healthcare professionals and deaf patients has been particularly challenging during the COVID-19 pandemic. We have explored the possibility to automatically translate phrases that are frequently used in the diagnosis and treatment of hospital patients, in particular phrases related to COVID-19, from Dutch or English to Dutch Sign Language (NGT). The prototype system we developed displays translations either by means of pre-recorded videos featuring a deaf human signer (for a limited number of sentences) or by means of animations featuring a computer-generated signing avatar (for a larger, though still restricted number of sentences). We evaluated the comprehensibility of the signing avatar, as compared to the human signer. We found that, while individual signs are recognized correctly when signed by the avatar almost as frequently as when signed by a human, sentence comprehension rates and clarity scores for the avatar are substantially lower than for the human signer. We identify a number of concrete limitations of the JASigning avatar engine that underlies our system. Namely, the engine currently does not offer sufficient control over mouth shapes, the relative speed and intensity of signs in a sentence (prosody), and transitions between signs. These limitations need to be overcome in future work for the engine to become usable in practice.
Original languageEnglish
Pages (from-to)35-57
Number of pages23
JournalUniversal Access in the Information Society
Volume23
Issue number1
Early online date29 Sept 2023
DOIs
Publication statusPublished - Mar 2024

Bibliographical note

Publisher Copyright:
© The Author(s) 2023.

Funding

We gratefully acknowledge financial support from the Netherlands Organization for Innovation in Healthcare (ZonMw, Grant No. 10430042010027), the Netherlands Organization for Scientific Research (NWO, Grant No. VI.C.201.014), and the European Research Council (Grant No. 680220). Since the time-frame and available resources for this research project were really quite different than for prototypical academic projects, we provide some details. Initial funding for the project was provided by an ad hoc funding scheme setup by the Netherlands Organization for Innovation in Healthcare (ZonMW) to address urgent COVID-related issues in the healthcare sector. The deadline for proposals in this funding scheme was two weeks after the call for proposals had been announced, funded projects had to start one month later, and had to be completed within six months, with a total budget of 25.000 euros. In this period, we designed and implemented the prototype system. Separate funding was used for the evaluation study.

FundersFunder number
Netherlands Organization for Innovation in Healthcare
European Research Council680220
ZonMw10430042010027
Nederlandse Organisatie voor Wetenschappelijk OnderzoekVI.C.201.014

    Keywords

    • Access to healthcare information
    • Avatar technology
    • Sign language
    • User study

    Fingerprint

    Dive into the research topics of 'Exploring automatic text-to-sign translation in a healthcare setting'. Together they form a unique fingerprint.

    Cite this