Exploring automatic text‑to‑sign translation in a healthcare setting

Authors Lyke Esselink, Floris Roelofsen, Jakub Dotlačil, Shane Mende-Gillings, Maartje de Meulder, Nienke Sijm, Anika Smeijers
Published in Universal Access in the Information Society
Publication date 2023
Research groups Speech and Language Therapy: Participation through communication
Type Article


Communication between healthcare professionals and deaf patients has been particularly challenging during the COVID-19 pandemic. We have explored the possibility to automatically translate phrases that are frequently used in the diagnosis and treatment of hospital patients, in particular phrases related to COVID-19, from Dutch or English to Dutch Sign Language (NGT). The prototype system we developed displays translations either by means of pre-recorded videos featuring a deaf human signer (for a limited number of sentences) or by means of animations featuring a computer-generated signing avatar (for a larger, though still restricted number of sentences). We evaluated the comprehensibility of the signing avatar, as compared to the human signer. We found that, while individual signs are recognized correctly when signed by the avatar almost as frequently as when signed by a human, sentence comprehension rates and clarity scores for the avatar are substantially lower than for the human signer. We identify a number of concrete limitations of the JASigning avatar engine that underlies our system. Namely, the engine currently does not offer sufficient control over mouth shapes, the relative speed and intensity of signs in a sentence (prosody), and transitions between signs. These limitations need to be overcome in future work for the engine to become usable in practice.

Language English
Published in Universal Access in the Information Society
Key words access to healthcare information, sign language, avatar technology, user study
Digital Object Identifier 10.1007/s10209-023-01042-6

Speech and Language Therapy: Participation through communication