Signing as Input for a Dictionary Query: Matching Signs Based on Joint Positions of the Dominant Hand

Manolis Fragkiadakis, Victoria Nyst, Peter van der Putten


Abstract
This study presents a new methodology to search sign language lexica, using a full sign as input for a query. Thus, a dictionary user can look up information about a sign by signing the sign to a webcam. The recorded sign is then compared to potential matching signs in the lexicon. As such, it provides a new way of searching sign language dictionaries to complement existing methods based on (spoken language) glosses or phonological features, like handshape or location. The method utilizes OpenPose to extract the body and finger joint positions. Dynamic Time Warping (DTW) is used to quantify the variation of the trajectory of the dominant hand and the average trajectories of the fingers. Ten people with various degrees of sign language proficiency have participated in this study. Each subject viewed a set of 20 signs from the newly compiled Ghanaian sign language lexicon and was asked to replicate the signs. The results show that DTW can predict the matching sign with 87% and 74% accuracy at the Top-10 and Top-5 ranking level respectively by using only the trajectory of the dominant hand. Additionally, more proficient signers obtain 90% accuracy at the Top-10 ranking. The methodology has the potential to be used also as a variation measurement tool to quantify the difference in signing between different signers or sign languages in general.
Anthology ID:
2020.signlang-1.11
Volume:
Proceedings of the LREC2020 9th Workshop on the Representation and Processing of Sign Languages: Sign Language Resources in the Service of the Language Community, Technological Challenges and Application Perspectives
Month:
May
Year:
2020
Address:
Marseille, France
Editors:
Eleni Efthimiou, Stavroula-Evita Fotinea, Thomas Hanke, Julie A. Hochgesang, Jette Kristoffersen, Johanna Mesch
Venue:
SignLang
SIG:
Publisher:
European Language Resources Association (ELRA)
Note:
Pages:
69–74
Language:
English
URL:
https://aclanthology.org/2020.signlang-1.11
DOI:
Bibkey:
Cite (ACL):
Manolis Fragkiadakis, Victoria Nyst, and Peter van der Putten. 2020. Signing as Input for a Dictionary Query: Matching Signs Based on Joint Positions of the Dominant Hand. In Proceedings of the LREC2020 9th Workshop on the Representation and Processing of Sign Languages: Sign Language Resources in the Service of the Language Community, Technological Challenges and Application Perspectives, pages 69–74, Marseille, France. European Language Resources Association (ELRA).
Cite (Informal):
Signing as Input for a Dictionary Query: Matching Signs Based on Joint Positions of the Dominant Hand (Fragkiadakis et al., SignLang 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.signlang-1.11.pdf