Indian Sign Language Interpreter
Overview:
Indian Sign Language (ISL), is the main sign language used in India, by anywhere between 1 million to 2.7 million people, most of them Hearing Impaired (HI). The number of certified interpreters of ISL, however, is in the low hundreds, indicating an acute shortage as compared to the HI population. Like other sign languages, it comes with distinctive features that make it very different from spoken languages. English comprehension among HI is low, making written content as well as closed captioning for videos not viable as a solution. Spoken and written content are, hence, very inaccessible to HI in India. Our project seeks to rectify that by automating the interpretation process, automatically generating translations of English content in ISL displayed using computer generated animation.
The interpretation pipeline consists of two major parts: the translation module which converts English to a textual representation of ISL, termed a ‘gloss’; followed by the animation module, which converts the ISL gloss to computer animation. The translation module depends on a rule based translation system. This is the only feasible option currently, given the lack of available data in ISL. The animation module has a vocabulary of 1300 signs with less non-manual features, and 300 more naturally animated signs with non-manual features like mouthing and head movement.
Data in sign languages in general is scarce, and in ISL in particular, the available data is very low. We have therefore had to generate some of the data required ourselves, consisting of English to ISL gloss parallel texts for translation, ISL performance videos of the corresponding sentences to aid in animation. The sentences for this parallel corpus were chosen to range from simpler structures to more complex and compound forms, based on language learning programs for English. There are currently around 200 of these sentence pairs in the corpus.
The project has always aimed at making the solution and the data generated onto an open source platform. To this end, we have created a GitHub page with the code available, as well as a demo on how to use the solution. A limited set of animations is also available directly on the page, with the rest of the animations made available on request. The translation corpus and performance videos are also available on request. We hope that this will help facilitate future research in the field, which is at a very new and unformed stage in India.
Expected Social impact:
Currently, for the estimated 10 to 20 lakh hearing impaired people in our country, the number of ISL interpreters are in the low hundreds. Spoken content is therefore inaccessible to them, be it lectures or meetings or speeches. We expect our solution to greatly help in levelling the field and making this content accessible to this section of the population, ensuring easier access to education and social interactions.
The female and male avatars that have been developed for the project
Publications
- Krishna, S., Jindal, A. R., & Jayagopi, D. Virtual Indian Sign Language Interpreter., In Proceedings of the 2020 4th International Conference on Vision, Image and Signal Processing (pp. 1-5)
- Krishna S, Vignesh V, Jayagopi. Sign Language Animation Through 3D Pose Lifting, accepted to Crossmodal Social Animation (XSAnim) workshop, In Proceedings of the 2021 International Conference on Computer Vision (ICCV), Montreal.
TEAM MEMBERS:
Dr. Dinesh Babu J: Associate Professor – IIITB
Mr. Shyam Krishna: MS-R Student –IIITB
Ms. Aswathy Alok
Special thanks to Friends For Inclusion (https://friendsforinclusion.org) for organizing outreach activities and collaboration with the HI and disabilities groups.
