Indian Sign Language Interpreter
Results of User Survey on ISL Interpreter Performance
An online user survey was conducted to study the efficacy of the virtual sign language interpreter that converts speech and textual inputs into ISL. Having addressed several of the issues and feedback that we received from the community, the team decided to conduct a user survey to study the understandability of the solution built so far in November 2020. The survey was on field for a month, from 20-November-2020 to 20-December-2020.
The survey provided encouraging results, with very high scores on three out of the four quantitative dimensions, including the dimension of Understandability (96.67% of words/ sentences were understood well by the user groups), Clarity (100% of users found the ISL words clear), and on familiarity (90% of words were correctly guessed by the users). However, the insights revealed that the solution needs to work on ISL grammar in sentences, with users providing detailed feedback on the grammar.
Category 1: a set of 10 signs which were labelled i.e., the sign names were provided. User then provided feedback on the animation.
Category 2: a set of 10 unlabelled signs. Users were asked to guess the given ISL word and provide feedback on ease of understanding.
Category 3: a set of 10 short phrases/sentences. The users were told what the given phrase/ sentence is and were requested to indicate if they were able to understand and if the ISL grammar was correct.
The team targeted to get feedback from at least 20 qualified users. This included deaf ISL users and ISL professionals. The survey was on field for a month, from 20-November-2020 to 20-December-2020. As on 21 December 2020, we received a total of 31 responses, out of which 26 were qualified respondents and 5 were from those who did not understand ISL, which were subsequently removed from the final analyses.
Objective and dimensions:
The survey was designed to gather insights at four dimensions, that is:
• Understandability: if the users understand the output presented by the avatar.
• Clarity: if the users find it easy to understand the words presented by the avatar.
• Familiarity: if the users can identify the words presented by the avatar.
• Grammar: if the output phrases were grammatically correct / understandable.
Survey Results and Insights:
- For this dimension, all words / sentences presented in the survey was taken in account for analyses.
- The solution scored extremely high on the dimension of Understandability, with 96.67% of the users indicating they understood the given words/ sentences by the digital avatar.
- On analysing the 3.3% which were not understood, we found that this was because the users were not familiar with a given sign or the sentences given in the question was a complex sentence.
- 100% of the users positively responded to this dimension and found the words presented by the avatar clear and understandable. This, yet again, is a strong indicator of liking towards the solution.
- This was evaluated based on the ISL words presented in category 2 of the questionnaire, where the users were able to guess the given ISL word. (Please note we accepted the responses as correct if the users provided a word/ phrase that is closest to the meaning of the given word).
- The solution scored well on the familiarity dimension as well, with 90% of the words being correctly guessed by most users.
This dimension was evaluated based on the sentences presented in category 3 of the questionnaire, where the users were presented a set of 10 short sentences.
- About 60% of sentences received the highest response. Several users found the sentences grammatically incorrect and provided detailed feedback on the correct order of signs. It was observed that the translation model worked well for simple sentences but struggled when it came to longer or complex sentences.
- Overall, the users feel that the prospective solution will be beneficial for them.
- The users are not happy with the translation model and have indicated that the order of words while converting from an English sentence into ISL sentence, is often incorrect. thereby diluting the meaning of the sentence.
- The users are responding well to the two newly created male and female avatars, named as Sanket and Mudra. At the same time, users provided several areas of improvement for the characters including changing the colours of Mudra’s clothes, re-validating signs, especially in those cases where there is more than one sign for a given word.
We would like to thank all the users who participated in this survey. The names are not listed for privacy purposes. In addition, we would like to acknowledge Anuj Jain Secretary – National Association of the Deaf, Neeraj Agarwal – ISL Expert, Sarika naik – EVP & CMO Capgemini India, Kasturi Sridhar – Independent Sign Language Interpreter for evangelize this survey with the users and encouraging them to take the survey.
Note: This is an abridged version of the report available on the Friends For Inclusion blog – https://friendsforinclusion.org/2021/01/07/findings-from-the-user-survey-to-understand-the-efficacy-of-virtual-interpreter-for-indian-sign-language-2/
Dr. Dinesh Babu J: Associate Professor – IIITB
Mr. Ankit Rajiv Jindal, Founder & Chairman – Friends For Inclusion
Mr. Mahesh R: 3D Animation & Software Developer – Friends For Inclusion
Mr. Shyam Krishna: MS-R Student – International Institute of Information technology
Mr. Rahul K: 3D Animation & Software Developer – Friends For Inclusion
Mr. Shubham Kumar: MS-R Student – International Institute of Information technology
Ms. Kasturi Shridhar: Independent Sign Language Interpreter
Mr. Nataraj R: Independent Sign Language Interpreter
Ms. Ranjini Ramanujam: Manager – Infosys Limited
Mr. Alok Bhaisare
Ms. Aswathy Alok
Special thanks to Friends For Inclusion (https://friendsforinclusion.org) for organizing outreach activities and collaboration with the HI and disabilities groups.