Indian Sign Language Interpreter

Results of User Survey on ISL Interpreter Performance

An online user survey was conducted to study the efficacy of the virtual sign language interpreter that converts speech and textual inputs into ISL. Having addressed several of the issues and feedback that we received from the community, the team decided to conduct a user survey to study the understandability of the solution built so far in November 2020. The survey was on field for a month, from 20-November-2020 to 20-December-2020.
The survey provided encouraging results, with very high scores on three out of the four quantitative dimensions, including the dimension of Understandability (96.67% of words/ sentences were understood well by the user groups), Clarity (100% of users found the ISL words clear), and on familiarity (90% of words were correctly guessed by the users). However, the insights revealed that the solution needs to work on ISL grammar in sentences, with users providing detailed feedback on the grammar.

Question categories:

Category 1: a set of 10 signs which were labelled i.e., the sign names were provided. User then provided feedback on the animation.
Category 2: a set of 10 unlabelled signs. Users were asked to guess the given ISL word and provide feedback on ease of understanding.
Category 3: a set of 10 short phrases/sentences. The users were told what the given phrase/ sentence is and were requested to indicate if they were able to understand and if the ISL grammar was correct.

Field work:

The team targeted to get feedback from at least 20 qualified users. This included deaf ISL users and ISL professionals. The survey was on field for a month, from 20-November-2020 to 20-December-2020. As on 21 December 2020, we received a total of 31 responses, out of which 26 were qualified respondents and 5 were from those who did not understand ISL, which were subsequently removed from the final analyses.

Objective and dimensions:

The survey was designed to gather insights at four dimensions, that is:
• Understandability: if the users understand the output presented by the avatar.
• Clarity: if the users find it easy to understand the words presented by the avatar.
• Familiarity: if the users can identify the words presented by the avatar.
• Grammar: if the output phrases were grammatically correct / understandable.

Clarity

• 100% of the users positively responded to this dimension and found the words presented by the avatar clear and understandable. This, yet again, is a strong indicator of liking towards the solution.

Grammar

• This dimension was evaluated based on the sentences presented in category 3 of the questionnaire, where the users were presented a set of 10 short sentences.

• About 60% of sentences received the highest response. Several users found the sentences grammatically incorrect and provided detailed feedback on the correct order of signs. It was observed that the translation model worked well for simple sentences but struggled when it came to longer or complex sentences.

Conclusion

• Overall, the users feel that the prospective solution will be beneficial for them.
• The users are not happy with the translation model and have indicated that the order of words while converting from an English sentence into ISL sentence, is often incorrect. thereby diluting the meaning of the sentence.
• The users are responding well to the two newly created male and female avatars, named as Sanket and Mudra. At the same time, users provided several areas of improvement for the characters including changing the colours of Mudra’s clothes, re-validating signs, especially in those cases where there is more than one sign for a given word.

Acknowledgements:

We would like to thank all the users who participated in this survey. The names are not listed for privacy purposes. In addition, we would like to acknowledge Anuj Jain Secretary – National Association of the Deaf, Neeraj Agarwal – ISL Expert, Sarika naik – EVP & CMO Capgemini India, Kasturi Sridhar – Independent Sign Language Interpreter for evangelize this survey with the users and encouraging them to take the survey.

Note: This is an abridged version of the report available on the Friends For Inclusion blog – https://friendsforinclusion.org/2021/01/07/findings-from-the-user-survey-to-understand-the-efficacy-of-virtual-interpreter-for-indian-sign-language-2/