Deep Learning-Based Sign Language Recognition from Videos and Cross-Lingual Translation to Indian Vernaculars

A new two-stage deep learning pipeline for sign language recognition and translation has been developed, utilizing a fine-tuned VideoMAE video transformer for classifying Indian sign language videos into English labels, followed by translation into Hindi, Telugu, and Bengali using Meta AI's NLLB-200 model. The classification model achieved 99% training accuracy and 78% validation accuracy on a 13-class subset of the AI4Bharat Indian Sign Language corpus, processing 16-frame clips at 224 x 224 resolution. This work addresses the lack of automated tools for low-resource Indian languages, highlighting significant implications for accessibility and communication within the deaf and hard-of-hearing community while acknowledging its limitations and future development needs.