Deep Learning Based Sign Language Recognition

Abstract

Sign Language constitutes the primary means of communication for the deaf and hard-of-hearing. Sign Language Recognition is a complex task, which lies at the intersection of computer vision and language modeling. Manual and non-manual cues such as expression, hand shape, and body orientation occur in parallel and play a meaningful role in the articulation of the sign. In this thesis, we study this problem extensively by leveraging recent deep learning approaches. In the first section, we focus on 3D Hand and Body Pose estimation and report quantitative and qualitative results. In the second section, we explore the task of continuous sign language recognition and how expressive 3D skeleton and parameterizations of the human body can be exploited in conjunction with graph convolutions in order to effectively solve our task. We also compare our results with successful architectures, such as transformers and LSTM attention encoder-decoders. We report competitive performance on the Phoenix 2014-T dataset.