Portuguese Sign Language Recognition from Depth Sensing Human Gesture and Motion Capture

Student: Carlos Sotelo
Supervisor: Carlos Silva (DEI) e Miguel Dias (Microsoft)

Abstract

Just like spoken languages, Sign Languages (SL) have evolved over time, featuring their own grammar and vocabulary, and thus, they are considered real languages. The major difference between SL and other languages is that the first one is signed and the second one is spoken, meaning that SL is a visual language. SL is the most common type of language among deaf people as no sense of hearing is required to understand it.
The main motivation of this dissertation is to build a bridge to ease the communication between those who are deaf (and hard-of-hearing) and those not familiarized with SL. We propose a system whose main feature is the absence of intrusion, discarding the usage of glove like devices or a setup with multiple cameras. We achieved this using the Kinect One sensor from Microsoft. Using a single device, we can acquire both depth and colour information, yet this system makes usage only on the depth information.
Four experimental situations have been performed: simple posture recognition, movement postures recognition, sign recognition using only hand path information, and sign recognition using hand path and hand configuration information. The first and third experimental classes were conducted, in order to confirm the feature extraction method’s eligibility while the second and fourth experiments were conducted to address our hypothesis. Accuracy rates reached 87.4% and 64.2% for the first and second experiments, respectively. In the experiments concerning signs, accuracy rates of 91.6% for hand path data only, and 81.3% for hand path and hand configuration data were achieved.

Thesis Download (PDF)