Keypoint detection in an image for the communication system of sign language speakers

Author: Guram Chaganava
Keywords: Sign language, keypoint detection, Dataset, Transfer learning
Annotation:

The communication system will enable sign language speakers to establish continuous communication. The speaker will see a three-dimensional avatar on the screen that will repeat the signs performed by the interlocutor. One of the main parts of the communication system is to get some kind of information about signs from the image. Based on the conducted studies, keypoint detection was selected as a relatively optimal method. Testing of ready-made detectors has shown that their use for the communication system would be ineffective. So, we decided to create a detector relevant to the objectives of the task. For this, we built the dataset, which contained photos showing one signing person standing in front of the camera. Using this dataset, we have performed transfer learning on deep learning model Mobilenet. It can be concluded that the accuracy and speed of the trained detector are sufficient to establish continuous communication.