Learning multiview deep features from skeletal sign language videos for recognition

Abstract

The most challenging objective in machine translation of sign language has been the machine’s inability to learn interoccluding finger movements during an action process. This work addresses the problem of teaching a deep learning model to recognize differently oriented skeletal data. The multi-view 2D skeletal sign language video data is obtained using 3D motion-captured system. A total of 9 signer views were used for training the proposed network and the 6 for testing and validation. In order to obtain multi-view deep features for recognition, we proposed an end-to-end trainable multistream convolutional neural network (CNN) with late feature fusion. The fused multiview features are then inputted to a two-layer dense and a decision making softmax. The proposed CNN employs numerous layers to characterize view correspondence to generate maximally discriminative features. This study is important to understand the effects of multiview data processing by CNNs for sign language recognition in decoding joint spatial information. Further, deeper perspectives were developed into multiview processing of CNNs by applying skeletal action data