Sign language recognition with multi feature fusion and ANN classifier

Abstract

Extracting and recognizing complex human movements such as sign language gestures from video sequencesis a challenging task. In this paper this kind of a difficult problem is approached with Indian sign language (ISL) videos.A new segmentation algorithm is developed by fusion of features from discrete wavelet transform (DWT) and local binarypattern (LBP). A 2D point cloud is formed from fused features, which represent the local hand shapes in consecutivevideo frames. We validate the proposed feature extraction model with state of the art features such as HOG, SIFTand SURF for each sign video on the same ANN classifier. We found that the Haar-LBP fused features represent signvideo data in better manner compared to HOG, SIFT and SURF. This is due to the combination of global and localfeatures in our proposed feature matrix. The extracted features input the artificial neural network (ANN) classifier withlabels forming the corresponding words. The proposed ANN classifier is tested against state of the art classifiers such asAdaboost, support vector machine (SVM) and other ANN methods on different features extracted from the ISL dataset.The classifiers were tested for accuracy and correctness in identifying the signs. The ANN classifier that produced arecognition rate of 92.79% was obtained with maximum training instances, which was far greater than the existing workson sign language with other features and ANN classifier on our ISL dataset