Efficient Kinect Sensor-based Kurdish Sign Language Recognition Using Echo System Network
Sign language assists in building communication and bridging gaps in understanding. Automatic sign language recognition (ASLR) is a field that has recently been studied for various sign languages. However, Kurdish sign language (KuSL) is relatively new and therefore researches and designed datasets on it are limited. This paper has proposed a model to translate KuSL into text and has designed a dataset using Kinect V2 sensor. The computation complexity of feature extraction and classification steps, which are serious problems for ASLR, has been investigated in this paper. The paper proposed a feature engineering approach on the skeleton position alone to provide a better representation of the features and avoid the use of all of the image information. In addition, the paper proposed model makes use of recurrent neural networks (RNNs)-based models. Training RNNs is inherently difficult, and consequently, motivates to investigate alternatives. Besides the trainable long short-term memory (LSTM), this study has proposed the untrained low complexity echo system network (ESN) classifier. The accuracy of both LSTM and ESN indicates they can outperform those in state-of-the-art studies. In addition, ESN which has not been proposed thus far for ASLT exhibits comparable accuracy to the LSTM with a significantly lower training time.
Abdul, Z.K., Al-Talabani, A.K. and Ramadan, D.O., 2020. A hybrid temporal feature for gear fault diagnosis using the long short term memory. IEEE Sensors Journal, 20(23), pp.14444-14452.
Almasre, M.A. and Al-Nuaim, H., 2016. Areal-time letter recognition model for arabic sign language using Kinect and leap motion controller v2. International Journal of Advanced Engineering, Management and Science, 2(5), p.239469.
Al-Talabani, A., Sellahewa, H. and Jassim, S.A., 2015. Emotion recognition from speech: Tools and challenges. Mobile multimedia/image processing, security, and applications. International Society for Optics and Photonics, 9497, p.94970N.
Awata, S., Sako, S. and Kitamura, T., 2017. Japanese sign language recognition based on three elements of sign using kinect v2 sensor. In: International Conference on Human-computer Interaction. Springer, Berlin, Germany, pp.95-102.
Bianchi, F.M., Livi, L. and Alippi, C., 2016. Investigating echo-state networks dynamics by means of recurrence analysis. IEEE Transactions on Neural Networks and Learning Systems, 29(2), pp.427-439.
Bianchi, F.M., Scardapane, S., Løkse, S. and Jenssen, R., 2020. Reservoir computing approaches for representation and classification of multivariate time series. IEEE Transactions on Neural Networks and Learning Systems, 23(5), pp.2169-2179.
Capilla, D.M., 2012. Sign Language Translator Using Microsoft Kinect Xbox 360 tm. Department of Electrical Engineering and Computer Science, University of Tennessee, Tennessee.
Čerňanský, M. and Tiňo, P., 2007. Comparison of echo state networks with simple recurrent networks and variable-length Markov models on symbolic sequences. In: International Conference on Artificial Neural Networks. Springer, Berlin, Germany, pp.618-627.
Chai, X., Li, G., Lin, Y., Xu, Z., Tang, Y., Chen, X. and Zhou, M., 2013. Sign language recognition and translation with kinect. IEEE Conference on Automatic, 655, p.4.
Chen, Q., Georganas, N.D. and Petriu, E.M., 2008. Hand gesture recognition using Haar-like features and a stochastic context-free grammar. IEEE Transactions on Instrumentation and Measurement, 57(8), pp.1562-1571.
Dos Santos Anjo, M., Pizzolato, E.B. and Feuerstack, S., 2012. A Real-time System to Recognize Static Gestures of Brazilian Sign Language (Libras) Alphabet Using Kinect. IHC, Citeseer, pp.259-268.
El-Bendary, N., Zawbaa, H.M., Daoud, M.S., Hassanien, A.E. and Nakamatsu, K., 2010. Arslat: Arabic sign language alphabets translator. In: International Conference on Computer Information Systems and Industrial Management Applications. IEEE, United States, pp.590-595.
Gao, L., Li, H., Liu, Z., Liu, Z., Wan, L. and Feng, W., 2021. RNN-transducer based Chinese sign language recognition. Neurocomputing, 434, pp.45-54.
Gilorkar, N.K. and Ingle, M.M., 2014. A review on feature extraction for Indian and American sign language. International Journal of Computer Science and Information Technologies, 5(1), pp.314-318.
Hashim, A.D. and Alizadeh, F., 2018. Kurdish sign language recognition system. UKH Journal of Science and Engineering, 2(1), pp.1-6.
Hossein, M.J. and Ejaz, M.S., 2020. Recognition of Bengali sign language using novel deep convolutional neural network. In: 2nd International Conference on Sustainable Technologies for Industry 4.0 (STI), 2020. IEEE, United States, pp.1-5.
Karami, A., Zanj, B. and Sarkaleh, A.K., 2011. Persian sign language (PSL) recognition using wavelet transform and neural networks. Expert Systems with Applications, 38(3), pp.2661-2667.
Katilmis, Z. and Karakuzu, C., 2021. ELM based two-handed dynamic turkish sign language (TSL) word recognition. Expert Systems with Applications, 2021, pp.115213.
Kratimenos, A., Pavlakos, G. and Maragos, P., 2021. Independent sign language recognition with 3d body, hands, and face reconstruction. In: ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, United States, pp.4270-4274.
Kumar, P., Saini, R., Roy, P.P. and Dogra, D.P., 2018. A position and rotation invariant framework for sign language recognition (SLR) using Kinect. Multimedia Tools and Applications, 77(7), pp.8823-8846.
Lang, S., Block, M. and Rojas, R., 2012. Sign language recognition using kinect. In: International Conference on Artificial Intelligence and Soft Computing. Springer, Berlin, Germany, pp.394-402.
Lee, C.K., Ng, K.K., Chen, C.H., Lau, H.C., Chung, S. and Tsoi, T., 2021. American sign language recognition and training method with recurrent neural network. Expert Systems with Applications, 167, pp.114403.
Lee, G.C., Yeh, F.H. and Hsiao, Y.H., 2016. Kinect-based Taiwanese signlanguage recognition system. Multimedia Tools and Applications, 75(1), pp.261-279.
Li, D., Rodriguez, C., Yu, X. and Li, H., 2020. Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. WACV, United States, pp.1459-1469.
Li, X., He, M., Li, H. and Shen, H., 2021. A Combined Loss-based Multiscale Fully Convolutional Network for High-resolution Remote Sensing Image Change Detection. IEEE Geoscience and Remote Sensing Letters, United States.
Li, X., Mao, C., Huang, S. and Ye, Z., 2017. Chinese sign language recognition based on shs descriptor and encoder-decoder lstm model. In: Chinese Conference on Biometric Recognition. Springer, United States, pp.719-728.
Li, Y., 2012. Hand gesture recognition using Kinect. In: 2012 IEEE International Conference on Computer Science and Automation Engineering. IEEE, United States, pp.196-199.
Liu, T., Zhou, W. and Li, H., 2016. Sign language recognition with long shortterm memory. In: 2016 IEEE International Conference on Image Processing (ICIP). IEEE, United States, pp.2871-2875.
Livi, L., Bianchi, F.M. and Alippi, C., 2017. Determination of the edge of criticality in echo state networks through Fisher information maximization. IEEE Transactions on Neural Networks and Learning Systems, 29(3), pp.706-717.
Liwicki, S. and Everingham, M., 2009. Automatic recognition of fingerspelled words in british sign language. In: 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. IEEE, United States, pp.50-57.
Lukoševičius, M., 2012. A practical guide to applying echo state networks. In: Neural Networks: Tricks of the Trade. Springer, Berlin, Germany.
Maass, W., Natschläger, T. and Markram, H., 2002. Real-time computing without stable states: A new framework for neural computation based on perturbations. Neural Computation, 14(11), pp.2531-2560.
Mahmood, M.R., Abdulazeez, A.M. and Orman, Z., 2018. Dynamic hand gesture recognition system for kurdish sign language using two lines of features. In: 2018 International Conference on Advanced Science and Engineering (ICOASE). IEEE, United States, pp.42-47.
Mittal, A., Kumar, P., Roy, P.P., Balasubramanian, R. and Chaudhuri, B.B., 2019. A modified LSTM model for continuous sign language recognition using leap motion. IEEE Sensors Journal, 19(16), pp.7056-7063.
Pandita, S. and Narote, S., 2013. Hand gesture recognition using SIFT. Proceedings of the IEEE International Conference on Computational Intelligence and Security, 2(1), p.4.
Prabakaran, D. and Shyamala, R., 2019. Areview on performance of voice feature extraction techniques. In: 2019 3rd International Conference on Computing and Communications Technologies (ICCCT). IEEE, United States, p.221-231.
Prasad, M., Kishore, P., Kumar, E.K. and Kumar, D.A., 2016. Indian sign language recognition system using new fusion based edge operator. Journal of Theoretical and Applied Information Technology, 88(3), p.574-583.
Preeti Amatya, K.S. and Meixner, G., 2018. Translation of sign language into text using kinect for windows v2. In: The Eleventh International Conference on Advances in Computer-human Interactions. ACHI, United States.
Rastgoo, R., Kiani, K. and Escalera, S., 2020. Hand sign language recognition using multi-view hand skeleton. Expert Systems with Applications, 150, p.113336.
Truong, V.N., Yang, C.K. and Tran, Q.V., 2016. A translator for American sign language to text and speech. In: 2016 IEEE 5th Global Conference on Consumer Electronics. IEEE, United States, pp.1-2.
Verma, H.V., Aggarwal, E. and Chandra, S., 2013. Gesture recognition using kinect for sign language translation. In: 2013 IEEE 2nd International Conference on Image Information Processing (ICIIP-2013). IEEE, United States, pp.96-100.
Wasenmüller, O. and Stricker, D., 2016. Comparison of kinect v1 and v2 depth images in terms of accuracy and precision. In: Asian Conference on Computer Vision, 2016. Springer, United States, pp.34-45.
Wikipedia., 2019. Kurdish Sign Language. Available from: https://www.en.wikipedia.org/wiki/Kurdish_Sign_Language. [Last accessed on 2019 Mar 28].
World Health Organization., 2020. Deafness and Hearing Loss. Available from: https://www.who.int/news-room/fact-sheets/detail/deafness-and-hearing-loss. [Last accessed on 2020 Mar 01].
Copyright (c) 2021 Sami F. Mirza, Abdulbasit K. Al-Talabani
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License [CC BY-NC-SA 4.0] that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).