Efficient Kinect Sensor-based Kurdish Sign Language Recognition Using Echo System Network

Sami F. Mirza; Abdulbasit K. Al-Talabani

doi:10.14500/aro.10827

Sami F. Mirza Department of Computer Science, Faculty of Science, Soran University, Soran, Kurdistan Region – F.R. Iraq https://orcid.org/0000-0002-0382-027X
Abdulbasit K. Al-Talabani Department of Software Engineering, Faculty of Engineering, Koya KOY45, Kurdistan Region – F.R. Iraq http://orcid.org/0000-0001-6328-204X

Keywords: Deep learning, Echo system network, Long short-term memory, Microsoft Kinect v2 Sensor, Recurrent neural network, Sign language

Abstract

Sign language assists in building communication and bridging gaps in understanding. Automatic sign language recognition (ASLR) is a field that has recently been studied for various sign languages. However, Kurdish sign language (KuSL) is relatively new and therefore researches and designed datasets on it are limited. This paper has proposed a model to translate KuSL into text and has designed a dataset using Kinect V2 sensor. The computation complexity of feature extraction and classification steps, which are serious problems for ASLR, has been investigated in this paper. The paper proposed a feature engineering approach on the skeleton position alone to provide a better representation of the features and avoid the use of all of the image information. In addition, the paper proposed model makes use of recurrent neural networks (RNNs)-based models. Training RNNs is inherently difficult, and consequently, motivates to investigate alternatives. Besides the trainable long short-term memory (LSTM), this study has proposed the untrained low complexity echo system network (ESN) classifier. The accuracy of both LSTM and ESN indicates they can outperform those in state-of-the-art studies. In addition, ESN which has not been proposed thus far for ASLT exhibits comparable accuracy to the LSTM with a significantly lower training time.

Downloads

Download data is not yet available.

Author Biographies

Sami F. Mirza, Department of Computer Science, Faculty of Science, Soran University, Soran, Kurdistan Region – F.R. Iraq

Sami Farman Mirza is currently an assistant researcher at the computer science department, Faculty of science, Soran University. He has got a B.Sc. in Computer Science, Soran University. His research interest is in computer vision and video processing. He has participated in various IT projects such as translating Moodle to Kurdish. He is currently a master student in the computer science department at Soran University.

Abdulbasit K. Al-Talabani, Department of Software Engineering, Faculty of Engineering, Koya KOY45, Kurdistan Region – F.R. Iraq

Abdulbasit Al-Talabani is an Assistant Prof. at the Department of Software Engineering, Faculty of Engineering, Koya University. He has a B.Sc. in mathematics at Salahadin University/Iraq, M.Sc. in Computer Science, Koya University, Iraq, and a PhD degree at applied computing, Buckingham University, UK. His research interest is in machine learning, speech processing and computer vision.

References

Abdul, Z.K., Al-Talabani, A.K. and Ramadan, D.O., 2020. A hybrid temporal feature for gear fault diagnosis using the long short term memory. IEEE Sensors Journal, 20(23), pp.14444-14452.

Almasre, M.A. and Al-Nuaim, H., 2016. Areal-time letter recognition model for arabic sign language using Kinect and leap motion controller v2. International Journal of Advanced Engineering, Management and Science, 2(5), p.239469.

Al-Talabani, A., Sellahewa, H. and Jassim, S.A., 2015. Emotion recognition from speech: Tools and challenges. Mobile multimedia/image processing, security, and applications. International Society for Optics and Photonics, 9497, p.94970N.

Awata, S., Sako, S. and Kitamura, T., 2017. Japanese sign language recognition based on three elements of sign using kinect v2 sensor. In: International Conference on Human-computer Interaction. Springer, Berlin, Germany, pp.95-102.

Bianchi, F.M., Livi, L. and Alippi, C., 2016. Investigating echo-state networks dynamics by means of recurrence analysis. IEEE Transactions on Neural Networks and Learning Systems, 29(2), pp.427-439.

Bianchi, F.M., Scardapane, S., Løkse, S. and Jenssen, R., 2020. Reservoir computing approaches for representation and classification of multivariate time series. IEEE Transactions on Neural Networks and Learning Systems, 23(5), pp.2169-2179.

Capilla, D.M., 2012. Sign Language Translator Using Microsoft Kinect Xbox 360 tm. Department of Electrical Engineering and Computer Science, University of Tennessee, Tennessee.

Čerňanský, M. and Tiňo, P., 2007. Comparison of echo state networks with simple recurrent networks and variable-length Markov models on symbolic sequences. In: International Conference on Artificial Neural Networks. Springer, Berlin, Germany, pp.618-627.

Chai, X., Li, G., Lin, Y., Xu, Z., Tang, Y., Chen, X. and Zhou, M., 2013. Sign language recognition and translation with kinect. IEEE Conference on Automatic, 655, p.4.

Chen, Q., Georganas, N.D. and Petriu, E.M., 2008. Hand gesture recognition using Haar-like features and a stochastic context-free grammar. IEEE Transactions on Instrumentation and Measurement, 57(8), pp.1562-1571.

Dos Santos Anjo, M., Pizzolato, E.B. and Feuerstack, S., 2012. A Real-time System to Recognize Static Gestures of Brazilian Sign Language (Libras) Alphabet Using Kinect. IHC, Citeseer, pp.259-268.

El-Bendary, N., Zawbaa, H.M., Daoud, M.S., Hassanien, A.E. and Nakamatsu, K., 2010. Arslat: Arabic sign language alphabets translator. In: International Conference on Computer Information Systems and Industrial Management Applications. IEEE, United States, pp.590-595.

Gao, L., Li, H., Liu, Z., Liu, Z., Wan, L. and Feng, W., 2021. RNN-transducer based Chinese sign language recognition. Neurocomputing, 434, pp.45-54.

Gilorkar, N.K. and Ingle, M.M., 2014. A review on feature extraction for Indian and American sign language. International Journal of Computer Science and Information Technologies, 5(1), pp.314-318.

Hashim, A.D. and Alizadeh, F., 2018. Kurdish sign language recognition system. UKH Journal of Science and Engineering, 2(1), pp.1-6.

Hossein, M.J. and Ejaz, M.S., 2020. Recognition of Bengali sign language using novel deep convolutional neural network. In: 2nd International Conference on Sustainable Technologies for Industry 4.0 (STI), 2020. IEEE, United States, pp.1-5.

Karami, A., Zanj, B. and Sarkaleh, A.K., 2011. Persian sign language (PSL) recognition using wavelet transform and neural networks. Expert Systems with Applications, 38(3), pp.2661-2667.

Katilmis, Z. and Karakuzu, C., 2021. ELM based two-handed dynamic turkish sign language (TSL) word recognition. Expert Systems with Applications, 2021, pp.115213.

Kratimenos, A., Pavlakos, G. and Maragos, P., 2021. Independent sign language recognition with 3d body, hands, and face reconstruction. In: ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, United States, pp.4270-4274.

Kumar, P., Saini, R., Roy, P.P. and Dogra, D.P., 2018. A position and rotation invariant framework for sign language recognition (SLR) using Kinect. Multimedia Tools and Applications, 77(7), pp.8823-8846.

Lang, S., Block, M. and Rojas, R., 2012. Sign language recognition using kinect. In: International Conference on Artificial Intelligence and Soft Computing. Springer, Berlin, Germany, pp.394-402.

Lee, C.K., Ng, K.K., Chen, C.H., Lau, H.C., Chung, S. and Tsoi, T., 2021. American sign language recognition and training method with recurrent neural network. Expert Systems with Applications, 167, pp.114403.

Lee, G.C., Yeh, F.H. and Hsiao, Y.H., 2016. Kinect-based Taiwanese signlanguage recognition system. Multimedia Tools and Applications, 75(1), pp.261-279.

Li, D., Rodriguez, C., Yu, X. and Li, H., 2020. Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. WACV, United States, pp.1459-1469.

Li, X., He, M., Li, H. and Shen, H., 2021. A Combined Loss-based Multiscale Fully Convolutional Network for High-resolution Remote Sensing Image Change Detection. IEEE Geoscience and Remote Sensing Letters, United States.

Li, X., Mao, C., Huang, S. and Ye, Z., 2017. Chinese sign language recognition based on shs descriptor and encoder-decoder lstm model. In: Chinese Conference on Biometric Recognition. Springer, United States, pp.719-728.

Li, Y., 2012. Hand gesture recognition using Kinect. In: 2012 IEEE International Conference on Computer Science and Automation Engineering. IEEE, United States, pp.196-199.

Liu, T., Zhou, W. and Li, H., 2016. Sign language recognition with long shortterm memory. In: 2016 IEEE International Conference on Image Processing (ICIP). IEEE, United States, pp.2871-2875.

Livi, L., Bianchi, F.M. and Alippi, C., 2017. Determination of the edge of criticality in echo state networks through Fisher information maximization. IEEE Transactions on Neural Networks and Learning Systems, 29(3), pp.706-717.

Liwicki, S. and Everingham, M., 2009. Automatic recognition of fingerspelled words in british sign language. In: 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. IEEE, United States, pp.50-57.

Lukoševičius, M., 2012. A practical guide to applying echo state networks. In: Neural Networks: Tricks of the Trade. Springer, Berlin, Germany.

Maass, W., Natschläger, T. and Markram, H., 2002. Real-time computing without stable states: A new framework for neural computation based on perturbations. Neural Computation, 14(11), pp.2531-2560.

Mahmood, M.R., Abdulazeez, A.M. and Orman, Z., 2018. Dynamic hand gesture recognition system for kurdish sign language using two lines of features. In: 2018 International Conference on Advanced Science and Engineering (ICOASE). IEEE, United States, pp.42-47.

Mittal, A., Kumar, P., Roy, P.P., Balasubramanian, R. and Chaudhuri, B.B., 2019. A modified LSTM model for continuous sign language recognition using leap motion. IEEE Sensors Journal, 19(16), pp.7056-7063.

Pandita, S. and Narote, S., 2013. Hand gesture recognition using SIFT. Proceedings of the IEEE International Conference on Computational Intelligence and Security, 2(1), p.4.

Prabakaran, D. and Shyamala, R., 2019. Areview on performance of voice feature extraction techniques. In: 2019 3rd International Conference on Computing and Communications Technologies (ICCCT). IEEE, United States, p.221-231.

Prasad, M., Kishore, P., Kumar, E.K. and Kumar, D.A., 2016. Indian sign language recognition system using new fusion based edge operator. Journal of Theoretical and Applied Information Technology, 88(3), p.574-583.

Preeti Amatya, K.S. and Meixner, G., 2018. Translation of sign language into text using kinect for windows v2. In: The Eleventh International Conference on Advances in Computer-human Interactions. ACHI, United States.

Rastgoo, R., Kiani, K. and Escalera, S., 2020. Hand sign language recognition using multi-view hand skeleton. Expert Systems with Applications, 150, p.113336.

Truong, V.N., Yang, C.K. and Tran, Q.V., 2016. A translator for American sign language to text and speech. In: 2016 IEEE 5th Global Conference on Consumer Electronics. IEEE, United States, pp.1-2.

Verma, H.V., Aggarwal, E. and Chandra, S., 2013. Gesture recognition using kinect for sign language translation. In: 2013 IEEE 2nd International Conference on Image Information Processing (ICIIP-2013). IEEE, United States, pp.96-100.

Wasenmüller, O. and Stricker, D., 2016. Comparison of kinect v1 and v2 depth images in terms of accuracy and precision. In: Asian Conference on Computer Vision, 2016. Springer, United States, pp.34-45.

Wikipedia., 2019. Kurdish Sign Language. Available from: https://www.en.wikipedia.org/wiki/Kurdish_Sign_Language. [Last accessed on 2019 Mar 28].

World Health Organization., 2020. Deafness and Hearing Loss. Available from: https://www.who.int/news-room/fact-sheets/detail/deafness-and-hearing-loss. [Last accessed on 2020 Mar 01].