A Deep Hybrid Model for Human-Computer Interaction Using Dynamic Hand Gesture Recognition

Brindha Ramalingam; Geetha Angappan

doi:10.24423/cames.559

Brindha Ramalingam Annamalai University, Annamalai Nagar, Chidambaram, India
Geetha Angappan Annamalai University, Annamalai Nagar, Chidambaram, India

DOI: http://dx.doi.org/10.24423/cames.559

Abstract

Dynamic hand gestures attract great interest and are utilized in different fields. Among these, man-machine interaction is an interesting area that makes use of the hand to provide a natural way of interaction between them. A dynamic hand gesture recognition system is proposed in this paper, which helps to perform control operations in applications such as music players, video games, etc. The key motivation of this research is to provide a simple, touch-free system for effortless and faster human-computer interaction (HCI). As this proposed model employs dynamic hand gestures, HCI is achieved by building a model with a convolutional neural network (CNN) and long short-term memory (LSTM) networks. CNN helps in extracting important features from the images and LSTM helps to extract the motion information between the frames. Various models are constructed by differing the LSTM and CNN layers. The proposed system is tested on an existing EgoGesture dataset that has several classes of gestures from which the dynamic gestures are utilized. This dataset is used as it has more data with a complex background, actions performed with varying speeds, lighting conditions, etc. This proposed hand gesture recognition system attained an accuracy of 93%, which is better than other existing systems subject to certain limitations.

Keywords

dynamic hand gesture, human-computer interaction, long short-term memory, convolutional neural network.,

References

1. R.M. Gurav, P.K. Kadbe, Real time finger tracking and contour detection for gesture recognition using OpenCV, [in:] 2015 International Conference on Industrial Instrumentation and Control (ICIC), pp. 974–977, IEEE, 2015, doi: 10.1109/IIC.2015.7150886.
2. H. Tang, H. Liu, W. Xiao, N. Sebe, Fast and robust dynamic hand gesture recognition via key frames extraction and feature fusion, Neurocomputing, 331(C): 424–433, 2019, doi: 10.1016/j.neucom.2018.11.038.
3. N.A. Ibraheem, R.Z. Khan, M.M. Hasan, Comparative study of skin color-based segmentation techniques, International Journal of Applied Information Systems (IJAIS), 5(10): 24–34, 2013, doi: 10.5120/ijais13-450985.
4. M. Alhussein, K. Aurangzeb, S.I. Haider, Hybrid CNN-LSTM model for short-term individual household load forecasting, IEEE Access, 8: 180544–180557, 2020, doi: 10.1109/ACCESS.2020.3028281.
5. R.M. Prakash, T. Deepa, T. Gunasundari, N. Kasthuri, Gesture recognition and finger-tip detection for human computer interaction, [in:] 2017 International Conference on Innovations in Information, Embedded and Communication Systems (ICIIECS), pp. 1–4, IEEE, 2017, doi: 10.1109/ICIIECS.2017.8276056.
6. M. Soliman, F. Mueller, L. Hegemann, J.S. Roo, C. Theobalt, J. Steimle, Finger input: Capturing expressive single-hand thumb-to-finger microgestures, [in:] 2018 International Conference on Interactive Surfaces and Spaces (ICISS), pp. 177–187, ACM, 2018, doi: 10.1145/3279778.3279799.
7. F. Chen, J. Deng, Z. Pang, M. Baghaei Nejad, H. Yang, G. Yang, Finger angle-based hand gesture recognition for smart infrastructure using wearable wrist-worn camera, Applied Sciences, 8(3): 369, 2018, doi: 10.3390/app8030369.
8. N.L. Hakim, T.K. Shih, S.P.K. Arachchi, W. Aditya, Y.C. Chen, C.Y. Lin, Dynamic hand gesture recognition using 3DCNN and LSTM with FSM context-aware model, Sensors, 19(24): 5429, 2019, doi: 10.3390/s19245429.
9. O. Kopuklu, A. Gunduz, N. Kose, G. Rigoll, Real-time hand gesture detection and classification using convolutional neural networks, [in:] 2019 International Conference on Automatic Face & Gesture Recognition (ICAFGR), pp. 1–8, IEEE, 2019, doi: 10.48550/ARXIV.1901.10323.
10. S. Sridhar, F. Mueller, A. Oulasvirta, C. Theobalt, Fast and robust hand tracking using detection-guided optimization, [in] 2015 Conference on Computer Vision and Pattern Recognition, pp. 3213–3221, IEEE, 2015, doi: 10.1109/cvpr.2015.7298941.
11. C. Cao, Y. Zhang, Y. Wu, H. Lu, J. Cheng, Egocentric gesture recognition using recurrent 3d convolutional neural networks with spatiotemporal transformer modules, [in:] 2017 International conference on computer vision (ICCV), pp. 3763–3771, IEEE, 2017, doi: 10.1109/ICCV.2017.406.
12. H. Gammulle, S. Denman, S. Sridharan, C. Fookes, Two stream LSTM: A deep fusion framework for human action recognition, [in:] 2017 Winter Conference on Applications of Computer Vision (WACV), pp. 177–186, IEEE, 2017, doi: 10.1109/WACV.2017.27.
13. M. Baccouche, F. Mamalet, C. Wolf, C. Garcia, A. Baskurt, Sequential deep learning for human action recognition, [in:] 2011 International workshop on human behavior understanding, pp. 29–39, Springer, 2011, doi: 10.1007/978-3-642-25446-8_4.
14. M. Loey, G. Manogaran, M.H.N. Taha, N.E.M. Khalifa, A hybrid deep transfer learning model with machine learning methods for face mask detection in the era of the COVID-19 pandemic, Measurement, 167: 108288, 2021, doi: 10.1016/j.measurement.2020.108288.
15. A. Agrawal, R. Raj, S. Porwal, Vision-based multimodal human-computer interaction using hand and head gestures, [in:] 2013 Conference on Information & Communication Technologies, pp. 1288–1292, IEEE, 2013, doi: 10.1109/CICT.2013.6558300.
16. C. Wang, Z. Liu, S.C. Chan, Super pixel-based hand gesture recognition with kinect depth camera, IEEE transactions on multimedia, 17(1): 29–39, 2014, doi: 10.1109/TMM.2014.2374357.
17. M.D. Zeiler, R. Fergus, Visualizing and understanding convolutional networks, [in:] 2014 European Conference on computer vision (ECCV), 8689: 818–833, Springer, 2014, doi: 10.1007/978-3-319-10590-1_53.
18. Y. Zhang, C. Cao, J. Cheng, H. Lu, EgoGesture: A new dataset and benchmark for egocentric hand gesture recognition, IEEE Transactions on Multimedia, 20(5): 1038–1050, 2018, doi: 10.1109/TMM.2018.2808769.
19. R.P. Sharma, G.K. Verma, Human computer interaction using hand gesture, Procedia Computer Science, 54: 721–727, 2015, doi: 10.1016/j.procs.2015.06.085.
20. K. Simonyan, A. Zisserman, Two-stream convolutional networks for action recognition in videos, [in:] 2014 International Conference on Neural Information Processing Systems, Vol. 1, pp. 568–576, 2014, doi: 10.48550/ARXIV.1406.2199.
21. L. Chao, J. Tao, M. Yang, Y. Li, Z. Wen, Long short-term memory recurrent neural network based encoding method for emotion recognition in video, [in:] 2016 International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2752–2756, IEEE, 2016, doi: 10.1109/ICASSP.2016.7472178.
22. K. Manisha, K. Artik, Automatic hand gesture recognition using hybrid meta-heuristicbased feature selection and classification with dynamic time warping, Computer Science Review, 39: 100320, 2021, doi: 10.1016/j.cosrev.2020.100320.
23. A. Mujahid et al., Real-time hand gesture recognition based on deep learning YOLOv3 model, Applied Sciences, 11(9): 4164, 2021, doi: 10.3390/app11094164.
24. C. Li, S. Li, Y. Gao, X. Zhang, W. Li, A two-stream neural network for pose-based hand gesture recognition, IEEE Transactions on Cognitive and Developmental Systems, 40: 2021, doi: 10.1109/TCDS.2021.3126637.
25. T. Xianlun, Y. Zhenfu, P. Jiangping, H. Bohui, W. Huiming, J. Li, Selective spatiotemporal features learning for dynamic gesture recognition, Expert Systems with Applications, 169: 4499, 2021, doi: 10.1016/j.eswa.2020.114499.
26. EgoGesture Dataset, National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, http://www.nlpr.ia.ac.cn/iva/yfzhang/datasets/egogesture.html.