Development of Thai sign language recognition model using deep learning for communication between student teachers and student with special needs: application of CNN and RNN models
Article Sidebar
Main Article Content
Abstract
This research aims to 1) analyze the Thai sign language used by teachers to communicate with students with special needs in the classroom, and 2) develop and evaluate the performance of a Thai sign language recognition model using deep learning. The research subjects include special education teachers in schools for the deaf and volunteers proficient in sign language. The research tools include a questionnaire on the frequency of sign language sentences. Data analysis is divided into two parts: Part 1 involves analyzing the Thai sign language used by teachers to communicate with students with special needs in the classroom using mean and percentage from the questionnaire data. Part 2 involves developing and evaluating the performance of a Thai sign language recognition model using deep learning. The researcher selected four deep learning models combining CNN and RNN to develop a model that recognizes 20 Thai sign language sentences using video data recorded with volunteers. The models' performance was compared in terms of accuracy, model size, and training time, to identify the most effective model for evaluating the overall and individual sentence recognition using accuracy, precision, recall, and F1-score.
The research findings revealed that 1) the sign language used in the classroom tends to be shorter and more concise than spoken language, with reinforcement phrases being the most commonly used; 2) the development of a Thai sign language recognition model using the EfficientNetB0 combined with GRU achieved the highest performance in terms of accuracy, model size, and training time; and 3) the evaluation of the Thai sign language recognition model's performance using a test dataset showed that the model achieved an accuracy of 73%. Sentences that the model recognized well included "proud," "are you ready?" "look at the picture," and "understand," while sentences it struggled with included "beautiful work," "can you do it?" and "right or wrong."
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
The content and information contained in the published article in the Journal of Educational Measurement Mahasarakham University represent the opinions and responsibilities of the authors directly. The editorial board of the journal is not necessarily in agreement with or responsible for any of the content.
The articles, data, content, images, etc. that have been published in the Journal of Educational Measurement Mahasarakham University are copyrighted by the journal. If any individual or organization wishes to reproduce or perform any actions involving the entirety or any part of the content, they must obtain written permission from the Journal of Educational Measurement Mahasarakham University.
References
Amatanon, V., Chanhang, S., Naiyanetr, P., & Thongpang, S. (2014). Sign language-Thai alphabet conversion based on Electromyogram (EMG). In 2014 7th Biomedical Engineering International Conference (BMEiCON) (pp. 1–4). IEEE. https://doi.org/10.1109/BMEiCON.2014.7017398
Balaha, M. M., El-Kady, S., Balaha, H. M., Salama, M., Emad, E., Hassan, M., & Saafan, M. M. (2022). A vision-based deep learning approach for independent-users Arabic sign language interpretation. Multimedia Tools and Applications, 82(5), 6807-6826. https://doi.org/10.1007/s11042-022-13423-9
Chaikaew, A. (2022). An Applied Holistic Landmark with Deep Learning for Thai Sign Language Recognition. In 2022 37th International Technical Conference on Circuits/Systems, Computers and Communications (ITC-CSCC) (pp. 1046-1049). IEEE. https://doi.org/10.1109/ITC-CSCC55581.2022.9895052
Chansri, C., & Srinonchat, J. (2016). Hand gesture recognition for Thai sign language in complex background using fusion of depth and color video. Procedia Computer Science, 86, 257-260. https://doi.org/10.1016/j.procs.2016.05.113
Gedkhaw, E. (2022). The Performance of Thai Sign Language Recognition with 2D Convolutional Neural Network Based on NVIDIA Jetson Nano Developer Kit. TEM Journal, 11(1), 411-419. https://doi.org/10.18421/TEM111-52
Huang, J., & Chouvatut V. (2024). Video-Based Sign Language Recognition via ResNet and LSTM Network. Journal of Imaging, 10(6), 149. https://doi.org/10.3390/jimaging10060149
Kelly, J. F., McKinney E. L., & Swift, O. (2020). Strengthening teacher education to support deaf learners. International Journal of Inclusive Education, 26(13), 1289–1307. https://doi.org/10.1080/13603116.2020.1806366
Klomsae, A., Auephanwiriyakul, S., & Theera-Umpon, N. (2017). A Novel String Grammar Unsupervised Possibilistic C-Medians Algorithm for Sign Language Translation Systems. Symmetry, 9(12), 321. https://doi.org/10.3390/sym9120321
Pariwat, T., & Seresangtakul, P. (2021). Multi-stroke thai finger-spelling sign language recognition system with deep learning. Symmetry, 13(2), 262. https://doi.org/10.3390/sym13020262
Puranik, V., Gawande, V., Gujarathi, J., Patani, A., & Rane, T. (2022). Video-based Sign Language Recognition using Recurrent Neural Networks. In 2022 2nd Asian Conference on Innovation in Technology (ASIANCON) (pp. 1-6). IEEE. https://doi.org/10.1109/ASIANCON55314.2022.9909061
Saengsri, S., Niennattrakul, V., & Ratanamahatana, C. A. (2012). TFRS: Thai finger-spelling sign language recognition system. In 2012 second international conference on digital information and communication technology and it's applications (DICTAP) (pp. 457-462). IEEE. https://doi.org/10.1109/DICTAP.2012.6215407
Sharma, S., & Singh, S. (2023). ISL recognition system using integrated mobile-net and transfer learning method. Expert Systems with Applications, 221, 119772. https://doi.org/10.1016/j.eswa.2023.119772
Sripairojthikoon, N., & Harnsomburana, J. (2019). Thai Sign Language Recognition Using 3D Convolutional Neural Networks. In Proceedings of the 2019 7th International Conference on Computer and Communications Management (pp. 186-189). Association for Computing Machinery. https://doi.org/10.1145/3348445.3348452
Vachirapipop, M., Soymat, S., Tiraronnakul, W., & Hnoohom, N. (2017, November). Sign Translation with Myo Armbands. In 2017 21st International Computer Science and Engineering Conference (ICSEC) (pp. 1-5). IEEE. https://doi.org/10.1109/ICSEC.2017.8443836
Vijitkunsawat, W., Racharak, T., Nguyen, C., & Le Minh, N. (2023). Video-Based Sign Language Digit Recognition for the Thai Language: A New Dataset and Method Comparisons. In Proceedings of the 12th International Conference on Pattern Recognition Applications and Methods (pp. 775-782). SciTePress. https://doi.org/10.5220/0011643700003411
Boonya, R. (2008). Sign Language: Language of the Deaf. Journal of Ratchasuda Institute, 4(1), 77-94. (in Thai)
Buangamdee, C. (2017) A Translation of Thai Language to Thai Sign Language with Rule-based Technique [Master’s thesis]. Naresuan University. (in Thai)
Jaruwanawat, A. (2007). Continuous Thai Sign Language Recognition Using Hiden Markov Models [Master’s thesis]. King Mongkut Institute of Technology Ladkrabang. (in Thai)
Petchporee, S. (2005). A Study on Sign Language Vocabulary Memory of Hearing Impaired Children in Prathomsuksa 1 by Minor Game [Master’s thesis]. Srinakarinwirot University. (in Thai)
Plianwong, N., & Kangkachit, T. (2021). Thai Sign Language Translation System Through Few Shot Learning. DPU Graduate Studies Journal, 10(2), 119-132. (in Thai)
Sakulsujirapa, B. (2011). Development of a Thai Sign Language Recognition System using Hiden Markov Models [Master’s thesis]. Prince of Songkla University. (in Thai)
Sriboonruang, P., Tanachai, P., Khaggathog, K. Vijitkunsawat, W., & Anunvrapong, P. (2022). Software Sign Language Translator to Text and Speech by Using the Landmarks Technique of MediaPipe. The Journal of Industrial Technology : Suan Sunandha Rajabhat University, 10(2), 66-76. (in Thai)
Veerasakulthong, W. (2007). Thai Hand Sign Recognition [Master’s thesis]. National Institute of Development Administration. (in Thai)
