Designing Automated Scoring System of Open-Ended Test by Providing Automatic Feedback to Diagnose Mathematical Proficiency Levels through Machine Learning
Article Sidebar
Main Article Content
Abstract
The study aimed to (1) analyze students' response patterns for the design of the prototypical innovation, including students' misconceptions, determining cut scores for mathematical proficiency levels, and providing the automated feedback based on misconceptions and variables used to make predictions through machine learning; and (2) design the scoring system for the open-ended question test through machine learning. Design research was applied in this study. The sample comprised 495 grade 7 students. The research instrument was an open-ended test on the topic of Statistics and Probability through diagnostic tools in an online testing system, “eMAT-Testing.” The collected data were analyzed using a multidimensional random coefficients multinomial model (MRCMLM). The results are reported below.
1. Regarding the analysis of the students' response patterns based on the secondary data, it was found that (1) the students had 4 types of misconceptions: 1) misused data; 2) distorted theorems, rules, formulas, definitions and properties; 3) unverified solutions; and 4) misinterpreted language and mathematical symbols. In both the mathematical process dimension (MAP) and the conceptual structures dimension (SLO), the students’ most common misconception among the 4 types was the misused data. (2) On determining cut scores with defined criterion zones on the Wright map for diagnosing mathematical proficiency levels through Machine learning, it was found that the mathematic proficiency of MAP was divided into five levels based on four cut scores, ranging from the lowest to the highest as follows: -0.99 ,0.26, 0.44 and 0.61 respectively. In the same manner, that of SLO was distinguished into five levels according to four cut scores, including -0.70, 0.15, 0.76 and 1.39 respectively. Such cut scores can be employed to determine proficiency level ranges, scale scores, and raw scores as criteria for assessment of mathematical proficiency levels in each dimension. (3) And regarding the variables from secondary data used to generate the equation to predict mathematical proficiency levels through machine learning, the results showed that the independent variables that would affect the generation of the predictive equation were the mathematics GPA in the previous semester, the total GPA of the previous semester, the number of hours of tuition, and that of hours of self-study on mathematics. In addition, the dependent variables included raw scores of MAP, raw scores of SLO, and the total score of both dimensions.
2. The results also showed that the design of the scoring system for the open-ended question test through machine learning featured 5 parts, namely input, processing, output, automated feedback, and assessment reporting. Based on the qualitative assessment of the system design conducted by 5 experts through a focus group discussion, it covered 5 standards, including utility, feasibility, propriety, accuracy, and evaluation accountability.
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
The content and information contained in the published article in the Journal of Educational Measurement Mahasarakham University represent the opinions and responsibilities of the authors directly. The editorial board of the journal is not necessarily in agreement with or responsible for any of the content.
The articles, data, content, images, etc. that have been published in the Journal of Educational Measurement Mahasarakham University are copyrighted by the journal. If any individual or organization wishes to reproduce or perform any actions involving the entirety or any part of the content, they must obtain written permission from the Journal of Educational Measurement Mahasarakham University.
References
Adams, R. J., Wilson, M., & Wang, W. C. (1997). The multidimensional mandom coefficients multinomial logit model. Applied Psychological Measurement, 21(1), 1-23.
Brookhart, S. M., Moss, C. M., & Long, B. A. (2008). Teacher inquire into formative assessment practices among Title I reading teachers and literacy coaches. http://www.castl.duq.edu/Castl_TechReports.htm
Demars, C. (2010). Item Response Theory: Understanding Statistics Measurement. Oxford University Press.
Finnegan, D., Kauppinen, A., & Wärnsby, A. (2015). Automated Feedback in a Blended Learning Environment: Student Experience and Development. In G. Rijlaarsdam (Eds.), M. Deane & T. Guasch. (Vol. Eds.), Studies in Writing: Vol. x, Learning and teaching writing online: Strategies for success (pp. 31-45). Brill.
Fisher, D., Grant, M., Frey, N., & Johnson, C. (2007). Taking formative assessments schoolwide. Educational Leadership, 65(4), 64–68.
Fisher, D., & Frey. N. (2011). The formative assessment action plan. ASCD.
Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77, 81-112.
Junpeng, P., Krotha, J., Chanayota, K., Tang, K. N., & Wilson, M. (2019). Constructing Progress Maps of Digital Technology for Diagnosing Mathematical Proficiency. Journal of Education and Learning, 8(6), 90-102
Shalev-Shwartz, S., & Ben-David, S. (2014). Understanding Machine learning: From theory to algorithms. Cambridge university press
Stiggins, R. J., & Chappuis, J. (2011). An Introduction to Student-Involved Assessment FOR Learning (6th Ed.). Pearson.
Van Der Kleij, F. M., Feskens, R. C., & Eggen, T. J. (2015). Effects of feedback in a computer based learning environment on students’ learning outcomes: A meta-analysis. Review of Educational Research, 85(4), 475-511.
Varasunun, P. (2011). The Program Evaluation Standards (3rd Ed.) in 2010. Journal of Research Methodology, 24(2), 273-278.
Wilson, M. (2005). Constructing measures: An item response modeling approach. Routledge.
Wu, M. L., Adams, R. J., & Wilson, M. R. (2007). ACER Conquest: Generalized item response modelling software. ACER Press.
Yadav, R., Tiruwa, A., & Suri, P. K. 2017. Internet Based Learning (IBL) in Higher Education: A Literature Review. Journal of International Education in Business,10(2), 102-129.
Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2010). The program Evaluation standards: A guide for evaluators and evaluation users (3rd Ed.). Sage.
Adulpongpaisal, S. (2009). Construction of Mathematics Diagnostic Test on Relations and Functions for MathayomSuksa 5 Students [Master’s thesis]. Chiang Mai University. (in Thai)
Chinjunthuk, S., & Junpeng, P. (2020). Assessment Guidelines for Student’s Personalized Mathematical Proficiency Development. Journal of Educational Measurement, Mahasarakham University, 26(1), 47- 64. (in Thai)
Junpeng, P., Marwiang, M., Chinjunthuk, S., Suwannatrai, P., Krotha, J., Chanayota, K., Tawarungruang, C., Thuanman, J., Tang K. N., & Wilson M. (2020). Developing Students’ Mathematical Proficiency Level Diagnostic Tools through Information Technology in Assessment for Learning Report. The Thailand Research Fund and Khon Kaen University. (in Thai)
Junpeng, P. (2018). Applications of the multidimensional item response theory of research (4th Ed.). Khon Kaen University Printing. (in Thai)
Ministry of Education. (2017). Indicators and subjects of learning the core Learning group Mathematics according to the basic education core curriculum (Revised version 2017). Agricultural Co-operative Federation of Thailand Limited. (in Thai)
Office of the Basic Education Commission. (2020). Instruction manual learning mathematics (revised version 2020) According to the core curriculum of basic education version 2017. Agricultural Co-operative Federation of Thailand Limited. (in Thai)
Sukkrom, P. & Makanong, A. (2015). A Study of Misconceptions and Mistakes in Mathematics of Eleventh Grade Students in Schools Under the Secondary Educational Service Area Office 1 and 2. OJED, 10(4), 599-611. (in Thai)
Thamcharoen, T. (2019). Fault Detection and Diagnosis of Machine Downtime in Slider Attachment Process using Machine learning Technique [Master’s thesis]. Suranaree University of Technology. (in Thai)
The National Institute of Educational Testing Service. (2021). The result of the national basic educational testing (O-NET) Mathayom 3 Year 2020. http://www.niet.or.th (in Thai)
Wongwanit, S. (2020). Design Research in Education. Chulalongkorn University. (in Thai)