Reliability of Modified Essay Question Test for Measuring Critical Thinking Skills in Mathematics under Different Testing Conditions: An Application of Generalizability

Abstract

This thesis was aimed at 1) developing a Modified Essay Question test (MEQ) for measuring mathematical critical thinking skills and a double-layer scoring rubric for ninth grade students 2) checking the quality of the double -layer scoring rubric for ninth grade students and 3) comparing the reliability of the modified essay question for measuring mathematical critical thinking skills with different scoring methods through an application of generalizability theory. The sample was comprised of 90 ninth grade students. Research instruments included the essay question test for mathematical critical thinking skills and three types of scoring rubrics. Cronbach’s alpha, Pearson’s Product Moment Correlation and G–Coefficient were used to analyze the data. Results were as follows: 1) Each question of the modified essay questions test was in accordance with the process of mathematical critical thinking. The 1st step of the double -layer scoring rubric concentrated on a detailed assessment of validity, an understanding of what was being measured, and communication in mathematics. Scores obtained from the 1st step were converted into those of the 2nd step according to the assigned criteria. 2) The double layer scoring rubric yielded content validity, which conformed to the answering guideline and reliability of scoring of an individual assessor as well as between assessors was at a high level. 3) The double layer scoring rubric was of higher precision than the small scaled analytical method and the Knox scoring method with value 0.1, respectively.