Development of the Adjustment of Cut-off Scores Method Based on Angoff’s Concept by Applying the Item Response Theory
Keywords:
method development, adjustment of the cut-off scores, Angoff's conceptAbstract
This research objectives were to develop a method to adjust the cut-off scores based on Angoff's concept by applying the item response theory, and to compare the cut-off scores based on the traditional Angoff’s concept with the newly developed model. The subject was classified according to experiences and education levels of the judges. The research method was divided into two phases: 1) studying and developing a method for determining the cut-off scores based on the item response theory, and then 2) comparing the cut-off scores based on the traditional Angoff's concept with the newly developed model. The research results showed that: (1) the adjustment of the cut-off scores method based on Angoff's concept by applying the item response theory was comprised of five steps: 1) selecting the test, 2) analyzing the test based on the item response theory, 3) sorting the test, 4) training on how to determine the cut-off scores, and 5) the judge approximated the probability of considering the answer to each item under 3 rounds of judgment; (2) when comparing the cut-off scores based on the traditional Angoff’s concept with the newly developed model by applying item response theory, the judges with different experiences and educational levels, the results were not different. However, there were two strand groups: Science and Social Studies, and Religion and Culture of the judges with different education levels that gave different cut-off scores with statistical significance at the level of 0.05. (p. <0.05)
Downloads
References
ศิริชัย กาญจนวาสี. (2555). ทฤษฎีการทดสอบแนวใหม่. (พิมพ์ครั้งที่ 4). กรุงเทพมหานคร : โรงพิมพ์ แห่งจุฬาลงกรณ์มหาวิทยาลัย.
Barman, A. (2008). Standard setting in student assessment: Is a defensible method yet to come?. Annals Academy of Medicine Singapore. 37(11), pp. 957-563.
Berk, R. A. (1986). A Consumer’s Guide to Setting Performance Standards on Criterion- Referenced Test. Review of Educational Research. 56(Spring), pp. 137-172.
Carlson, J., Tomkowiak, J., & Stilp, C. (2009). Using the Angoff Method to Set Defensible Cutoff Scores for Standardized Patient Performance Evaluations in PA Education. Journal of Physician Assistant Education (Physician Assistant Education Association). 20(1), pp. 15-23.
Chapman, B. (2014). Angovian Methods for Standard Setting in Medical Education: Can They Ever Be Criterion Referenced?. International Journal of Learning, Teaching and Educational Research. 4(1). pp. 1-26.
Cizek, G. J. (2001). Conjectures on the rise and call of standard setting: An introduction to context and practice. Setting Performance Standards: Concepts, Methods and Perspectives. pp. 3-17.
Cizek, G.J., & Bunch, M.B. (2007). Standard Setting: A Guide to Establishing and Evaluating Performance Standard on Tests. Thousand Oaks : SAGE Publications.
Cohen-Schotanus, J., & van der Vleuten, C. P. (2010). A standard setting method with the best performing students as point of reference: practical and affordable. Medical Teacher. 32(2), pp. 154-160.
Dagogo, J. P. (2014). Comparison of Outcome of Students’ Performance Using the Standard Setting Method with the Absolute Grading Method in Preclinical Examination. Nigerian Journal of Physiological Sciences. 29(2), pp. 103-106.
Ferdous, A. & Plake, B. S. (2005). The use of subsets of test questions in an Angoff standard setting method. Educational and Psychological Measurement. 65(2), pp. 185-201.
Gronlund, N.E. (1982). Construting Achievement Test. (3rd ed). Englewood Cliffs, New Jersey : Prentice Hall.
Hambleton, R. K., & Pitoniak, M. J. (2006). Setting performance standards. In R. Brennan (Ed.), Educational Measurement. (4th ed., pp. 433–470). Westport, CT: Praeger Publishers.
Hejri, S. M., & Jalili, M. (2014). Standard setting in medical education: fundamental concepts and emerging challenges. Medical Journal of the Islamic Republic of Iran. 28 (34), pp. 1-6.
Hess, B., Subhiyah, R. G., & Giordano, C. (2007). Convergence between cluster analysis and the Angoff method for setting minimum passing scores on credentialing examinations. Evaluation & the Health Professions. 30(4), pp. 362-375.
Impara, J. C., & Plake, B. S. (1998). Teachers’ Ability to Estimate Item Difficultly: A Test of The Assumptions in the Angoff Standard Setting Method. Journal of Education Measurement. 35 (1), pp. 69-81.
Jalili, M., Hejri, S. M., & Norcini, J. J. (2011). Comparison of two methods of standard setting: the performance of the three-level Angoff method. Medical Education. 45 (12), pp. 1199-1208.
Kane, M.T. (1994). Validating the performance standards associated with passing scores. Review of Educational Research. 64(3), pp. 425-461.
Lee, G. & Lewis, D. M. (2008). A generalizability theory approach to standard error estimates for bookmark standard settings. Educational and Psychological Measurement. 68(4), pp. 603-620.
Pant, H. A., Rupp, A. A., Tiffin-Richards, S. P., & Köller, O. (2009). Validity issues in standard-setting studies. Studies in Educational Evaluation. 35(2-3), pp. 95-101.
Peterson, C. H., Schulz, E. M., & Engelhard Jr., G. (2011). Reliability and validity of bookmark-based methods for standard setting: Comparisons to Angoff-based methods in the National Assessment of Educational Progress. Educational Measurement: Issues and Practice. 30(2), pp. 3-14.
Shulruf, B., Wilkinson, T., Weller, J., Jones, P., & Poole, P. (2016). Insights into the Angoff method: results from a simulation study. BMC Medical Education. 16(1), pp. 134-148. Smith, R. W., Davis-Becker, S. L., & O’Leary, L. S. (2014). Combining the best of Two Standard Setting Methods: the Ordered Item Booklet Angoff. Journal of Applied Testing Technology. 15(1), pp. 18-26.
Zieky, M. J., Perie, M., & Livingston, S. A. (2008). Cut Scores: A Manual for Setting Standards of Performance on Educational and Occupational Tests. Educational Testing Service.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2021 Dhonburi Rajabhat University
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
บทความที่ได้รับการตีพิมพ์เป็นลิขสิทธิ์ของ มหาวิทยาลัยราชภัฏธนบุรี
ข้อความที่ปรากฏในบทความแต่ละเรื่องในวารสารวิชาการเล่มนี้เป็นความคิดเห็นส่วนตัวของผู้เขียนแต่ละท่านไม่เกี่ยวข้องกับมหาวิทยาลัยราชภัฏธนบุรีและบุคลากรท่านอื่นๆในมหาวิทยาลัยฯ แต่อย่างใด ความรับผิดชอบองค์ประกอบทั้งหมดของบทความแต่ละเรื่องเป็นของผู้เขียนแต่ละท่าน หากมีความผิดพลาดใดๆ ผู้เขียนแต่ละท่านจะรับผิดชอบบทความของตนเองแต่ผู้เดียว
