Validity and Reliability Assessment of Cross-Culturally Adapted Bebras Test for Evaluating Computational Thinking in Junior High School Students
Keywords:
computational thinking, cross-cultural adaptation, Rasch model, BebrasAbstract
This study aimed to examine the validity and reliability of the Bebras test, which was cross-culturally adapted and adjusted to fit the Thai context. Both Classical Test Theory (CTT) and Modern Test Theory (MTT) were employed to analyze the data. The sample comprised of 115 grade 9 students. The research instrument was the Bebras test, a standardized assessment developed by Vilnius University in Lithuania. Data analysis employed CTT, yielding an average difficulty index of 0.57 and an average discrimination index of 0.42. Additionally, the Rasch model was applied to assess the students’ abilities and item difficulties. The findings revealed that the median student ability and item difficulty were closely aligned, indicating that the test was well-suited to the students’ ability levels. The weighted likelihood estimation reliability (WLE Reliability) was 0.659, with infit and outfit values falling within an acceptable range, demonstrating high validity. The ability levels ranged from -2.5 to 3.5 logits, and the fit indices were consistent with empirical data. Furthermore, the distribution of item difficulty and student ability, as illustrated by the Wright map, suggested that the test could effectively assess students with a wide range of abilities. Based on these findings, the study recommends that if the test is to be used with students from other educational settings, a preliminary pilot test should be conducted to evaluate whether the test is suitable for those students and to adjust test that is better aligned with their ability levels.
References
ภาษาไทย
ชลิตา ธัญญะคุปต์. (2564, 15 กรกฎาคม). วิทยาการคำนวณ: วิชาบังคับสำหรับนักเรียนไทย. คลังความรู้ SciMath. https://www.scimath.org/article-technology/item/12351-2021-07-01-05-42-38?%20utm_source
ศิริชัย กาญจนวาสี. (2556). ทฤษฎีการทดสอบแบบดั้งเดิม (พิมพ์ครั้งที่ 7). สำนักพิมพ์แห่งจุฬาลงกรณ์มหาวิทยาลัย.
สถาบันส่งเสริมการสอนวิทยาศาสตร์และเทคโนโลยี (สสวท.). (2562). คู่มือการใช้หลักสูตรรายวิชาพื้นฐานวิทยาศาสตร์ กลุ่มสาระการเรียนรู้วิทยาศาสตร์ สาระเทคโนโลยี (วิทยาการคำนวณ) ระดับประถมศึกษาและมัธยมศึกษา. https://uatscimath.ipst.ac.th/ebook-technology/item/8376-2560-2551
สํานักงานปลัดกระทรวงการอุดมศึกษา วิทยาศาสตร์ วิจัย และนวัตกรรม. (2567). ข้อมูลนักเรียนโรงเรียนสาธิต. https://info.mhesi.go.th/stat_satit.php?search_year=2567
สำนักงานเลขาธิการสภาการศึกษา. (2564). แนวทางการส่งเสริมการจัดการเรียนการสอนวิทยาการคำนวณและการโค้ดดิ้ง เพื่อพัฒนาทักษะผู้เรียนในศตวรรษที่ 21. เซ็นจูรี.
ภาษาอังกฤษ
Behr, D. (2017). Assessing the use of back translation: The shortcomings of back translation asa quality testing method. International Journal of Social Research Methodology, 20(6), 573-584. https://doi.org/10.1080/13645579.2016.1252188
Bond, T. G., & Fox, C. M. (2015). Applying the Rasch model: Fundamental measurement in the human sciences (3rd ed.). Routledge. https://doi.org/10.4324/9781315814698
Creswell, J. W., & Creswell, J. D. (2017). Research design: Qualitative, quantitative, and mixed methods approaches (4th ed.). Sage.
Dagiene, V., & Futschek, G. (2008). Bebras international contest on informatics and computer literacy: Criteria for good tasks. In R. T. Mittermeir & M. M. Sysło (Eds.), Informatics education - supporting computational thinking (pp. 19–30). Springer. https://doi.org/10.1007/978-3-540-69924-8_2
Dagiene, V., Gülbahar, Y., Grgurina, N., López-Pernas, S., Saqr, M., Apiola, M., & Stupuriene, G. (2023). Computing education research in schools. In B. Sentance, E. Barendsen, & C. Schulte (Eds.), Past, present and future of computing education research: A global perspective (pp. 481–520). Springer.
Embretson, S. E., & Reise, S. P. (2013). Item response theory for psychologists. Psychology.
Faul, F., Erdfelder, E., Buchner, A., & Lang, A. G. (2009). Statistical power analyses using G* power 3.1: Tests for correlation and regression analyses. Behavior research methods, 41(4), 1149-1160. https://doi.org/10.3758/BRM.41.4.1149
Futschek, G., & Moschitz, J. (2011). Learning algorithmic thinking with tangible objects eases transition to computer programming. In R. Mittermeir & M. Syslo (Eds.), Lecture notes in computer science: Vol. 7013. Informatics in schools: Contributing to 21st Century education (pp. 124–134). Springer. https://doi.org/10.1007/978-3-642-24722-4_14
Grover, S., & Pea, R. (2013). Computational thinking in k–12: A review of the state of the field. Educational Researcher, 42(1), 38-43. https://doi.org/10.3102/0013189X12463051
Gülbahar, Y., Kalelioğlu, F., Doğan, D., & Karataş, E. (2020). Bebras: A social approach for concept based learning of informatics and computational thinking. Ankara University Journal of Faculty of Educational Sciences (JFES), 53(1), 241-272. https://dergipark.org.tr/en/pub/auebfd/issue/53411/560771
Haladyna, T. M., & Downing, S. M. (2004). Construct irrelevant variance in high-stakes testing. Educational Measurement: Issues and Practice, 23(1), 17-27. https://doi.org/10.1111/j.1745-3992.2004.tb00149.x
Hambleton R. K., Merenda P. F., & Spielberger C. D. (2004). Adapting educational and psychological tests for cross-cultural assessment. Psychology. https://doi.org/10.4324/9781410611758
House, J. (2015). Translation as communication across languages and cultures. Routledge. https://doi.org/10.4324/9781315668956
Izu, C., Mirolo, C., Settle, A., Mannila, L., & Stupuriene, G. (2017). Exploring Bebras tasks content and performance: A multinational study. Informatics in Education, 16(1), 39-59. https://doi.org/10.15388/infedu.2017.03
Linacre, J. M. (2002). What do infit and outfit, mean-square and standardized mean? Rasch Measurement Transactions, 16(2), 878. https://www.rasch.org/rmt/rmt162f.htm
Meza, F., Vásquez, A., Espinoza, J., & Pérez, J. (2024). Validation of a Bebras-based test to assess computational thinking abilities in first-year college students. In S. Ben, S. A. Jeffrey, B. Lina, R. A. Samuel, & S. Libby (Eds.), Proceedings of the 55th ACM technical symposium on computer science education V.2. SIGCSE 2024 (pp. 123–129). Association for Computing Machinery. https://doi.org/10.1145/3626253.3635523
Shute, V. J., Sun, C., & Asbell-Clarke, J. (2017). Demystifying computational thinking. Educational Research Review, 22, 142-158. https://www.sciencedirect.com/science/article/pii/S1747938X17300350
Testa, S., Toscano, A., & Rosato, R. (2018). Distractor efficiency in an item pool for a statistics classroom exam: Assessing its relation with item cognitive level classified according to Bloom’s taxonomy. Frontiers in Psychology, 9, 1585. https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2018.01585/full
Wiebe, E., London, J., Aksit, O., Mott, B. W., Boyer, K. E., & Lester, J. C. (2019). Development of a lean computational thinking abilities assessment for middle grades students. In H. K. Elizabeth, P. A. Manuel, H. Sarah, & Z. Jian (Eds.), Proceedings of the 50th ACM technical symposium on computer science education. SIGCSE 2019 (pp. 456–461). Association for Computing Machinery. https://dl.acm.org/doi/abs/10.1145/3287324.3287390
Wing, J. M. (2006). Computational thinking. Communications of the ACM, 49(3), 33-35. https://doi.org/10.1145/1118178.1118215
Wright, B. D., & Masters, G. N. (1982). Rating scale analysis. MESA
Wright, B. D., & Stone, M. H. (1979). Best test design: Rasch measurement. MESA.
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
