The Development of a Multidimensional Forced-Choice Situational Judgment Test for Assessing Undergraduate Students’ Expectations Toward Blended Learning in the 21st Century
Article Sidebar
Main Article Content
Abstract
This study aimed to (1) develop a multidimensional forced-choice situational judgment test to assess undergraduate students’ expectations toward 21st-century blended learning and (2) examine the quality of the instrument in terms of response balance, validity, and reliability. Validity evidence was gathered through content validity assessment, criterion-related validation, and construct validity using a multitrait-multimethod (MTMM) matrix. Reliability was assessed using internal consistency and test-retest methods. The instrument development involved four groups: three purposively selected samples, consisting of experts in educational measurement and evaluation, higher education instructors, and undergraduate students, along with one stratified random sample of 120 undergraduate students, with academic faculties serving as strata. The final version of the instrument comprised 25 items incorporating visual stimuli, representing five distinct scenarios, each measuring five dimensions of blended learning expectations. The response balance analysis revealed no initial signs of response bias. Correlations between scores obtained from the developed instrument and external measures provided supportive evidence of criterion-related validity. Construct validity, assessed using the MTMM matrix, demonstrated higher correlations between different traits measured by the same method, thereby supporting convergent validity. Internal consistency was found to be at an acceptable level, and test-retest reliability indicated a high average correlation coefficient across the full instrument.
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
The content and information contained in the published article in the Journal of Educational Measurement Mahasarakham University represent the opinions and responsibilities of the authors directly. The editorial board of the journal is not necessarily in agreement with or responsible for any of the content.
The articles, data, content, images, etc. that have been published in the Journal of Educational Measurement Mahasarakham University are copyrighted by the journal. If any individual or organization wishes to reproduce or perform any actions involving the entirety or any part of the content, they must obtain written permission from the Journal of Educational Measurement Mahasarakham University.
References
Bonk, C. J., Graham, C. R., Cross, J., & Moore, M. G. (2005). The handbook of blended learning: Global perspectives, local designs. Pfeiffer.
Brown, A., & Maydeu-Olivares, A. (2011). Item response modeling of forced-choice questionnaires. Educational and Psychological Measurement, 71(3), 460–502.
Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56(2), 81–105.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.
George, D., & Mallery, P. (2003). SPSS for Windows step by step: A simple guide and reference (4th ed.). Allyn & Bacon.
Hontangas, P. M., de la Torre, J., Ponsoda, V., Leenen, I., Morillo, D., & Abad, F. J. (2015). Comparing traditional and IRT scoring of forced-choice tests. Applied Psychological Measurement, 39(8), 598–612.
Hooper, A. C., Cullen, M. J., & Sackett, P. R. (2006). Operational threats to the use of SJTs: Faking, coaching, and retesting issues. In J. A. Weekley & R. E. Ployhart (Eds.), Situational judgment tests: Theory, measurement, and application (pp. 205–232). Lawrence Erlbaum Associates.
Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings (2nd ed.). SAGE.
Krosnick, J. A., & Alwin, D. F. (1987). An evaluation of a cognitive theory of response-order effects in survey measurement. Public Opinion Quarterly, 51(2), 201–219.
Lakho, S., Jalbani, D., Memon, I., Siraj, S., & Ali, A. (2023). Blended enriched virtual model for the prediction of students’ performance using probabilistic-based model. In Proceedings of the 13th International Conference on Advances in Computing, Communications and Informatics (ICACCI) (pp. 138–144). Springer.
Lee, P., Joo, S. H., & Stark, S. (2021). Detecting DIF in multidimensional forced-choice measures using the Thurstonian item response theory model. Organizational Research Methods, 24(4), 739–771.
Lee, P., Joo, S., & Son, M. (2024). Detecting careless respondents in multidimensional forced choice data: An application of lz person-fit statistic to the TIRT model. Journal of Business and Psychology, 39, 541–564.
Muhlisoh, E. D., Santihastuti, A., & Wahjuningsih, E. (2020). Students’ perceptions of flipped approach in EFL classroom: A survey research. Journal of Education, Teaching, and Learning, 5(2), 393–400.
Peterson, E., & Irving, S. (2008). Secondary school students' conceptions of assessment and feedback. Learning and Instruction, 18(3), 238–250.
Prayitno, E. H., Masunah, J., & Milyartini, R. (2023). Implementation of enriched virtual learning model through online guidance to create film production in videography class. In Proceedings of the Fifth International Conference on Arts and Design Education (ICADE 2022) (pp. 531–540). Atlantis Press.
Sackett, P. R., & Yang, H. (2000). Correction for range restriction: An expanded typology. Journal of Applied Psychology, 85(1), 112–118.
Salleh, F. I. M., Baharum, H. I., & Shamsudin, S. (2017). Comparative study between flipped learning and flex model in English as a second language classroom. Advanced Science Letters, 23(4), 2663–2666.
Sass, R., Frick, S., Reips, U.-D., & Wetzel, E. (2020). Taking the test taker’s perspective: Response process and test motivation in multidimensional forced-choice versus rating scale instruments. Assessment, 27(3), 572–584.
Schünemann, A. L., & Ziegler, M. (2023). Use the force! Adaptation of response formats: From rating scale to multidimensional forced choice. Psychological Test Adaptation and Development, 4(1), 218–234.
Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257–285.
Swift, V., & Peterson, J. B. (2019). Contextualization as a means to improve the predictive validity of personality models. Personality and Individual Differences, 144, 153–163.
Ward, M. K., & Meade, A. W. (2023). Dealing with careless responding in survey data: Prevention, identification, and recommended best practices. Annual Review of Psychology, 74(1), 1–20.
Kanjanawasee, S. (2013). Classical test theory. Chulalongkorn University. (in Thai)
Yimyam, S., Jaruwatcharapanichkul, A., Charoensanti, C., Intharangkura Na Ayudhya, A., Chuto, P., & Chaloemsuk, N. (2015). The development of blended learning management to enhance 21st-century learning skills. Nursing Journal, 42(Special Issue), 129–140. (in Thai)
