Development of a Multidimensional Assessment Scale of Transformative Competency Levels for Lower Secondary School Students Based on Construct Modeling Approach

Abstract

This research is a part of the development of a scale for assessing transformative competencies using real-time digital automation platform. It specifically aimed to develop and validate the quality of a multidimensional assessment scale of transformative competency levels for lower secondary school students. This study used a construct modeling approach as the basis for four processes of development: (1) developing the construct maps of transformative competency assessment; (2) developing the situational questions based on the construct maps; (3) scoring the learning outcomes; and (4) analyzing the construct modeling. All of the four processes were conducted under the Design Research methodology. The group of test takers employed for the process of verifying the quality of the assessment scale consisted of 487 lower secondary school students. The data from their responses were analyzed and considered through the Multidimensional Random Coefficients Multinomial Logit Model (MRCML) and the Wright Map. The findings were as follows:

(1) The results of developing the multidimensional assessment scale of transformative competency levels for lower secondary school students based on the construct maps were of three dimensions: (1) creating new value; (2) taking responsibility; and (3) reconciling tensions and dilemmas. Each dimension consisted of five levels starting from the highest level of 5, or the level of extended thinking (Advanced Level), to the lowest one or the level of responding with inconsistent or negative answers (No-relevance Level). They were developed into 30 multiple choice-situational questions with 5 choices each and scoring scales of 0 to 4.

(2) The results of scale validation were acceptable according to the standards for educational and psychological testing. (2.1) In terms of the validity evidence, the assessment scale was consistent with the validity evidence as the content validity evidence of the assessment items indicated the list of questions covered the transformative competency assessment. The response process validity evidence from the interviews with the test takers after the assessment showed that they understood the questions exactly as the researcher asked them. The internal structure validity evidence and the determination of cut-off score were found to be conformed to the empirical data (= 72.21, df =5, p = .01). When it was tested by the statistical test as in Likelihood-Ratio, the multidimensional model was found to be most consistent with the data (G² =33741.91, AIC =33993.91, BIC =34080.54) because the values of G², AIC and BIC were lower than the unidimensional model (G² = 33814.12, AIC = 34056.12, BIC = 34139.31). (2.2) For the reliability evidence, the EAP/PV reliability values in the dimensions of creating new value, taking responsibility, and reconciling tensions and dilemmas were 0.80, 0.83 and 0.82, respectively. The reliability of internal consistency of the total scale was 0.87; and the mean of the standard error of measurement (SEM) in the dimensions of creating new value, taking responsibility, and reconciling tensions and dilemmas were 0.03, 0.03, and 0.02, respectively. (2.3) The results of the item fit statistics analysis on the OUTFIT MNSQ and INFIT MNSQ values of the question items from the multidimensional model analysis indicated that the OUTFIT MNSQ of the scale were between 0.77 and 1.73, while the INFIT MNSQ ranged from 0.86 to 1.30. Almost all questions were within the acceptable criteria.