Journal of Educational Measurement Mahasarakham University

Applying Generative Artificial Intelligence in Educational Research Instrument Development: A Methodological Framework

2026-04-04T21:08:37+07:00

The development of educational research instruments is critical to the quality, credibility, and interpretability of research findings. However, many researchers continue to face challenges in constructing instruments that are both theoretically grounded and methodologically sound. At the same time, recent advances in Generative Artificial Intelligence (GenAI), particularly tools based on Large Language Models (LLMs) and Natural Language Processing (NLP), have created new opportunities to support construct definition, indicator specification, preliminary item drafting, and early-stage language review. Nevertheless, the application of these technologies to educational research instrument development still lacks a clearly articulated methodological framework grounded in educational measurement and evaluation principles.
This conceptual paper proposes a methodological framework for applying Generative Artificial Intelligence to educational research instrument development. The framework integrates four core components: educational measurement and evaluation foundations, instrument development processes, the role of Artificial Intelligence as a methodological support mechanism, and Human-in-the-Loop decision-making. Within this framework, AI is positioned as a methodological assistant rather than a substitute for scholarly judgment. It may support construct clarification, indicator development, preliminary item generation, and early-stage quality review, while responsibility for decisions concerning validity, reliability, measurement precision, measurement fairness, and ethical appropriateness remains with researchers and domain experts.
The proposed framework offers a structured, transparent, and theoretically grounded approach to integrating AI into instrument development. It further emphasizes that the academic value of AI-assisted instrument development depends not merely on procedural efficiency, but on its alignment with sound measurement and evaluation principles and responsible human oversight. This paper thus provides a preliminary methodological contribution for educational researchers, research instructors, and graduate students, while also offering a foundation for future empirical validation of AI-assisted instrument development practices in education.

Factors Correlated with Digital Competency Among Youth in the 21st Century: A Meta-Analysis

2025-12-05T16:06:24+07:00

Digital competency (DC) is widely recognized as a foundational skill for 21st-century citizens, encompassing technical proficiency, critical thinking, and responsible use of digital tools. This study conducted a meta-analysis to quantitatively synthesize the relationship between various factors and youth DC, following the PRISMA guidelines. The meta-analysis included 42 eligible studies comprising 49 effect sizes focusing on youth (ages 12–24). A random-effects model revealed a large, significant positive correlation between the examined factors and youth digital competency (r=.525, Fisher's z = 0.583, p<.001). However, extremely high heterogeneity was detected (I²=99.49%). Key subgroup analysis was performed comparing factor types: Internal/Individual Variables (e.g., self-efficacy, cognitive skills) showed an estimated average correlation of z=0.475, which was comparable in magnitude to environmental, social norms, and other variables (e.g., parental mediation, SES), which showed z=0.481. The findings confirm the overall importance of both intrinsic and extrinsic factors in shaping DC, while the high variance suggests a strong need to identify contextual and methodological moderators.

Establishing CEFR-Referenced Cut Scores for the Royal Police Cadet Academy Test of English Proficiency (RPCA-TEP) Using the Modified Angoff Method

2025-12-19T19:49:43+07:00

The Royal Police Cadet Academy Test of English Proficiency (RPCA-TEP) was designed to evaluate the English language proficiency of police cadets in accordance with the mandate of the Higher Education Commission, which requires English proficiency testing aligned with the Common European Framework of Reference (CEFR). This study aimed to align RPCA-TEP scores with CEFR proficiency levels. Using the Modified Angoff method, 13 subject-matter experts provided judgments on the likelihood of borderline candidates at CEFR levels A2, B1, and B2 answering test items correctly. The analysis established cut-off scores out of 120, with candidates scoring 26–55 classified as A2, 56–82 as B1, and 83–120 as B2. The findings highlight the potential of the RPCA-TEP as a CEFR-referenced assessment tool and recommend further refinement using CEFR ability descriptors to guide future test development.

Psychometric Evaluation of the Emotion Regulation Instrument for Teacher Education Institution Students Utilizing AI in Scientific Writing

2026-02-19T22:05:30+07:00

This study examined the validity and reliability of an emotion regulation measurement instrument for university students using artificial intelligence (AI) in scientific writing, involving a sample of 100 respondents. Data were analysed using Confirmatory Factor Analysis (CFA) within the Covariance-Based Structural Equation Modelling (CB-SEM) framework, implemented using AMOS with Maximum Likelihood estimation. The adapted instrument consists of 18 items across two dimensions: Cognitive Reappraisal (Items 1–13) and Expressive Suppression (Items 14–18), expanded contextually from the original 10-item Gross & John’s ERQ. The measurement model demonstrated good overall fit (/df = 1.31; RMSEA = 0.056; CFI = 0.955; TLI = 0.949; IFI = 0.956). All 18 indicators loaded significantly (p < 0.001) with standardized factor loadings ranging from 0.57 to 0.76, confirming convergent validity. Construct Reliability (CR) was also computed and found to meet the acceptable threshold (CR ≥ 0.70). Overall, the instrument demonstrates adequate psychometric properties and provides a sound basis for assessing emotion regulation in the context of AI-assisted scientific writing.

Development and Validation of an Academic Engagement Scale for Undergraduate Students in Distance Learning

2026-03-20T14:20:07+07:00

Distance education at open universities faces significant challenges, including high attrition rates driven by social isolation and the demands of self-directed learning. A context-specific instrument is essential for understanding the mechanisms that support student persistence. This study aimed to develop and validate an academic engagement scale, with development referring to the refinement of an existing framework. Drawing on constructivist and motivational theories, academic engagement was conceptualized as a second-order construct comprising cognitive, emotional, and behavioral dimensions. Data were collected from 440 students at Sukhothai Thammathirat Open University using cluster sampling based on academic disciplines (Social Sciences, Health Sciences, and Science and Technology). Confirmatory Factor Analysis (CFA) was conducted to examine the hierarchical structure, while Cronbach’s alpha, Average Variance Extracted (AVE), and the Heterotrait–Monotrait (HTMT) ratio were used to assess reliability and construct validity.

The model fit indices indicated an acceptable fit to the data (/df = 3.22, RMSEA = .07, CFI = .92, NFI = .91, and SRMR = .04). All dimensions demonstrated satisfactory reliability and validity, with HTMT values below .85, supporting the distinctiveness of the constructs. Cognitive engagement showed strong associations within the model, highlighting the role of self-regulated learning in distance education. The findings provide empirical support for a multidimensional structure of academic engagement and contribute a context-specific measurement tool that can be applied to support student engagement and retention in an open university distance learning context.

Development Guidelines for Lifelong Learning Among Retirees in Guangdong Province, People’s Republic of China: A Practitioner-oriented Inquiry

2026-06-19T17:28:38+07:00

China's accelerating population ageing, the State Council's 2024 Silver Economy initiative, and the expansion of the Seniors University network have repositioned community elder learning centers as a strategic site for late-life development. This study examined how such centers can convert empirical evidence on the determinants of lifelong learning into culturally embedded guidance for practice. The study pursued four objectives: to examine the level of the factors influencing lifelong learning, to examine the level of lifelong learning across its five dimensions, to analyze the predictive contribution of the institutional, family, and learner domains, and to propose practitioner-oriented development guidelines. Using a sequential explanatory mixed-methods design, the research surveyed 384 retirees aged 60 years and above in Guangdong Province, selected through multistage random sampling, with a 60-item Likert questionnaire (overall reliability = .985) measuring three antecedent domains—institutional, family, and learner—and five outcome dimensions of lifelong learning. Quantitative data were analyzed using means, standard deviations, and stepwise multiple regression, and the interview data were analyzed thematically. The three antecedent domains and overall lifelong learning were all rated at a high level. Stepwise multiple regression confirmed all three domains as significant positive predictors, jointly explaining 68.0% of the variance (R² = .680, p < .01), with the learner domain dominant ( = .312). Semi-structured interviews with seven purposively selected experts were analyzed thematically, and the draft guidance was appraised for feasibility and utility by 12 senior learning-center administrators. The study delivered 24 practitioner-oriented development guidelines distributed across the three domains, with exemplar guidance illustrating their operation within Guangdong's 4-2-1 family structure and WeChat-saturated communication environment. Contributions include a relational reconception of the elder learning center, an implementation sequence aligned with the Silver Economy policy architecture, and a feasibility profile flagging the Silver Credit accumulation system as the guideline most dependent on cross-sectoral coordination.

The Mediating Role of Trust in the Relationship between Principal Leadership and Teachers’ Job and Teaching Satisfaction: Evidence from Hong Kong’s 2022 PISA Data

2026-01-30T14:18:45+07:00

This study employed a survey research design utilizing data from the Programme for International Student Assessment (PISA). The purposes of this study were: 1) to validate the proposed structural equation model of teachers’ job and teaching satisfaction 2) to analyze the direct and indirect effects of principal leadership on teachers’ job and teaching satisfaction, and 3) to investigate the mediating role of teachers’ trust in the relationship between principal leadership and teacher satisfaction. The sample consisted of 2,335 teachers from 155 schools, derived from the two-stage sampling design employed in the PISA framework. The instrument used was a 19-item Likert-scale questionnaire, with an overall reliability coefficient of 0.885. Data were analyzed using structural equation modeling (SEM) and bootstrapping techniques to test the significance of mediation effects. The results revealed that the proposed model demonstrated a good fit with the empirical data. Principal leadership had a significant direct effect on teachers’ trust ( = 0.50). Teachers’ trust exerted significant positive effects on job satisfaction ( = 0.76) and teaching satisfaction ( = 0.43). However, principal leadership had no significant direct effects on either dimension of satisfaction. Instead, it exerted indirect effects through trust on job satisfaction ( = 0.38) and teaching satisfaction ( = 0.21), both of which were statistically significant at the 0.05 level.

In conclusion, teachers’ trust served as a full mediator in the relationship between principal leadership and teachers’ satisfaction. The findings highlight the pivotal role of trust in educational organizations, particularly in Hong Kong’s highly competitive and hierarchically structured school system.

Development of a Climate-Mapped Educational Risk Index for Thailand

2026-01-25T12:54:55+07:00

Education systems worldwide, including Thailand's, face increasing vulnerability to climate change–related disasters, particularly in rural and socioeconomically disadvantaged areas. Although global policy frameworks emphasize protecting children from environmental risks, existing indices lack spatial granularity and do not directly incorporate educational outcomes into their structure. This study aims to (1) develop a spatial index of educational risk linked to climate conditions—the Climate-Mapped Educational Risk Index (CMER)—and (2) analyze the relationships between this index and geographic, school, and student factors. The study integrated school-level O-NET scores from four core subjects across 30,112 schools during 2017–2021 with high-resolution climate data from the SEACLID/CORDEX-SEA project covering the same period relative to a baseline of 1970–2007. The CMER Index was developed using Principal Component Analysis of extreme climate variables and student achievement, with schools classified by risk levels using ROC-based criteria. Elastic Net regression was then applied to analyze relationships between CMER scores and risk factors across educational service areas, followed by the construction of a Risk–Importance Matrix (RIM) to support policy decision-making.

The results revealed that the duration of extreme precipitation events was the only dimension showing a significant structural relationship with student achievement. Schools in the Northern, Northeastern, Central, and Southern regions exhibited higher risk levels. RIM analysis indicated that educational factors amplify risk to varying degrees across different areas, underscoring the need for context-specific interventions. The CMER Index and RIM provide a data-driven framework for identifying climate-linked educational risks and targeting resource allocation to build a more equitable and climate-resilient education system.

Meta-evaluation of Final Evaluation Reports by Save the Children Australia

2026-03-20T11:18:02+07:00

This research aims to evaluate the quality of final project evaluation reports produced by Save the Children Australia using the principles of meta-evaluation. It employs a documentary research approach, assessing 14 final evaluation reports prepared by Save the Children Australia between 2015 and 2023. The primary data collection tool was a meta-evaluation checklist, and the collected data were analyzed using descriptive statistical methods with the software Jamovi. The findings indicate that the 14 final evaluation reports produced by Save the Children Australia were generally designed with a systematic evaluation methodology. Most reports employed a mixed-methods approach and clearly specified the evaluation objectives, indicators, and data analysis procedures. When assessed against Stufflebeam’s meta-evaluation standards, most of the reports demonstrated very good quality in terms of utility, feasibility, and propriety. However, the accuracy standard showed considerable variation in scores, particularly regarding the meta-evaluation component, which was rarely included in the reports. This suggests opportunities for improving the quality and completeness of evaluation reports in the future.

The Study of Competency-Based Learning Assessment Problems of Grade 6 Mathematics Teachers in Schools under the Kalasin Primary Educational Service Area Office

2026-03-20T11:18:45+07:00

This research investigated and compared the conditions of problems regarding competency-based learning assessment among Grade 6 mathematics teachers in schools under the Kalasin Primary Educational Service Area Office, Thailand. Samples comprised 222 teachers selected through stratified random sampling. Research instrument used was a questionnaire with reliability coefficient of 0.99 and a focus group discussion form (IOC = 1.00). Data were analyzed using descriptive statistics, F-tests, and content analysis.

Results showed that: 1) problems in assessing core and subject-specific competencies were at moderate levels, with the most problematic areas being active citizenship (core) and mathematical communication and presentation (subject-specific); 2) teachers with different academic ranks showed significant differences at the .05 level in assessing subject-specific competencies, teachers with different experience levels also differed significantly at the .05 level in assessing subject-specific competencies, and teachers in schools of different sizes did not differ significantly for either core or subject-specific competencies; and 3) focus group findings revealed that teachers lacked clear understanding of curriculum-defined competencies, struggled with designing complex indicators, and faced time constraints in documentation, making it difficult to create comprehensive assessment tools with clear criteria and descriptors, which resulted in assessments lacking adequate validity.

Systemic Teacher Development for Enhancing Scientific Literacy Competency of Grade 6 Students Based on the PISA 2025 Framework Using the PCACK-4D Model: Diagnostic Research and Needs Assessment

2026-04-18T16:46:58+07:00

This study aimed to: 1) diagnose critical learning management issues based on O-NET Science results for Grade 6 students over three academic years (2021–2023); 2) analyze critical test items using the Revised Bloom's Taxonomy and the PISA 2025 scientific literacy competency framework; and 3) conduct a needs assessment and propose guidelines for systemic teacher development using the PCACK-4D model. A mixed-methods design employing diagnostic research and needs assessment was utilized. The target group comprised 22 participants—11 school administrators and 11 Grade 6 science teachers—from 11 schools under a Quality Development Center, Chaiyaphum Primary Educational Service Area Office 1, selected through purposive sampling. Research instruments included the PROFILE program, a Revised Bloom's Taxonomy × PISA 2025 Matrix analysis form, and semi-structured interviews (IOC = 0.67–1.00). Data were analyzed using T-scores, percentages, content analysis, and data triangulation.

The findings revealed that: 1) All learning standards were classified at Level L1; the most critically recurring standards were SC 1.2, SC 2.1, and SC 2.3, each yielding the highest weighted score (10 points). 2) The most critical test item was Standard SC 2.1, Indicator P.4/1 (academic year 2021), with an average correct-response rate of only 6.33%—below chance-level performance—indicating severe deficiencies in the Evaluating cognitive level and the PISA 2025 competency of designing and evaluating scientific inquiry processes. 3) The qualitative needs assessment identified systemic gaps across four dimensions—administration, context, pedagogy, and knowledge—leading to the design of six Knowledge Stack innovations under the PCACK-4D model, serving as Organizational Memory to sustain knowledge transfer and systemic learning management despite personnel turnover.

Development of a Scientific Explanation Competency Test on Sound Waves for Upper Secondary School Students

2026-04-17T14:37:00+07:00

The objective of this research was to develop a scientific explanation competency test on the topic of sound waves for upper secondary school students. The development process comprised five stages: 1) defining the operational definitions of scientific explanation competency regarding sound waves; 2) conducting content analysis and determining the test structure and number of items; 3) constructing test items and specific scoring rubrics; 4) evaluating the validity and suitability of the instrument; and 5) refining the test for pilot testing to determine its psychometric properties, including difficulty, discrimination, reliability, and the quality of the scoring rubrics.

The results indicated that the developed extended-response subjective test possessed content validity and was suitable for implementation. The instrument was pilot tested with a sample of 40 Grade 11 students from a large secondary school in Phrae Province, who had previously studied sound waves but had no formal training in constructing scientific explanations. The analysis revealed that the difficulty index ranged from 0.469 to 0.500, and the discrimination index ranged from 0.325 to 0.363. The test achieved a Cronbach’s alpha coefficient of 0.719, while the Intraclass Correlation Coefficient for inter-rater reliability of the scoring rubrics ranged from 0.766 to 0.913. Consequently, the developed test is an effective tool for assessing the scientific explanation competency of upper secondary school students regarding sound waves.

Development of an Instrument to Measure Artificial Intelligence Technological Pedagogical Content Knowledge (AI-TPACK) Competencies of Secondary School Teachers in Phitsanulok Province

2026-05-14T12:57:40+07:00

This research presents the development of an instrument to measure the Artificial Intelligence Technological Pedagogical Content Knowledge (AI-TPACK) competencies of secondary school teachers in Phitsanulok province. A Situational Judgment Test (SJT) format was applied to overcome the limitations of traditional self-report measures and assess professional judgment in real-world scenarios. Data were collected from a sample of 405 secondary school teachers selected through multi-stage random sampling. The developed instrument consisted of 4 main components and 16 behavioral indicators, totaling 45 items utilizing a 4-level rubric scoring system. The quality examination revealed that all items exhibited discrimination power greater than 0.20, and the overall internal consistency reliability was high at 0.917. Furthermore, the Second-Order Confirmatory Factor Analysis (CFA) confirmed strong construct validity and an excellent fit with the empirical data (Chi-square = 105.743, df = 86, p-value = 0.073, CFI = 0.992, TLI = 0.989, RMSEA = 0.024, SRMR = 0.028). Local norms were established using normalized T-scores, categorizing competencies into five levels. The assessment showed that the majority of secondary school teachers possessed a moderate level of AI-TPACK competency (40.25%), followed by high (25.68%) and low (23.70%) levels. This research provides a standardized measurement tool and essential information for planning teachers' professional development in the artificial intelligence era.

Evaluation of Mathematics for Life and Community through the Sufficiency Economy Project Case Study of Ban Nongle School

2026-05-11T12:32:02+07:00

This project evaluation was conducted based on Tyler’s Evaluation Model (1949). The objectives of this evaluation were to assess: 1) project objectives, 2) implementation plans, 3) developmental guidelines, 4) project execution, 5) project outcomes, and 6) project impacts. The data sources comprised one administrator, six teachers, one janitor, 43 students (Grades 1–6), 40 parents, and academic achievement records from the 2024 and 2025 academic years. The evaluation instruments comprised assessment forms, interview schedules, behavioral observation scales, questionnaires, and attitude scales. The evaluation findings are as follows 1) The objectives were highly congruent with student development needs and parental expectations regarding mathematical skills, demonstrating a strong alignment with the mathematics curriculum integrated with Sufficiency Economy principles. 2) The activity formats were found to be appropriate and well-structured. 3) The project was well-supported by the readiness of instructional media, equipment, personnel, and facilities. 4) All eight activities were considered effective as they utilized learning processes rooted in authentic contexts. However, some limitations were noted, particularly time constraints for complex activities, student presentations, and the need for greater calculation speed and accuracy. 5) Students demonstrated a "Good" level of mathematical process skills and collaborative abilities. Overall, student satisfaction and attitudes toward mathematics were at a "High" level. And 6) The impact assessment indicated an improvement in mathematics academic performance among students in Grades 1–4. Most participants exhibited a heightened awareness of financial saving and were able to effectively apply calculation and statistical knowledge to their daily lives.