Peer Assessment of Item Relevance in Mathematics Ability Test Instruments Designed by Mathematics Education Students: A Gregory Coefficient Approach

Abstract

Designing test items that are relevant to their intended indicators is a core competency for prospective researchers and mathematics educators, yet empirical evidence from peer assessments of student-designed instruments remains limited. This study examined item–indicator relevance in mathematics ability test instruments developed by mathematics education students using peer assessment and the Gregory coefficient. A descriptive quantitative approach with a document-analysis design was employed, involving 15 constructed-response test instruments comprising 150 items assessed by two peers. Relevance ratings on a four-point scale were analyzed using the Gregory technique, descriptive statistics, and categorical agreement percentages in Microsoft Excel. The mean Gregory coefficient across instruments was 0.91; ten instruments achieved coefficients of 1.00, and twelve had coefficients ≥ 0.80. For the nine instruments with traceable item-level scores, the analysis covered 180 ratings, of which 77% were assigned the maximum score of 4. The mean rating was 3.76, and categorical agreement was 96.7%. For the remaining six instruments, coefficients were available only in student summary tables, with a mean of 0.83. The analysis also identified discrepancies between rating sheets and summaries, identical summaries, and misinterpretations of the coefficient. Although peer assessment provided useful feedback and indicated generally high item relevance, ceiling effects and reporting inconsistencies limited confidence in the evidence. The findings highlight the need for clear relevance rubrics, rater calibration, verification of summary calculations, and complementary expert assessment to strengthen instrument development and evaluation.