The Quality of English Test Items for Senior High School Students: Validity, Cognitive Level, and Language Skills Analysis
Article Sidebar
Page Numbers:
3725-2746
Digital Object Identifier:
10.58578/mikailalsys.v4i2.11264
Viewed: 0 times
Downloaded: 0 times
LYAS Publisher cordially invites qualified professionals to serve as Editors or Reviewers. Your expertise will make an important contribution to maintaining and strengthening the academic quality of our publications. Interested applicants are kindly requested to complete the application form at the following link: Editors & Reviewers
Please feel free to contact us if you need any further information about the submission process or if you have any additional questions.
Authors:Hevriani Sevrika1*, Yenni Rozimela2, Syahrul R3
Copyright :Authors retain copyright and grant the journal right of first publication.
Main Article Content
Abstract
Although educational assessment has received considerable scholarly attention, comprehensive evaluations of teacher-made English examinations integrating validity, cognitive demand, and language-skill representation remain limited. This study evaluated the quality of a senior high school English summative examination by examining these three dimensions simultaneously. A quantitative descriptive design was employed to analyze all 50 multiple-choice and essay items included in the examination through total sampling. Data were obtained through documentation. Content validity was assessed based on curriculum alignment, empirical validity was examined using Pearson product–moment correlation, cognitive levels were classified according to Bloom’s Revised Taxonomy (C1–C6), and language-skill representation was analyzed using percentage-based descriptive statistics. The findings showed that 38 items (76%) were valid, whereas 12 items (24%) were invalid because of ambiguous wording, poorly constructed distractors, and insufficient alignment with instructional objectives. The cognitive-level distribution was dominated by lower-order thinking skills, particularly understanding (C2; 32%) and remembering (C1; 28%). Higher-order thinking skills were substantially underrepresented, comprising analyzing (C4; 12%), evaluating (C5; 6%), and creating (C6; 2%). The examination also emphasized receptive skills and structural knowledge, with reading comprehension accounting for 48% of the items, followed by grammar (20%) and vocabulary (14%). Productive and communicative skills received considerably less coverage, including writing (10%), listening (6%), and speaking (2%). These findings demonstrate that acceptable item validity alone does not ensure a cognitively balanced and communicatively authentic language assessment. This study contributes to the English language assessment literature by providing a multidimensional evaluation of teacher-made examinations in secondary education. It also highlights the need for teachers, schools, and policymakers to conduct continuous item validation, strengthen assessment literacy, increase higher-order cognitive demands, and achieve a more balanced representation of receptive, productive, and communicative language skills.
Citation Metrics:
Downloads
Article Details
Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
References
Adem, H., & Berkessa, M. (2022). A case study of EFL teachers’ practice of teaching speaking skills vis-à-vis the principles of communicative language teaching (CLT). Cogent Education, 9(1), 2087458. https://doi.org/10.1080/2331186X.2022.2087458
Al Mamari, M. R. H., Al-Mekhlafi, A. M., & Al-Barwani, T. A. (2018). Is communicative language teaching being tested communicatively? An analysis of English tests in Oman. English Language Teaching, 11(11), 83–91. https://doi.org/10.5539/elt.v11n11p83
Astrid, A., Hasanah, A., & Syafryadin. (2022). Integrating higher order thinking skills (HOTS) into English language teaching for elementary school students: Teachers’ perspectives and challenges. 3L: The Southeast Asian Journal of English Language Studies, 28(3), 217–230. https://doi.org/10.17576/3L-2022-2803-14
Baharuddin, L. (2024). Implementasi Metode Pembelajaran Berbasis Proyek dalam Pembelajaran Bahasa Inggris di MAN 1 Ternate. JUANGA: Jurnal Agama dan Ilmu Pengetahuan, 10(1), 37–48. https://www.e-jurnal.staibabussalamsula.ac.id/index.php/JUANGA/article/view/140
Boral, K., Mandal, S. K., Kumbhakar, M., & Dutta, S. K. (2025). Analysis of higher order thinking skills of intended mathematics learning outcomes in India. MIER Journal of Educational Studies Trends and Practices, 15(1), 146–176. https://doi.org/10.52634/mier/2025/v15/i1/2856
Brookhart, S. M. (2010). How to assess higher-order thinking skills in your classroom. ASCD.
Cook, D. A., & Hatala, R. (2016). Validation of educational assessments: A primer for simulation and beyond. Advances in Simulation, 1, 31. https://doi.org/10.1186/s41077-016-0033-y
Dewi, H. H., Damio, S. M., & Sukarno. (2023). Item analysis of reading comprehension questions for English proficiency test using Rasch model. REID (Research and Evaluation in Education), 9(1), 24–36. https://doi.org/10.21831/reid.v9i1.53514
Effendi, T., & Mayuni, I. (2022). Examining teacher-made English test in a language school. LADU: Journal of Languages and Education, 2(2), 67–76. https://doi.org/10.56724/ladu.v2i2.109
Emmiyati, N., Asnur, S. M., & Ahmad. (2025). Validity, reliability, and practicality of test items in an internet-based English proficiency test for English laboratory. IJEE (Indonesian Journal of English Education), 11(2), 197–206. https://doi.org/10.15408/ijee.v11i2.37276
Firdaus, Yusuf, Y. Q., Samad, I. A., & Delavari, H. (2025). Enhancing speaking skills in higher education through Canvas LMS: A study in Banda Aceh. English Review: Journal of English Education, 13(2), 407–420. https://doi.org/10.25134/erjee.v13i2.11561
Fitrawati, & Safitri, D. (2021). Students’ grammatical errors in essay writing: A pedagogical grammar reflection. International Journal of Language Education, 5(2), 74–88. https://doi.org/10.26858/ijole.v5i2.15241
Fitriani, N., Rahmadewi, S., Nurhadi, & Shuhada. (2025). Development and role of measurement tools in educational evaluation. JMPI: Jurnal Manajemen, Pendidikan dan Pemikiran Islam, 3(1), 78–97. https://doi.org/10.71305/jmpi.v3i1.233
Fitriani, R., Iskandar, I., & Suseno, M. (2023). The implementation of higher order thinking skills (HOTS) in English language teaching: A case of Indonesian senior high school EFL teachers. Didaktika: Jurnal Kependidikan, 12(4), 739–750. https://doi.org/10.58230/27454312.303
Gebremariam, H. T., & Gedamu, A. D. (2023). Primary school teachers’ assessment for learning practice for students’ learning improvement. Frontiers in Education, 8, 1145195. https://doi.org/10.3389/feduc.2023.1145195
Hartanti, T. P. (2022). Analysis of students’ speaking skill at ninth grade in Junior High School 2 Jambi City [Undergraduate thesis, Batanghari University]. http://repository.unbari.ac.id/1959/
Im, G.-H., Shin, D., & Cheng, L. (2019). Critical review of validation models and practices in language testing: Their limitations and future directions for validation research. Language Testing in Asia, 9, 14. https://doi.org/10.1186/s40468-019-0089-4
Jaya, S., Kusmiarti, R., & Paulina, Y. (2026). Developing critical reading behaviors through guided close reading of research articles. Indonesian Journal of Educational Development, 6(4), 1569–1587. https://doi.org/10.59672/ijed.v6i4.5698
Jismulatif, Dahnilsyah, Delfi, S., & Purwanti, I. T. (2022). The representation of foreign and Indonesian cultures in English textbooks for junior high school in Indonesia. Al-Ishlah: Jurnal Pendidikan, 14(3), 4217–4230. https://doi.org/10.35445/alishlah.v14i1.2016
Kaçar, K. (2023). Commonly used techniques in testing foreign language skills and communicative language teaching. Sinerji Uluslararası Alan Eğitimi Araştırmaları Dergisi, 4(2), 68–85. https://doi.org/10.54971/synergy.1381532
Karim, S. A., Sudiro, S., & Sakinah, S. (2021). Utilizing test items analysis to examine the level of difficulty and discriminating power in a teacher-made test. EduLite: Journal of English Education, Literature and Culture, 6(2), 256–269. https://doi.org/10.30659/e.6.2.256-269
Khoirunnisa, A., & Widodo, E. (2019). Students’ difficulties in comprehending narrative text. TELL: Teaching of English Language and Literature Journal, 7(2), 65–74. https://doi.org/10.30651/tell.v7i2.3441
Lee, H. L. J. (2012). SMARTies: Using a board game in the English classroom for edutainment and assessment. Malaysian Journal of ELT Research, 8(1), 1–35. https://meltajournals.com/index.php/majer/article/view/616
Mutiah, S. D. (2023). English teachers’ assessment literacy and online formative assessment implementation in teaching writing: A mixed-method study [Thesis, UIN Syarif Hidayatullah Jakarta]. https://repository.uinjkt.ac.id/dspace/handle/123456789/66568
Putriani, Ilham, Rahmaniah, R., & Humaira. (2025). The effect of peer feedback on English writing skills among EFL students. ELTIN Journal: Journal of English Language Teaching in Indonesia, 13(2), 433–442. https://e-journal.stkipsiliwangi.ac.id/index.php/eltin/article/view/5991
Ripoll Y Schmitz, L. M., & Sonnleitner, P. (2025). Evaluating AI-generated vs. human-written reading comprehension passages: An expert SWOT analysis and comparative study for an educational large-scale assessment. Large-Scale Assessments in Education, 13, 20. https://doi.org/10.1186/s40536-025-00255-w
Saputra, H. T., & Pujiati, H. (2021). Cognitive domain of revised Bloom taxonomy in English student book. JELT: Journal of English Language and Literature Teaching, 6(1), 61–66. http://ejournal.unima.ac.id/index.php/jellt/article/view/2816
Säuberli, A., & Clematide, S. (2024). Automatic generation and evaluation of reading comprehension test items with large language models. Proceedings of the Third Workshop on Tools and Resources for People with REAding DIfficulties, 22–37. https://aclanthology.org/2024.readi-1.3
Suharyadi, & Kusumaningrum, S. R. (2025). Linking item analysis to teaching and learning: What English test data tell us about teachers’ competencies. Jurnal Pengabdian Masyarakat dan Riset Pendidikan, 4(2), 9109–9116. https://doi.org/10.31004/jerkin.v4i2.3116
Sulistio, P. H. (2023). Language assessment literacy: Cases of pre-service English teachers. IJLECR: International Journal of Language Education and Cultural Review, 9(1), 1–10. https://doi.org/10.21009/ijlecr.v9i1.33032
Sürücü, L., & Maslakçı, A. (2020). Validity and reliability in quantitative research. Business & Management Studies: An International Journal, 8(3), 2694–2726. https://doi.org/10.15295/bmij.v8i3.1540
Syukur, B. A., & Nurlaily, A. F. (2025). Rasch model-based evaluation of TOEFL listening items: Analyzing difficulty, discrimination, and fit. Jurnal Smart, 11(2), 176–191. https://doi.org/10.52657/js.v11i2.2931
Tarigan, A. F. A., Simanjuntak, P. H. T. M., Sitopu, C. D., Purba, J. H., & Silalahi, D. E. (2025). The implementation of English language assessment based on language assessment and evaluation principles at senior high school. Jurnal Pendidikan Bahasa, 15(3), 447–454. https://doi.org/10.37630/jpb.v15i3.3412
Tavakol, M., & Wetzel, A. (2020). Factor analysis: A means for theory and instrument development in support of construct validity. International Journal of Medical Education, 11, 245–247. https://doi.org/10.5116/ijme.5f96.0f4a
Winna, & Sabarun. (2023). The language assessment in teaching-learning English. DIAJAR: Jurnal Pendidikan dan Pembelajaran, 2(4), 413–419. https://doi.org/10.54259/diajar.v2i4.1894
Wolf, M. K. (Ed.). (2020). Assessing English language proficiency in U.S. K–12 schools. Routledge. https://doi.org/10.4324/9780429491689
Zhao, C. (2023). Theorizing the language ability and language assessment. US-China Foreign Language, 21(7), 261–266. https://doi.org/10.17265/1539-8080/2023.07.002