A pilot study on the development and validation of AI literacy test items for grade 7 to grade 9 students
Abstract
To provide K-12 students with an understanding of AI’s potential opportunities and challenges, AI education has been incorporated into the technology curriculum. Most existing methods for assessing students’ AI literacy rely on self-assessment questionnaires, which may not offer an objective measure of their understanding. This study, therefore, sought to develop and validate a 25 -item multiple-choice test tailored for Hong Kong secondary school students to gauge their AI literacy. A total of 144 students from six secondary schools participated in a pilot test. Item Response Theory (IRT) incorporating a Markov Chain Monte Carlo (MCMC) algorithm, was developed to estimate the difficulty and discrimination indices of test items. The 3-Parameter Logistic (3-PL) model was selected for its effectiveness in providing a more nuanced understanding of item characteristics. Preliminary analysis suggested that the test warrants further revision and enhancement, as evidenced by a Kuder-Richardson Formula 20 (KR-20) reliability coefficient of 0.68. Nonetheless, all test items exhibited satisfactory discrimination capabilities, indicating their proficiency in differentiating among varying levels of respondent aptitude. Thus, this study can provide a foundation for developing a standardized AI literacy test in the future.
Authors: Yifan Chen, King Woon Yau, Ching Sing Chai, Thomas K. F. Chiu, Helen M. L. Meng, Irwin King, Savio W.H. Wong, Yeung Yam
Published in: International Symposium on Educational Technology (2024)