Journal

A Data-Based Simulation Study of Reliability for an Adaptive Assessment Based on Knowledge Space Theory

Publication Information

Authors:

  • Christopher Doble, McGraw-Hill Education/ALEKS Corporation
  • Jeffrey Matayoshi, McGraw-Hill Education/ALEKS Corporation
  • Eric Cosyn, McGraw-Hill Education/ALEKS Corporation
  • Hasan Uzun, McGraw-Hill Education/ALEKS Corporation
  • Arash Karami, McGraw-Hill Education/ALEKS Corporation

Pages:

  • 258-282

Keywords:

  • Knowledge space theory, Simulation, Reliability, Adaptive assessment

Abstract:

  • A large-scale simulation study of the assessment effectiveness of a particular instantiation of knowledge space theory is described. In this study, data from more than 700,000 actual assessments in mathematics using the ALEKS (Assessment and LEarning in Knowledge Spaces) software were used to determine response probabilities for the same number of simulated assessments, for the purpose of examining reliability. Several measures of reliability were examined, borrowed from existing psychometric approaches, with an eye toward developing measures for evaluating reliability for adaptive assessments. The results are compared to analogous results for assessments having mathematics content overlapping that of the ALEKS assessment, and some consequences and future directions are discussed.