A Critique of Kodaganallur, Weitz and Rosenthal, "A Comparison of Model-Tracing and Constraint-Based Intelligent Tutoring Paradigms"
In IJAIED
16 (3)
Publication information
Abstract
Research on intelligent tutoring systems (ITS) has produced many systems, but only a handful of
design principles. From its start in the late 1970s, researchers have recognized that design principles should ideally be derived from psychological insights into the cognitive processes underlying the acquisition of cognitive skills, but attempts to base design philosophies on psychological principles are still few and far between (Ohlsson, 1991). Two such philosophies have come to be associated with the labels model-tracing (MT), which is based on the ACT-R theory of human cognition (Anderson, 2005; Anderson & Lebiere, 1998), and constraint-based modelling (CBM), which has grown out of our own work on learning (Ohlsson, 1993; 1996a; 1996b; Ohlsson & Rees, 1991). Given two design philosophies, it is useful to conduct systematic comparisons to ascertain the importance and implications of their differences.
In a recent IJAIED article, Kodaganallur, Weitz and Rosenthal (2005), henceforth referred to as
KWR, undertake a comparison between MT and CBM. They built two tutoring systems for the same
domain, one based on MT and one on CBM, and they report their observations and reflections with
respect to a catalogue of issues. For every issue but one they find some weakness or potential problem with the CBM approach and also some reason for believing that any corresponding or related problem with MT will not be too difficult to overcome. They conclude that CBM tutors are "less resource intensive" to build than MT tutors, but also that MT has a wider range of application ("an MTT can be built for every domain in which a CBMT can be built, but the reverse doesn't hold") and that "the remediation provided by an MTT will be superior", the more so the more complex the learner's task (p. 141). They offer these conclusions with the purpose of providing "guidance for others interested in building intelligent tutoring systems" (p. 118).
ITS researchers should beware of their guidance, because there are several problems with their
paper. The most serious is that KWR have multiple misconceptions regarding CBM which led them to
make suboptimal choices in the implementation of their CBM system. As a result, several of their
conclusions are wrong. Their methodology has multiple flaws. They draw sweeping conclusions about
the two ITS paradigms on the basis of a limited system-building effort. The domain model for their
CBM tutor contains 43 constraints (p. 128) and the model for their MT tutor contains 76 rules (p. 137).
In comparison, SQL-Tutor, a constraint-based system, contains more than 700 constraints, and the
published figures for the expert modules in MT tutors for topics like geometry and programming are in excess of 400 rules. Because strengths and weaknesses of system architectures tend to be exacerbated with increased system complexity, general conclusions based on KWR's two toy systems have the potential to mislead the field of ITS research.
In the following, we group the flaws in their paper into three sections. We first address KWR's
misconceptions regarding the constraint representation that is the core of the CBM approach and the
suboptimal implementation decisions these misconceptions caused. In the following two sections, we
discuss their conclusions with respect to the range of application and remediation. We then critique
how they conducted their comparison. We end with some reflections on how this sort of comparison
ought to be conducted.
