Summary: A formative evaluation of AnimalWatch
, Tom Murray1,3
, Joseph E. Beck2
, Beverly P. Woolf1
, Carole R. Beal1
Universityof Massachusetts, Amherst- 2
Carnegie Mellon University - 3
Abstract. We present the results of a deep formative evaluation of AnimalWatch, an
intelligent tutor for arithmetic. Students learned with AnimalWatch, and had a positive
experience with it. Still, we detected AnimalWatch selected too easy problems. We
proposed different hypotheses for this behavior and analyzed the performance of each
ITS component, by observing the interactions of hundreds of students with the system.
We conclude with a report of lessons learned that would make ITS better.
Evaluating Intelligent Tutoring Systems (ITS) is not an easy task due to their dynamic
nature. Most quantitative ITS evaluations in the past consisted of pre to post test
improvements (Koedinger, 97; Luckin, 99]. These evaluations are appropriate for ITS in