Dartmouth AI Tutor Hits 0.71-1.30 SD Effect Size — 90% Voluntary Uptake

A new AI-powered interactive textbook platform called Phosphor, deployed across 151 students in Dartmouth College’s Introductory Statistics course, achieved a final-exam effect size of between 0.71 and 1.30 standard deviations — large by any observational standard for educational interventions. The platform’s adoption rate was 90.2 per cent voluntary, ungraded, against a student-reported reading-compliance baseline of 10-15 per cent. The paper, presented at the iTextbooks’26 Workshop in Seoul on June 28, is one of the first rigorous deployments of an LLM-graded formative assessment system in a real university course.

🔍 THE BOTTOM LINE

The study’s central finding is not that “AI helps students learn.” It is more specific and more useful than that: constructed-response questions graded by Claude Sonnet 4.6 against rubric criteria drove the learning effect, while multiple-choice-only quizzes produced negligible dosage-to-performance relationships. When the platform switched from CRQ to MCQ-only for Module 2, the dosage slope went slightly negative. When CRQ was re-introduced for Module 3, the cumulative signal returned. The format that demands active generation is the format that works. AI makes grading that format at scale feasible for the first time.

What Phosphor Actually Does

Phosphor, developed by Jonah Bard at Dartmouth, embeds LLM-graded formative assessment directly into instructional content. Each lesson includes a bank of 15-20 exercises. Students take quizzes of four randomly selected questions. Multiple-choice questions are auto-graded; constructed-response questions are graded by Claude Sonnet 4.6 against instructor-defined, question-specific rubric criteria. The grading prompt receives the student’s response alongside the question text, a model answer, and explicit grading criteria, and returns a correctness judgment with an explanation.

The test bank consists of 40 per cent CRQ and 60 per cent MCQ. Content is not gated — students may freely read and take quizzes regardless of past results. Quizzes permit unlimited retries. The platform also includes a RAG-based chat sidebar allowing students to ask questions while reading, with responses restricted to the boundaries of the course curriculum.

The Numbers That Matter

The deployment covered three sections of MATH 010 (Introductory Statistics) at Dartmouth in Spring 2026. Starting at 151 students, finishing at 143. The platform was presented as entirely optional, ungraded — an alternative to traditional course readings.

The results, as reported in the paper:

90.2 per cent of enrolled students engaged with the platform at least once
48.1 per cent aggregate quiz completion rate (lower bound on reading, since passing assumes having read the lesson)
75.6 per cent of student-lesson pairs reached (upper bound on reading compliance)
Against a 10-15 per cent student- and instructor-reported baseline reading compliance for this course
Students who passed all three Module Reviews scored 7.1 points higher on the Final (Cohen’s d = 0.66, p < 0.0001)
The Tobit model places the gap between full and zero engagement at 14.7 points on a 0-100 scale (1.30 SD) before adjustment
Controlling for prior achievement via midterm scores, the gap attenuates to 8.0 points (0.71 SD)

Why Constructed Response Is the Active Ingredient

The study’s most methodologically interesting finding comes from a natural variation in quiz design between modules. Module 1 used mixed MCQ + CRQ. Module 2, in response to student feedback that the CRQ auto-grader was “rigid and discouraging,” went MCQ-only. Module 3 restored CRQ after exam analysis suggested MCQ-only quizzes provided negligible learning benefits.

The results are striking. For Module 1, each additional lesson completion was associated with roughly 1.6 additional percentage points on the first midterm (p < 0.001, R² = 0.123). For Module 2 (MCQ-only), the apparent positive slope among all users reflects only the zero-versus-nonzero distinction — among students with at least one completion, the slope is slightly negative (R² = 0.001), indicating no dosage relationship. For the cumulative Final, each completion is associated with about 0.4 additional points (R² = 0.091), essentially unchanged when zero-completion students are excluded.

This is consistent with the testing-effect literature: Kang et al. (2007) found that short-answer quizzes with feedback produced stronger retention than multiple-choice quizzes (d = 0.41). The Dartmouth study’s contribution is showing that LLM grading makes this format feasible at scale — a capability that appears pedagogically significant rather than merely convenient.

The Engagement Paradox

The engagement numbers challenge a common assumption in AI-education discourse. Bastani et al. (2025), in a randomised controlled trial with nearly 1,000 students, demonstrated that unfettered access to GPT-4 actually harmed subsequent performance by 17 per cent when the tool was removed — students used it as a crutch rather than a learning aid. Only a version with carefully designed pedagogical guardrails mitigated these negative effects. A 2026 HEPI survey found that 94 per cent of university students reported using generative AI on assessed work, up from 53 per cent two years earlier.

Phosphor’s design responds to this: AI is integrated into the content delivery system, not offered as an external chat tool. The RAG chat assistant saw minimal usage — only 72 total queries, with only 14 students submitting more than one. Students reported that general-purpose LLMs were faster and more capable, and that the reference content was “sufficient” such that they did not generate enough questions during reading to justify a separate chat interface. This is consistent with Khan Academy’s report that only 15 per cent of users regularly engage with their supplementary chatbot. Integrating AI through assessment, feedback, and progress tracking — not chat — appears to be the more productive design.

What the Study Cannot Claim

The author is explicit about the limitations. This is an observational study of a pilot deployment at a single selective institution, and lacks randomised controls. Self-selection is the central threat: students who complete more quizzes may be more motivated or higher-performing generally. The joint Tobit models control for prior achievement, generating a lower and upper bound, but the cross-module lesson contrast remains confounded by content domain, timing, and the simultaneous introduction of the Module Review. The all-reviews-passed group is the most self-selected in the study.

The 0.71 SD figure should be read as a conservative lower bound — likely over-adjusted, because the cumulative Final re-tests content already assessed at the midterms, so the midterm control absorbs learning that Phosphor itself may have produced earlier in the term. The 1.30 SD figure is selection-inflated. The true effect lies somewhere between. Even at the conservative bound, this is a large effect for an educational intervention.

❓ FAQ

What is Phosphor and who built it?

Phosphor is a digital learning platform that integrates LLM-graded formative assessment directly into instructional content. It was developed by Jonah Bard at Dartmouth College and deployed across three sections of MATH 010 (Introductory Statistics) in Spring 2026. The platform embeds AI-graded quizzes into the reading workflow, making active recall a structural feature of the learning experience. Constructed-response questions are graded by Claude Sonnet 4.6 against instructor-defined rubric criteria.

What is a 0.71-1.30 SD effect size in practical terms?

One standard deviation on the Final Exam was 2.72 points on the 24-point scale. An effect size of 0.71 SD means full-engagement students scored roughly 8 points higher (out of 100) than zero-engagement students after controlling for prior midterm performance. An effect size of 1.30 SD — the unadjusted figure — means roughly 14.7 points. In education research, effects above 0.40 SD are generally considered substantial; most educational technology interventions show effects in the 0.10-0.30 range.

Why did MCQ-only quizzes not work?

The study’s natural variation in quiz format revealed that lesson-level dosage tracked exam performance under constructed-response quizzes but not under multiple-choice quizzes. When Module 2 switched to MCQ-only, the dosage-to-performance slope went slightly negative (R² = 0.001). When CRQ was re-introduced in Module 3, the cumulative signal returned. This is consistent with the testing-effect literature: short-answer questions with feedback produce stronger retention than multiple-choice because they demand active generation, not recognition.

Could this work in New Zealand universities?

Yes, and the implications are direct. The platform runs as a web application, the LLM grading uses a standard API (Claude Sonnet 4.6), and the content was custom-authored grounded in open educational resources. The barrier is not technical — it is curricular. Any institution willing to author lesson-level quiz banks with rubric criteria for CRQ grading could deploy a similar system. The study’s author plans to deploy Phosphor with a completion requirement attached to course grade, which will increase engagement and provide cleaner dosage-performance measurement. NZ universities piloting AI-augmented learning could replicate the design without building new infrastructure.

🔍 THE BOTTOM LINE

The Dartmouth Phosphor study is the first rigorous deployment of an LLM-graded formative assessment system in a real university course, and the results are large enough to demand attention: 0.71-1.30 SD on final exam performance, 90 per cent voluntary adoption against a 10-15 per cent baseline. But the study’s most important finding is not the headline effect size. It is that the format of assessment — constructed-response, not multiple-choice — is the active ingredient. AI makes grading that format at scale feasible for the first time. That is the structural change. Everything else is implementation.