- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper asks what runtime supervision adds to an LLM tutor beyond prompting. In the supervised setup, an LLM classifier labels each proposed tutor action; fixed conversation-dependent rules check it alongside deterministic citation and audit checks, and a failure triggers one revision. That revision is released even if violations remain.
The study uses two tutors from one developer (gpt-5.6-terra and gpt-oss-120b), 20 conceptual statistics scenarios with scripted learners, 1,440 dialogues, and 800 stability probes. A three-judge cross-family panel scores four TEAS dimensions. A planted-error examination (520 items) and a blind TA sample decide which dimensions are confirmatory.
Critical strengths
- Clean isolation of the runtime contribution. Matching initial prompts across U/W (all 560 hashes verified) and running the full B0-P-U-W ladder separates what prompting, context, and supervision each contribute.
- The resulting finding is practically useful: instructions and context account for most of the measurable gain, while supervision mainly enforces structural compliance at a substantial cost.
- The planted-error examination uses harmless-edit controls, unfavorable treatment of missing responses, and a frozen prompt-repair round. Failure demotes a dimension to descriptive status. This is a meaningful methodological contribution for LLM-judge-based tutoring evaluation.
Critical weaknesses
- The central question is unanswered. The pooled W–U pedagogy comparison uses only 11 of 20 scenarios after complete-case dropping.
- The score is a coarse integer built from nested majority votes. As a result, 14 of 22 model/scenario contributions are ties.
- Limited novelty. Supervised generation with regeneration already exists (CodeHelp, MWPTutor, Kadir 2026, Pisan 2026). The novelty therefore rests on the controlled comparison and judge examination rather than the architecture.
- Scope is narrow: scripted learners, two tutors from one developer, 20 conceptual scenarios in one domain, no learning outcomes, and a shared runtime/evaluation classifier model.
Questions for authors
- The analysis plan was “deposited externally after execution and before results were inspected.” What was registered before data collection, and what motivated the amended family?
- On the evaluation sample, how often does the runtime gate fail to flag replies that the judges or humans mark as ordering violations, particularly explain-then-check turns?