- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Eduardo trains language-model tutors with multi-turn RL against a frozen student. Its learning signal compares a measured cold solve rate with performance on a near-transfer problem after tutoring, retaining only the student's turns in the post-test context. Separate factual-correctness and no-handover gates, plus an efficiency discount, shape the trajectory reward. The paper constructs 8,671 source/variant pairs and trains 4B–27B tutors across two architectural families. Evaluation combines held-out simulated gains with MathDial/Eedi responses, MathTutorBench, and TutorMoments using different evaluation models. The results show useful pedagogical-score improvements and lower thinking-token use, while the 4B leave-one-out study suggests that the gates, transfer test, and masking influence different behaviors. The paper also reports limitations, including leakage remaining substantial for smaller tutors and loss of support-fading behavior.
Critical strengths
- A concrete, inspectable objective addressing an important reward-design failure. Sections 3.3–3.6 replace same-problem/full-dialog evaluation with masked near transfer and explicitly separate gain, behavioral gates, and efficiency. The no-gates ablation is especially informative: leakage rises from 31% to 61% while measured gain stays similar. This demonstrates why optimizing the student proxy alone is not enough and makes the paper valuable even where the proxy fails.
- Substantial external evaluation and candid negative findings. The work spans model sizes/families and uses evaluation instruments distinct from the training judge, including an independent simulated student in TutorMoments. It reports cases without improvement, base-model truncation issues, continued leakage at small sizes, and the decline of support fading. The strong MathDial results and external rubric improvements support a useful open-model training recipe, without needing to interpret every simulator score as human learning.
Critical weaknesses
- Masking tutor turns does not guarantee that telling cannot improve the reward. Sections 3.2–3.4 and the abstract describe student-notes-only evaluation as preventing cognitive offloading. However, a student can repeat a tutor-provided method or derivation in its own turns; that material survives the mask and can help on a near-transfer variant. Masking establishes a channel restriction, not that surviving content was independently constructed. The separate leakage gate is therefore doing essential work, and similar average gains with/without the gates do not identify the causal effect of telling. The paper itself notes copying-related behavior in Section 5.2. The empirical objective remains useful, but “prevents” and “telling no longer raises the gain” should be weakened unless supported by an explicit mediation or transcript audit.
- Mechanistic conclusions are stronger than the ablation design supports. The leave-one-out study uses one seed, one size, and one checkpoint per arm, which the limitations acknowledge. The no-masking arm also has substantial cap overruns/truncation, so its longer thinking and lower scores cannot cleanly identify a pedagogical mechanism. Both gates are removed together, and every arm shares a token-efficiency term. Moreover, the fixed gate reward is −0.2 (Appendix B), while valid negative gain can in principle be lower; this is a finite penalty, not a hard feasibility constraint. These details do not invalidate the recipe but qualify the claimed necessity and independence of its components. Existing bootstrap intervals over problems should not be interpreted as variation across training runs.
Questions for authors
- Clarification: How should the no-offloading claim be interpreted when students repeat tutor-supplied reasoning in their own turns? Do existing transcripts/audits distinguish reproduction from independent construction?
- Clarification: Was each ablation checkpoint selected by the same validation rule, and how much of the no-masking deficit remains in completed, non-overrun dialogs according to existing logs?
- Clarification: Since a gate failure receives −0.2 whereas a gate-passing negative gain can be smaller, is the intended mechanism a soft penalty rather than a strict constraint? What existing reward statistics characterize this case?
- Existing-evidence clarification: Were the gates calibrated against any human annotations, and are gate disagreement/error rates available? Additional seeds would strengthen future claims but are not demanded during rebuttal.
3 comments
Thank you for your response. After reconsideration, I have decided to keep my score unchanged.
Thank you for engaging. Please see our global comment, which takes a step back to lay out the novel contributions (and the model ranks in the top 3 on an independent leaderboard on which the model is not trained directly, well above every open-weight model on the leaderboard).
Response
Thank you for your review. We agree with your precision points and will update our phrasing accordingly.
1. Channel Masking vs. Anti-Dictation Gates (Q1 & W1)
Gemini-3.1-Projudge, pedagogy scores reach 4.50/5 on the 494 held-out dialogs (Table 1). Additionally, we ran a dictation audit on 200 TutorMoments dialogs using aGemini-3.1-Projudge, which flagged 2 of 200 dialogs, with 0 cases of solution dictation upon inspection (both read problem text or a basic definition aloud: "Can you say 34.04 out loud?"; "What number is two steps left from zero?").2. Ablation Checkpoint Selection &
w/o MaskingOverruns (Q2 & W2)Eduardo-4B,w/o Gates,w/o Masking,w/o Near-Transfer) were selected by the same rule under the identical protocol in Table 4 and Appendix A.w/o Maskingstays under budget. Instead note that MathDial (n=500) and Eedi (n=500) evaluate all items at a single mid-conversation turn, where multi-turn sequence truncation cannot accumulate. Even there,w/o Maskingtrails fullEduardo-4Bsubstantially: MathDial RM 7.92→6.48 (Win 78.0) and Eedi RM 8.83→7.48 (Win 72.4), while thinking tokens increase 3.5–5.7x.3. Fixed Penalty vs. Hard Constraint (Q3)
4. Gate Design & Human Alignment (Q4)
Qwen3.6-27B).Qwen3.6-27B, we evaluate with an independent judge (Gemini-3.1-Pro), Ped-RM, and in our new Section 6 Teacher Study with 7 experienced STEM teachers, who judgedEduardo-27Blower on solution handover (2 vs. 5 rounds) and over-helping (2 vs. 6 rounds) than its base model, with few factual errors (1 vs. 2 rounds).