- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to train language models that can solve problems into mathematics teachers that can adapt to the needs of different students. SHERPA builds student archetypes on top of a weak, frozen student backbone model, using a guidance gate that blocks disclosure of the final answer and an adaptive gate that enforces a designated instructional preference. The teacher goes through preparation, multi-turn tutoring, and testing on the same problem, and is optimized with reinforcement learning on the difference between the student's probability of answering correctly before and after tutoring. Training covers four archetypes, while three additional preferences are held out for out-of-distribution evaluation. Rejected teacher messages never enter the student's context, but the corresponding feedback is retained in the teacher's history; only after passing both gates is an instruction forwarded to the student model for substantive reasoning. The reward is masked at the turn level according to the gate outcomes, and advantages are constructed through leave-one-out centering.
Critical strengths
-
Well-positioned problem with solid motivation. "Being able to solve a problem ≠ being able to teach it" is a real and important issue, and the authors shift the training signal from "predefined criteria of good teaching" to "students' learning outcomes."
-
The gated student archetype design is clever and well-argued. The student-side gating makes preferences consequential for the interaction while preserving the weak student backbone's actual problem-solving process, resulting in a controllable training environment.
-
The benefits are validated across a reasonably broad range of settings, and the evidence does not rely solely on simulated students from the same model family.
Critical weaknesses
-
The measure of "learning outcomes" is closer to "answer transmission" than to transferable learning. This is the paper's most central methodological concern. At test time, the student is given the original problem together with the tutoring dialogue history and asked to solve it again, and the reward is computed against a no-history baseline. This means the reward measures "whether the student can re-solve the same problem with the dialogue in context" — essentially the transmission of that problem's solution — rather than the ability to solve new problems independently without any dialogue history.
-
What the paper empirically demonstrates is mainly the ability to "respond to explicit instructional requests," which remains a considerable distance from real-world settings, where students often cannot articulate their own needs and may even be unaware of their preferences.
-
The reliability of the gating and judging LLMs is not analyzed. (a) The teacher may overfit to the idiosyncrasies of the 8B judge (reward hacking against the gates), yet the paper contains no analysis of agreement between gate judgments and human judgments; (b) the impact of the train/eval judge mismatch on the direction of the results is not discussed. In addition, the guidance gate's prompt is clearly designed for numeric or computable-expression answers; how non-numeric answers in MATH (e.g., intervals, symbolic expressions) are handled is not explained, and the distribution of answer types among the problems retained after filtering is not reported.
-
Baselines and controls could be strengthened. (a) The only RL baseline is the authors' re-implementation of PedagogicalRL on AReaL following the original paper, rather than an official checkpoint, and concurrent work such as PEARL is compared only in prose, without experimental comparison; (b) the human evaluation compares only against the untrained base model, with no pairwise comparisons against PedagogicalRL or stronger general-purpose models (e.g., Gemini).
Questions for authors
-
Across the student-improvement measurements, in what fraction of trajectories has the correct answer (or an equivalent expression) already appeared in the student-visible context during tutoring? Could you report the "clean" improvement after excluding such trajectories, as well as a transfer test on new problems (without dialogue history)?
-
If the preference-specific feedback pools used in training were replaced with bare complaints that do not name the strategy, how much would SHERPA_DP's results change across the board? Has the teacher genuinely learned to probe actively (e.g., asking exploratory questions first) rather than waiting for explicit instructions?
-
What is the agreement rate between the gate judge (8B) and human judgments on the six adaptive-gate criteria? Is it possible to measure the trained teacher's overfitting to the judge (e.g., by re-judging late-training trajectories with the 27B judge and checking whether the pass rate drops)?
-
Please clarify how the guidance gate handles non-numeric answers, and report the distribution of answer types in the filtered training and evaluation sets.
6 comments
Response to Reviewer 8Qr7 (2/3)
W3. Reliability of the gating and judging LLMs
Thanks for the detailed comments.
(a) To check whether the teacher overfits to the training judge, we re-judged the training trajectories with the 27B judge using the same gate requests. From the first 200 to the last 200 training steps, the 8B pass rate rises from 41.2% to 64.9%, and the 27B pass rate also rises, from 50.9% to 57.7%, so the gain is not limited to the training judge. Training against a judge is expected to induce some fitting to it, so we use the held-out 27B judge for all reported results to avoid this bias. On 120 human-annotated gate decisions covering all six criteria, the 27B judge agrees with the human labels on 82.4% of decisions (κ = 0.65) and the 8B judge on 76.4% (κ = 0.52); see Q3 for details.
(b) The train/eval judge mismatch is intentional: we use a different judge for evaluation to avoid the bias of using the same judge for both training and evaluation. Under the 8B judge, SHERPA_DP reaches 72.2% accuracy versus 47.8% for the base model; under the 27B judge, it reaches 69.0% versus 48.6%. SHERPA_DP outperforms the base model on all seven archetypes under both judges, so the direction of the results is consistent. For how the guidance gate handles non-numeric answers and the answer-type distribution, please see Q4.
W4. Baselines and human evaluation
Thanks for the suggestions.
(a) The official PedagogicalRL checkpoint (TutorRL-7B) is built on Qwen2.5-7B-Instruct, so we re-implement PedagogicalRL on Qwen3-8B to compare with SHERPA on the same backbone. We additionally evaluate the official checkpoint in our student environment; its MathTutorBench result is the one reported in its paper.
| Teacher | Backbone | Student accuracy | MathTutorBench Ped. Avg. |
|---|---|---|---|
| TutorRL-7B (official) | Qwen2.5-7B-Instruct | 40.4 | 59.3 |
| PedagogicalRL (ours) | Qwen3-8B | 52.7 | 63.2 |
| SHERPA_DP | Qwen3-8B | 69.0 | 79.2 |
We discuss PEARL but do not reproduce it because it has not released its code or checkpoints.
(b) With our limited budget, we recruit 96 teachers for 960 judgments and focus the human evaluation on comparing SHERPA_DP with its base model to assess whether SHERPA training improves teaching quality as judged by human teachers. PedagogicalRL and Gemini 3.8 Flash are compared in our automated evaluations (Tables 1, 2, 5 and 6).
Response to Reviewer 8Qr7 (3/3)
Q1. Answer leakage and performance without dialogue history
Thanks for the question. The guidance gate blocks teacher messages that reveal the answer, so those messages never enter the student's context. To check the messages that pass the gate, we use a GPT model to audit every teacher message visible to the student and flag those that state the final answer or an equivalent expression. We count only answers given by the teacher, since the student working out the answer during tutoring is the intended outcome. The table reports the share of dialogues containing such a message, and the results after excluding these dialogues:
| Teacher | Dialogues with leaked answer | Improvement (all → clean) | Accuracy (all → clean) |
|---|---|---|---|
| SHERPA_DP | 4.5% | 33.1 → 32.7 | 69.0 → 68.6 |
| Base | 4.1% | 12.4 → 11.6 | 48.6 → 47.5 |
| SHERPA_NP | 1.7% | 16.6 → 16.3 | 52.0 → 51.5 |
| PedagogicalRL | 5.9% | 16.8 → 15.7 | 52.7 → 51.2 |
The results barely change after excluding these dialogues. For the transfer test, since the student model has no memory beyond its context, testing without any dialogue history would reduce to the no-teaching baseline. We therefore follow EduardoRL's approach of masking all teacher messages in the dialogue history, so that students rely only on their own reasoning to solve transfer tasks. As reported in W1, SHERPA still improves student accuracy on new problems by 15.8 points in this setting, compared with 5.8 points for the base model.
Q2. Implicit feedback and active probing
Thanks for the question. We replace the preference-specific feedback with replies in which students only describe where they are stuck, without naming the teaching strategy (see W2). Across three runs with different reply wordings, SHERPA_DP outperforms the base model by 21.9 points on average, close to its 24.4-point advantage with the original feedback. Regarding active probing, our setting does not require the teacher to ask exploratory questions first. The teacher teaches normally and adapts when the student is not satisfied with its guidance. This is similar to a real teacher who starts with a default teaching style and adjusts it when the student does not understand.
Q3. Agreement with human judgments and overfitting to the judge
Thanks for the question.
(a) Human agreement. We annotated 120 gate decisions from the main evaluation, stratified over the six criteria, SHERPA$_\text{DP}$ versus the base model, and the judge's verdict. Reweighted to the full evaluation, the 8B training judge agrees with the human labels on 76.4% of decisions (κ = 0.52), and the 27B evaluation judge on 82.4% (κ = 0.65).
(b) Overfitting. We re-judged the gate requests from the first and last 200 training steps (about 6,960 turns each) with the 27B judge. The 27B pass rate rises on all three training preferences, and overall from 50.9% to 57.7% (8B: 41.2% → 64.9%). Agreement between the 8B and 27B judges also slightly increases (73.2% → 76.8%, κ 0.47 → 0.51). This measurement shows limited overfitting of the teacher to the 8B judge, and we report all results in the paper with the 27B judge for fairness.
Q4. Non-numeric answers and answer-type distribution
Thanks for the question. The guidance gate uses an LLM judge that compares the ground-truth answer with the teacher message. It is required to compute and normalize numerical expressions, so the comparison goes beyond exact string matching. In our evaluation, the judge also blocks leaks involving non-numeric answers. For example, for SHERPA_DP, it blocks leaks in 26 of the 70 dialogues with interval answers and in 28 of the 84 dialogues with matrix answers. We analyzed the answer types in the filtered training and evaluation sets and found that non-numeric answers account for 13.7% and 15.2%, respectively. The detailed distribution is:
| Answer type | Test (528) | Train (759) |
|---|---|---|
| Integer | 61.7% | 62.2% |
| Fraction | 15.7% | 14.5% |
| Decimal | 1.5% | 2.1% |
| Closed-form number (radicals, π, etc.) | 5.9% | 7.5% |
| List / set | 3.2% | 1.7% |
| Text | 2.7% | 1.7% |
| Expression / equation | 2.5% | 3.2% |
| Matrix / vector | 2.3% | 1.3% |
| Tuple (a, b) | 2.1% | 1.8% |
| Interval | 1.9% | 2.6% |
| Other (base-n numbers, ratios) | 0.6% | 1.3% |
We will add this clarification and the distribution to the revised paper.
Response to Reviewer 8Qr7 (1/3)
We thank reviewer 8Qr7 for recognizing our motivation, student archetype design, and broad empirical validation.
W1. Performance on related problems
Thanks for raising this important point. SHERPA improves students' transferable learning rather than superficial answer transmission. To validate this, we additionally evaluate students on unseen problems after tutoring, following EduardoRL (Macina et al., 2026). These problems are generated by an LLM with different numbers or contexts and different final answers, while preserving the concepts, solution steps and difficulty of the original problems. Since the student model does not retain information beyond its context, removing the dialogue entirely would not capture the effect of tutoring. We therefore follow their setting of masking all teacher messages in the dialogue history to test whether the improvement persists even when students rely only on their own reasoning to solve transfer tasks. The table shows how much student accuracy on transfer problems improves after tutoring, with the full history (full) and with teacher messages masked (teacher masked).
On unseen variants, SHERPA outperforms the base model by 14.2 points with the full dialogue and by 10.0 points with the teacher turns masked. This shows that the gains go beyond re-solving the same problem. In addition, as reported in Q1, the teacher states the final answer in only 4.5% of SHERPA’s dialogues, and excluding these dialogues barely changes the results. We will add these results to the revised paper.
W2. Inferring students’ needs from implicit feedback
Thanks for the insightful comment. Student simulation is itself an active research area. We build our students on established LLM-based simulation and further design the gates to make them behave more realistically. We agree that real students often cannot articulate their own needs. To address this concern, we additionally run an evaluation in which explicit requests for a teaching strategy are replaced with replies where students only describe their confusion. To test robustness to wording, we ran the evaluation three times, each with a different set of implicit replies. Here we use the 8B judge to reduce cost, with all other settings unchanged.
SHERPA’s overall advantage over the base model is 24.4 percentage points with explicit feedback and 21.9 points on average with implicit feedback. Most of the gain is retained across different wordings, suggesting that SHERPA can adapt to needs expressed implicitly rather than relying solely on explicit requests for a teaching strategy. Students who are unaware of their own preferences remain an important next step, which we discuss as future work in Sec. 5. We will add these results to the revised paper.