acceptodds
Under review as a conference paper at ICLR 2027

Sherpa: Teaching LLMs to Teach Adaptively

Abstract

Large language models (LLMs) have become increasingly capable problem solvers, but being able to solve a problem is not the same as being able to teach it. Existing approaches to training LLMs as teachers rely on demonstrations, preference data, or predefined pedagogical criteria that specify what good teaching looks like. However, these signals are often not grounded in individual student learning outcomes, where effective teaching strategies can vary substantially across learners. To address this, we introduce Sherpa, a multi-turn reinforcement learning framework that instantiates multiple student archetypes with LLMs conditioned on distinct learning preferences and trains a teacher model to adapt its instruction by directly maximizing their learning outcomes. Teacher LLMs trained with Sherpa improve instructed students' performance across all archetypes by an average of 20.5 percentage points. Under MathTutorBench's evaluation, Sherpa raises the overall pedagogy score from 52.5% to 79.2%, indicating better teaching responses. Our human studies show that the trained teacher is preferred over the base model in 79.6% of pairwise comparisons. Together, Sherpa trains LLM teachers to adapt to diverse simulated students and become better aligned with human teachers, paving the road towards AI tutors teaching real students.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, reinforcement learning, adaptive teaching, intelligent tutoring systems, student archetypes

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8Qr7OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to train language models that can solve problems into mathematics teachers that can adapt to the needs of different students. SHERPA builds student archetypes on top of a weak, frozen student backbone model, using a guidance gate that blocks disclosure of the final answer and an adaptive gate that enforces a designated instructional preference. The teacher goes through preparation, multi-turn tutoring, and testing on the same problem, and is optimized with reinforcement learning on the difference between the student's probability of answering correctly before and after tutoring. Training covers four archetypes, while three additional preferences are held out for out-of-distribution evaluation. Rejected teacher messages never enter the student's context, but the corresponding feedback is retained in the teacher's history; only after passing both gates is an instruction forwarded to the student model for substantive reasoning. The reward is masked at the turn level according to the gate outcomes, and advantages are constructed through leave-one-out centering.

Critical strengths
  1. Well-positioned problem with solid motivation. "Being able to solve a problem ≠ being able to teach it" is a real and important issue, and the authors shift the training signal from "predefined criteria of good teaching" to "students' learning outcomes."

  2. The gated student archetype design is clever and well-argued. The student-side gating makes preferences consequential for the interaction while preserving the weak student backbone's actual problem-solving process, resulting in a controllable training environment.

  3. The benefits are validated across a reasonably broad range of settings, and the evidence does not rely solely on simulated students from the same model family.

Critical weaknesses
  1. The measure of "learning outcomes" is closer to "answer transmission" than to transferable learning. This is the paper's most central methodological concern. At test time, the student is given the original problem together with the tutoring dialogue history and asked to solve it again, and the reward is computed against a no-history baseline. This means the reward measures "whether the student can re-solve the same problem with the dialogue in context" — essentially the transmission of that problem's solution — rather than the ability to solve new problems independently without any dialogue history.

  2. What the paper empirically demonstrates is mainly the ability to "respond to explicit instructional requests," which remains a considerable distance from real-world settings, where students often cannot articulate their own needs and may even be unaware of their preferences.

  3. The reliability of the gating and judging LLMs is not analyzed. (a) The teacher may overfit to the idiosyncrasies of the 8B judge (reward hacking against the gates), yet the paper contains no analysis of agreement between gate judgments and human judgments; (b) the impact of the train/eval judge mismatch on the direction of the results is not discussed. In addition, the guidance gate's prompt is clearly designed for numeric or computable-expression answers; how non-numeric answers in MATH (e.g., intervals, symbolic expressions) are handled is not explained, and the distribution of answer types among the problems retained after filtering is not reported.

  4. Baselines and controls could be strengthened. (a) The only RL baseline is the authors' re-implementation of PedagogicalRL on AReaL following the original paper, rather than an official checkpoint, and concurrent work such as PEARL is compared only in prose, without experimental comparison; (b) the human evaluation compares only against the untrained base model, with no pairwise comparisons against PedagogicalRL or stronger general-purpose models (e.g., Gemini).

Questions for authors
  1. Across the student-improvement measurements, in what fraction of trajectories has the correct answer (or an equivalent expression) already appeared in the student-visible context during tutoring? Could you report the "clean" improvement after excluding such trajectories, as well as a transfer test on new problems (without dialogue history)?

  2. If the preference-specific feedback pools used in training were replaced with bare complaints that do not name the strategy, how much would SHERPA_DP's results change across the board? Has the teacher genuinely learned to probe actively (e.g., asking exploratory questions first) rather than waiting for explicit instructions?

  3. What is the agreement rate between the gate judge (8B) and human judgments on the six adaptive-gate criteria? Is it possible to measure the trained teacher's overfitting to the judge (e.g., by re-judging late-training trajectories with the 27B judge and checking whether the pass rate drops)?

  4. Please clarify how the guidance gate handles non-numeric answers, and report the distribution of answer types in the filtered training and evaluation sets.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 8Qr7 (1/3)

We thank reviewer 8Qr7 for recognizing our motivation, student archetype design, and broad empirical validation.

W1. Performance on related problems

Thanks for raising this important point. SHERPA improves students' transferable learning rather than superficial answer transmission. To validate this, we additionally evaluate students on unseen problems after tutoring, following EduardoRL (Macina et al., 2026). These problems are generated by an LLM with different numbers or contexts and different final answers, while preserving the concepts, solution steps and difficulty of the original problems. Since the student model does not retain information beyond its context, removing the dialogue entirely would not capture the effect of tutoring. We therefore follow their setting of masking all teacher messages in the dialogue history to test whether the improvement persists even when students rely only on their own reasoning to solve transfer tasks. The table shows how much student accuracy on transfer problems improves after tutoring, with the full history (full) and with teacher messages masked (teacher masked).

SettingNon.Att.Sub.Ctr.Cau.Stp.Ver.IDOODOvl.
SHERPA_DP, variant (full)23.826.924.514.727.98.323.122.519.821.3
Base, variant (full)8.36.98.66.911.53.94.27.76.57.2
SHERPA_DP, variant (teacher masked)19.020.420.111.320.43.116.217.713.315.8
Base, variant (teacher masked)7.75.28.15.08.53.42.96.54.95.8

On unseen variants, SHERPA outperforms the base model by 14.2 points with the full dialogue and by 10.0 points with the teacher turns masked. This shows that the gains go beyond re-solving the same problem. In addition, as reported in Q1, the teacher states the final answer in only 4.5% of SHERPA’s dialogues, and excluding these dialogues barely changes the results. We will add these results to the revised paper.

W2. Inferring students’ needs from implicit feedback

Thanks for the insightful comment. Student simulation is itself an active research area. We build our students on established LLM-based simulation and further design the gates to make them behave more realistically. We agree that real students often cannot articulate their own needs. To address this concern, we additionally run an evaluation in which explicit requests for a teaching strategy are replaced with replies where students only describe their confusion. To test robustness to wording, we ran the evaluation three times, each with a different set of implicit replies. Here we use the 8B judge to reduce cost, with all other settings unchanged.

SettingNon.Att.Sub.Ctr.Cau.Stp.Ver.IDOODOvl.
SHERPA_DP, explicit (original)78.778.779.469.175.255.868.676.566.672.2
Base, explicit (original)53.146.050.647.953.642.041.349.445.647.8
SHERPA_DP, implicit79.8 ± 1.074.3 ± 2.373.7 ± 0.548.1 ± 1.669.0 ± 1.138.0 ± 0.365.4 ± 1.468.9 ± 0.757.5 ± 0.464.0 ± 0.2
Base, implicit52.3 ± 0.538.6 ± 1.847.6 ± 2.736.7 ± 0.644.7 ± 0.537.0 ± 1.138.2 ± 0.343.8 ± 0.940.0 ± 0.342.2 ± 0.6

SHERPA’s overall advantage over the base model is 24.4 percentage points with explicit feedback and 21.9 points on average with implicit feedback. Most of the gain is retained across different wordings, suggesting that SHERPA can adapt to needs expressed implicitly rather than relying solely on explicit requests for a teaching strategy. Students who are unaware of their own preferences remain an important next step, which we discuss as future work in Sec. 5. We will add these results to the revised paper.

AuthorsOpenReview ↗

Response to Reviewer 8Qr7 (2/3)

W3. Reliability of the gating and judging LLMs

Thanks for the detailed comments.

(a) To check whether the teacher overfits to the training judge, we re-judged the training trajectories with the 27B judge using the same gate requests. From the first 200 to the last 200 training steps, the 8B pass rate rises from 41.2% to 64.9%, and the 27B pass rate also rises, from 50.9% to 57.7%, so the gain is not limited to the training judge. Training against a judge is expected to induce some fitting to it, so we use the held-out 27B judge for all reported results to avoid this bias. On 120 human-annotated gate decisions covering all six criteria, the 27B judge agrees with the human labels on 82.4% of decisions (κ = 0.65) and the 8B judge on 76.4% (κ = 0.52); see Q3 for details.

(b) The train/eval judge mismatch is intentional: we use a different judge for evaluation to avoid the bias of using the same judge for both training and evaluation. Under the 8B judge, SHERPA_DP reaches 72.2% accuracy versus 47.8% for the base model; under the 27B judge, it reaches 69.0% versus 48.6%. SHERPA_DP outperforms the base model on all seven archetypes under both judges, so the direction of the results is consistent. For how the guidance gate handles non-numeric answers and the answer-type distribution, please see Q4.

W4. Baselines and human evaluation

Thanks for the suggestions.

(a) The official PedagogicalRL checkpoint (TutorRL-7B) is built on Qwen2.5-7B-Instruct, so we re-implement PedagogicalRL on Qwen3-8B to compare with SHERPA on the same backbone. We additionally evaluate the official checkpoint in our student environment; its MathTutorBench result is the one reported in its paper.

TeacherBackboneStudent accuracyMathTutorBench Ped. Avg.
TutorRL-7B (official)Qwen2.5-7B-Instruct40.459.3
PedagogicalRL (ours)Qwen3-8B52.763.2
SHERPA_DPQwen3-8B69.079.2

We discuss PEARL but do not reproduce it because it has not released its code or checkpoints.

(b) With our limited budget, we recruit 96 teachers for 960 judgments and focus the human evaluation on comparing SHERPA_DP with its base model to assess whether SHERPA training improves teaching quality as judged by human teachers. PedagogicalRL and Gemini 3.8 Flash are compared in our automated evaluations (Tables 1, 2, 5 and 6).

AuthorsOpenReview ↗

Response to Reviewer 8Qr7 (3/3)

Q1. Answer leakage and performance without dialogue history

Thanks for the question. The guidance gate blocks teacher messages that reveal the answer, so those messages never enter the student's context. To check the messages that pass the gate, we use a GPT model to audit every teacher message visible to the student and flag those that state the final answer or an equivalent expression. We count only answers given by the teacher, since the student working out the answer during tutoring is the intended outcome. The table reports the share of dialogues containing such a message, and the results after excluding these dialogues:

TeacherDialogues with leaked answerImprovement (all → clean)Accuracy (all → clean)
SHERPA_DP4.5%33.1 → 32.769.0 → 68.6
Base4.1%12.4 → 11.648.6 → 47.5
SHERPA_NP1.7%16.6 → 16.352.0 → 51.5
PedagogicalRL5.9%16.8 → 15.752.7 → 51.2

The results barely change after excluding these dialogues. For the transfer test, since the student model has no memory beyond its context, testing without any dialogue history would reduce to the no-teaching baseline. We therefore follow EduardoRL's approach of masking all teacher messages in the dialogue history, so that students rely only on their own reasoning to solve transfer tasks. As reported in W1, SHERPA still improves student accuracy on new problems by 15.8 points in this setting, compared with 5.8 points for the base model.

Q2. Implicit feedback and active probing

Thanks for the question. We replace the preference-specific feedback with replies in which students only describe where they are stuck, without naming the teaching strategy (see W2). Across three runs with different reply wordings, SHERPA_DP outperforms the base model by 21.9 points on average, close to its 24.4-point advantage with the original feedback. Regarding active probing, our setting does not require the teacher to ask exploratory questions first. The teacher teaches normally and adapts when the student is not satisfied with its guidance. This is similar to a real teacher who starts with a default teaching style and adjusts it when the student does not understand.

Q3. Agreement with human judgments and overfitting to the judge

Thanks for the question.

(a) Human agreement. We annotated 120 gate decisions from the main evaluation, stratified over the six criteria, SHERPA$_\text{DP}$ versus the base model, and the judge's verdict. Reweighted to the full evaluation, the 8B training judge agrees with the human labels on 76.4% of decisions (κ = 0.52), and the 27B evaluation judge on 82.4% (κ = 0.65).

(b) Overfitting. We re-judged the gate requests from the first and last 200 training steps (about 6,960 turns each) with the 27B judge. The 27B pass rate rises on all three training preferences, and overall from 50.9% to 57.7% (8B: 41.2% → 64.9%). Agreement between the 8B and 27B judges also slightly increases (73.2% → 76.8%, κ 0.47 → 0.51). This measurement shows limited overfitting of the teacher to the 8B judge, and we report all results in the paper with the 27B judge for fairness.

Q4. Non-numeric answers and answer-type distribution

Thanks for the question. The guidance gate uses an LLM judge that compares the ground-truth answer with the teacher message. It is required to compute and normalize numerical expressions, so the comparison goes beyond exact string matching. In our evaluation, the judge also blocks leaks involving non-numeric answers. For example, for SHERPA_DP, it blocks leaks in 26 of the 70 dialogues with interval answers and in 28 of the 84 dialogues with matrix answers. We analyzed the answer types in the filtered training and evaluation sets and found that non-numeric answers account for 13.7% and 15.2%, respectively. The detailed distribution is:

Answer typeTest (528)Train (759)
Integer61.7%62.2%
Fraction15.7%14.5%
Decimal1.5%2.1%
Closed-form number (radicals, π, etc.)5.9%7.5%
List / set3.2%1.7%
Text2.7%1.7%
Expression / equation2.5%3.2%
Matrix / vector2.3%1.3%
Tuple (a, b)2.1%1.8%
Interval1.9%2.6%
Other (base-n numbers, ratios)0.6%1.3%

We will add this clarification and the distribution to the revised paper.

Reviewer_UiGXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript proposes a multi-turn reinforcement learning framework for training an LLM tutor with simulated students who have different instructional preferences. The work addresses an important problem and reports encouraging results. However, the interpretation of student learning and adaptive teaching requires further support.

Critical strengths

Strengths

  1. The study addresses an important problem between solving a mathematics question and teaching someone to solve it. A model producing the correct answer may fail to help a learner reason independently. The introduction explains this motivation clearly and identifies a meaningful gap in current methods for training LLM tutors.

  2. The structure from preparation to test is clear and maps onto a teaching episode. Separating the teacher, student backbone, and gating mechanisms makes the framework easy to understand. The study also explains how accepted and rejected teacher messages affect subsequent interactions.

  3. The main improvement is substantial. Average post-tutoring accuracy rises from 48.6% with the base teacher to 69.0% with SHERPA, exceeding the PedagogicalRL baseline.

  4. The study evaluates on MathTutorBench, which provides evidence beyond the simulation environment. The average measures increases from 52.5% to 79.2%, with gains in scaffolding and instruction following.

Critical weaknesses

Weaknesses

  1. Presentation: Several tables use dense abbreviations whose definitions are distributed across captions or surrounding text, and some appendix tables reuse labels introduced earlier. Please define abbreviations consistently.

  2. The study emphasizes adaptation to students’ latent needs, but gate rejection produces feedback that states the preferred teaching approach. The human study also appends explicit instructional requests. It would be helpful to test replies that imply a need without directly naming the teaching approach, to see whether SHERPA can infer students’ needs or mainly follow explicit requests.

  3. After tutoring, the simulated student answers the same problem while retaining the accepted dialogue in its context, so the improved accuracy may reflect the use of problem-specific hints rather than transferable

  4. Some measures decline after training. Socratic questioning falls from 25.3% to 21.9%, and mistake location from 37.1% to 30.3%. The unseen Step archetype reaches 51.8% accuracy and has a lower gate-pass rate than the base model. So it is needed to fully analysis and discuss the situation.

  5. The study does not test the learning effect of real students after interacting with SHERPA. Human teachers compare the quality of individual responses, which is valuable but differs from measuring learning outcomes. Therefore, the current experiments can’t support the claimed statement to improve students’ actual learning outcomes. The conclusions about real-world tutoring should be qualified, or the study should evaluate students’ subsequent independent performance.

  6. In Appendix B.5, the authors sample 50 problems without replacement from the 528-problem evaluation set to conduct the analysis. I wonder the rationale of this sampling. Also, in the following ‘simulator variants’, the authors randomly select one of six fixed responses. What are the rationales for the random sampling and how to ensure result replication?

  7. Rigor. The evaluated dataset is limited in MathTutorBench, making reviewers concerned about its generalization across diverse teaching scenarios. Suggest to evaluate in more diverse datasets. Moreover, the baseline models are limited, with only one major baseline named for each experiment. Other baselines look more like ablation models, rather than SOTA models from prior work. Tests under diverse LLM families are also missing, making generalization unclear.

Questions for authors

Please see the weakness part.

6 comments
AuthorsOpenReview ↗

Response to Reviewer UiGX (1/3)

We thank reviewer UiGX for recognizing our motivation, clear framework, and strong empirical results.

W1. Clarifying table abbreviations

Thanks for pointing this out. To improve readability, we will add text at the beginning of the experiments section to introduce abbreviations used in the tables. For less frequently used abbreviations, we will also explain them in the corresponding paragraphs.

W2. Inferring students’ needs from implicit feedback

Thanks for the suggestion. We replace the rejection feedback with replies in which students describe where they are stuck without naming a teaching approach. To test robustness to wording, we ran the evaluation three times, each with a different set of implicit replies. Here we use the 8B judge to reduce cost, with all other settings unchanged.

SettingNon.Att.Sub.Ctr.Cau.Stp.Ver.IDOODOvl.
SHERPA_DP, explicit (original)78.778.779.469.175.255.868.676.566.672.2
Base, explicit (original)53.146.050.647.953.642.041.349.445.647.8
SHERPA_DP, implicit79.8 ± 1.074.3 ± 2.373.7 ± 0.548.1 ± 1.669.0 ± 1.138.0 ± 0.365.4 ± 1.468.9 ± 0.757.5 ± 0.464.0 ± 0.2
Base, implicit52.3 ± 0.538.6 ± 1.847.6 ± 2.736.7 ± 0.644.7 ± 0.537.0 ± 1.138.2 ± 0.343.8 ± 0.940.0 ± 0.342.2 ± 0.6

SHERPA’s overall advantage over the base model is 24.4 percentage points with explicit feedback and 21.9 points on average with implicit feedback. Most of the gain is retained across different wordings, suggesting that SHERPA can adapt to needs expressed implicitly rather than relying solely on explicit requests for a teaching strategy. We will add these results to the revised paper.

W3. Performance on related problems

Thanks for raising this point. To test whether the improvement transfers beyond the tutored problem, we additionally evaluate students on unseen problems after tutoring, following EduardoRL (Macina et al., 2026). These problems are generated by an LLM with different numbers or contexts and different final answers, while preserving the concepts, solution steps and difficulty of the original problems. We also follow their setting of masking all teacher messages in the dialogue history to test whether the improvement persists even when students rely only on their own reasoning to solve transfer tasks. The table shows how much student accuracy on transfer problems improves after tutoring, with the full history (full) and with teacher messages masked (teacher masked).

SettingNon.Att.Sub.Ctr.Cau.Stp.Ver.IDOODOvl.
SHERPA_DP, variant (full)23.826.924.514.727.98.323.122.519.821.3
Base, variant (full)8.36.98.66.911.53.94.27.76.57.2
SHERPA_DP, variant (teacher masked)19.020.420.111.320.43.116.217.713.315.8
Base, variant (teacher masked)7.75.28.15.08.53.42.96.54.95.8

On unseen variants, SHERPA outperforms the base model by 14.2 points with the full dialogue and by 10.0 points with the teacher turns masked. This shows that the gains go beyond re-solving the same problem. We will add these results to the revised paper.

AuthorsOpenReview ↗

Response to Reviewer UiGX (2/3)

W4. Declines on Step, Mistake Location and Socratic Questioning

Thanks for pointing this out. We will add the following analysis to the revised paper.

(1) Step. The ID preferences favor detailed explanations, while Step favors a short nudge (one step, then stop), so training on ID students can hurt Step. This is most visible for the Subgoal-only teacher, which drops to 40.8% on Step (base 47.8%) (Table 5). Training on multiple preferences mitigates this: SHERPA_DP is the only trained teacher that outperforms the base model on Step (51.8%). Although its Step gate-pass rate is slightly lower, each accepted message helps much more (13.9 vs. 4.9 points per round, Table 8). We intentionally hold out three archetypes for OOD evaluation; in practice, all archetypes could be included in training, which we expect would improve performance further.

(2) Mistake Location and Socratic Questioning. Both scores decline for most trained teachers, including PedagogicalRL (Mistake Location 18.7%, Socratic Questioning 21.9%). We examined the scoring mechanisms and found that reference-based scoring can produce large score differences for semantically equivalent responses. For example, Socratic Questioning uses BLEU against reference questions: “How old is Jane?” scores 61.5, while the equivalent “What is Jane's age?” scores 5.0. We find that 93.3% of the reference questions start with “How many/much/old/long”, compared with 64.4% for the base model and 57.6% for SHERPA_DP. This shift in wording can therefore contribute to the lower Socratic Questioning score without indicating worse teaching. Our student-performance evaluations and human teachers’ judgments provide complementary evidence of improved teaching quality.

Overall, SHERPA offers a new perspective on training teacher LLMs, learning adaptive teaching directly from students' outcomes, and it already achieves substantial gains in simulated students' performance, on MathTutorBench, and in human teachers' judgments. We hope to address the remaining challenges, such as the decline in some of the model's original abilities like mistake location, in future work.

W5. Learning outcomes of real students

Thanks for this valuable comment. We agree that human teachers’ judgments of response quality differ from real students’ learning outcomes. However, we would like to clarify that, since we have not evaluated learning outcomes with real students, we do not claim improvements in those outcomes. We view this as an important goal requiring further effort and leave it to future work. In this work, we demonstrate substantial improvements in simulated students’ performance and in teaching quality as judged by human teachers. We will make this scope clearer in the revised paper. For simulated students, we also evaluate performance on new problems after tutoring (see W3), showing that the improvement extends beyond the tutored problems.

AuthorsOpenReview ↗

Response to Reviewer UiGX (3/3)

W6. Sampling and robustness of the simulator analysis

Thanks for the questions. We sample 50 problems due to limited compute budget: evaluating all 6×6 strategy–preference pairs for two student designs already requires 3,600 dialogues with an API teacher. To check that the findings do not depend on this sample, we repeat the analysis on 50 new problems with no overlap with the original sample, keeping all other settings unchanged. The table reports improvement and complaint rate (%) for matched (diagonal) and mismatched (off-diagonal) strategy–preference pairs.

SettingDiagonalOff-diagonalGap
Ours, improvement (original 50)53.322.1+31.2
Prompted, improvement (original 50)51.351.30.0
Ours, complaint rate (original 50)7.075.7−68.7
Prompted, complaint rate (original 50)29.928.9+1.0
Ours, improvement (new 50)49.517.2+32.2
Prompted, improvement (new 50)45.845.2+0.6
Ours, complaint rate (new 50)9.975.7−65.9
Prompted, complaint rate (new 50)33.129.2+4.0

The results on the new problems are consistent with the paper. A paired bootstrap on the original 50 problems also gives an improvement gap of +31.2 [26.5, 36.0] for our design and 0.0 [−3.5, 3.4] for prompted students.

We use six replies to match the training setting, where each preference has six feedback replies. Randomly sampling among them provides wording diversity and helps prevent overfitting to a single sentence. In our main evaluation, 65.7% of dialogues with rejections contain no repeated complaint. Both the 50 problems and the sampled replies are drawn with fixed random seeds.

W7. Generalization across datasets, baselines and model families

Thanks for the suggestions. MathTutorBench is already OOD to our training setting, and such benchmarks that evaluate LLM tutors on math problems are still few. In addition, our human study also uses an unseen data distribution. Together, these evaluations demonstrate SHERPA’s generalization through both benchmark performance and human teachers’ judgments. Related methods are also few. Besides PedagogicalRL, we discuss PEARL (Chang et al., 2026), but do not reproduce it because it has not released its code or checkpoints. For different model families, we cover two aspects. On the student side, we evaluate with Gemma-3-1B-IT and Llama-3.1-8B-Instruct students (Appendix C.2.3, Table 11), where SHERPA_DP raises overall accuracy from 14.3% to 21.7% and from 37.5% to 53.5%, respectively. On the teacher side, we train a Ministral-3-8B-Instruct teacher with SHERPA (Appendix C.2.5, Table 13), which raises overall accuracy from 39.6% to 55.0%.

Reviewer_eZL5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SHERPA uses multi-turn RL to train an LLM tutor, rewarding it based on how much simulated students improve at solving math problems. It creates different types of students using gates that only allow teaching responses that match their learning preferences.

Critical strengths

According to the experimental results shown in Tab. 1, 11, and 12, the idea of training tutors based on student improvement is intuitive and works well. SHERPA clearly outperforms PedagogicalRL and the no-preference variant. The gains also hold for unseen student types and different student models, with consistent results across training seeds.

Critical weaknesses

The gates reject responses that do not match predefined teaching strategies, even if they could still help the student. The student also explicitly tells the tutor what kind of guidance it wants (Sec. 2.2; Appendix D.1.3). This makes it unclear whether the tutor actually learns to identify students’ needs or simply learns to follow these signals. The human study does not fully address this, as student preferences are also explicitly stated in the generated requests.

Questions for authors

Can the authors evaluate (i) tutoring with the adaptive gate removed at test time, using students who express needs naturally, and (ii) performance on related unseen problems after tutoring, without the dialogue history? These experiments would help clarify whether SHERPA learns to adapt to students’ needs and improves their actual learning, rather than just fitting the simulator or helping students solve the same problem.

5 comments
AuthorsOpenReview ↗

Response to Reviewer eZL5 (1/2): Weaknesses

We thank reviewer eZL5 for recognizing the effective and generalizable training approach of our paper.

W1. Unclear if tutor learns from student needs or gating signals

Thanks for raising this point. Indeed, some rejected responses could still help a real student, but here we intentionally construct single-preference archetypes. To verify whether the tutor relies on explicit signals, we design an experiment in which students express their preferences implicitly: we replace the rejection feedback with replies that only describe where the student is stuck, without naming any teaching strategy. To test robustness to wording, we ran the evaluation three times, each with a different set of implicit replies (e.g., “I’m doing the math, but I have no idea where this is going.”). Here we use the 8B judge to reduce cost, with all other settings unchanged.

SettingNon.Att.Sub.Ctr.Cau.Stp.Ver.IDOODOvl.
SHERPA_DP, explicit (original)78.778.779.469.175.255.868.676.566.672.2
Base, explicit (original)53.146.050.647.953.642.041.349.445.647.8
SHERPA_DP, implicit79.8 ± 1.074.3 ± 2.373.7 ± 0.548.1 ± 1.669.0 ± 1.138.0 ± 0.365.4 ± 1.468.9 ± 0.757.5 ± 0.464.0 ± 0.2
Base, implicit52.3 ± 0.538.6 ± 1.847.6 ± 2.736.7 ± 0.644.7 ± 0.537.0 ± 1.138.2 ± 0.343.8 ± 0.940.0 ± 0.342.2 ± 0.6

SHERPA’s overall advantage over the base model is 24.4 percentage points with explicit feedback and 21.9 points on average with implicit feedback. Most of the gain is retained across different wordings, suggesting that SHERPA can adapt to needs expressed implicitly rather than relying solely on explicit requests for a teaching strategy. Results with the adaptive gate removed are reported in Q1. We will add these results to the revised paper.

AuthorsOpenReview ↗

Response to Reviewer eZL5 (2/2): Questions

Q1. Evaluating tutoring outcome with natural student needs

Thanks for the suggestion. To evaluate tutoring with the adaptive gate removed at test time, we follow Sec. 3.4 and directly prompt the student model with each preference, without any adaptive gate. All other settings follow the main evaluation.

SettingNon.Att.Sub.Ctr.Cau.Stp.Ver.IDOODOvl.
SHERPA_DP, prompted (no gate)76.975.077.475.474.076.076.576.275.575.9
Base, prompted (no gate)50.053.055.651.255.153.649.852.552.852.6
SHERPA_DP, gated (Table 1)76.278.272.760.777.151.866.472.065.169.0
Base, gated (Table 1)53.250.748.444.154.447.841.549.147.948.6

Without the gate, SHERPA still clearly outperforms the base model (75.9% vs. 52.6%, +23.3 points), similar to its advantage with the gate (+20.4 points). This is consistent with our finding in Sec. 3.4 that prompted students behave similarly regardless of their preferences: across the seven prompted students, SHERPA_DP's accuracy varies only from 74.0% to 77.4%, compared with 51.8% to 78.2% for gated students. This is why we use the gate to make preferences consequential in training and in the main evaluation.

Q2. Performance on related problems

Thanks for the suggestion. We additionally evaluate students on related unseen problems after tutoring, and find that SHERPA still clearly outperforms the base model. Following EduardoRL (Macina et al., 2026), these problems are generated by an LLM with different numbers or contexts and different final answers, while preserving the concepts, solution steps and difficulty of the original problems. Since the student model has no memory beyond its context, testing “without the dialogue history” would reduce to the no-teaching baseline. We therefore also follow their setting of masking all teacher messages in the dialogue history, so that students rely only on their own reasoning to solve transfer tasks. The table shows how much student accuracy on transfer problems improves after tutoring, with the full history (full) and with teacher messages masked (teacher masked).

SettingNon.Att.Sub.Ctr.Cau.Stp.Ver.IDOODOvl.
SHERPA_DP, variant (full)23.826.924.514.727.98.323.122.519.821.3
Base, variant (full)8.36.98.66.911.53.94.27.76.57.2
SHERPA_DP, variant (teacher masked)19.020.420.111.320.43.116.217.713.315.8
Base, variant (teacher masked)7.75.28.15.08.53.42.96.54.95.8

On unseen variants, SHERPA outperforms the base model by 14.2 points with the full dialogue and by 10.0 points with the teacher turns masked. This shows that SHERPA improves students’ learning beyond the tutored problems to unseen ones. We will add these results to the revised paper.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer eZL5,

Thank you again for your time and constructive feedback. In our responses, we added an evaluation with implicit student feedback, an evaluation with the adaptive gate removed at test time, and a transfer evaluation on unseen problems. As the discussion period is coming to a close, we would like to check whether our responses have addressed your concerns. We would be happy to provide any further clarification.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.