- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies outcome optimization in open-ended LLM tutoring, motivated by the fact that optimizing for student outcomes can reward tutors for revealing answers ("disclosure inflation"). The authors use an offline DPO pipeline on StudyChat to compare three optimization objectives: pedagogical process, turn-level outcome, and trajectory-level outcome with MCTS look-ahead. Their main finding is that outcome optimization, even with look-ahead, does not outperform directly optimizing the pedagogical process, which achieves the highest simulated long-horizon KC mastery and lowest answer leakage. While comparing the look-ahead setup with outcome-level and turn-level optimization is interesting, several claimed contributions overlap with prior work, the methodology relies on an offline DPO setup without fair comparisons to state-of-the-art on-policy RL or online DPO approaches, and the evaluation setup is limited.
Critical strengths
- The paper is motivated by an important problem in educational dialogues: optimizing for immediate student success can inadvertently reward answer disclosure.
- Controlled comparison of three DPO preference signals across two open-weight models (Qwen-2.5-7B and SmolLM3-3B)
Critical weaknesses
- Offline DPO only, no SOTA baselines (main weakness): All candidate and scores are pre-generated offline, so only off-policy learning is tested. There is no comparison to on-policy RL/GRPO approaches from prior work (Dinucu-Jianu et al., 2025) or online DPO approaches with look-ahead such as CollabLLM [2] (see Section 3.1.2 on Forward Sampling), where preference pairs are build from the current policy’s own samples scored with multi-turn lookahead. The appendix N does not address this. Since disclosure inflation is a form of reward hacking, it depends on the optimizer, and offline DPO over eight fixed candidates cannot discover new exploits. I suggest the authors either a) reframe the paper as a study of offline DPO preference construction for tutoring, compared against Scarlatos et al. (2025c) and the frontier tutors in Table 2, or b) compare each objective under SFT, offline DPO, and online DPO or GRPO. The code is also not available to check the data generation and MCTS pipeline.
- Overlapping contributions with prior work: C1 and C3 are essentially the same (identifying disclosure inflation and measuring task success vs. answer disclosure), and this trade-off has already been studied in prior works (e.g., Dinucu-Jianu et al., 2025 Figure 1 and Table 2; in [1] Figure 5 and 6). The same holds for KC mastery estimation from dialogues (Scarlatos et al., 2024). The actual contribution is the comparison of the three objectives.
- Engagement with prior work and imprecise notation: The paper groups diverse prior works into the same bucket without distinguishing their core ML assumptions (on-policy vs. off-policy, online vs. offline) and single-turn vs. multi-turn optimization setups. Rubrics design and the reward weights (Equations 1–4) and thresholds are not justified beyond manual selection.
- Evaluation reliability and uninformative transfer results: Evaluation is mostly in-domain and single-turn (with fixed cutoff points), while the long-horizon evaluation relies entirely on prompted LLM student simulators. The quality of the LLM judge (GPT-4o) and its agreement with human judgments are not reported. The human evaluation only measures turn-level preferences rather than actual learning gains, is small (48 situations, 3 annotators), and lacks details on participant backgrounds. Finally, Table 3 does not add meaningful insights, as the differences between conditions are very small (within 1.4 percentage points) and nearly identical to the untrained Base Tutor.
- KC granularity and single-concept action assumption: The actual difficulty is breaking an assignment into KCs at the right granularity, which is not discussed. Additionally, each tutor action targets a single KC, which is unrealistic for coursework requiring a combination of concepts simultaneously (e.g., synthesizing multiple ideas or higher-level tasks in Bloom's taxonomy). Relying on a predefined taxonomy to pre-generate M=8 strategy-conditioned utterances does not scale well and compute for scoring candidaes is not reported.
Questions for authors
- You use three broad tutor action strategy families (probing, telling, and focusing). Why were these three chosen? They appear to be inspired by MathDial [1], but this connection is neither motivated nor cited.
- Equation 4: how is regression mathematically defined and computed?
- What is meant by "zero-shot ETH" in the simulator configurations, and what does ETH stand for in this context?
- Why did you choose older models for the main training backbone (Qwen-2.5-7B) and primary LLM judge (GPT-4o), especially given that newer models are used in Table 2? Is there a specific reason for this?
- How do you validate the granularity of the automatically extracted KCs, and how can your action formulation handle turns that require synthesizing multiple KCs at once?
- Could you report the agreement between the LLM-as-a-judge and human annotators, and provide background details (e.g., teaching experience) for the three annotators in Section 6.5?
[1] Macina, J., Daheim, N., Chowdhury, S., Sinha, T., Kapur, M., Gurevych, I. and Sachan, M., 2023, December. Mathdial: A dialogue tutoring dataset with rich pedagogical properties grounded in math reasoning problems. In Findings of the Association for Computational Linguistics: EMNLP 2023 (pp. 5602-5621). [2] Wu, S., Galley, M., Peng, B., Cheng, H., Li, G., Dou, Y., Cai, W., Zou, J., Leskovec, J. and Gao, J., 2025. Collabllm: From passive responders to active collaborators. In Proceedings of ICML 2025.
11 comments
Response to Reviewer ChHr Part 2/8
Lack of a supervised fine-tuning (SFT) baseline.
We thank the reviewer for this suggestion. We added five SFT baselines to examine whether the observed differences come from the optimization objective or the preference-training procedure. Unlike problem-oriented tutoring datasets such as MathDial, where final answer correctness can serve as supervision, our setting involves open-ended dialogues with multiple knowledge components (KCs). We therefore construct SFT data using simulated tutoring trajectories and vary how we select the training examples.
SFT setup. We sample 1,000 training contexts and generate four 8-exchange tutor–student rollouts per context using GPT-5.5 at 4 different runs. We use a neutral tutoring instruction without explicitly encouraging answer withholding. For each rollout, we compute pedagogical quality (P), estimated KC mastery gain (G), and answer leakage (L). We filter trajectories using the same factual accuracy, response quality, KC alignment, and leakage thresholds as our preference-data pipeline.
We train five SFT variants using the same Qwen2.5-7B-Instruct backbone and LoRA configuration:
- SFT-Chosen: Uses the chosen responses from the original preference pairs, allowing a direct SFT–DPO comparison.
- SFT-Unf: Randomly selects one valid rollout per context, without reward-based ranking.
- SFT-A: Selects the rollout with the highest mean pedagogical reward (P).
- SFT-B: Selects the rollout with the highest estimated KC mastery gain (G).
- SFT-AB: Selects the rollout maximizing the sum of context-normalized P and G, breaking ties by lower leakage.
We train for two epochs with a learning rate of 1e-4, applying the SFT loss only to tutor tokens, and select checkpoints using held-out validation loss. We evaluate all variants using the same 32-turn mastery protocol across five student simulators.
Results.
| Tutor | Final mastered KCs ↑ |
|---|---|
| Pedagogically Optimized (DPO-A) | 18.92 |
| SFT-Chosen | 16.24 |
| SFT-Unf | 16.15 |
| SFT-A | 16.38 |
| SFT-B | 15.16 |
| SFT-AB | 15.89 |
Two observations emerge. First, the pedagogically selected SFT-A achieves higher mastery than outcome-selected SFT-B (16.38 vs. 15.16), consistent with our main finding that optimizing estimated learner outcomes does not necessarily improve long-horizon mastery. Combining process and outcome scores in SFT-AB also does not improve over process-only selection (15.89 vs. 16.38).
Second, SFT-Unf performs comparably to SFT-A (16.15 vs. 16.38), suggesting that much of the benefit of these SFT variants may come from distilling the teacher-generated tutoring trajectories rather than reward-based selection alone. The DPO-trained pedagogical tutor achieves substantially higher mastery (18.92) than all five SFT variants.
Importantly, SFT-A, SFT-B, and SFT-AB are reward-selected SFT baselines rather than objective-neutral SFT methods. We therefore interpret these results as an additional comparison of training and data-selection strategies, rather than a controlled causal isolation of the reward objective. Overall, the results are consistent with our central observation: selecting tutoring behavior based on estimated learner outcomes alone does not guarantee better independently assessed mastery.
Response to Reviewer ChHr Part 3/8
W2: Overlapping contributions with prior work.
We agree with the reviewer that the trade-off between student success and answer disclosure has been studied in prior work (Dinucu-Jianu et al., 2025; Macina et al., 2023). Similarly, KC-based mastery estimation from tutoring dialogues builds on Scarlatos et al., which we acknowledge. We also agree that C1 and C3 overlap and will merge them in the revised manuscript.
However, we would like to highlight an important distinction between the settings studied in prior work and ours.
Prior works such as Dinucu-Jianu et al. (2025) and Macina et al. (2023) primarily study mathematical problem-solving tasks, where student outcomes can be evaluated against known correct answers. In these settings, the trade-off between solving success and answer disclosure can be explicitly measured and balanced.
Our work focuses on open-ended, natural tutoring dialogues, where learner outcomes are not directly verifiable. For example, in StudyChat, students work on AI coursework involving multiple concepts, programming tasks, and reasoning skills. Unlike mathematical problems with a known answer, student learning cannot be measured through a single correctness signal. Therefore, outcome-based training must rely on estimated learner progress from dialogue, KC-based assessments, or simulated student responses.
This creates an additional challenge beyond the known success–disclosure trade-off. When the outcome itself is estimated from tutor-influenced interactions, the reward may incorrectly interpret answer disclosure as learner progress. We refer to this as disclosure inflation.
We provide empirical evidence of this effect in Appendix K. Among 14,904 scored candidates, those credited only by the IRT-based outcome channel show higher answer leakage (0.180 vs. 0.078) and telling rates (50.1% vs. 28.0%). This suggests that the outcome reward itself can favor revealing answers rather than eliciting independent understanding.
Building on this observation, our main contribution is a systematic comparison of three training objectives in this open-ended setting:
- Process vs. outcome optimization: The pedagogical objective achieves higher simulated long-horizon mastery than both turn-level and trajectory-level outcome optimization. Our additional matched-budget experiment further shows that the advantage over turn-level optimization remains (+2.87 mastered KCs, 95% CI: +1.39 to +4.47).
- Effect of longer-term planning: Although MCTS-based look-ahead improves over turn-level outcome optimization, it still does not outperform the pedagogical objective (0.81 vs. 0.85 simulated mastery).
- Analysis of the reward mechanism: We examine how the outcome reward can favor answer disclosure and how different reward objectives influence the resulting tutoring behavior.
We agree that our Big-Math and FinQA evaluations build on existing protocols for measuring task success and disclosure, and we will make this attribution clearer.
Therefore, our novelty is not in identifying the answer-disclosure trade-off or introducing KC-based mastery estimation. Rather, we investigate the reliability of outcome-based training when learner progress is not directly verifiable, and show how this affects the relative effectiveness of different optimization objectives. We will revise our contribution statements to make this distinction clearer.
Response to Reviewer ChHr Part 1/8
We thank the reviewer for the detailed and constructive feedback. Below, we address the concerns and questions raised in the review.
We agree with the reviewer that the choice of optimizer can influence how strongly a model exploits a particular reward signal. Our primary experiments use offline DPO, and we acknowledge that our findings should not be generalized to all online RL settings. We will clarify this scope in the revised paper.
Scope of our contribution. Our main objective is to investigate which reward signal should be used to train LLM tutors when learner outcomes cannot be directly verified. We compare pedagogical-process, turn-level outcome, and trajectory-level outcome rewards while keeping the underlying optimization framework fixed.
This setting is related to Scarlatos et al. (2025c), which uses DPO for tutor training, and the offline DPO configuration of CollabLLM (Wu et al., 2025), where preferences are constructed using simulated future outcomes. Our trajectory-level objective similarly uses MCTS-based look-ahead to estimate future learner progress. However, unlike CollabLLM, we do not evaluate online DPO or PPO variants that continuously generate new responses from the updated policy.
The findings are still relevant to reward hacking. We provide two additional observations:
We acknowledge that the GRPO experiment is limited to one outcome objective, four completions per prompt, less than one epoch of training, and a single run. Therefore, we do not consider it a controlled comparison between online and offline optimization.
We will revise the paper to make clear that our main findings concern reward construction under offline preference optimization. Whether online optimization amplifies or reduces disclosure inflation remains an open question that requires further controlled experiments.