acceptodds
Under review as a conference paper at ICLR 2027

Solved ≠ Learned: When Outcome Optimization Rewards Answer Disclosure in LLM Tutors

Abstract

We study outcome optimization in open-ended LLM tutoring, where learner improvement is not directly verifiable and must be estimated from dialogue or simulation. In this setting, the tutor can also influence the evidence used to measure progress by revealing solution-relevant information. We refer to this as disclosure inflation. We adapt three representative state-of-the-art LLM tutor training objectives—pedagogical-process, turn-level outcome, and trajectory-level outcome optimization on open-ended coursework-based tutor–student dialogues while keeping the training setup fixed. The Pedagogically Optimized Tutor achieves the highest long-horizon KC mastery (0.85), compared with trajectory-level (0.81) and turn-level outcome optimization (0.72), despite using no simulated look-ahead or trajectory-level mastery reward. It also shows the lowest answer disclosure in goal-oriented evaluation. Trajectory-level planning improves over turn-level optimization, but does not outperform the process objective. On a separate verifiable mathematics evaluation, tutoring improves accuracy from 14.7% to 26.7%. These results show that optimizing a model-estimated learner outcome does not necessarily optimize independently measured learner mastery.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM Tutoring, reward hacking, evaluation, LLM-as-a-judge

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ChHrOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies outcome optimization in open-ended LLM tutoring, motivated by the fact that optimizing for student outcomes can reward tutors for revealing answers ("disclosure inflation"). The authors use an offline DPO pipeline on StudyChat to compare three optimization objectives: pedagogical process, turn-level outcome, and trajectory-level outcome with MCTS look-ahead. Their main finding is that outcome optimization, even with look-ahead, does not outperform directly optimizing the pedagogical process, which achieves the highest simulated long-horizon KC mastery and lowest answer leakage. While comparing the look-ahead setup with outcome-level and turn-level optimization is interesting, several claimed contributions overlap with prior work, the methodology relies on an offline DPO setup without fair comparisons to state-of-the-art on-policy RL or online DPO approaches, and the evaluation setup is limited.

Critical strengths
  • The paper is motivated by an important problem in educational dialogues: optimizing for immediate student success can inadvertently reward answer disclosure.
  • Controlled comparison of three DPO preference signals across two open-weight models (Qwen-2.5-7B and SmolLM3-3B)
Critical weaknesses
  • Offline DPO only, no SOTA baselines (main weakness): All candidate and scores are pre-generated offline, so only off-policy learning is tested. There is no comparison to on-policy RL/GRPO approaches from prior work (Dinucu-Jianu et al., 2025) or online DPO approaches with look-ahead such as CollabLLM [2] (see Section 3.1.2 on Forward Sampling), where preference pairs are build from the current policy’s own samples scored with multi-turn lookahead. The appendix N does not address this. Since disclosure inflation is a form of reward hacking, it depends on the optimizer, and offline DPO over eight fixed candidates cannot discover new exploits. I suggest the authors either a) reframe the paper as a study of offline DPO preference construction for tutoring, compared against Scarlatos et al. (2025c) and the frontier tutors in Table 2, or b) compare each objective under SFT, offline DPO, and online DPO or GRPO. The code is also not available to check the data generation and MCTS pipeline.
  • Overlapping contributions with prior work: C1 and C3 are essentially the same (identifying disclosure inflation and measuring task success vs. answer disclosure), and this trade-off has already been studied in prior works (e.g., Dinucu-Jianu et al., 2025 Figure 1 and Table 2; in [1] Figure 5 and 6). The same holds for KC mastery estimation from dialogues (Scarlatos et al., 2024). The actual contribution is the comparison of the three objectives.
  • Engagement with prior work and imprecise notation: The paper groups diverse prior works into the same bucket without distinguishing their core ML assumptions (on-policy vs. off-policy, online vs. offline) and single-turn vs. multi-turn optimization setups. Rubrics design and the reward weights (Equations 1–4) and thresholds are not justified beyond manual selection.
  • Evaluation reliability and uninformative transfer results: Evaluation is mostly in-domain and single-turn (with fixed cutoff points), while the long-horizon evaluation relies entirely on prompted LLM student simulators. The quality of the LLM judge (GPT-4o) and its agreement with human judgments are not reported. The human evaluation only measures turn-level preferences rather than actual learning gains, is small (48 situations, 3 annotators), and lacks details on participant backgrounds. Finally, Table 3 does not add meaningful insights, as the differences between conditions are very small (within 1.4 percentage points) and nearly identical to the untrained Base Tutor.
  • KC granularity and single-concept action assumption: The actual difficulty is breaking an assignment into KCs at the right granularity, which is not discussed. Additionally, each tutor action targets a single KC, which is unrealistic for coursework requiring a combination of concepts simultaneously (e.g., synthesizing multiple ideas or higher-level tasks in Bloom's taxonomy). Relying on a predefined taxonomy to pre-generate M=8 strategy-conditioned utterances does not scale well and compute for scoring candidaes is not reported.
Questions for authors
  • You use three broad tutor action strategy families (probing, telling, and focusing). Why were these three chosen? They appear to be inspired by MathDial [1], but this connection is neither motivated nor cited.
  • Equation 4: how is regression mathematically defined and computed?
  • What is meant by "zero-shot ETH" in the simulator configurations, and what does ETH stand for in this context?
  • Why did you choose older models for the main training backbone (Qwen-2.5-7B) and primary LLM judge (GPT-4o), especially given that newer models are used in Table 2? Is there a specific reason for this?
  • How do you validate the granularity of the automatically extracted KCs, and how can your action formulation handle turns that require synthesizing multiple KCs at once?
  • Could you report the agreement between the LLM-as-a-judge and human annotators, and provide background details (e.g., teaching experience) for the three annotators in Section 6.5?

[1] Macina, J., Daheim, N., Chowdhury, S., Sinha, T., Kapur, M., Gurevych, I. and Sachan, M., 2023, December. Mathdial: A dialogue tutoring dataset with rich pedagogical properties grounded in math reasoning problems. In Findings of the Association for Computational Linguistics: EMNLP 2023 (pp. 5602-5621). [2] Wu, S., Galley, M., Peng, B., Cheng, H., Li, G., Dou, Y., Cai, W., Zou, J., Leskovec, J. and Gao, J., 2025. Collabllm: From passive responders to active collaborators. In Proceedings of ICML 2025.

11 comments
AuthorsOpenReview ↗

Response to Reviewer ChHr Part 1/8

We thank the reviewer for the detailed and constructive feedback. Below, we address the concerns and questions raised in the review.

W1: Offline DPO only, without on-policy or online baselines.

We agree with the reviewer that the choice of optimizer can influence how strongly a model exploits a particular reward signal. Our primary experiments use offline DPO, and we acknowledge that our findings should not be generalized to all online RL settings. We will clarify this scope in the revised paper.

Scope of our contribution. Our main objective is to investigate which reward signal should be used to train LLM tutors when learner outcomes cannot be directly verified. We compare pedagogical-process, turn-level outcome, and trajectory-level outcome rewards while keeping the underlying optimization framework fixed.

This setting is related to Scarlatos et al. (2025c), which uses DPO for tutor training, and the offline DPO configuration of CollabLLM (Wu et al., 2025), where preferences are constructed using simulated future outcomes. Our trajectory-level objective similarly uses MCTS-based look-ahead to estimate future learner progress. However, unlike CollabLLM, we do not evaluate online DPO or PPO variants that continuously generate new responses from the updated policy.

The findings are still relevant to reward hacking. We provide two additional observations:

  1. The outcome reward can favor disclosure before optimization. In Appendix K, among 14,904 scored candidates, responses credited only by the IRT-based outcome channel show higher answer leakage (0.180 vs. 0.078) and telling rates (50.1% vs. 28.0%). This indicates that the issue is not limited to the DPO optimizer; the reward signal itself can favor responses that disclose more information.
  2. A preliminary online GRPO experiment shows a similar pattern. In Appendix N, GRPO training with the outcome reward and an explicit leakage penalty achieves 0.672 ± 0.067 final simulated mastery, close to the DPO-trained outcome tutor (0.685) and below the DPO-trained pedagogical tutor (0.851).

We acknowledge that the GRPO experiment is limited to one outcome objective, four completions per prompt, less than one epoch of training, and a single run. Therefore, we do not consider it a controlled comparison between online and offline optimization.

We will revise the paper to make clear that our main findings concern reward construction under offline preference optimization. Whether online optimization amplifies or reduces disclosure inflation remains an open question that requires further controlled experiments.

AuthorsOpenReview ↗

Response to Reviewer ChHr Part 2/8

Lack of a supervised fine-tuning (SFT) baseline.

We thank the reviewer for this suggestion. We added five SFT baselines to examine whether the observed differences come from the optimization objective or the preference-training procedure. Unlike problem-oriented tutoring datasets such as MathDial, where final answer correctness can serve as supervision, our setting involves open-ended dialogues with multiple knowledge components (KCs). We therefore construct SFT data using simulated tutoring trajectories and vary how we select the training examples.

SFT setup. We sample 1,000 training contexts and generate four 8-exchange tutor–student rollouts per context using GPT-5.5 at 4 different runs. We use a neutral tutoring instruction without explicitly encouraging answer withholding. For each rollout, we compute pedagogical quality (P), estimated KC mastery gain (G), and answer leakage (L). We filter trajectories using the same factual accuracy, response quality, KC alignment, and leakage thresholds as our preference-data pipeline.

We train five SFT variants using the same Qwen2.5-7B-Instruct backbone and LoRA configuration:

  • SFT-Chosen: Uses the chosen responses from the original RAR_A preference pairs, allowing a direct SFT–DPO comparison.
  • SFT-Unf: Randomly selects one valid rollout per context, without reward-based ranking.
  • SFT-A: Selects the rollout with the highest mean pedagogical reward (P).
  • SFT-B: Selects the rollout with the highest estimated KC mastery gain (G).
  • SFT-AB: Selects the rollout maximizing the sum of context-normalized P and G, breaking ties by lower leakage.

We train for two epochs with a learning rate of 1e-4, applying the SFT loss only to tutor tokens, and select checkpoints using held-out validation loss. We evaluate all variants using the same 32-turn mastery protocol across five student simulators.

Results.

TutorFinal mastered KCs ↑
Pedagogically Optimized (DPO-A)18.92
SFT-Chosen16.24
SFT-Unf16.15
SFT-A16.38
SFT-B15.16
SFT-AB15.89

Two observations emerge. First, the pedagogically selected SFT-A achieves higher mastery than outcome-selected SFT-B (16.38 vs. 15.16), consistent with our main finding that optimizing estimated learner outcomes does not necessarily improve long-horizon mastery. Combining process and outcome scores in SFT-AB also does not improve over process-only selection (15.89 vs. 16.38).

Second, SFT-Unf performs comparably to SFT-A (16.15 vs. 16.38), suggesting that much of the benefit of these SFT variants may come from distilling the teacher-generated tutoring trajectories rather than reward-based selection alone. The DPO-trained pedagogical tutor achieves substantially higher mastery (18.92) than all five SFT variants.

Importantly, SFT-A, SFT-B, and SFT-AB are reward-selected SFT baselines rather than objective-neutral SFT methods. We therefore interpret these results as an additional comparison of training and data-selection strategies, rather than a controlled causal isolation of the reward objective. Overall, the results are consistent with our central observation: selecting tutoring behavior based on estimated learner outcomes alone does not guarantee better independently assessed mastery.

AuthorsOpenReview ↗

Response to Reviewer ChHr Part 3/8

W2: Overlapping contributions with prior work.

We agree with the reviewer that the trade-off between student success and answer disclosure has been studied in prior work (Dinucu-Jianu et al., 2025; Macina et al., 2023). Similarly, KC-based mastery estimation from tutoring dialogues builds on Scarlatos et al., which we acknowledge. We also agree that C1 and C3 overlap and will merge them in the revised manuscript.

However, we would like to highlight an important distinction between the settings studied in prior work and ours.

Prior works such as Dinucu-Jianu et al. (2025) and Macina et al. (2023) primarily study mathematical problem-solving tasks, where student outcomes can be evaluated against known correct answers. In these settings, the trade-off between solving success and answer disclosure can be explicitly measured and balanced.

Our work focuses on open-ended, natural tutoring dialogues, where learner outcomes are not directly verifiable. For example, in StudyChat, students work on AI coursework involving multiple concepts, programming tasks, and reasoning skills. Unlike mathematical problems with a known answer, student learning cannot be measured through a single correctness signal. Therefore, outcome-based training must rely on estimated learner progress from dialogue, KC-based assessments, or simulated student responses.

This creates an additional challenge beyond the known success–disclosure trade-off. When the outcome itself is estimated from tutor-influenced interactions, the reward may incorrectly interpret answer disclosure as learner progress. We refer to this as disclosure inflation.

We provide empirical evidence of this effect in Appendix K. Among 14,904 scored candidates, those credited only by the IRT-based outcome channel show higher answer leakage (0.180 vs. 0.078) and telling rates (50.1% vs. 28.0%). This suggests that the outcome reward itself can favor revealing answers rather than eliciting independent understanding.

Building on this observation, our main contribution is a systematic comparison of three training objectives in this open-ended setting:

  1. Process vs. outcome optimization: The pedagogical objective achieves higher simulated long-horizon mastery than both turn-level and trajectory-level outcome optimization. Our additional matched-budget experiment further shows that the advantage over turn-level optimization remains (+2.87 mastered KCs, 95% CI: +1.39 to +4.47).
  2. Effect of longer-term planning: Although MCTS-based look-ahead improves over turn-level outcome optimization, it still does not outperform the pedagogical objective (0.81 vs. 0.85 simulated mastery).
  3. Analysis of the reward mechanism: We examine how the outcome reward can favor answer disclosure and how different reward objectives influence the resulting tutoring behavior.

We agree that our Big-Math and FinQA evaluations build on existing protocols for measuring task success and disclosure, and we will make this attribution clearer.

Therefore, our novelty is not in identifying the answer-disclosure trade-off or introducing KC-based mastery estimation. Rather, we investigate the reliability of outcome-based training when learner progress is not directly verifiable, and show how this affects the relative effectiveness of different optimization objectives. We will revise our contribution statements to make this distinction clearer.

Reviewer_qy3oOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies disclosure inflation in LLM tutoring. A tutor can reveal solution information that improves apparent learner outcomes without establishing independent understanding. It compares pedagogical-process, immediate-outcome and trajectory-level objectives on coursework tutoring dialogues. Process-trained tutors show less answer disclosure and higher estimated long-horizon mastery. Trajectory optimization improves on immediate-outcome optimization but does not surpass the process objective. Checks include a second backbone, five learner formulations, alternative judges, blinded human ratings and a verifiable mathematics transfer test. The transfer test improves over untutored performance but does not clearly distinguish the objectives.

Critical strengths

[+] The distinction between task completion and independent understanding matters for evaluating tutors.

[+] A second backbone and five learner formulations test whether the findings depend on one model or simulator.

[+] Blinded human ratings and a verifiable mathematics test check behavior beyond the main simulated mastery scores.

Critical weaknesses

[-] Process and outcome training use 3,729 versus 484 preference pairs, and Appendix H mentions different templates. The comparison does not isolate the reward objective.

[-] Transcript-based mastery may favor the elicitation reward. Alternative judges reuse these transcripts, while transfer and human ratings do not independently confirm greater learning.

[-] It is unclear whether students or sessions overlap across splits. The numerical DPO training settings are also missing.

Questions for authors
  1. Does the gap remain with the same candidates, training contexts, preference counts, and optimizer updates?
  2. Can fresh KC-aligned questions, answered without the tutoring transcript, distinguish the trained tutors?
  3. Were splits disjoint by student or session? Please report their sizes and the DPO settings.
6 comments
AuthorsOpenReview ↗

Response to Reviewer qy3o

We thank the reviewer for the constructive feedback and for finding the importance of the distinction of task completion vs learning for the AI tutor evaluation. Below, we address the concerns and questions raised in the review.

W1: The comparison does not isolate the reward objective due to differences in preference pairs, candidates, and prompt templates.

We thank the reviewer for pointing out this concern. We agree that differences in training data and optimization settings could affect the comparison. In the main experiments, we monitored validation loss during training and used validation-based checkpoint selection to avoid overfitting, resulting in different training durations. The corresponding optimizer step counts were 418, 275, and 590, respectively. To further address this concern, we conducted an additional controlled experiment where we retrained the pedagogical tutor (Reward A) using the same data budget, candidate pool, prompt format, and DPO configuration as the outcome-optimized tutor (Reward B).

We matched the following settings:

ParameterReward BMatched Reward A
Preference pairs484484
Training / validation pairs436 / 48436 / 48
Candidate pool8 generated responsesSame 8 generated responses
DPO epochs55
Optimizer steps275275

Both models use the same Qwen2.5-7B backbone, learning rate (5e-6), DPO beta (0.1), effective batch size (8), and LoRA configuration. Thus, the main difference in this additional experiment is the reward used to construct preference pairs.

Results. We evaluated the matched Reward A model on 40 tutoring contexts with two simulated student configurations over 32 turns. The results are summarized below.

TutorFinal mastered KCs
Reward A (full, 3,729 pairs)18.92
Reward A (matched, 484 pairs)18.30
Reward B (484 pairs)15.30
Base Tutor13.38

Interestingly, the matched Reward A model still outperforms Reward B by 3 mastered KCs (95% CI: +1.39 to +4.47, p = 0.0007). The matched model also retains approximately 83% of the original Reward A advantage over Reward B.

We further compared mastery gains rather than final mastery alone. The matched Reward A model maintains a significant improvement over Reward B (+2.59 KCs, p = 0.0016).

These results suggest that the observed advantage of pedagogical optimization cannot be explained solely by the larger preference dataset or differences in training configurations. Even with a substantially reduced and matched training budget, the pedagogical reward continues to outperform the outcome-based reward in our simulated long-horizon evaluation.

We acknowledge that the evaluations still rely on simulated students and model-based mastery judgments. We will include this additional controlled experiment in the revised manuscript.


W2: Transcript-based mastery may favor the elicitation reward, while transfer and human evaluations do not independently confirm greater learning.

We agree with the reviewer that transcript-based mastery estimation may favor tutors that encourage students to explain their reasoning. This is a valid limitation of model-based evaluation, and we acknowledge it in our paper.

To complement the transcript-based mastery evaluation, we conducted a separate transfer experiment using fresh questions, where the learner was evaluated on problems different from those discussed during tutoring. We also examined post-test performance using only the student's turns, rather than the complete tutoring transcript, to reduce the influence of direct tutor-provided information.

As reported in Section 6.4 (Table 3), tutoring improves accuracy from 14.7% without tutoring context to 25.3–26.7% across the four tutors. These improvements are statistically significant compared with the no-tutoring baseline (p < 0.05). However, the differences between the tutoring objectives are small (at most 1.4 percentage points) and do not establish a significant advantage for any particular tutor.

This is also why we explicitly state in Section 6.4 that we use this experiment as evidence that tutoring provides transferable task-relevant information, rather than as evidence for differences between optimization objectives.

We therefore do not claim that the transfer experiment independently confirms the higher mastery of the pedagogical tutor. Instead, our findings highlight two separate observations: (1) the tutoring sessions can improve subsequent task performance, and (2) the choice of optimization objective affects the tutoring behavior and model-estimated long-horizon mastery.

We believe that a larger and a longitudinal evaluation using independent KC-aligned assessments, ideally with real students, is necessary to establish whether the observed mastery differences translate into greater independent learning. We scoped it for future research.


AuthorsOpenReview ↗

Contd. Response to Reviewer qy3o

W3/Q3: Dataset splits and DPO training configurations

We thank the reviewer for pointing out the missing details regarding data splits and training configurations.

1. Dataset splits and overlap

In StudyChat, we define a session as an individual tutoring dialogue (chatId) and a student by their unique userId. Our filtered dataset contains 4,051 tutoring contexts from 390 sessions and 58 students. Further the data were filtered by the three different reward objectives to be consumed by the training of Tutors.

The following table summarizes the data used for each objective and the trajectory evaluation.

ObjectiveSplitContextsSessionsStudents
RAR_ATrain2,86637358
RAR_AValidation31818550
RBR_BTrain43620255
RBR_BValidation484138
MCTS-RA+BR_{A+B}Train95028653
MCTS-RA+BR_{A+B}Validation504629
Trajectory evaluationHeld-out403626

The splits were constructed at the tutoring-context level rather than the student or session level. Therefore, while RAR_A, RBR_B, and RA+BR_{A+B} have no exact context overlap with trajectory evaluation, students and sessions can appear across these splits.

2. DPO training configurations

All three objectives use Qwen2.5-7B with 4-bit NF4 QLoRA and the same LoRA configuration (rank 16, alpha 32, dropout 0.05). The DPO settings are summarized below.

ParameterRAR_ARBR_BRA+BR_{A+B}
DPO beta0.10.10.1
Learning rate5e-65e-65e-6
Epochs155
Effective batch size888
Optimizer steps418275590

We additionally use sigmoid DPO loss, AdamW optimizer, cosine learning-rate scheduling with 10% warmup, bf16 precision, seed 42, and a maximum sequence length of 1536. LoRA is applied to the query, key, value, output, gate, up, and down projection modules.

The number of training epochs differs across objectives (1 for RAR_A, 5 for RBR_B, and 5 for RA+BR_{A+B}). We monitored validation loss during training and used validation-based checkpoint selection to avoid overfitting, resulting in different training durations. The corresponding optimizer step counts were 418, 275, and 590, respectively.

Although the number of epochs differs, the effective batch size and core optimization hyperparameters remain identical across objectives.

AuthorsOpenReview ↗

Please review our responses

Dear Reviewer qy3o,

Since we are fast approaching the end of the discussion period can you kindly have a look at the responses and reconsider your scores.

Best, Authors

Reviewer_ygy1OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The authors present a method for optimizing a LLM as a tutor using either a reward that prioritizes features of interventions that are expected to be effective (“instructional quality, elicitation of learner reasoning” etc), or the immediate student performance or the expected long term reward using simulated lookahead.

Figure 3 looks at tutoring behavior but it is not so surprizing pedagogical does better on things like answer withholding since that is part of the reward objective for the pedagogical model.

Critical strengths

It is interesting that the expected long term reward optimizer does not beat the pedagogical reward in simulated students. This will depend on the simulator quality, and it is interesting that in contrast to the other results, people preferred the trajectory-level tutors over the pedagogical optimized tutor.

Critical weaknesses

These results aren’t so surprizing given Google’s worm on LearnLM (which moved from an initial interested/focus on supervised learning and RL to a focus on prompting for behaviors expected to be pedagogically effective). Similar, Khanmigo also uses a pedagogical prompt style approach.

There are no real world experiments with students, and the overall novelty is limited.

Questions for authors

Why do you think people preferred the trajectory optimized tutors?

3 comments
AuthorsOpenReview ↗

Response to Reviewer ygy1 Part 1/2

We thank the reviewer for the constructive feedback and for finding some of our results interesting. Below, we address the concerns and questions raised in the review.

W1: The results are not surprising given LearnLM and Khanmigo, and Limited Novelty.

We agree with the reviewer that the importance of pedagogically guided tutoring is already recognized in existing systems such as LearnLM and Khanmigo. Our findings are also consistent with this broader understanding. However, we would like to clarify that our main contribution is not simply to show that pedagogical tutoring performs better, but to investigate when and why optimizing learner outcomes may fail to improve actual learning. We provide a systematic mechanism to validate the effectiveness of different AI based tutoring strategies. Recent research has increasingly explored optimizing LLM tutors using immediate learner performance or longer-term simulated outcomes ([1] Scarlatos et al., 2025; [2] Shi et al., 2026). While this is a reasonable direction, it raises an important question: Does improving the measured learner outcome necessarily lead to better learning?

Our paper examines this question through a controlled empirical comparison of three optimization objectives. We highlight three key findings:

  • Outcome optimization does not necessarily improve long-term mastery. The pedagogically optimized tutor achieves higher simulated mastery (0.85) than trajectory-level (0.81) and turn-level (0.72) outcome optimization. The trend remains consistent across five learner simulator configurations and a second model backbone.

  • Longer-term planning does not fully address this limitation. Although MCTS-based look-ahead improves over turn-level outcome optimization, it still performs below the pedagogical objective.

  • The outcome reward itself can favor answer disclosure. In Appendix K, Our analysis of 14,904 candidates shows that responses rewarded only by the IRT-based outcome signal have higher answer leakage (0.180 vs. 0.078) and telling rates (50.1% vs. 28.0%). This suggests that the reward may favor responses that help learners complete the immediate task without necessarily improving their independent understanding.

We also observe a similar distinction in our human evaluation. While outcome-optimized tutors were preferred for immediate assistance (37.5% each), the pedagogical tutor was preferred for supporting independent understanding (43.8%). Further analysis shows that this difference is more prominent when the tutors differ in how much answer information they reveal (14% vs. 43%).

Thus, while the benefits of pedagogically guided tutoring are not new, our work provides empirical evidence of a specific failure mode in outcome-based optimization, which we refer to as disclosure inflation. We believe this distinction is important, as recent research increasingly uses estimated learner outcomes to train and evaluate LLM tutors.

Question: Why did people prefer the trajectory-optimized tutors?

We believe this is mainly because outcome-optimized tutors provide more direct help, which annotators may find more useful immediately. However, this preference changes when the focus shifts to building independent understanding. In our human evaluation (48 situations), both trajectory-level and turn-level tutors were preferred equally for immediate help (37.5% each), compared to 25% for the pedagogical tutor. However, when asked which response better supports understanding, the pedagogical tutor became the most preferred (43.8%). We further analyzed the individual judgments and found an interesting pattern. In 28 situations where the tutors differed in how much of the answer they revealed, the pedagogical tutor's preference increased from 14% (immediate help) to 43% (understanding). In contrast, when the tutors provided similar levels of disclosure (20 situations), there was almost no change (40% vs. 45%). The trajectory-level tutor also provided partial solutions more frequently (22.9%) than the turn-level tutor (12.5%), while revealing complete answers less often (58.3% vs. 70.8%). This may explain why its responses were perceived as helpful without always providing the full solution. These findings suggest that immediate preference is influenced by how much direct assistance a tutor provides, which does not necessarily translate into better support for independent learning. However, this depends on the student's request. For example, directly providing code may be appropriate when explicitly requested, while simply withholding an answer without sufficient guidance is not necessarily better tutoring.

References:

[1] Training LLM-based tutors to improve student learning outcomes in dialogues. AIED 2025,

[2] Beyond pedagogical principles: Multi-horizon preference optimization for efficient socratic tutoring. ACL 2026

AuthorsOpenReview ↗

Response to Reviewer ygy1 Part 2/2

W2: No real-world experiments with students.

We agree that the evaluation with real students is important to validate our findings from simulated learners. We already provided a turn-level human evaluation in Section 6.5. To address your concern on the actual learning validation, we conducted a small-scale pilot study with 8 university participants (4 B.Tech graduates and 4 CS PhD students) on stock-market investing, an out-of-domain topic for our tutors. All participants had a CS background and no formal finance education.

Knowledge components and question selection. We used FinanceMath (Zhao et al., 2024), containing 1,200 finance problems with expert-written Python solutions. Since its 864-term knowledge bank is not publicly available, we extracted six KCs from the formulas used in equity-related problems using Claude Opus 5.5, followed by validation from an M.Com. graduate. These cover stock returns, earnings and valuation ratios, dividend valuation, systematic risk, portfolio diversification, and stock-market indices.

For each KC, we searched the FinanceMath questions and tables using relevant financial terms, obtaining 14–55 candidate problems. We manually selected 20 questions, ensuring that each tested one KC, could be solved using its formulas, and had no overlap between pre-test and post-test.

We assigned difficulty based on the reference solution programs:

  • Simple: One formula with directly available inputs (1–2 arithmetic steps).
  • Medium: Requires intermediate calculations, table aggregation, or identifying relevant inputs.
  • Hard: Requires combining multiple formulas or non-trivial reasoning steps.

We selected 8 pre-test questions (5 simple, 2 medium, 1 hard) and 12 post-test questions (4 per difficulty level), with each KC appearing at least once in the pre-test and twice in the post-test. We independently verified all answers against the FinanceMath ground truth and corrected six minor textual inconsistencies without changing any numerical values.

Experimental protocol. Each participant completed the pre-test, attended a tutoring session with one of the three optimized tutors (RAR_A: 3 participants, RBR_B: 3, RA+BR_{A+B}: 2), and completed the post-test independently. Sessions ended after covering all six KCs or reaching 32 turns. Tutors received only the KC descriptions, without access to any test questions. Participants could use calculators but no external resources.

Results. The pre-test confirmed limited prior knowledge: six participants scored 1/8 and two scored 0/8.

Post-testRAR_A (n=3)RBR_B (n=3)RA+BR_{A+B} (n=2)
Simple12/1210/128/8
Medium6/125/123/8
Hard1/120/120/8
KCs covered per student6, 6, 65, 5, 46, 6

We observe that all three tutors helped students solve most simple questions, but performance dropped on medium and hard questions. Interestingly, RAR_A and RA+BR_{A+B} covered all six KCs, whereas RBR_B covered only four or five within the same turn budget. This is consistent with our simulated observations that turn-level outcome optimization may focus more on individual KCs at the cost of overall coverage.

We acknowledge that the sample is too small to establish statistically significant differences between tutors. However, this pilot provides initial evidence from real learners using independently verified post-test questions rather than transcript-based mastery judgments. We also recognize that immediate post-test improvement alone may not fully capture learning effectiveness, particularly knowledge retention and independent reasoning. A larger controlled longitudinal study would be needed to examine these aspects, which we leave for future work.

References:

[3] FinanceMATH: Knowledge-Intensive Math Reasoning in Finance Domains. ACL 2024

AuthorsOpenReview ↗

Please review our responses

Dear Reviewer ygy1,

Since we are fast approaching the end of the discussion period can you kindly have a look at the responses and reconsider your scores.

Best, Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We thank all reviewers for their thoughtful and constructive feedback. Below, we summarize what the reviewers found promising and how we addressed their main concerns during the discussion period.

What the reviewers found promising

  • Reviewer ygy1 found it interesting that trajectory-level outcome optimization with look-ahead does not outperform the pedagogical reward with simulated students. They also found the human-preference results interesting, and rated the paper clear and easy to follow, with no substantive soundness concerns.
  • Reviewer qy3o noted that the distinction between task completion and independent understanding matters for evaluating tutors. They also valued our robustness checks: a second backbone, five learner simulator formulations, blinded human ratings, and a verifiable mathematics transfer test.
  • Reviewer ChHr found the paper well motivated by an important problem in educational dialogues, namely that optimizing for immediate student success can inadvertently reward answer disclosure. They also highlighted the controlled comparison of three preference signals across two open-weight models, and rated the presentation clear and easy to follow.

Key concerns and how we addressed them

ConcernReviewer(s)How we addressed it
Limited novelty (LearnLM, Khanmigo, prior disclosure trade-off work)ygy1, ChHrClarified that our contribution is studying outcome-based training when learning is not directly verifiable. Showed that the outcome reward itself favors disclosure (leakage 0.180 vs. 0.078; telling 50.1% vs. 28.0%). Merged C1 and C3 in the revision.
Unequal training budget (3,729 vs. 484 pairs, different templates)qy3oNew matched experiment (same pairs, candidates, template, and optimizer steps): the pedagogical tutor still leads (18.30 vs. 15.30 mastered KCs; 95% CI +1.39 to +4.47).
Offline DPO only; no online or on-policy baselinesChHrClarified the scope as reward construction under offline preference optimization. A preliminary online GRPO run shows the same pattern (0.672 vs. 0.851 mastery).
No SFT baselineChHrAdded five SFT variants. The DPO pedagogical tutor (18.92 KCs) outperforms all of them (best 16.38), and pedagogical selection beats outcome selection (16.38 vs. 15.16).
No evaluation with real studentsygy1, ChHrRan a pilot with 8 participants on an out-of-domain finance topic with independent post-tests. The pedagogical and trajectory-level tutors covered all six KCs vs. four or five for the turn-level tutor (preliminary).
Reliance on simulators and LLM judges; judge–human agreementqy3o, ChHrResults hold across five simulators. Alternative judges (Claude Opus 4.8, GPT-5.4) preserve the ranking (r = 0.93, 0.94). Reported human–GPT-4o agreement (71%, κ=0.58\kappa=0.58) and annotator backgrounds.
Transcript-based mastery may favor elicitation; transfer test does not separate tutorsqy3o, ChHrAcknowledged this limitation. Clarified that Table 3 shows transferable information (14.7% to 25.3–26.7%), not a ranking of objectives. The pilot uses independent post-test questions.
Missing details (data splits, DPO settings, PregressionP_{\text{regression}}, reward weights, "Zero-Shot ETH", compute)qy3o, ChHrReported splits and DPO settings, defined PregressionP_{\text{regression}}, added weight-sensitivity results (96.2% of preferences unchanged), renamed the simulator, and added a compute breakdown.
KC granularity, single-KC actions, missing MathDial citationChHrDescribed KC extraction and validation (91% source-verified; 90% accepted after minor edits). 19% of KCs are higher-order, and MCTS tracks multiple KCs. Added the MathDial citation.

We have uploaded the revised manuscript incorporating these additional experiments, clarifications, and limitations. Changes are highlighted in blue (Reviewer qy3o) and magenta (Reviewer ChHr), and our replies to each reviewer list where each change appears. We thank the reviewers again for helping us strengthen the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.