acceptodds
Withdrawn from ICLR 2027

Hard Negatives Should Match the Student: Teacher–Student–Judge Preference Distillation for Agentic Tool Use

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Open-weight language models lag behind frontier systems on "agentic" workflows: long-horizon planning, strict tool-schema adherence, and policy compliance under adversarial user pressure. We present the Teacher–Student–Judge (TSJ) pipeline, which synthesizes Direct Preference Optimization (DPO) pairs by letting the target model (Student) compete against a stronger Teacher inside simulated multi-turn conversations, with a Judge that selects winners and synthesizes hard negatives against the Student's own successful replies. Fine-tuning Qwen3-30B-A3B on 7,747 TSJ pairs improves Tau-Bench Airline from 32% to 44% and Retail from 54% to 61%; general capabilities remain within 1.3 points across four evaluations (MMLU, GSM8K, and HumanEval improve, while IFEval declines 0.9 points). Attribution: four-condition ablations on Qwen3-4B associate removing the Judge-synthesized-negative slice with a 2.2-point lower macro-average; size-matched control scores 1.0 point above distillation-only. Generality and its boundary: the recipe fails when Granite-4.1-8B is trained on Qwen-generated data, but succeeds (+5.8 points, +21% relative) when regenerated with Granite as Student. At identical dataset size, Granite-matched synthesized negatives beat negative-free distillation by +4.1 points. Offline vs. online: at matched scale, offline TSJ-DPO matches online GRPO trained directly in the executable Tau environment (0.432 vs. 0.419 macro-average). These results position student-anchored preference pairs, rather than generic teacher demonstrations, as the highest-leverage data for agentic preference optimization.

Primary area:
reinforcement learning
Keywords:
preference optimization, direct preference optimization, agentic language models, tool use, synthetic data generation, hard negative mining, LLM alignment, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_sGDhOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a synthetic preference-data pipeline for multi-turn tool use. A Judge selects between teacher and student responses and synthesizes rejected responses when the student wins, producing more capable pairs for preference optimization. It claims that preference data should match the student’s behavior distribution, evaluated through experiments covering data-construction ablations, cross-family transfer, and an offline-versus-online comparison.

Critical strengths

Not much.

Critical weaknesses
  1. Poor presentation.
    The manuscript contains inconsistent or insufficiently contextualized statements and apparent draft remnants that make its claims difficult to assess. Examples include the abrupt qualification at L109-110, revision-oriented commentary in the Table 1 caption (L217-220), references to an "earlier draft" and removed results in Appendix A.3 ("This replaces ..."; "An earlier draft listed ... We removed them ..."), and an unexplained internal script reference in Appendix I (L904: experiments/trial7_ctx8k/hardness.py). Collectively, these issues suggest that the manuscript has not been adequately consolidated and checked, undermining confidence in the reliability of its presentation. Substantial rewriting and consistency checks are needed before its contributions can be assessed reliably.

  2. Limited novelty.
    The proposed approach uses standard DPO, with its main contribution lying in more capable preference-pair construction. Closely related ideas have been explored in prior work, including LLM-based preference judgments [1], synthetic rejected trajectories for tool use [2], and the benefits of on-policy preference data [3]. The student-win negative-generation branch and cross-family matching experiments offer a specific extension of these ideas, while the methodological advance appears incremental.

References
[1] Yuan et al. Self-Rewarding Language Models. ICML, 2024.
[2] Jung et al. DiaTool-DPO: Multi-Turn Direct Preference Optimization for Tool-Augmented Large Language Models. SIGDIAL, 2025.
[3] Tajwar et al. Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data. ICML, 2024.

Questions for authors

See weaknesses above.

Reviewer_ikvROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the issues of erroneous execution and failure to adhere to specifications in multi-turn tool-use scenarios involving open-weight language models. The authors propose a "Teacher–Student–Judge" preference data construction pipeline: when the teacher model outperforms the student, the teacher’s response is paired with the student’s erroneous response; when the student model performs better, its response is retained, while a negative example is synthesized by a "judge" model. This data is then used for training via standard Direct Preference Optimization (DPO). The primary contributions include this student-behavior-anchored preference data construction mechanism and the core premise that preference data must be specifically aligned with the target student model. Using two synthetic datasets, the authors conducted extensive evaluations on Qwen and Granite models—including benchmarking, ablation studies with controlled sample sizes, cross-model data swapping, and comparisons with online Group Relative Policy Optimization (GRPO). Results demonstrate that a 30B-parameter flagship model achieves a 6.4 percentage point improvement in macro-average performance on the τ²-bench benchmark and significantly reduces the number of interaction turns terminated due to attempts to invoke non-existent tools, indicating that this approach effectively improves certain aspects of tool-use behavior.

Critical strengths

There is a clear improvement, with key performance gains supported by statistical evidence. On the τ²-bench benchmark, the 30B model shows an average improvement of 6.4 percentage points, with a 95% confidence interval of [2.2, 10.5] percentage points based on task-level paired bootstrapping. Furthermore, the number of interaction turns aborted due to calls to non-existent tools dropped from 67–75 per 114 turns to just 1–2. These results demonstrate that the proposed approach effectively mitigates specific tool-use errors that otherwise hinder task execution.

The construction of preference data is highly targeted and offers valuable cross-model validation. TSJ incorporates both the actual errors made by the "student" model and synthetic negative examples derived from the student’s successful, correct responses—thereby ensuring that the student’s own behavioral patterns inform the training data. The authors validated this design through cross-model experiments (using two distinct model families) and ablation studies with controlled sample sizes. Notably, a mixed-data configuration—incorporating the student’s own outputs—yielded a 4.5 percentage-point improvement over a DPO baseline that relied solely on "teacher-winning" samples, with a confidence interval that excludes zero. This provides compelling evidence in support of the proposed data construction strategy.

Critical weaknesses

The ablation of non-isolated negative examples fails to provide sufficient support for the paper’s primary attribution of performance gains. Although the "control" and "distillation" conditions utilize the same sample size, the introduction of a synthetic branch alters multiple factors simultaneously: it shifts the source of positive examples from teacher responses to the student model’s own correct outputs, while also modifying input states, response formats, and sample difficulty. Consequently, the observed gains may stem from the retention of the student’s own correct responses or shifts in data composition, rather than from the matching of negative examples per se. The absence of a controlled experiment—one that holds the context and positive examples constant while varying only the source of the negative examples—prevents the authors from disentangling these potential explanations, thereby undermining the credibility of their core contribution.

Furthermore, the proposed mechanism regarding "style shortcuts" lacks direct empirical validation, and the design of the development sets further weakens the supporting evidence. Because development sets are sampled from specific data slices—partitioned by individual rows rather than complete dialogues—the validation distributions differ across conditions, and prefixes from the same dialogue may overlap. Thus, the observation that a model exhibits "lower development loss but poorer downstream transfer" is insufficient to identify style shortcuts, nor does it prove that synthetic negative examples have successfully eliminated them. The paper fails to rule out alternative explanations through interventions such as using a shared, independent development set, standardizing stylistic features, or matching response lengths. As a result, the paper’s primary mechanistic claims remain based on indirect inference, which is insufficient to justify a recommendation for acceptance.

Questions for authors

When the context and positive examples are held constant and only the source of the negative examples is varied, do the negative examples matched to a specific student still yield significant gains?

When using a shared development set partitioned by complete conversations—while controlling for response length and style—is there still direct evidence to support the explanation that "synthetic negative examples disrupt style-based shortcuts"?

Reviewer_vedROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Teacher–Student–Judge (TSJ), a pipeline for generating synthetic preference data for agentic tool use. A teacher and student respond to the same simulated multi-turn context, and a judge selects the better response. When the teacher wins, the student’s response becomes the negative; when the student wins, the judge synthesizes a negative paired with the student’s response.

Critical strengths

The paper starts from a reasonable concern: stylistic differences between chosen and rejected responses may enable shortcut learning, allowing the model to distinguish response sources rather than learn task-relevant correctness.

Critical weaknesses

The paper frames its core contribution as “Negatives Should Match the Student,” but the current ablations do not isolate the effect of negative–student matching.

  1. For student-winning cases, the paper should compare alternative negatives while keeping the student’s winning response fixed as the positive: the teacher’s response, an error-inducing rewrite of the student’s response, an error-inducing rewrite of the teacher’s response, and an unrelated response.
  2. The paper should also compare against a variant in which all candidate responses are sampled from the student, with positive and negative responses selected based on correctness.
Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.