- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes a synthetic preference-data pipeline for multi-turn tool use. A Judge selects between teacher and student responses and synthesizes rejected responses when the student wins, producing more capable pairs for preference optimization. It claims that preference data should match the student’s behavior distribution, evaluated through experiments covering data-construction ablations, cross-family transfer, and an offline-versus-online comparison.
Critical strengths
Not much.
Critical weaknesses
-
Poor presentation.
The manuscript contains inconsistent or insufficiently contextualized statements and apparent draft remnants that make its claims difficult to assess. Examples include the abrupt qualification at L109-110, revision-oriented commentary in the Table 1 caption (L217-220), references to an "earlier draft" and removed results in Appendix A.3 ("This replaces ..."; "An earlier draft listed ... We removed them ..."), and an unexplained internal script reference in Appendix I (L904: experiments/trial7_ctx8k/hardness.py). Collectively, these issues suggest that the manuscript has not been adequately consolidated and checked, undermining confidence in the reliability of its presentation. Substantial rewriting and consistency checks are needed before its contributions can be assessed reliably. -
Limited novelty.
The proposed approach uses standard DPO, with its main contribution lying in more capable preference-pair construction. Closely related ideas have been explored in prior work, including LLM-based preference judgments [1], synthetic rejected trajectories for tool use [2], and the benefits of on-policy preference data [3]. The student-win negative-generation branch and cross-family matching experiments offer a specific extension of these ideas, while the methodological advance appears incremental.
References
[1] Yuan et al. Self-Rewarding Language Models. ICML, 2024.
[2] Jung et al. DiaTool-DPO: Multi-Turn Direct Preference Optimization for Tool-Augmented Large Language Models. SIGDIAL, 2025.
[3] Tajwar et al. Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data. ICML, 2024.
Questions for authors
See weaknesses above.