- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The individual Qwen incompatibility was reported previously in vLLM issue #32926. The authors' contribution is to quantify it as an evaluation-validity failure, decompose its causes, and follow it into interactive evaluation and Reinforcement Learning.
Critical strengths
The problem is easily ignored yet important: different root causes resulting in the same error really confuse humans sometimes. The motivation for solving this has real-world impact.
The experimental design focuses on single-factor interpretation, which can help rigorously identify the problems.
Critical weaknesses
- I am pretty sure AI largely generated this paper with little sign of human writing. (1) First of all, the core of the paper, i.e., the abstract, is completely generated by AI, as detected by GPTZero. (2) Figure 1 contains a watermark in the bottom left corner; based on my experience using vibe coding tools, I am pretty confident that AI generated the Figure without human revision. Even though people use AI to plot very often, polishing up the code to make it better is always necessary. (3) Some obscure descriptions, e.g., "Model emission", "Function Calling uses 2,048 output tokens". I am skeptical about whether the authors really put sincere effort into this paper.
Questions for authors
For BFCL, the authors said that 16,384 is the max_model_length, i.e., the input token plus the output token. But both parts are not specified; what exactly are they?
General response
We thank all three reviewers for their time. We are glad that the reviewers found the problem practically important (Magp, yeZE), the template-by-parser factorial and single-factor design useful (Magp, yeZE), and the fixed-output replay and the separation of experience access from task performance convincing (6EJ2).
We have uploaded a revised PDF and the anonymous supplementary package. All changes are additions or clarifications; no reported number has changed. The pdfdiff shows every edit.
Anonymous verification package (6EJ2). The package described in Appendix B was omitted from the original upload. It is now attached as supplementary material. It contains the statistical code and inputs, the BFCL fixed-output replay bundle, the τ-bench trajectories, the RL accepted-lineage event checks, and the native-FC evaluation records. We checked it for identifying information before upload. New analysis: does the policy use the restored feedback during training? (Magp) Using the retained training rollouts of the repaired arm, we measured within-trajectory feedback use across all 16,844 executed calls. The model does act on feedback: after a failed first test it calls the tool again in 75.9% of cases, changes the code in 67.9% of those retries, and reaches a passing test in 14.4%. This in-training repair rate does not increase over the 150 steps (16.1%, 14.4%, 13.6%, 13.0%, 15.0% in 30-step bins), which is consistent with the absent held-out gain. The analysis is summarized in Section 5.5 and detailed in the new Appendix G.11 (Table 34). Presentation (yeZE, 6EJ2). All five figures are now vector graphics typeset in the paper's font. Figure 1 is redrawn in a larger 2×2 layout, and its explanatory footnote, which had been rendered at an unreadable size, is moved into the caption. We now gloss "emission" as "raw model output" at its first use in the abstract and introduction, and we added the exact BFCL token budgets (Appendix G.4). Scope (Magp, 6EJ2). We made the limits of the scale and RL claims more explicit in Section 6. The paper does not claim that this failure is prevalent in modern deployments, that censoring generally grows with scale, or that the null RL result shows equivalence.