- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Replay-Verified Factor-Level Distillation (RVFD) for repairing failed tool-use traces. A black-box teacher proposes a local edit to one of four schema-aligned factors—function, argument key, argument value, or stop/missing call—and the edit is retained as a supervised fine-tuning (SFT) target only if applying it in isolation makes the trace pass a checker. The student is then trained using standard token-level SFT and evaluated either via greedy decoding without test-time assistance or via beam search with replay verification. On the constructed BFCL-CF benchmark, RVFD improves the greedy repair success rate of Qwen2.5-7B and Qwen2.5-14B students by approximately 5.4 percentage points over direct teacher-output SFT. The augmented factor-level model also outperforms the whole-trace RFT/STaR baseline, although the two methods use different training-set sizes. Test-time replay further improves accuracy. The paper additionally evaluates naturally occurring BFCL outputs, alternative proposal providers, CAR/CausalFlow selectors, and stateful tau2 tasks. The results suggest that replay can be effective for constructing local repair supervision.
Critical strengths
A clear and practically useful separation of proposal, verification, and learning. The paper addresses a genuine problem in tool-use distillation: a teacher-generated repair can be fluent and schema-valid while remaining behaviorally incorrect. Treating the teacher as a proposal generator and using executable checking as the label-selection mechanism is a simple but practical design. The paper also clearly distinguishes the student's greedy-decoding performance from its performance with test-time replay, thereby avoiding conflation between learned model capability and verifier-assisted search. This separation substantially improves the interpretability of the experimental results.
Broad and notably transparent evaluation. The paper evaluates two student model sizes, three random seeds, multiple training baselines, selector comparisons on a fixed candidate pool, different teacher models, naturally occurring BFCL outputs, and stateful tau2 tasks. Importantly, it transparently reports settings in which the proposed method does not outperform the alternatives: RVFD ties CAR-point and oracle-step CausalFlow on the shared candidate pool; factor-local repair underperforms whole-trace rewriting on natural errors; and success on tau2 requires structured assistance. Despite the remaining attribution issues, this transparency gives the paper meaningful informational value.
Critical weaknesses
Limited methodological novelty and no demonstrated independent value of factor-level supervision.
The core RVFD pipeline asks a teacher to generate local edits, uses checker/replay to filter out invalid edits, and then performs standard token-level SFT on the retained edits. This is largely a direct combination of execution-filtered distillation, rejection sampling, and verifier-guided data curation. The paper introduces no new learning objective, optimization algorithm, or credit-assignment mechanism; the proposed factor taxonomy primarily decomposes the tool-call schema into function, argument key, argument value, and stop/missing call. Therefore, the current version does not establish a substantive methodological contribution beyond existing verifier-based data-filtering paradigms.
More importantly, the experiments do not demonstrate an independent benefit from the factor-level representation itself. In the most direct comparison with execution-filtered whole-trace training, the factor-level condition uses 1,023 training examples whereas the whole-trace condition uses only 681; their development sets likewise contain 256 and 170 examples, respectively. The reported 3.06-percentage-point gain could therefore result from having more examples, shorter targets, different token or optimization-update budgets, or different augmentation compositions rather than from factor-level credit assignment. The purportedly balanced comparison in Table 2 also does not establish whether the methods use exactly the same failed examples, candidate pools, proposal budgets, response constraints, accepted candidates, and supervision-token budgets. Statistical significance cannot eliminate these protocol confounds.
The benchmark and proposal protocol are closely aligned with the method, and split independence is unclear.
BFCL-CF introduces exactly one corruption to a known schema-aligned factor. Appendix C indicates that the teacher is told which factor to repair and is required to return only a compact edit object; Algorithm 2 rejects edits outside the specified factor. The primary checker tests equality with a canonical reference trace rather than behavior under live API execution. This differs materially from natural repair settings, where the erroneous step and factor are generally unknown, multiple errors may interact, and multiple semantically valid repairs may exist. This setup also weakens a central motivation of the paper—that factor-level supervision avoids incidental or conflicting changes in whole responses—because the main RVFD proposal protocol already restricts the teacher to producing a single compact edit for an oracle-specified factor.
In addition, the paper reports a proposal-disjoint split but does not state whether all corruptions derived from the same original BFCL request or reference trace, or examples sharing the same API schema, are assigned to a single split. If sibling corruptions derived from the same clean trace appear across the training and test sets, the reported generalization performance may be substantially overestimated.
Questions for authors
Split independence: What is the exact split key used for BFCL-CF? Please report the overlap among the training, development, and test sets at the following levels: original BFCL example or request ID, canonical clean reference trace, function/API name, and complete tool schema. Can the authors provide an evaluation in which all corrupted examples derived from the same clean BFCL example are assigned exclusively to a single split? If sibling corruptions appear across different splits, this would substantially lower my assessment of the paper.
Strictly matched factor-level versus whole-trace comparison: Can the authors compare replay-accepted factor-level targets with replay-accepted whole-trace targets using exactly the same original failed examples and accepted candidate pool, the same number of training examples, matched proposal and replay budgets, matched numbers of optimizer updates, and preferably matched supervision-token budgets? Please also clarify whether the three balanced settings in Table 2 use exactly the same inputs and accepted examples. Such a comparison is necessary to determine whether the factor decomposition has empirical value; however, even a positive result would not, by itself, address the lack of substantive novelty relative to existing verifier-based data-filtering paradigms.
Dependence on oracle localization and output constraints: How much of the reported performance is retained when the proposal model must jointly identify the erroneous step or factor and generate the repair? Alternatively, can the authors separately ablate (i) oracle factor localization, (ii) constrained edit-schema generation, (iii) replay filtering, and (iv) factor-level training targets? This would help determine whether the gains arise from replay-verified factor-level credit assignment or primarily from substantially simplifying the proposal task.