acceptodds
Under review as a conference paper at ICLR 2027

Replay-Verified Factor-Level Distillation for Tool-Trace Repair

Abstract

Distilling tool use from a black-box teacher is complicated by a simple failure mode: a repair can be fluent and schema-valid yet remain incorrect under execution. We introduce replay-verified factor-level distillation (RVFD), a procedure that treats teacher outputs as proposals rather than ground truth. RVFD decomposes a proposed repair into schema-aligned factors, applies each edit to the failed trace, and retains an edit as supervision only when replay reaches the desired outcome. On BFCL-CF, full-parameter Qwen2.5-7B and 14B students trained with RVFD improve greedy repair success over direct teacher-output SFT by 0.0547 and 0.0534, respectively, without a teacher or verifier at test time. An augmented factor student reaches 0.9199 greedy success, compared with 0.8893 for whole-trace execution-filtered SFT; this comparison also differs in training-set size. On naturally occurring errors in archived BFCL outputs, local repair raises AST accuracy from 0.5939 to 0.6774, while whole-trace rewriting with beam4 and checker access reaches . Optional test-time replay improves accuracy further, but does not surpass CAR-point or oracle-step CausalFlow on a shared candidate pool. These results support replay as a source of local credit assignment for tool-trace repair, rather than as a general solution to end-to-end agent learning. Code and data are available at https://anonymous.4open.science/r/rvfd-aaai27/.

Primary area:
reinforcement learning
Keywords:
knowledge distillation, tool use, counterfactual repair, execution feedback, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_rw8tOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Replay-Verified Factor-Level Distillation (RVFD) for repairing failed tool-use traces. A black-box teacher proposes a local edit to one of four schema-aligned factors—function, argument key, argument value, or stop/missing call—and the edit is retained as a supervised fine-tuning (SFT) target only if applying it in isolation makes the trace pass a checker. The student is then trained using standard token-level SFT and evaluated either via greedy decoding without test-time assistance or via beam search with replay verification. On the constructed BFCL-CF benchmark, RVFD improves the greedy repair success rate of Qwen2.5-7B and Qwen2.5-14B students by approximately 5.4 percentage points over direct teacher-output SFT. The augmented factor-level model also outperforms the whole-trace RFT/STaR baseline, although the two methods use different training-set sizes. Test-time replay further improves accuracy. The paper additionally evaluates naturally occurring BFCL outputs, alternative proposal providers, CAR/CausalFlow selectors, and stateful tau2 tasks. The results suggest that replay can be effective for constructing local repair supervision.

Critical strengths

A clear and practically useful separation of proposal, verification, and learning. The paper addresses a genuine problem in tool-use distillation: a teacher-generated repair can be fluent and schema-valid while remaining behaviorally incorrect. Treating the teacher as a proposal generator and using executable checking as the label-selection mechanism is a simple but practical design. The paper also clearly distinguishes the student's greedy-decoding performance from its performance with test-time replay, thereby avoiding conflation between learned model capability and verifier-assisted search. This separation substantially improves the interpretability of the experimental results.

Broad and notably transparent evaluation. The paper evaluates two student model sizes, three random seeds, multiple training baselines, selector comparisons on a fixed candidate pool, different teacher models, naturally occurring BFCL outputs, and stateful tau2 tasks. Importantly, it transparently reports settings in which the proposed method does not outperform the alternatives: RVFD ties CAR-point and oracle-step CausalFlow on the shared candidate pool; factor-local repair underperforms whole-trace rewriting on natural errors; and success on tau2 requires structured assistance. Despite the remaining attribution issues, this transparency gives the paper meaningful informational value.

Critical weaknesses

Limited methodological novelty and no demonstrated independent value of factor-level supervision.

The core RVFD pipeline asks a teacher to generate local edits, uses checker/replay to filter out invalid edits, and then performs standard token-level SFT on the retained edits. This is largely a direct combination of execution-filtered distillation, rejection sampling, and verifier-guided data curation. The paper introduces no new learning objective, optimization algorithm, or credit-assignment mechanism; the proposed factor taxonomy primarily decomposes the tool-call schema into function, argument key, argument value, and stop/missing call. Therefore, the current version does not establish a substantive methodological contribution beyond existing verifier-based data-filtering paradigms.

More importantly, the experiments do not demonstrate an independent benefit from the factor-level representation itself. In the most direct comparison with execution-filtered whole-trace training, the factor-level condition uses 1,023 training examples whereas the whole-trace condition uses only 681; their development sets likewise contain 256 and 170 examples, respectively. The reported 3.06-percentage-point gain could therefore result from having more examples, shorter targets, different token or optimization-update budgets, or different augmentation compositions rather than from factor-level credit assignment. The purportedly balanced comparison in Table 2 also does not establish whether the methods use exactly the same failed examples, candidate pools, proposal budgets, response constraints, accepted candidates, and supervision-token budgets. Statistical significance cannot eliminate these protocol confounds.

The benchmark and proposal protocol are closely aligned with the method, and split independence is unclear.

BFCL-CF introduces exactly one corruption to a known schema-aligned factor. Appendix C indicates that the teacher is told which factor to repair and is required to return only a compact edit object; Algorithm 2 rejects edits outside the specified factor. The primary checker tests equality with a canonical reference trace rather than behavior under live API execution. This differs materially from natural repair settings, where the erroneous step and factor are generally unknown, multiple errors may interact, and multiple semantically valid repairs may exist. This setup also weakens a central motivation of the paper—that factor-level supervision avoids incidental or conflicting changes in whole responses—because the main RVFD proposal protocol already restricts the teacher to producing a single compact edit for an oracle-specified factor.

In addition, the paper reports a proposal-disjoint split but does not state whether all corruptions derived from the same original BFCL request or reference trace, or examples sharing the same API schema, are assigned to a single split. If sibling corruptions derived from the same clean trace appear across the training and test sets, the reported generalization performance may be substantially overestimated.

Questions for authors

Split independence: What is the exact split key used for BFCL-CF? Please report the overlap among the training, development, and test sets at the following levels: original BFCL example or request ID, canonical clean reference trace, function/API name, and complete tool schema. Can the authors provide an evaluation in which all corrupted examples derived from the same clean BFCL example are assigned exclusively to a single split? If sibling corruptions appear across different splits, this would substantially lower my assessment of the paper.

Strictly matched factor-level versus whole-trace comparison: Can the authors compare replay-accepted factor-level targets with replay-accepted whole-trace targets using exactly the same original failed examples and accepted candidate pool, the same number of training examples, matched proposal and replay budgets, matched numbers of optimizer updates, and preferably matched supervision-token budgets? Please also clarify whether the three balanced settings in Table 2 use exactly the same inputs and accepted examples. Such a comparison is necessary to determine whether the factor decomposition has empirical value; however, even a positive result would not, by itself, address the lack of substantive novelty relative to existing verifier-based data-filtering paradigms.

Dependence on oracle localization and output constraints: How much of the reported performance is retained when the proposal model must jointly identify the erroneous step or factor and generate the repair? Alternatively, can the authors separately ablate (i) oracle factor localization, (ii) constrained edit-schema generation, (iii) replay filtering, and (iv) factor-level training targets? This would help determine whether the gains arise from replay-verified factor-level credit assignment or primarily from substantially simplifying the proposal task.

Reviewer_r3vTOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents RVFD, a distillation method that only utilizes teacher-proposed edits that pass the checker. The teacher proposes fixes to function names, argument keys/values, or missing calls from failed tool traces of the student. Correct edits are used to train the student through standard SFT. RVFD improves controlled repair performance over direct teacher-output and response-filtered baselines.

Critical strengths
  1. The paper proposes a straightforward supervised method for distillation. Teacher edits are used for distillation only when they pass the checker.
Critical weaknesses
  1. Novelty is limited. Filtered distillation is established in STaR and other literature, as the authors acknowledge. RVFD is an application of existing techniques to tool repair.

  2. Evaluation does not support the efficacy of the method. The comparison against execution-filtered SFT cannot be trusted due to an unfair comparison. The authors also acknowledge this and state that "an equal-data rerun is needed to estimate the independent effect of factor granularity". There is no direct evidence of the efficacy of removing incidental trace content. On natural failures, greedy whole-trace rewriting achieves 0.7832 versus 0.6774 for local repair. The benchmark shows improvement with RVFD, but its design is flawed. It is constructed using single-factor corruptions, which align with RVFD's assumptions.

Questions for authors
  1. Many evaluation limitations are acknowledged in Section 7. Could you address them? Including the equal data comparison between RVFD and execution-filtered SFT.
  2. Can you show the benefit of factor-level supervision by itself?
  3. Can you show the benefit of removing incidental traces through an ablation?
  4. What evidence do you have to prove that the benchmark is valid and fair? It seems to favor RVFD.
  5. The paper states that the splits are proposal-disjoint. What about the task/reference level?
  6. Does the student know the corrupted factor at evaluation?
  7. Why is a teacher needed? You already have the correct and corrupted traces. Can you show the benefit of having a teacher?
  8. How does RVFD perform on realistic failures that have multiple errors?
  9. Can you show the results using live tool execution rather than reference trace matching?
Reviewer_DcZmOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RVFD, which turns black-box teacher edits for failed tool-use traces into factor-level SFT labels only when executable replay confirms that the single edit fixes the failure. It builds BFCL-CF, a factor-balanced repair benchmark, and evaluates 7B/14B students on BFCL-CF, natural BFCL failures, cross-provider teachers, and tau2 tasks. Results show consistent gains from replay-validated factor targets over direct teacher SFT, while test-time replay adds further gains at extra execution cost and stateful tau2 remains difficult.

Critical strengths

Clear and simple formulation. RVFD cleanly separates proposal, replay validation, and learning, and the replay-checked factor label is easy to implement and reproduce. Good experimental discipline. The paper explicitly separates train-time label construction from test-time replay, and reports false adoption, rescue, and harm rather than only aggregate success. Honest analysis of limits. The authors show that RVFD ties CAR-point and CausalFlow-oracle on a fixed candidate pool, locating the selection ceiling rather than overclaiming a new selection rule.

Critical weaknesses

The main benchmark is synthetic and narrow. BFCL-CF injects single-factor corruptions, while real failures often involve multiple interacting errors, hidden state, and irreversible actions; the natural BFCL study is only a checker-gated overlay. The key factor-vs-whole-trace comparison is confounded. The augmented comparison uses 1,023 factor rows versus 681 whole-trace rows, so the reported 3.06-point gap cannot isolate the effect of factor granularity. Scale and selection contribution are limited. The proposal set is small, only Qwen2.5-7B/14B and three seeds are used, and RVFD matches oracle-style selectors once candidates are fixed. Stateful and cost analyses remain weak. Tau2 standalone student performance is low and the hybrid relies on fallback, while end-to-end replay latency, API cost, and coverage bottlenecks are not systematically analyzed.

Questions for authors

See weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.