acceptodds
Under review as a conference paper at ICLR 2027

BranchWise: A Path Is Not a Policy — Unfolding Successful Trajectories into Model-Tailored Decision Boundaries

Abstract

Tool-use agents make sequential decisions among multiple possible actions, yet successful trajectories expose only one realized path. Direct supervised fine-tuning (SFT) underutilizes their latent supervision: long trajectories entangle local decisions and expose only one reference branch per state, leaving alternative failure directions implicit; static demonstrations do not explicitly reveal model-specific errors; and uniform token-level supervision does not explicitly target the regions that distinguish behaviors. When trained on general tool-use data, SFT can therefore transfer inconsistently and even underperform the base model. We propose BranchWise, a post-training framework that unfolds successful trajectories into model-tailored local decision boundaries. It decomposes trajectories into state-conditioned decisions and constructs verified counterfactuals from tool constraints and execution evidence. These counterfactuals are fused with verifiable error branches generated by probing the initial policy to form compact local decision boundaries. BranchWise localizes comparisons to decision forks and combines evidence-calibrated reference-branch supervision with margin-aware listwise ranking to emphasize insufficiently separated boundaries. With all methods trained from the same general tool-use corpus rather than benchmark-specific supervision, BranchWise outperforms Vanilla SFT across -Bench, BFCL V3, ToolHop, and ToolTalk-Hard. It achieves the best overall performance among all compared methods, demonstrating stronger and more consistent downstream transfer.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Tool-Use Agents, Agent Post-Training, Preference Optimization, Decision Boundary Learning, Model-Aware Supervision

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_EsYROpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes BranchWise, a post-training framework for tool-use agents that converts successful trajectories into state-conditioned positive actions and verified counterfactual negative branches. The method combines rule and execution-evidence-derived counterfactuals with errors sampled from the initial policy, then trains with evidence-weighted SFT and a localized, margin-aware ranking loss.

Critical strengths

The paper identifies a concrete limitation of trajectory-only SFT: a successful trajectory specifies an observed action but not the local alternatives that a policy must reject. The proposed framework is useful and well connected to tool-use settings.

Critical weaknesses
  1. Table 8 has severe layout issues.
  2. The article only conducted experiments using Qwen3-4B and Qwen3-8B. It cannot be determined whether this method is only effective on Qwen-series LLMs.
  3. This article mentions many related works such as CausalFlow, but there is no comparison.
Questions for authors
  1. Can you provide more experimental results besides Qwen-series LLMs?
  2. Can you provide more baseline results such as CausalFlow?
29 comments
AuthorsOpenReview ↗

Main Response — Part 01/07

** Main Response — Part 01/07**

During this rebuttal, we added extensive controlled experiments, ablations, and in-depth analyses and redrew the paper's analysis figures, further improving the experimental validation, technical argumentation, and presentation of BranchWise. We believe this additional evidence makes the paper more complete, rigorous, and persuasive. We sincerely hope that the reviewer will consider these improvements when reassessing the paper's score. Your recognition would also encourage us to continue pursuing more rigorous, interesting, and valuable research.

Thank you for recognizing the issue studied in this paper: successful trajectories show only the actions actually taken, without specifying which alternative local actions the model should reject. We agree that the original submission's evaluation on the Qwen family alone is insufficient to establish transfer to other models, and that the related-work comparison and presentation of Table 8 also need improvement.

In response, we conducted six new training experiments during the rebuttal:

  • E1: Same-Data Action-DPO, E2: Same-Data SimPO, and E3: Same-Data Uniform Listwise, testing the role of the optimization objective (settings and results in Section 2.3).
  • E4: Action-DPO Using Only Model-Generated Errors, testing the role of error sources (settings and results in Section 2.3).
  • E5: Llama Standard SFT and E6: Llama Full BranchWise, testing cross-model gains (settings and results in Sections 1.2–1.4).

We also completed one Base evaluation of the untrained Llama backbone as the shared starting point for E5 and E6 (Sections 1.2–1.3). We additionally include a methodological comparison with CausalFlow and one new analysis, A5: Evaluation Protocol and Aggregation Audit (Sections 3.1–3.3).

We first answer the two Questions for Authors, then address the three Critical Weaknesses individually, for a total of five issues. Overlapping issues are cross-referenced by section number within this document.

1. Response to Question 1: New Llama experiments show that multi-step tool-use gains are not restricted to Qwen

Your original question: Can you provide more experimental results besides Qwen-series LLMs?

1.1 Summary of our response

Yes. We conducted two new Llama-3.1-8B-Instruct training experiments, E5: Standard SFT and E6: Full BranchWise, and completed one Base evaluation using the original untrained Llama model as the common starting point. The three settings use the same model-family-compatible tool-call templates and scoring pipeline and are evaluated on four benchmarks. Full settings and results appear in Sections 1.2–1.4.

Compared with SFT on the same backbone, BranchWise improves all seven aggregate metrics across the four benchmarks. Tau2 and ToolHop Mandatory/Free increase by 7.91, 15.48, and 13.77 pp, respectively, and BFCL Overall increases by 4.76 pp. This directly supports the effectiveness of local decision supervision on another model family.

1.2 New Training Experiments E5–E6 and Base Evaluation: Objectives, data, and setup

Experimental objective. Determine whether local error supervision beyond successful trajectories and the complete training objective remain helpful when changing the model family.

New comparisons. This section includes two new training experiments: E5, Llama Standard SFT, which learns only correct trajectories; and E6, Llama Full BranchWise, which adds verified local error branches, evidence-weighted supervision, and margin-aware ranking to the same correct trajectories. A Base evaluation of the original untrained Llama serves as their common starting point. All three use Llama-3.1-8B-Instruct. Full results are in Section 1.3 and detailed analyses in Section 1.4.

AuthorsOpenReview ↗

Main Response — Part 02/07

Main Response — Part 02/07

ControlSetting
Backbone and initializationOriginal Llama-3.1-8B-Instruct; SFT and BranchWise both initialize from this model
Correct trajectoriesSame 10,200: 9,000 ToolMind mixed tool-use trajectories + 1,200 Tau2 training-domain state trajectories
SFT objectiveStandard SFT loss on correct completions only, retaining history as context, without evidence weights or error-branch loss
BranchWise objectiveEvidence-reliability-weighted correct-branch SFT plus margin-aware joint local ranking over correct and incorrect branches sharing the same state
Update budgetBoth use 1,200 optimizer steps, effective batch size 32, maximum length 8,192
OptimizationLearning rate 2e-4; LoRA r=64r=64, α=128\alpha=128, dropout =0.05=0.05
Templates and parsingLlama-compatible multi-tool-call history template and tool-call parser
EvaluationBFCL V3, Tau2, ToolHop, ToolTalk-Hard

Model-specific error construction. We retain verifiable rule/execution-grounded branches but do not treat Qwen's model errors as Llama errors. For the full method, we resample from the original Llama four times at each of 4,800 training-side candidate states, yielding 19,200 responses. Verification and deduplication yield 1,870 distinct errors across 1,185 states, which are then fused with evidence-constructed branches. The data checks pass, and no test examples from the four benchmarks are read during construction.

How the full method learns. At each decision state, the same history and tool context are preserved, and ranking scores are computed only in the local regions where correct and incorrect actions diverge. Correct-branch SFT weights are determined by evidence reliability. One correct action is jointly ranked against several errors, pushing insufficiently separated branches toward the target score margin. E5 and E6 therefore differ in error supervision, local comparison, and the training objective. Section 1.3 compares the complete methods; Section 2.3 tests the objective and error sources through controlled experiments.

Compatibility and evaluation checks. All three settings use the formal Llama-compatible evaluation pipeline. The remaining evaluation settings are the same as in the paper.

1.3 New Training Experiments E5–E6 and Base Evaluation: Results

MetricLlama BaseE5 Vanilla SFTE6 BranchWiseE6 − E5
BFCL V3 Overall26.3641.1745.94+4.76 pp
Tau2 Pass@115.1114.7522.66+7.91 pp
ToolHop Direct7.147.248.14+0.90 pp
ToolHop Mandatory7.3421.3136.78+15.48 pp
ToolHop Free10.4520.5034.27+13.77 pp
ToolTalk-Hard Success16.0018.0020.00+2.00 pp
ToolTalk-Hard Soft Success47.3947.5547.60+0.05 pp

1.4 Analysis: Cross-model gains and sustained tool interaction

The following Tau2 domains and three BFCL components further show where the improvements arise:

ComponentLlama BaseE5 Vanilla SFTE6 BranchWise
Tau2 Airline30.006.0018.00
Tau2 Retail7.0227.1933.33
Tau2 Telecom16.676.1414.04
BFCL Non-Live Overall39.1846.2749.07
BFCL Live33.4162.3769.61
BFCL Multi-Turn6.5014.8819.12
  1. BranchWise's benefits extend to Llama under the same training conditions. E5 and E6 share the backbone, 10,200 correct trajectories, and update budget. Full BranchWise exceeds standard SFT on all seven aggregate metrics. This directly addresses whether the method is effective only on Qwen.

  2. Tasks requiring sequential decisions benefit most. Mandatory and Free improve by 15.48 and 13.77 pp, Tau2 by 7.91 pp, and BFCL Multi-Turn by 4.25 pp, with Long Context increasing from 15.00% to 23.50%. These tasks require continued action selection over multi-step interactions, for which local error branches provide supervision beyond successful paths.

  3. BranchWise mitigates the regressions of standard SFT in some domains. SFT reduces Airline from 30.00% for Base to 6.00%, and Telecom from 16.67% to 6.14%. BranchWise recovers them to 18.00% and 14.04%, while further increasing Retail from 27.19% to 33.33%. The three-domain success count grows from 41/278 for SFT to 63/278, indicating better decisions in other domains while learning from training data.

AuthorsOpenReview ↗

Main Response — Part 03/07

Main Response — Part 03/07

  1. Aggregate and component results support each other. BFCL Non-Live Overall, Live, and Multi-Turn all improve over SFT, increasing Overall from 41.17% to 45.94%. Direct and ToolTalk-Hard also improve, though less than the tool-interaction settings. These results support cross-family gains in this controlled same-backbone comparison; Airline, Telecom, and parallel calls still exhibit gaps relative to Base.

Revision plan. We will include the full non-Qwen comparison table, BFCL and Tau2 components, and explanations of Llama-specific error construction and compatible templates in the paper.

2. Response to Question 2: CausalFlow comparison and controlled training tests of BranchWise's design

Your original question: Can you provide more baseline results such as CausalFlow?

2.1 Summary and comparison approach

Thank you for suggesting a more direct comparison with related work. We add two components: first, an account of CausalFlow's commonalities and technical differences relative to BranchWise (Section 2.2); second, four new training experiments—E1: Same-Data Action-DPO, E2: Same-Data SimPO, E3: Same-Data Uniform Listwise, and E4: Action-DPO Using Only Model-Generated Errors—to test optimization objectives and error sources separately (settings and results in Section 2.3).

2.2 Analysis of CausalFlow

We analyzed CausalFlow's design and implementation. CausalFlow starts from failed executions, intervenes by replacing candidate steps, and verifies through subsequent execution and feedback whether a failure was repaired, yielding paired incorrect and repaired actions. These pairs can also be used for offline preference learning or reward modeling. Both approaches therefore exploit local error information; the distinction concerns the starting point of supervision, the verification question, and the learning method.

ComparisonCausalFlowBranchWise
Supervision starting pointExisting failed executions and their repairsSuccessful reference trajectories, tool/execution evidence, and initial-model errors
Verification questionWhether replacing a candidate step repairs the eventual resultWhether an alternative action under the same state violates checkable conditions
Learning objectVerified incorrect/repaired action pairsLocal branch groups fusing multi-source errors with weighted SFT and margin-aware ranking

Role of our design. At locations with sufficient local evidence, BranchWise can recover failure directions from successful trajectories without first requiring complete failed executions and successful repairs. It then combines these alternative actions with actual model errors and specifies ranking positions and objectives. Thus, supervision construction and model updates focus on decision differences at the same state.

Conclusion: A complete CausalFlow comparison would involve failure collection, intervention-based repair, verification, and training, with a final score reflecting the entire pipeline. We instead prioritize E1–E4 to distinguish the respective effects of error construction and the ranking objective in our study.

2.3 New Training Experiments E1–E4: Separate tests of optimization objectives and error sources

We add four training experiments:

  • E1: Same-Data Action-DPO. Retains the complete BranchWise error pool but replaces ranking with a local pairwise preference objective relative to the initial model.
  • E2: Same-Data SimPO. Retains the same local positive/negative actions but uses a reference-free pairwise preference objective.
  • E3: Same-Data Uniform Listwise. Retains joint comparison across multiple errors, sets the positive target margin to zero, and uses ordinary softmax contrast.
  • E4: Action-DPO Using Only Model-Generated Errors. Uses the same local DPO as E1 but removes evidence-constructed errors and retains only responses actually generated by the initial model and verified as incorrect, for comparison against the complete pool.

Experimental objective. Test separately whether the full method's gains can be explained by an existing local preference objective or by simply adding model failures.

Experimental setup.

Reviewer_wpKROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents BranchWise, an offline post-training framework for tool-using agents. It decomposes successful trajectories into state-conditioned local decisions and constructs counterfactual negatives filtered by verification rules, using tool constraints, execution evidence, and one-time sampling from the initial policy. Training combines evidence-weighted reference-branch SFT with margin-aware listwise ranking over localized decision regions, while the negative pool remains fixed during training. The authors train Qwen3-4B/8B on ToolMind and τ\tau-bench trajectories and evaluate downstream performance on τ2\tau^2-Bench, BFCL V3, ToolHop, and ToolTalk-Hard.

Critical strengths

1. The negative construction considers practical constraints in tool use. The method does not treat every action that differs from the demonstration as incorrect. Instead, it filters negatives through tool constraints, execution conditions, or the Action Contract. Combining evidence-derived counterfactuals with errors actually generated by the initial model is a targeted supervision design.

2. The reported improvements on multi-step tool use are noteworthy. In Table 1, the 8B model improves from Vanilla SFT's 27.94% and 26.23% to 40.10% and 41.21% on ToolHop Mandatory and Free, respectively. Improvements are also reported on BFCL Multi-Turn and ToolTalk-Hard. The evaluation covers several forms of tool interaction and includes a 4B model as a second scale.

3. The paper includes several component ablations and behavioral analyses. Table 2 separately removes the two negative sources, Fork Localization, and Margin-Aware Ranking. Figures 5–6 examine ranking-supervision coverage and feedback incorporation. These results provide information beyond final task success rates and help guide further analysis of why the method works.

Critical weaknesses

1. The advantage of the additional designs over existing local preference learning still lacks direct evidence. Section 2 already discusses step-wise preference learning in TP-LLaMA and student-action sampling at expert states in Agentic-DPO. Its distinctions mainly concern evidence-constrained negative construction, multi-source fusion, and localized optimization, whereas the external baselines in the main table mostly cover data selection, token filtering, and SFT objective modifications. The “w/o Fork Localization” ablation in Table 2 does compare full-branch and local-region scoring, which should be acknowledged. What remains missing is a standard action-level preference objective applied to the same negative data, to establish whether the proposed optimization adds further value. Also, “w/o Margin-Aware Ranking” is described only as removing low-margin prioritization, without the replacement loss or implementation rule, making this ablation difficult to interpret.

2. Several results cannot yet be reconciled with the stated evaluation protocol. First, Figure 4(c) gives Telecom success rates of 33.3%/35.1% for BranchWise/Base, whereas Table 1 reports 34.21%/33.33%, reversing the direction of improvement. The paper does not explain whether these results come from different runs. Second, if Overall is pooled over the 50/114/114 tasks listed in A.6.1, LESS's domain scores of 6.50%/41.50%/27.75% give approximately 29.57%, rather than the 27.25% in Table 1. No aggregation formula is provided that explains this difference. Another point needing clarification is that Table 8 specifies three training runs and two evaluations per run for 8B BranchWise. If each evaluation contains all 50 Airline tasks, uses binary success outcomes, and is weighted equally, the success rate must be (100k/300)%(100k/300)\% for an integer kk, which cannot round to the reported 25.50%. These observations indicate that the run provenance, effective denominators, or aggregation description needs clarification or correction; they do not by themselves establish which underlying results are wrong.

3. Key verification and decontamination procedures lack checkable details. A.2.8 describes held-out taxonomy validation and mentions coverage and the OTHER rate, but provides neither the actual proportions nor the validation sample size. A.2.1 does provide 2,000 Contracts, a Wilson-lower-bound threshold-selection rule, and the resulting threshold of 0.8. What is missing is not the calibration idea, but the exact computation of ρt\rho_t and the retained count, correct count, and observed lower bound at that threshold. The main text also refers to A.3.1 for instance-level decontamination and replacement, but that section describes only corpus composition and state-segment extraction, without matching rules or removal and replacement counts. Since training includes 1,200 state segments from τ\tau-bench Retail, source-task provenance and instance-level separation records are particularly important for checking the claimed absence of overlap with the τ2\tau^2 test set. The issue is that these claims are difficult to verify, not that incorrect labels or data leakage have been established.

Questions for authors

1. Beyond the existing Fork Localization ablation, could the authors hold the positive–negative action data, backbone, reference-branch SFT term, and parameter-update budget fixed, and replace only the ranking term with action-level DPO or SimPO to test the added benefit of the proposed optimization on the same supervision data? Please also provide the exact loss or implementation rule for “w/o Margin-Aware Ranking”.

2. Please identify the runs or checkpoints used for Figure 4(c) and Table 1, and specify the Overall aggregation formula. For the 25.50% Airline score, please state the evaluation repeats, effective task denominators, task filtering, and averaging procedure. Per-run success counts and valid task counts would help reconcile these differences.

3. Please provide the taxonomy-validation sample size and coverage/OTHER rate, the Contract-confidence computation and calibration statistics at the selected threshold, and the decontamination matching rules, removal and replacement counts, and mapping from the 1,200 state segments to their original tasks.

39 comments
AuthorsOpenReview ↗

Main Response — Part 01/08*

Main Response — Part 01/08

During this rebuttal, we added extensive controlled experiments, ablations, and in-depth analyses and redrew the paper's analysis figures, further strengthening the experimental validation, technical argumentation, and presentation of BranchWise. We believe these additional results make the paper more complete, rigorous, and persuasive. We sincerely hope the reviewer will take these improvements into account when reassessing the paper's score. Your recognition would encourage us to keep pursuing rigorous, interesting, and valuable research.

Thank you for acknowledging multi-source error construction, the results on multi-step tool interaction, and the component ablations, and for your concrete comments on the evaluation protocol and data auditability. We agree that the original submission needs same-data preference-objective comparisons to further support the optimization design, as well as clearer reporting of run provenance, denominators, aggregation formulas, and verification statistics.

To address these issues, we conducted three additional training experiments during rebuttal: E1, Same-Data Action-DPO; E2, Same-Data SimPO; and E3, Same-Data Uniform Listwise (settings and results in Sections 1.2–1.4). We also conducted five new analyses: A1, Error Taxonomy Statistics (Section 3.2); A2, Contract Threshold Audit (Section 3.4, calculation method in Section 3.3); A3, Decontamination Audit (Section 3.5); A4, State-Segment Provenance Audit (Section 3.6); and A5, Evaluation Protocol and Aggregation Audit (Sections 2.2–2.5). We first answer the three Questions for Authors and then address the three Critical Weaknesses individually. Overlapping issues retain separate entries and are cross-referenced to section numbers within this document.

E1–E3 are the Action-DPO, SimPO, and Uniform Listwise comparisons described below. A1–A5 are new analyses of existing data and records. The settings, results, and statistical definitions are presented in this reply.

1. Response to Question 1: Same-data preference-objective comparisons and a fully specified Uniform Listwise experiment

Your question (summary): With positive/negative action data, the backbone, reference-branch SFT, and update budget fixed, can replacing only the ranking term with Action-DPO or SimPO verify the additional value of the optimization design? Please also specify the exact replacement loss or implementation for w/o Margin-Aware Ranking.

1.1 Summary of our response

Thank you for noting that the existing Decision-Fork Localization ablation (w/o Fork Localization, which replaces local comparison with full-branch comparison) cannot substitute for same-data objective comparisons. We therefore conducted two new training experiments: E1, local Action-DPO using a pairwise preference objective relative to the initial model; and E2, local SimPO using a reference-free pairwise preference objective. Both share the positive/negative actions, local comparison region, evidence-reliability-weighted SFT, and training budget, replacing only the preference term. Settings and full results are in Sections 1.2–1.3.

In addition, we conducted one more training experiment, E3: Same-Data Uniform Listwise, which jointly compares the correct branch with multiple incorrect branches at the same state under an ordinary softmax contrastive objective without a positive target margin, to test whether joint comparison of multiple negatives is sufficient. Full results are in Section 1.3, and the formula, parameters, and relationship to the original ablation are in Section 1.4.

1.2 New Training Experiments E1–E3: Objectives, settings, and controls

Experimental objective. With local supervision held fixed, distinguish whether error data are helpful from whether the full training objective offers additional value.

Experimental setup. All three experiments initialize from the original Qwen3-8B and train to a fixed checkpoint-1200.

AuthorsOpenReview ↗

Main Response — Part 02/08

Main Response — Part 02/08

Control factorShared setting
Training trajectoriesSame frozen 10,200 trajectories: 9,000 ToolMind mixed examples + 1,200 Tau2 training-domain state trajectories
Positive/negative actionsSame error pool; compare only local regions where correct and incorrect actions diverge, i.e., contrast_span
Correct-branch SFTSame evidence weighting, mask deciding which correct-response tokens receive supervision, and sampling stream
Update budget1,200 optimizer steps, effective batch 32, maximum length 8,192
Learning rate and LoRALearning rate 2e-4; r=64r=64, α=128\alpha=128, dropout =0.05=0.05
Error-branch exposureAt most three error branches per position;
EvaluationSame settings as the paper

Experimental content. The three new training experiments are E1: Same-Data Action-DPO, E2: Same-Data SimPO, and E3: Same-Data Uniform Listwise.

ComparisonReplaced objectiveParameters and aggregation
E1 Action-DPOLocal reference-relative DPOβ=1\beta=1; original Qwen3-8B as reference; average pairs within states, then average states
E2 SimPOLocal reference-free preference lossβ=2\beta=2, γ/β=0.5\gamma/\beta=0.5; likewise average within states to avoid overweighting states with more errors
E3 Uniform ListwiseSoftmax contrast between the correct branch and all selected incorrect branchesT=1T=1, margin =0=0; normalize by log⁡(1+K)\log(1+K); no reference policy; full formula in Section 1.4

These comparisons share the local scoring region and primarily test optimization terms. The effect of localization itself is still tested by the original Fork Localization ablation.

Existing localization evidence. In the original paper, removing Decision-Fork Localization reduces Tau2 from 40.24 for the full method to 34.89, ToolHop Free from 41.21 to 32.67, and ToolTalk-Hard Success from 28.00 to 16.00. This ablation changes the comparison region; E1–E3 preserve the region and change only the objective.

Formal evaluation setting. Same as the paper.

1.3 New Training Experiments E1–E3: Results and analysis

MetricE1 Action-DPOE2 SimPOE3 Uniform ListwiseBranchWise
BFCL V3 Overall65.1263.8164.7665.49
Tau2 Pass@135.9734.5335.6140.24
ToolHop Direct7.648.247.948.94
ToolHop Mandatory39.6040.1039.2040.10
ToolHop Free39.6039.4036.2841.21
ToolTalk-Hard Success26.0024.0020.0028.00
ToolTalk-Hard Soft56.2554.6651.0257.71

BranchWise uses the corrected aggregated results from the paper.

Analysis of results.

Following your suggestion, these experiments fix the data, backbone, correct-branch SFT, and update budget, and replace only the ranking term, directly testing the additional value of the full optimization objective.

  1. On the same local supervision, BranchWise's ranking objective yields higher interaction success. Action-DPO (E1) and SimPO (E2) compare the correct action with each error pairwise and average losses. BranchWise jointly ranks multiple errors from the same state with a target margin. Its Tau2 gains are 4.27 and 5.71 pp, and ToolTalk-Hard Success gains are 2.00 and 4.00 pp, respectively. These gains indicate that the organization and optimization of action comparisons affect final task performance even on the same error data.

  2. The target margin provides further gains over ordinary joint comparison. Uniform Listwise (E3) retains joint comparison over multiple same-state errors but omits an explicit positive target margin. Full BranchWise improves Tau2, ToolHop Free, and ToolTalk-Hard Success by 4.63, 4.93, and 8.00 pp, supporting stronger differentiation between correct and confusable incorrect actions through the target margin. E3's full loss and parameters appear in Section 1.4.

  3. The new experiments complement rather than replace localization ablations. The original Fork Localization ablation changes the comparison scope, examining where action divergence should be trained; E1–E3 keep this scope unchanged and test which objective is more effective at those positions. These comparisons separately support where to compare and how to distinguish correct and incorrect actions, linking the roles of localization and the ranking objective to direct evidence.

We thus provide direct training results for the requested same-data comparisons: even when standard preference baselines share BranchWise's local supervision, the full ranking objective can further improve multi-step interaction success.

AuthorsOpenReview ↗

Main Response — Part 03/08

Main Response — Part 03/08

1.4 New Training Experiment E3: Exact Uniform Listwise loss and treatment of the original ablation

Motivation. You noted that w/o Margin-Aware Ranking was described only as removing low-margin prioritization without specifying its replacement objective. To provide a directly reproducible comparison, we conducted a new experiment, E3: Same-Data Uniform Listwise (introduced in Section 1.1), with its loss and parameters specified explicitly.

Scores and loss. For the correct branch and up to three incorrect branches at each state, we compute mean token log-probability only over the same local comparison region. Let q+q^+ be the correct-branch score, qk−q_k^- the kk-th incorrect-branch score, dk=q+−qk−d_k=q^+-q_k^-, and KK the number of selected errors. The formula is:

Luniform=ln⁡[1+∑kexp⁡(−dk/T)]ln⁡(1+K),T=1.L_{\mathrm{uniform}}=\frac{\ln\left[1+\sum_k\exp(-d_k/T)\right]}{\ln(1+K)},\qquad T=1.

Ltotal=Lweighted SFT+0.05×mean⁡statesLuniform.L_{\mathrm{total}}=L_{\mathrm{weighted\,SFT}}+0.05\times\operatorname{mean}_{\mathrm{states}}L_{\mathrm{uniform}}.

Implementation itemUniform Listwise rule
Target margin0
Explicit coefficients for incorrect branchesAll equal to 1
Additional margin-aware weightingDisabled
Low-margin sampling/resamplingDisabled
Scoring region, data, etc.Same as Section 1.2
Reference policyNot used

Ordinary softmax already assigns larger gradients to errors with higher current scores; this is not equivalent to enabling additional margin-aware weighting. As an implementation comparison, the frozen full BranchWise ranking term uses the same normalization and includes a target margin of 0.1 in the exponent, i.e., exp⁡((0.1−d)/T)\exp((0.1-d)/T).

Relationship to the original ablation. In addition to the original paper's ablation, we introduce the Uniform Listwise comparison (E3; full results in Section 1.3), fixing error branches, local comparison scope, and training budget while setting the target margin to zero to further examine margin-aware ranking. This experiment is reported separately in the revised manuscript.

2. Response to Question 2: Clarifying run provenance, Tau2 denominators and aggregation, and correcting scoring errors

Your question (summary): Do Figure 4(c) and Table 1 use different runs or checkpoints? How is Overall aggregated? How many evaluations and what denominator, filtering, and averaging procedure produce Airline 25.50%?

2.1 Summary of our response

Your questions involve separate issues. The values in Figure 4(c) can be traced to a specific single evaluation, whereas the main table aggregates repeated evaluations; they should not be presented as one statistic in the caption. Separately, four Overall entries in the original table disagreed with their three domain scores. Auditing confirmed errors in the script's Overall aggregation. We recomputed Overall from the total successes across three domains divided by 278, leaving domain scores unchanged. BranchWise's reported 40.25% was separately corrected to 40.24% after checking the display rounding. We apologize for these errors and reviewed other numbers throughout the paper, confirming that there are no further problems.

To make the settings and results auditable, we added new analysis A5: Evaluation Protocol and Aggregation Audit, examining run provenance, aggregation, corrections, and denominators from existing records (Sections 2.2–2.5). The actual Tau2 training/evaluation setting was confirmed as four training seeds with 18 complete evaluations per seed.

2.2 Single-run evidence in Figure 4(c) versus the main-table aggregate

Thank you for pointing out the discrepancy. We checked the complete evaluation records and found a single run matching the 33.3% and 35.1% values in Figure 4(c). These differ from Table 1 because the figure shows a single evaluation while the table reports the mean over repeated evaluations.

Statistical scopeBranchWiseOriginal Qwen3-8B
Single evaluation matching Figure 4(c)33.3%35.1%
Table 1 mean over repeated evaluations34.21%33.33%

In that single evaluation, BranchWise solved 38 Telecom problems and Base solved 40, two fewer for BranchWise. In the main-table average, BranchWise instead exceeds Base by 0.88 pp. Single-run interaction and sampling variability can reverse the comparison direction relative to repeated averages, so their statistical scopes must be clearly identified.

Reviewer_9dSoOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a training framework that teaches tool-use agents to distinguish correct trajectories from plausible mistakes, which may overcome some drawbacks of SFT successful trajectoroes. Training procedures combine weighted SFT with a contrastive ranking objective focused on the tokens where actions differ. Experiments with Qwen3-4B and Qwen3-8B show improved performance across four tool-use benchmarks with particularly strong gains in multi-step tool interactions.

Critical strengths
  • This paper targets at an important topic, in which tool learning by SFT can only learn from successful trajectores and lack negative signals.

  • Careful verification of negative examples. The authors carefully design the aspects of negative examples and generate from branches. Experiments show that the method is effective.

Critical weaknesses
  • Difference between DPO and contrastive learning. Learning from positive and negative samples is not a new topic. Both DPO and contrastive learning (CL) use the similar idea of comparing good and bad examples [1,2]. The authors should explain why BranchWise is significantly different from and better than DPO/CL, and comparing them in experiments. Also, beyond SFT, the author may also use DPO or other RL methods to test whether the counterfactual samples will help.

  • The contributions in this paper are more like engineering work. The research findings in this paper are unclear. Simply introducing unsuccessful trajectories is limited.

  • Words in all figures are too small to read.

[1] Agentic-DPO: From Imitation to Agentic Policy Optimization on Expert Trajectories [2] Advancing Tool-Augmented Large Language Models: Integrating Insights from Errors in Inference Trees. NeurIPS 2024

Questions for authors

See Weaknesses

24 comments
AuthorsOpenReview ↗

Main Response — Part 01/09

Main Response — Part 01/09

During this rebuttal, we have added extensive controlled experiments, ablation studies, and in-depth analyses, and redrawn the paper's analysis figures, further strengthening the experimental validation, technical arguments, and presentation of BranchWise. We believe that this new evidence makes the paper more complete, rigorous, and convincing. We sincerely hope that you will consider these improvements when reassessing your score for our paper. Your recognition would also encourage us to continue pursuing more rigorous, interesting, and valuable research.

Thank you for recognizing the training problem addressed by our work and for acknowledging the negative-sample verification design and the results on multi-step tool interaction. We agree that the original submission needs to explain BranchWise's contribution relative to existing preference-learning methods more directly and to distinguish the effects of data construction and optimization objectives through same-data comparisons. In response, we conducted four additional training experiments during the rebuttal: E1, Same-Data Action-DPO; E2, Same-Data SimPO; E3, Same-Data Uniform Listwise (settings and results in Section 2.2); and E4, Action-DPO Using Only Model-Generated Errors (settings and results in Section 2.3). We describe their settings, results, and supported conclusions in turn below.

1. Responses to Questions for Authors

1.1 Corresponding response locations

Your Questions for Authors correspond to three weaknesses. For ease of reference, we retain this guide and address them separately below:

Original commentLocation hereFocus of response
Weakness 1: Differences from DPO/CL, direct comparisons, and uses of counterfactual data2.1–2.4Methodological differences; three objective comparisons (E1–E3, Section 2.2); and one error-source comparison (E4, Section 2.3)
Weakness 2: Unclear research findings and whether the method merely adds failed trajectories3.1–3.3Advantages over existing methods, research motivation and design rationale, and validation from the original paper and new experiments
Weakness 3: Text in all figures is too small4.1–4.2Revisions to figure text, legends, and readability in the final PDF

2. Response to Weakness 1: Same-data comparisons support BranchWise's value beyond standard preference objectives

Your question (summary): DPO and contrastive learning already learn from positive and negative samples. How is BranchWise substantively different, and can it be compared directly? Can counterfactual samples also be used for DPO or other RL methods?

2.1 Core design, methodological differences, and direct evidence

Vanilla SFT directly learns recorded correct actions; T-SHIRT, DaaR, LESS, and GIST improve data use by selecting training trajectories, while Token Cleaning and Beyond Log Likelihood adjust token supervision and losses, respectively. These approaches do not explicitly supplement successful paths with verifiable error branches tied to the model's weaknesses.

Our three new same-data training experiments (E1: Action-DPO, E2: SimPO, and E3: Uniform Listwise) replace only the ranking term while sharing error branches, local comparison regions, and weighted SFT, yet still do not match the multi-step interaction performance of the full BranchWise method (Section 2.2). BranchWise supplements successful trajectories with verifiable, model-relevant local error supervision and additionally designs the comparison locations and ranking objective to make this supervision more effective for multi-step decision learning.

BranchWise unfolds a successful path into local decision boundaries at multiple decision states: it constructs diverse, verifiable failure directions from evidence of correct execution, fuses them with genuine errors made by the initial model, and focuses ranking on key action differences that the model has not yet adequately distinguished. Specifically:

  1. Decision-level supervision: We preserve the history and tool context at each step and construct evidence-supported error branches from the tool, argument, execution, and state conditions under which the correct action is valid. This Evidence-Grounded Counterfactual Boundary Expansion allows one successful demonstration to supervise both how to act correctly and which conditions must not be violated.
AuthorsOpenReview ↗

Main Response — Part 02/09

** Main Response — Part 02/09**

  1. Coverage of multiple failure directions: When sufficient evidence exists, error branches are constructed for six categories: arguments, state, action selection, task progress, error recovery, and termination. Margin-Aware Local Ranking jointly compares the correct action against multiple errors and pushes their scores apart, so supervision covers a related set of failure directions.

  2. Supervision targeted to model weaknesses: Before training, we sample and verify errors of the initial model once. Through Multi-Source Boundary Fusion, these errors are combined with branches constructed from tool constraints and execution evidence, balancing source diversity and branch budget. The resulting local decision boundaries include both verifiable failure directions and weaknesses of the particular model.

  3. Optimization focused on key differences: Decision-Fork Localization confines ranking to local differences that actually affect behavior. Current score gaps then determine greater optimization pressure for insufficiently separated branches. The supervision weight of the correct branch is adjusted according to the reliability of verification evidence, as in Evidence-Calibrated Reference-Branch Supervision.

To examine joint ranking and margin optimization, we conducted three same-data training experiments: E1, Action-DPO, with pairwise preferences relative to the initial model; E2, SimPO, with a reference-free pairwise preference objective; and E3, Uniform Listwise, with an ordinary joint contrastive objective comparing one correct action against several incorrect actions. All three fix the positive/negative branches, local comparison regions, weighted SFT, sampling and exposure schedule, and training budget, and replace only the preference/ranking term (full settings and results in Section 2.2).

To examine multi-source error supervision, we conducted one further training experiment, E4: Action-DPO Using Only Model-Generated Errors. It removes errors constructed from rules and execution evidence, retaining only answers actually generated by the initial model and verified to be incorrect, and compares them with E1, which uses the complete multi-source error pool (sampling, training settings, and full results in Section 2.3).

The experimental results distinguish the roles of these designs. Relative to Vanilla SFT, the paper's Qwen3-8B results improve Tau2 and ToolHop Mandatory/Free by 10.47, 12.16, and 14.98 pp, respectively. Even when the same error pool, localization, and weighted SFT are given to the new preference baselines, full BranchWise exceeds Action-DPO, SimPO, and Uniform Listwise on Tau2 by 4.27, 5.71, and 4.63 pp, respectively, and is also better on ToolHop Free and ToolTalk-Hard (Section 2.2). The former comparison supports the value of the complete decision-supervision approach; the latter directly supports the additional value of its ranking objective on shared supervision.

AuthorsOpenReview ↗

Main Response — Part 03/09

** Main Response — Part 03/09**

Work / baselineBaseline algorithmBranchWise designMain BranchWise results
Vanilla SFTStandard completion-only training on successful trajectories; no explicit supervision of alternative incorrect choices under the same stateConstructs reference and counterfactual branches at decision states to learn local decision boundariesTau2 40.24 / 29.77; ToolHop Free 41.21 / 26.23
T-SHIRTSelects the top 5% of trajectories by hierarchical informativeness; selection alone does not add error choices beyond successful pathsCompletes local failure directions from recorded correct behavior and its evidence, enriching each trajectory's supervisionTau2 40.24 / 29.41; ToolHop Free 41.21 / 32.36
DaaRSelects the top 20% of trajectories by diversity; selection does not explicitly create reference–counterfactual comparisons at each stateOrganizes verifiable counterfactual branches at a shared decision state through Multi-Source Boundary FusionTau2 40.24 / 34.26; ToolHop Free 41.21 / 27.57
LESSEstimates gradient influence using target validation data and selects the top 5% of trajectories; does not explicitly add local error directions within selected trajectoriesExpands verifiable errors from conditions supporting the correct action, increasing within-state action-discrimination supervisionTau2 40.24 / 29.57; gains on other benchmarks are given in the score details
GISTSelects the top 5% of trajectories using target data and projection; emphasizes sample relevance without explicit same-state failure branchesUses evidence-grounded boundary expansion and initial-policy error probing to construct model-relevant local decision boundariesTau2 40.24 / 29.57; gains on other benchmarks are given in the score details
Token CleaningPreserves context but supervises only the top 60% of response tokens; filtering is not equivalent to behavioral divergence between reference and counterfactual branches, and provides no incorrect-action comparisonUses Decision-Fork Localization and Evidence-Calibrated Reference-Branch Supervision to determine ranking regions and supervision weights separatelyTau2 40.24 / 29.57; gains on other benchmarks are given in the score details
Beyond Log LikelihoodUses the pp variant, replacing the NLL of correct tokens with 1−p1-p; does not explicitly constrain relative scores of correct and incorrect actionsJointly ranks locally correct and incorrect branches and uses a target margin to separate their scoresTau2 40.24 / 29.57; gains on other benchmarks are given in the score details
Agentic-DPOSamples student actions at expert states to construct state-conditioned preferences; error construction mainly depends on choices observed in samplingCombines evidence-grounded and model-generated counterfactual branches through Multi-Source Boundary Fusion, followed by localized margin-aware rankingSame-data DPO results are in E1, Section 2.2
TP-LLaMA / Inference TreesExtracts stepwise preferences from failed exploration in existing inference trees and then applies DPO; depends on collected trees and failed explorationExpands local errors from linear successful trajectories and checkable evidence without first requiring a complete failed-exploration treeExtends usable supervision sources;
Same-Data Action-DPO (E1; Section 2.2)Shares the error pool, localization, and weighted SFT; computes reference-relative preferences per pair and averages within states without joint listwise normalization over error groupsUses margin-aware within-state listwise ranking over the same local supervisionTau2 40.24 / 35.97; ToolTalk Success 28.00 / 26.00
Same-Data SimPO (E2; Section 2.2)Shares supervision and budget; uses a reference-free pairwise objective with a fixed margin and averages across multiple errorsJointly ranks errors in the same state with BranchWise's target marginTau2 40.24 / 34.53; ToolTalk Success 28.00 / 24.00; Mandatory tied
Same-Data Uniform Listwise (E3; Section 2.2)Retains joint softmax over multiple errors and score-dependent gradients but sets the target margin to zero, without an explicit positive target marginAdds a target margin under the same error grouping and normalization to emphasize insufficiently separated local boundariesTau2 40.24 / 35.61; ToolHop Free 41.21 / 36.28; ToolTalk Success 28.00 / 20.00

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.