acceptodds
Under review as a conference paper at ICLR 2027

The Environment Knows Why: Bidirectional Self-Distillation from Augmented Tool Feedback for Agentic RL

Abstract

Self-distillation with privileged context has emerged as a promising paradigm for agentic reinforcement learning, providing denser supervision for long-horizon credit assignment. However, existing methods typically construct such context for individual tasks or trajectories, limiting reuse and requiring repeated generation during training. We observe that rejected tool calls expose a reusable source of local supervision: the relevant tool contract is encoded in the environment implementation but often omitted from the returned feedback. Building on this observation, we introduce (*Bidirectional Environment-Augmented Distillation*), which combines task-agnostic environment augmentation with bidirectional feedback self-distillation. The former mines tool contracts offline and converts rejected feedback into reusable local guidance across tasks and trajectories. The latter uses the same augmented feedback bidirectionally: backward scoring provides hindsight correction for the rejected call, while forward scoring provides recovery guidance for the subsequent action. The resulting token-level teacher–student log-probability gaps serve as self-distillation signals for RL advantage reweighting. Extensive experiments demonstrate the effectiveness of across both in-distribution and out-of-distribution multi-turn tool-use settings, while analyses support the complementary roles of backward and forward supervision.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Tool Learning, Self-distillation

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_uKMSOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a bidirectional self-distillation framework, BEAD, for multi-round agent reinforcement learning. This method first extracts tool contracts from the environment implementation and builds enhanced feedback for rejected tool calls. These enhanced feedbacks are generated in the offline stage and can be reused between tasks and trajectories that trigger the same rejection pattern. Subsequently, BEAD uses the enhanced feedback for bidirectional teacher scoring: the reverse scoring provides post-event information on the rejected actions, while the forward scoring evaluates the subsequent recovery actions. The resulting token-level log probability gap is used to reweight the original reinforcement learning advantage value without changing its sign.

Critical strengths
  1. An effective method for constructing reusable supervision from the environment.
  2. Achieve continuous experimental improvement through detailed analysis.
Critical weaknesses
  1. The contributions of environmental enhancement and bidirectional scoring are not completely independent.
  2. The efficiency advantages and differences from related self-distillation methods still require more evidence.
Questions for authors

Q1. Contribution of Environment Augmentation.

Can the author conduct an ablation experiment on the complete two-way BEAD framework solely using the original rejection feedback without resorting to environmental enhancement? Additionally, what impact will removing the usefulness filtering based on replay have on performance? These comparisons will help distinguish the respective contributions of the extracted tool contracts and the feedback selection process.

Q2. Backward Correction and Advantage Reweighting.

Equation (8) maintains the original RL advantage's sign. Therefore, even if the backward teacher assigns negative context influence scores, the rejected actions with positive trajectory advantages will still receive positive updates. Has the author analyzed how backward scoring affects rejected actions under positive and negative advantages separately? Such analysis will help clarify the actual correction mechanism.

Q3. Computational Efficiency and Reusability.

Compared to the standard GRPO and self-distillation baselines, how much wall time and computational cost does BEAD require additionally? Can the author separately report the offline costs of contract generation, verification, and replay checks, excluding the overhead of online teacher scoring? Since reusable feedback is the core motivation, analyzing the depreciation cost of training trajectories will be particularly beneficial.

Q4. Comparison with Related Methods.

Can the author further clarify the differences between its approach and HINT-SD [1] and AHEAD [2], especially in terms of the source and reuse of privileged contexts and the resulting credit allocation mechanism? If feasible, conducting matching comparisons will help strengthen the assessment of novelty.

References

[1] Yeo et al., "HINT-SD: Targeted Hindsight Self-Distillation for Long-Horizon Agents." 2026. https://arxiv.org/abs/2605.17873

[2] Jin et al., "AHEAD: Adaptive Hindsight with Environment-Augmented Distillation for Agentic RL." 2026. https://arxiv.org/abs/2608.24114

4 comments
AuthorsOpenReview ↗

Response to Reviewer uKMS: Part 1

W1 The contributions of environmental enhancement and bidirectional scoring are not completely independent.

Q1 Contribution of Environment Augmentation. Can the author conduct an ablation experiment on the complete two-way BEAD framework solely using the original rejection feedback without resorting to environmental enhancement? Additionally, what impact will removing the usefulness filtering based on replay have on performance? These comparisons will help distinguish the respective contributions of the extracted tool contracts and the feedback selection process.

Thank you for this suggestion. We conducted the requested ablations on BFCL v4 with Qwen3-4B. The ablations for environment augmentation and scoring directions are already reported in Table 3 of our paper, and we additionally conducted an experiment removing the replay check.

Removing environment augmentation has a particular consequence for bidirectional scoring. Without augmentation, the teacher and student have identical contexts for forward scoring, resulting in a zero log-probability gap and no additional supervision. Backward scoring remains applicable because the original rejection feedback can still serve as hindsight for the rejected action. Therefore, the variant without augmentation effectively retains only backward scoring.

The results are shown below.

MethodBFCL v4 (ID) Avg.Δ Avg.
BEAD (Ours)51.56—
w/o forward (backward only)49.00−2.56
w/o augmentation46.22−5.34
w/o replay check51.33−0.23

These results show that contract-derived augmentation provides substantial additional supervision beyond the original rejection feedback, even under the same backward-only setting. In contrast, removing the replay check has little impact on performance, suggesting that the preceding generation and validation stages already produce useful augmentations. The replay check therefore serves mainly as an additional quality gate rather than the primary source of performance improvement.

W2 The efficiency advantages and differences from related self-distillation methods still require more evidence.

Q3 Computational Efficiency and Reusability. Compared to the standard GRPO and self-distillation baselines, how much wall time and computational cost does BEAD require additionally? Can the author separately report the offline costs of contract generation, verification, and replay checks, excluding the overhead of online teacher scoring? Since reusable feedback is the core motivation, analyzing the depreciation cost of training trajectories will be particularly beneficial.

Thank you for your question. We use Skill-SD as a representative self-distillation baseline and report the average wall-clock time per training step below.

MethodRecompute logprobA training step
GRPO88.7 s853.9 s
Skill-SD198.9 s968.5 s (+13.42%)
BEAD173.8 s934.2 s (+9.40%)

The additional online computation mainly comes from teacher log-probability computation. Rollout dominates RL training time in our setting, so this additional computation accounts for a relatively small fraction of the total runtime. When rollout and training are pipelined across separate machines, teacher scoring can overlap with subsequent rollouts, further reducing the added wall-clock overhead.

We also separately report the offline construction costs, excluding online teacher scoring.

Contract generation and verificationReplay checks
12.3 min7.2 min

These costs are incurred only once before RL training. The resulting augmentations are reusable across tasks and trajectories, allowing the fixed offline construction cost to be amortized over an increasing number of training trajectories without repeated generation or verification.

AuthorsOpenReview ↗

Response to Reviewer uKMS: Part 2

Q2 Backward Correction and Advantage Reweighting. Equation (8) maintains the original RL advantage's sign. Therefore, even if the backward teacher assigns negative context influence scores, the rejected actions with positive trajectory advantages will still receive positive updates. Has the author analyzed how backward scoring affects rejected actions under positive and negative advantages separately? Such analysis will help clarify the actual correction mechanism.

The advantage reweighting in Eq. (8) follows the direction-aware credit assignment in RLSD [1], where the environment reward determines the update direction and the privileged teacher adjusts its token-level magnitude. A positive advantage trajectory may still contain rejected tool calls. When backward scoring produces a negative log-probability gap δi,t<0\delta_{i,t}<0, Eq. (8) attenuates positive updates for A^i,t>0\hat A_{i,t}>0 and amplifies negative penalties for A^i,t<0\hat A_{i,t}<0, without reversing the original update direction.

To examine these effects separately, we recorded backward-scored rejected calls during training and divided them into positive and negative advantage groups. For each rejected call, we compute mˉ\bar m, the mean effective advantage multiplier over its action tokens, where mi,t=(1−λi,t)+λi,twi,tm_{i,t}=(1-\lambda_{i,t})+\lambda_{i,t}w_{i,t}. The results are shown below.

Backward-scored rejected callsA^>0\hat A>0A^<0\hat A<0
Fraction of calls with mˉ<1\bar m<182.1%18.9%
Fraction of calls with mˉ>1\bar m>117.6%80.8%

For rejected calls with positive advantages, 82.1% receive attenuated positive reinforcement, whereas for those with negative advantages, 80.8% receive amplified negative penalties. These findings are consistent with the predominantly corrective backward teacher signals observed in Figure 4(b).

These results clarify that BEAD's backward scoring does not directly assign negative advantages to rejected actions. Instead, it uses backward teacher evidence to adjust the magnitude of RL updates while preserving the update direction determined by the environment reward. For rejected calls with positive advantages, backward scoring predominantly attenuates positive reinforcement, whereas for those with negative advantages, it predominantly amplifies negative penalties.

Reference

[1] Chenxu Yang et al. Self-distilled RLVR. arXiv preprint arXiv:2604.03128, 2026.

AuthorsOpenReview ↗

Response to Reviewer uKMS: Part 3

Q4 Comparison with Related Methods. Can the author further clarify the differences between its approach and HINT-SD and AHEAD, especially in terms of the source and reuse of privileged contexts and the resulting credit allocation mechanism? If feasible, conducting matching comparisons will help strengthen the assessment of novelty.

Thank you for highlighting these related works. AHEAD[1] and HINT-SD[2] are concurrent works on self-distillation, both cited in our Introduction. Both demonstrate the value of hindsight-based self-distillation for fine-grained credit assignment. However, their corrective annotations are constructed for individual training trajectories. HINT-SD analyzes complete failed trajectories to identify actions related to the failure and generate targeted feedback, while AHEAD combines environment observations with corrective hints generated by an LLM for identified error steps. Such hindsight information is tied to the particular trajectory and requires repeated generation as new failures arise during training.

Regarding matched experiments, we could not find a publicly available training implementation of AHEAD. We reproduced HINT-SD using its released implementation to train Qwen3-4B in thinking mode on the same BFCL v4 training split used for BEAD. As shown in the table below, HINT-SD yielded modest gains over the vanilla model in this setting.

MethodBFCL v4 (ID) Avg. (%)BFCL v4 (OOD) Avg. (%)
Vanilla33.6719.16
HINT-SD37.33 (+3.66)20.17 (+1.01)
BEAD51.56 (+17.89)27.94 (+8.78)

BEAD addresses a complementary but previously overlooked source of supervision in tool-use environments, namely the underlying tool contracts. A rejected tool call reflects a violation of environment-defined constraints, such as parameter domains, value provenance, or state preconditions. Unlike trajectory-specific corrective hints, these constraints describe how tools operate and remain valid across different tasks and trajectories. BEAD therefore mines and validates these contracts offline, constructing reusable augmentations associated with rejection signatures rather than generating new explanations for individual failures.

This difference in supervision source also enables a distinct credit allocation mechanism. BEAD uses the same augmentation derived from tool contracts bidirectionally. Backward scoring provides hindsight correction for the rejected call, while forward scoring evaluates the subsequent action to guide recovery. In this way, BEAD provides complementary supervision for both the decision that violated the tool contract and the subsequent attempt to recover, using a shared, reusable environment-grounded signal.

The reuse of augmentations across tasks and trajectories is supported by our analysis in Figure 4(a). Moreover, as detailed in our response to Reviewer VpFC's W2, our analysis of construction costs estimates approximately $1,583.07 and 49 hours for online trajectory-specific annotation, compared with $6.97 and 12.3 minutes for BEAD's one-time offline generation. This highlights the efficiency of BEAD's offline construction in substantially reducing the cost of privileged-context generation.

Reference

[1] Jin et al., AHEAD: Adaptive Hindsight with Environment-Augmented Distillation for Agentic RL. arXiv preprint arXiv:2608.24114, 2026.

[2] Yeo et al., HINT-SD: Targeted Hindsight Self-Distillation for Long-Horizon Agents. arXiv preprint arXiv:2605.17873, 2026.

Thank you for the detailed rebuttal and additional experiments. The responses have adequately addressed my concerns.

Reviewer_VrT2OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

BEAD uses tool rejections to provide local supervision for multi-turn agent RL. It mines reusable tool contracts from environment code and uses them as privileged context for a self-teacher to score both the failed call and the subsequent recovery action. The resulting token-level signals reweight the RL advantage, while the acting policy receives ordinary feedback throughout training and inference. Experiments train on BFCL and evaluate held-out BFCL tasks, τ²-bench, and ACEBench-Agent, with ablations and analyses of rejection and recovery behavior.

Critical strengths
  1. A useful source of reusable step-level supervision. Mining tool contracts from environment implementations provides concrete explanations that can be reused across tasks. It can also provide step-level supervision on tool-call rejected steps to alleviate the sparse reward problem. The reuse analysis and augmentation ablation support the value of this information source. Keeping it on the teacher side also allows the learned policy to operate with ordinary tool feedback.

  2. Behavioral evidence supports the learning design. Using a rejection to supervise both the failed call and the subsequent recovery is well motivated. The observed reduction in rejected calls and improvement in recovery provide evidence that BEAD changes tool-use behavior in the intended direction, complementing the benchmark results. These analyses support the complete method, although they do not fully isolate the benefit of bidirectional scoring.

Critical weaknesses
  1. The choice of turn-level, rejection-triggered distillation is not validated. The contracts are task-agnostic, and the table is small and fixed (56 entries). A natural alternative is trajectory-level distillation that gives the teacher the relevant contracts, for example those for the task's API classes, throughout the trajectory. This would also supervise the reasoning before the first rejection and remove the need for the budget B. The paper motivates turn-level supervision only by arguing that task-level context is too coarse, and never compares against trajectory-level use of the same contracts. Without this comparison, it is unclear whether the gains come from the contracts themselves or from turn-level localization, which is the paper's main design choice beyond the contract source.

  2. The supervision is limited in both where it applies and what information it provides. Additional teacher supervision is confined to the failed action immediately before a matched tool rejection and the recovery action immediately after it, leaving other steps in the trajectory without this signal. Its content is also limited to contracts mined once from offline trajectories and frozen throughout training, with no mechanism to incorporate new error patterns as the policy evolves. The paper does not quantify how much of the trajectory or the changing rejection distribution this supervision covers.

  3. Transfer beyond BFCL is insufficiently explained, and applicability depends on access to environment code. The contracts are specific to BFCL's implementation and are not supplied at inference. Improvements on τ²-bench and ACEBench therefore require transfer of learned behavior, but the paper does not establish which behaviors transfer or why. Constructing the table also requires access to environment source code and a frontier LLM, limiting its applicability to black-box tools.

Questions for authors
  1. How many training seeds were used per method? Please report mean ± std (or CIs) over ≥3 training seeds for Table 1 (ID/OOD averages) and Table 3.
  2. How was the "BFCL skill context" used by the baselines constructed, and what information was available to its author or generator?
  3. The replay check keeps an augmentation only if it yields a "higher estimated probability of successful continuation". How is success measured? Does it use the BFCL checker or outcome reward on training tasks? If so, the statement that the pipeline "uses no ground-truth task solution" should be qualified.
  4. The OOD evaluations involve different degrees of distribution shift. BFCL Long Context retains the same APIs, while ACEBench and τ² include domains or state constraints related to those in training. Since augmentation is absent at inference, please analyze which learned behaviors transfer, quantify task and tool-contract overlap, and report case study about reuse of familiar constraints in augmentations (collected from BFCL) to OOD datasets.
  5. Since you "cap each trajectory at B distinct augmented feedback items to bound additional teacher computation", please clarify how you select the B items.
  6. What fraction of training-time rejections are not covered by the offline-constructed augmentations, and how does this fraction evolve as the policy improves?
  7. Since the contracts are task-agnostic, a natural alternative is trajectory-level distillation that places the contracts for the task's API classes in the teacher context throughout the trajectory. Please compare BEAD against this variant under matched settings, for example implemented with RLSD or Skill-SD using the same contracts.
  8. The augmentation table is fixed throughout training. Did you consider updating it online, for example by mining new signatures when uncovered rejections recur above a threshold, or by enumerating the guards directly from source? How would this affect coverage and performance?
6 comments
AuthorsOpenReview ↗

Response to Reviewer VrT2: Part 1

W1 The choice of turn-level, rejection-triggered distillation is not validated. The contracts are task-agnostic, and the table is small and fixed (56 entries). A natural alternative is trajectory-level distillation that gives the teacher the relevant contracts, for example those for the task's API classes, throughout the trajectory. This would also supervise the reasoning before the first rejection and remove the need for the budget B. The paper motivates turn-level supervision only by arguing that task-level context is too coarse, and never compares against trajectory-level use of the same contracts. Without this comparison, it is unclear whether the gains come from the contracts themselves or from turn-level localization, which is the paper's main design choice beyond the contract source.

Q7 Since the contracts are task-agnostic, a natural alternative is trajectory-level distillation that places the contracts for the task's API classes in the teacher context throughout the trajectory. Please compare BEAD against this variant under matched settings, for example implemented with RLSD or Skill-SD using the same contracts.

Thank you for this suggestion. We conducted an additional experiment with Qwen3-4B using an RLSD variant that performs trajectory-level distillation with the same tool contracts as BEAD's main experiment.

Specifically, we organized the augmentations by API class. For each training task, the teacher was provided with all contracts associated with the task's API classes as task-level privileged context throughout the trajectory.

The results are shown below and are also included in Appendix E.5 of our revision.

MethodBFCL v4 (ID) Avg.Δ Avg.
BEAD (Ours)51.56—
RLSD w/ task-level contracts47.89−3.67

Although trajectory-level distillation provides access to the same tool contracts throughout the trajectory, it performs worse than BEAD. This suggests that the benefit comes not only from the extracted contracts, but also from how the supervision is localized around actual rejections. In BEAD, each rejection identifies the relevant tool constraint, allowing the teacher to provide targeted hindsight correction for the rejected call and recovery guidance for the subsequent action, rather than conditioning every action on a broader set of contracts that may not be relevant to the current decision.

The ablations in Table 3 further support the complementary roles of backward and forward supervision. Together, these results support the effectiveness of BEAD's rejection-triggered bidirectional distillation over trajectory-level use of the same contracts.

AuthorsOpenReview ↗

Response to Reviewer VrT2: Part 2

W2 The supervision is limited in both where it applies and what information it provides. Additional teacher supervision is confined to the failed action immediately before a matched tool rejection and the recovery action immediately after it, leaving other steps in the trajectory without this signal. Its content is also limited to contracts mined once from offline trajectories and frozen throughout training, with no mechanism to incorporate new error patterns as the policy evolves. The paper does not quantify how much of the trajectory or the changing rejection distribution this supervision covers.

Q6 What fraction of training-time rejections are not covered by the offline-constructed augmentations, and how does this fraction evolve as the policy improves?

Q8 The augmentation table is fixed throughout training. Did you consider updating it online, for example by mining new signatures when uncovered rejections recur above a threshold, or by enumerating the guards directly from source? How would this affect coverage and performance?

Thank you for these concerns. We clarify the rationale for selective supervision and fixed offline contracts below. Rejection coverage and online augmentation updates are addressed in a separate reply.

Limited scope of additional teacher supervision (W2)

A tool rejection identifies a call that violates an environment requirement. The triggering action and the subsequent recovery action are therefore directly relevant to correcting the violation and restoring progress. We visualize 1,000 randomly sampled successful trajectories and 1,000 randomly sampled failed trajectories in Figure 11. Failed trajectories exhibit denser and more persistent rejection sequences, whereas successful trajectories often recover from rejections and resume successful tool use. These observations suggest that the actions immediately before and after a rejection can play a decisive role in the eventual success or failure of a trajectory, making them particularly valuable targets for additional supervision. BEAD targets these decisions through backward scoring of the rejected action and forward scoring of the recovery action. Importantly, other actions still receive supervision from the standard RL objective through the environment reward. BEAD provides additional teacher supervision specifically around rejections, where the violated tool constraint can be identified, rather than applying contracts that may not be relevant to every decision. Our additional experiment in the response to W1 and Q7 further supports this choice, showing that using the same contracts as privileged context throughout the trajectory performs worse than BEAD.

The additional supervision is selective. 43.8% of rollout trajectories receive teacher scoring, and 27.6% of action steps within those trajectories are scored. This concentrates teacher computation on relevant decisions and avoids scoring every action, while all action steps remain part of the underlying RL objective.

Measure0–910–2930–5960–99100–129130–149Overall
Trajectories scored51.1%49.7%43.7%42.6%40.7%42.6%43.8%
Steps scored within those trajectories32.9%31.6%27.9%26.1%25.8%26.3%27.6%

Supervision content from fixed offline contracts (W2)

Offline trajectories identify rejection signatures, while the corresponding guidance is grounded in environment source code, tool schemas, and cross-tool or state dependencies (§4.1 and Appendix C). These contracts describe requirements enforced by the environment rather than individual trajectories. Although policy evolution changes which errors are encountered, the requirements of a fixed tool implementation remain unchanged. Thus, the same contracts can continue to provide supervision across changing actions and rollout contexts. We further quantify the coverage of these fixed augmentations throughout training in our separate reply.

AuthorsOpenReview ↗

Response to Reviewer VrT2: Part 2 (Continued)

Coverage of the changing rejection distribution (W2, Q6)

Offline augmentations cover 99.22% of training-time rejections for Qwen3-4B and 98.98% for Qwen3-8B. Only 0.78% and 1.02% remain uncovered, respectively, among which the vast majority are Python exception representations produced by invalid invocations rather than explicitly defined environment rejection messages. Coverage remains high across all training stages.

Model0–910–2930–5960–99100–129130–149Overall
Qwen3-4B98.46%99.62%99.80%98.71%99.11%99.47%99.22%
Qwen3-8B97.77%99.45%99.18%99.24%99.15%98.29%98.98%

We also report augmented rejection events per optimization step to account for the different widths of the training bands. The change row compares the final band with the initial band.

Training-step bandState preconditionValue provenanceValue formParameter domainTotal
0–9521.8116.7165.129.1832.7
10–29487.276.0120.825.9710.0
30–59385.851.563.617.1518.1
60–99307.629.653.410.6401.1
100–129351.124.342.811.3429.6
130–149289.227.939.212.0368.4
Change−44.6%−76.1%−76.2%−58.8%−55.8%

All four categories decrease between the initial and final bands, with a 55.8% reduction in total events per step. State-precondition violations decline by 44.6%, while their relative share rises from 62.7% to 78.5% because the other categories decline faster. These measurements show changes in rejection volume and composition alongside persistently high coverage by the fixed table.

Online augmentation updates and their impact on coverage and performance (Q8)

The near-complete observed coverage supports our motivation for constructing augmentations offline from reusable environment knowledge. In the current BFCL setting, the small uncovered fraction leaves limited room for online mining to increase rejection coverage, although its effect on performance has not been directly evaluated.

We appreciate the reviewer's suggestion and see a promising application in environment scaling, where new environments are continually introduced during training. The augmentation table could be extended online with source-grounded contracts for these environments, using recurrent uncovered signatures or direct guard enumeration to guide discovery. Newly constructed augmentations could then be reused across tasks and trajectories, extending coverage and potentially improving performance. We will add these coverage analyses to the revision and discuss online updates in Appendix G(Limitations and Future Work).

Reviewer_VpFCOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes BEAD, a framework to improve multi-turn tool-use reinforcement learning. To address sparse rewards and difficult credit assignment, BEAD first mines "tool contracts" from the environment's source code offline to generate task-agnostic, augmented rejection feedback. It then uses this augmented feedback as privileged context for a teacher model to score actions bidirectionally: backward to correct the rejected call (hindsight) and forward to guide subsequent recovery. The resulting token-level teacher-student log-probability gaps are used to reweight the standard GRPO advantage. Validation is performed on BFCL v4 (in-distribution and out-of-distribution), τ-bench, and ACEBench using Qwen3-4B and Qwen3-8B, showing consistent improvements over strong baselines (GRPO, EnvTuning, Skill-SD, etc.), supported by comprehensive ablations and token-level routing analyses.

Critical strengths

Novel and highly reusable environment augmentation: The idea of mining tool contracts offline directly from the environment's source code to explain why a rejection occurred is highly creative. Because these contracts describe how tools work rather than how to solve a specific task, they are task-agnostic and can be reused across numerous trajectories and tasks, solving the problem of expensive, trajectory-specific privileged context generation.

Elegant bidirectional distillation design: Using the exact same augmented feedback to supervise both the rejected action (backward) and the recovery action (forward) is highly effective. The detailed token-level analysis (Figures 7 & 8) provides a rigorous mechanistic understanding of how forward scoring acts broadly on action tokens while backward scoring acts sparsely but sharply on reasoning tokens.

Critical weaknesses

Dependency on Environment Source Code: The offline augmentation heavily relies on access to the environment's source code (e.g., guards, tables, _load_scenario) to mine the tool contracts. This limits the immediate applicability of BEAD to proprietary, closed-source, or black-box APIs where only a generic error message (e.g., "400 Bad Request") is returned.

Offline Construction Overhead: The offline pipeline uses a heavy LLM (GPT-5.5, as mentioned in Appendix C) as both analyzer and validator, plus replay checks. While the paper notes it is done once, the computational and human effort required to run this pipeline for complex, large-scale API suites might be non-trivial and could be a bottleneck compared to purely online methods.

Several important related studies are missing, particularly on tool feedback for agentic RL, including Tool-Star and Agentic Reinforced Policy Optimization (ARPO). A comparison with these methods would make the current contribution stronger.

Questions for authors

If the environment is a black-box API (e.g., a third-party web service returning only generic error codes like HTTP 400 without descriptive text), can BEAD still construct meaningful augmentations using only the schema and generic errors, or does the method fundamentally require source code access?

How sensitive is the performance to the quality of the offline analyzer/validator (e.g., if a smaller, open-source model were used instead of GPT-5.5 for the offline mining process)?

3 comments
AuthorsOpenReview ↗

Response to Reviewer VpFC: Part 1

W1 Dependency on Environment Source Code: The offline augmentation heavily relies on access to the environment's source code (e.g., guards, tables, _load_scenario) to mine the tool contracts. This limits the immediate applicability of BEAD to proprietary, closed-source, or black-box APIs where only a generic error message (e.g., "400 Bad Request") is returned.

Q1 If the environment is a black-box API (e.g., a third-party web service returning only generic error codes like HTTP 400 without descriptive text), can BEAD still construct meaningful augmentations using only the schema and generic errors, or does the method fundamentally require source code access?

Thank you for raising these related concerns. We acknowledge that the current offline construction pipeline relies on environment source code to mine and validate tool contracts. Access to source code is preferable because it enables more comprehensive mining of tool constraints, such as preconditions and parameter value formats, thereby covering a broader range of environment rejections. However, BEAD's bidirectional self-distillation mechanism does not inherently require source code access, provided that meaningful augmentations can be constructed from the available information.

To investigate this, we conducted an additional experiment on BFCL v4 with Qwen3-4B, constructing offline augmentations using only tool schemas and generic rejection feedback without accessing the environment source code. We then trained Qwen3-4B using BEAD with these augmentations under the same training settings. We report the proportion of rejections covered by offline augmentations and the resulting performance. These results are also included in Appendix E.5 of our revision.

Augmentation ConstructionRejection CoverageBFCL v4 (ID) Avg.
GRPO—44.33
With source code99.2%51.56
Without source code73.5%49.89

Without source code, BEAD still covers 73.5% of training-time rejections and achieves an average score of 49.89. Access to source code further increases coverage to 99.2% and improves performance. These results indicate that BEAD can still benefit from augmentations constructed without source code, while access to the environment implementation substantially improves their coverage and effectiveness.

Nevertheless, we acknowledge that fully black-box APIs with limited schemas and uninformative error messages remain challenging, as their underlying constraints may not be reliably inferred. We have added this limitation to the Appendix G. Investigating reliable contract acquisition from API documentation and tool interactions, while reducing dependence on source code, is an important direction for future work.

AuthorsOpenReview ↗

Response to Reviewer VpFC: Part 2

W2 Offline Construction Overhead: The offline pipeline uses a heavy LLM (GPT-5.5, as mentioned in Appendix C) as both analyzer and validator, plus replay checks. While the paper notes it is done once, the computational and human effort required to run this pipeline for complex, large-scale API suites might be non-trivial and could be a bottleneck compared to purely online methods.

Thank you for raising this concern. We agree that complex, large-scale API suites can increase the upfront cost of mining and validating tool contracts. However, BEAD's offline construction cost primarily depends on the number of distinct rejection signatures requiring augmentation, rather than the number of training trajectories. In contrast, online trajectory-specific annotation requires repeated generation as new trajectories are collected during RL training. As a result, the cumulative cost of repeated online annotation can substantially exceed the one-time cost of offline construction.

We compare the construction costs of online trajectory-specific annotation and BEAD's offline augmentation on BFCL v4. For the online estimate, we collected 31,328 failed trajectories from a 150-step GRPO training run, randomly sampled 100 trajectories for annotation using GPT-5.5, and extrapolated the measured annotation cost and time to all 31,328 trajectories. Both methods use 4 concurrent API requests for the wall-clock time measurements. The results are summarized below.

MethodCostWall-clock Time
Online Method~$1,583.07~49 h
Offline Method (Ours)$6.9712.3 min

We also separately report the offline construction time for contract generation and verification, along with replay checks, excluding online teacher scoring.

Contract generation and verificationReplay checks
12.3 min7.2 min

These results show that BEAD substantially reduces annotation costs compared with online trajectory-specific annotation, demonstrating the efficiency of its reusable offline construction process.

Importantly, although larger API suites may involve more distinct tool contracts, each augmentation can be reused across tasks and trajectories whenever the same rejection signature occurs, as supported by our analysis in Figure 4(a). As RL training generates increasingly many trajectories, this reuse amortizes the one-time construction cost and avoids repeated annotation of the same underlying tool constraints. Therefore, the advantage of offline construction becomes particularly significant at scale, where recurring rejection patterns can be addressed without repeatedly incurring trajectory-specific annotation costs.

W3 Several important related studies are missing, particularly on tool feedback for agentic RL, including Tool-Star and Agentic Reinforced Policy Optimization (ARPO). A comparison with these methods would make the current contribution stronger.

Thank you for pointing out these relevant studies. We agree that Tool-Star and ARPO are important related works and will add both to the Related Work sections in the revision. We clarify their differences from BEAD below.

  • Tool-Star [1] combines cold-start SFT, hierarchical rewards, and self-critic preference optimization to improve multi-tool collaborative reasoning. Its RL training primarily reinforces tool-use behaviors through reward and preference signals.

  • ARPO [2] incorporates cold-start SFT and uses uncertainty after tool interactions to guide adaptive branching. It performs advantage attribution across shared and divergent rollout segments, improving exploration and turn-level credit assignment by estimating advantages from rollout outcomes.

Both methods improve agentic RL through more effective reward-based optimization. However, their RL-stage credit signals are primarily derived from rewards and sampled trajectories, without introducing external privileged information to guide credit attribution. In particular, neither explicitly leverages the underlying tool contracts encoded in the environment implementation.

In contrast, BEAD introduces additional supervision from the environment itself. It extracts task-agnostic tool contracts from the environment implementation and constructs augmented rejection feedback that can be reused across tasks and trajectories. By using this feedback as privileged teacher context, BEAD provides targeted credit attribution around critical rejection and recovery decisions. The resulting token-wise teacher signals further refine credit allocation within these steps. Thus, BEAD goes beyond refining credit estimation from rewards by introducing reusable environment knowledge to inform fine-grained credit assignment.

References

[1] Guanting Dong et al. Tool-Star: Empowering multi-tool collaborative web agent via reinforcement learning. SIGIR 2026: 301–312.

[2] Guanting Dong et al. Agentic reinforced policy optimization. ICLR 2026.

AuthorsOpenReview ↗

Response to Reviewer VpFC: Part 3

Q2 How sensitive is the performance to the quality of the offline analyzer/validator (e.g., if a smaller, open-source model were used instead of GPT-5.5 for the offline mining process)?

Thank you for this question. To assess the sensitivity to the offline analyzer and validator, we replaced GPT-5.5 with the smaller open-source model Qwen3.5-35B-A3B for both roles. We reconstructed the augmented feedback using this model and trained Qwen3-4B with the resulting augmentations under the same training settings. The results on BFCL are shown below and are also included in Appendix E.5 of our revision.

Analyzer/ValidatorBaseMiss FuncMiss ParamBFCL v4 (ID) Avg.
GPT-5.557.3352.0045.3351.56
Qwen3.5-35B-A3B58.0050.6744.6751.11

Replacing GPT-5.5 with Qwen3.5-35B-A3B results in only a 0.45-point decrease in average performance, with comparable results across all three subsets. This suggests that BEAD's performance is not strongly dependent on GPT-5.5 specifically, and that a smaller open-source model can also construct effective tool-contract augmentations.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary for AC

Dear Area Chair,

To help you quickly navigate our rebuttal discussion, we provide the following summary.

We sincerely thank all the reviewers for their time and constructive feedback. We are pleased that they unanimously recognized the value of our core contribution, turning reusable environment knowledge into local supervision for agentic RL.

Reviewers’ Acknowledgement

  • Reusable supervision from the environment (VpFC, VrT2, uKMS). VpFC described the idea as “highly creative” and highlighted its reuse across tasks and trajectories. VrT2 called it “a useful source of reusable step-level supervision,” while uKMS recognized its effectiveness.

  • Motivated bidirectional distillation (VpFC, VrT2). VpFC praised the elegant bidirectional distillation design, while VrT2 recognized the motivation for supervising both rejected calls and recovery.

  • Effectiveness supported by detailed analyses (VpFC, VrT2, uKMS). VpFC highlighted the mechanistic analyses. VrT2 noted that behavioral evidence supports the learning design, while uKMS recognized the experimental improvements.

Discussion Update. Reviewer VrT2 confirmed that the comparison using the same contracts and the coverage analysis addressed their main concerns, found the OOD examples helpful, and raised their score (3 → 4). Reviewer uKMS also confirmed that “The responses have adequately addressed my concerns”, and raised their score (3 → 4).

Summary of Responses and New Results

We addressed the reviewers’ main concerns through additional experiments and clarifications, summarized below.

1. Supervision around tool rejections (VrT2, uKMS)

We compared BEAD with an RLSD variant providing the teacher with the same tool contracts throughout the trajectory, supporting the benefit of supervision around actual rejections. An ablation without the replay check and existing ablations clarify the roles of environment augmentation and bidirectional scoring. We also analyzed how backward scoring adjusts RL update magnitudes under positive and negative advantages without reversing their direction.

See our responses to VrT2, Part 1, and uKMS, Parts 1 and 2.

2. Training stability and related methods (VrT2, uKMS)

Experiments with independent training seeds support BEAD’s consistent improvements over baselines and the contribution of forward supervision. We reproduced HINT-SD using its released implementation on the same training split, with BEAD performing better on ID and OOD evaluations. We also clarified differences from related methods in supervision sources, reuse, and credit attribution.

See our responses to VrT2, Part 4, and uKMS, Part 3.

3. Coverage of fixed offline augmentations (VrT2)

We quantified teacher-scored trajectories and actions and rejection coverage throughout training. Fixed augmentations maintain high coverage as the policy evolves, while teacher supervision remains selective. Other actions still receive the standard RL signal. We also discussed extending the augmentation table for new environments.

See our responses to VrT2, Part 2 and its continuation.

4. Transfer beyond BFCL (VrT2)

We analyzed recurring constraint types and added case studies showing how learned behaviors transfer to unseen environments, including establishing prerequisites, retrieving valid identifiers, correcting value formats, and selecting admissible arguments. These cases make the transfer explanation more concrete without requiring augmentation at inference.

See our response to VrT2, Part 3, and Appendix F.3.

5. Dependence on environment source code and frontier models (VpFC, VrT2)

We conducted additional experiments that construct augmentations without source code and replace the frontier analyzer and validator with a smaller open model. The results show that useful augmentations can still be obtained in these settings. We also clarified the remaining limitations for tools with insufficient schemas or uninformative feedback and discussed possible extensions based on API documentation and tool interactions, as well as online augmentation updates for environment scaling.

See our responses to VpFC, Parts 1 and 3, and Appendix E.5.

6. Construction cost and training efficiency (VpFC, uKMS)

We reported the offline cost of contract generation and verification, with replay checks listed separately, and measured the additional training overhead. We also estimated the cost of repeatedly generating annotations for individual trajectories. These analyses clarify the computational requirements and support the efficiency of reusing augmentations across tasks and trajectories, with modest additional training overhead.

See our responses to VpFC, Part 2, and uKMS, Part 1.

Best regards,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.