acceptodds
Under review as a conference paper at ICLR 2027

BRIDGE: Bridging Expert and Learner Trajectories for Black-box LLM Agent Distillation

Abstract

Distilling capable large language model (LLM) agents into smaller models offers a practical path toward efficient deployment, yet existing approaches typically specialize in either imitation from expert trajectories or correction on student-generated trajectories, leaving the two complementary supervision regimes disconnected. More fundamentally, both paradigms often couple trajectory provenance with supervision direction, treating teacher-generated behavior as the learning target regardless of its realized quality. We introduce BRIDGE, a black-box agent distillation framework that bridges expert and learner trajectories within a unified preference-learning framework while decoupling trajectory source from preference direction. BRIDGE constructs comparisons from both expert- and learner-induced histories and determines their labels from realized trajectory quality, allowing either teacher- or student-generated behavior to provide the positive learning signal. It further explores informative recurrent states and terminates local comparisons upon environment-state reconvergence, yielding finer-grained credit assignment while limiting unnecessary policy divergence. BRIDGE requires only teacher-generated text and environment feedback, without access to teacher logits or internal representations. Across WebShop, ALFWorld, and ScienceWorld, BRIDGE consistently outperforms strong imitation- and correction-based distillation baselines, and remains effective with DeepSeek-R1-Distill-Llama-8B students, demonstrating robust black-box distillation across heterogeneous model families. The anonymous code repository is available at https://anonymous.4open.science/r/bridge123/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agent, On-policy distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Bca9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces BRIDGE, a black-box agent distillation method that learns from both teacher- and student-induced trajectories. Instead of treating teacher behavior as the default target, BRIDGE compares teacher and student continuations using realized trajectory quality and uses the better one as the preferred response for DPO training. It further constructs local cross-policy comparisons and uses state reconvergence to localize preference supervision. Experiments show consistent improvements over prior distillation baselines and across different student model families.

Critical strengths
  1. The paper identifies a useful limitation of teacher-guided distillation: teacher behavior is not always the better target, especially on student-induced trajectories. BRIDGE addresses this by choosing the better continuation based on its actual trajectory quality, rather than whether it comes from the teacher or the student.
  2. The experimental results are consistent across settings, and the ablations support the main design choices. In particular, forcing teacher-generated behavior to always be preferred leads to clear performance drops, which directly supports the paper’s central motivation.
Critical weaknesses
  1. The paper does not report how often the student continuation actually outperforms the teacher. This is important because the main motivation of BRIDGE relies on such cases being common enough to matter. The Teacher+ ablation supports this indirectly, but a direct breakdown would make the central claim more convincing.
  2. The baseline comparison is somewhat limited. The experiments do not include a black-box distillation baseline or a preference-learning baseline; methods such as Black-Box OPD and Agentic-DPO would be natural comparisons.
Questions for authors
  1. How does the training cost of BRIDGE compare with the baselines? If BRIDGE uses more teacher calls or environment rollouts, how does it perform under the same training or rollout budget?
  2. How important is the proposed recurrent-state selection rule? The current method prioritizes shared states, lower-scoring circles, and more revisits. It would be useful to see whether these choices actually outperform random or simpler selection strategies.
3 comments

We sincerely thank the reviewer for the thoughtful feedback and valuable suggestions. We have conducted additional experiments and analyses to address these concerns.

W1. Frequency of student-preferred continuations

We appreciate this important suggestion. We have provided the student-win and teacher-win proportions among non-tied comparisons throughout training (figure).

Notably, the student-win rate increases from approximately 10% to 60–70% on WebShop and from nearly 0% to around 70% on ALFWorld, eventually surpassing teacher-win in both environments. ScienceWorld also exhibits a growing, albeit fluctuating, student-win rate. These findings complement the Teacher⁺ ablation, where always preferring teacher continuations substantially degrades performance, further supporting the importance of quality-based preference labeling.

W2. Black-box distillation and preference-learning baselines

We thank the reviewer for suggesting these comparisons. Black-Box OPD (GAD) was evaluated primarily on conversational tasks rather than interactive agent environments. Agentic-DPO uses a substantially different benchmark setting, and adapting its Policy-Preserving Augmentation (PPA) to our environments would require nontrivial schema-preserving trajectory transformations.

We have instead implemented ETO, a closely related black-box preference-learning baseline on the same three benchmarks. ETO achieves 59.45/87.96/63.40 on WebShop/ALFWorld/ScienceWorld, compared with 65.63/95.62/65.46 for BRIDGE. These improvements provide additional evidence for the effectiveness of BRIDGE's preference construction.

Q1. Training cost and budget efficiency

We appreciate this practical concern. We have provided evaluation score versus wall-clock training time curves for OPD, GRPO, and BRIDGE across all three environments (figure). All three methods use online training.

Despite additional teacher calls and environment interactions, BRIDGE demonstrates favorable training efficiency. It rapidly surpasses OPD and GRPO on WebShop, achieves higher performance than OPD within comparable training time on ALFWorld, and consistently reaches higher scores on ScienceWorld. These results demonstrate that the performance gains are not simply attributable to longer training, although equal wall-clock time does not imply identical rollout budgets.

Q2. Recurrent-state selection

We thank the reviewer for this suggestion. We conducted a Fully Random branching ablation, achieving 65.72/93.06/63.98, compared with 65.63/95.62/65.46 for BRIDGE. The proposed selection strategy improves ALFWorld and ScienceWorld by 2.56 and 1.48 points, respectively.

WebShop averages only 3.5 interaction steps, compared with 13.1/15.6 for ALFWorld/ScienceWorld, offering fewer opportunities for informative recurrent states. This explains why random selection performs comparably on WebShop, while guided selection is more beneficial in longer-horizon environments.

Thank you for the detailed response and additional experiments. They help address my concerns, though some limitations remain. I will maintain my rating of 3.

We thank you for the thoughtful follow-up and for acknowledging the additional experiments and clarifications. We appreciate your recognition of the improvements and your constructive feedback has been valuable in helping us strengthen the manuscript.

Reviewer_13YrOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents BRIDGE, a black-box agent distillation framework that constructs teacher–student comparisons under both expert- and learner-induced histories. It assigns preferences according to realized trajectory quality and trains the student with DPO. BRIDGE further introduces recurrent-state branching and reconvergence-based truncation for local comparisons. Experiments on WebShop, ALFWorld, and ScienceWorld datasets show improvements over baseline methods, with additional evaluation using a student from a different model family.

Critical strengths
  1. The idea of separating trajectory provenance from preference direction is reasonable and useful. It allows the student to learn from better behavior without assuming that teacher-generated responses are always preferable.

  2. The evaluation covers three interactive environments and two student model families. The teacher-preferred ablation and source-quality analysis provide useful evidence for quality-based supervision.

  3. The paper is generally easy to follow. Figures 1 and 2 clearly explain the motivation and comparison construction.

Critical weaknesses
  1. The technical novelty is not sufficiently established. BRIDGE uses standard DPO, while the main methodological distinction lies in preference construction. Comparisons with closely related agent preference-learning methods discussed in Section 2, such as Agentic-DPO and HPL, are missing.

  2. The ablations do not isolate the proposed mechanisms. Increasing KK changes both the comparison strategy and the amount of training data. There are no separate controls for history source, recurrent-state selection, or reconvergence truncation. Moreover, K=0K=0 already captures much of the improvement and performs best on WebShop.

  3. The justification for reconvergence-based credit assignment needs clarification. Reaching the same environment state does not imply identical future behavior, since the two branches retain different interaction histories and may have different remaining step budgets. Thus, truncation may discard consequences of earlier decisions.

Questions for authors
  1. Could you compare against agent preference-learning baselines and provide controlled ablations for teacher-only versus student-only histories, random versus recurrent-state branching, and comparisons with versus without reconvergence truncation, under matched data or interaction budgets?

  2. How is environment-state equality determined in each benchmark? How do you account for different histories and remaining interaction budgets when treating reconvergence as a credit-assignment boundary?

  3. What are the exact violation definitions and penalty weights? How sensitive are the results to these penalties, and how much of the violation-rate reduction remains without them?

  4. Equation (3) excludes comparison i=0i=0, although Equation (1) includes it as the root comparison. Is this a notation error? Please also correct “RIDGE” in Section 3.1.

3 comments

We sincerely thank the reviewer for the thoughtful feedback and constructive suggestions.

W1. Technical novelty and related baselines

Agentic-DPO constructs one-step preferences from expert histories, while HPL learns from offline preferences at multiple granularities. BRIDGE instead enables online, bidirectional teacher–student comparisons under both expert- and learner-induced histories, with quality-based preference labeling and reconvergence-based localization.

In our reproduction attempts, HPL's offline data collection alone required several days per environment, exceeding the available rebuttal time and computational budget. Agentic-DPO also employs a substantially different experimental setting, making direct comparison difficult. We therefore added ETO, a closely related preference-learning baseline evaluated on the same three benchmarks. ETO achieves 59.45/87.96/63.40 on WebShop/ALFWorld/ScienceWorld, compared with 65.63/95.62/65.46 for BRIDGE. These improvements suggest that BRIDGE's benefits extend beyond trajectory-level reward-based preference learning.

W2 & Q1. Controlled ablations

We appreciate the suggestion to better isolate the proposed mechanisms. We conducted three additional ablations:

MethodWebShopALFWorldScienceWorld
w/o Reconvergence65.2394.1662.82
Fully Random65.7293.0663.98
Teacher-only Histories65.1193.8064.22
BRIDGE65.6395.6265.46

These results support the effectiveness of reconvergence, guided branching, and bidirectional history coverage, particularly on longer-horizon tasks. WebShop averages only 3.5 interaction steps, versus 13.1/15.6 for ALFWorld/ScienceWorld, offering fewer opportunities for recurrent states and reconvergence. This also explains why K=0 performs competitively on WebShop.

W3 & Q2. Reconvergence and state equality

We appreciate this insightful question. State equality is determined using observations, admissible actions, and environment-provided state fingerprints. Step counts and remaining budgets are excluded, as requiring their equality would make reconvergence detection overly restrictive.

We do not assume that reconverged states guarantee identical future behavior. Instead, reconvergence serves as a practical boundary for localizing observed behavioral differences, with its empirical utility supported by the ablations above. More rigorous assessment of future-return equivalence, such as through Monte Carlo sampling, would introduce substantial additional computational overhead.

Q3. Violation penalties

We thank the reviewer for requesting clarification. Revised Sec. 4.1 and Appendix A.1 specify four violation types: malformed actions, unparseable reasoning, repeated/looping actions, and invalid/rejected actions. Each incurs a 0.01 penalty, accumulating when multiple types occur.

These violations are generally straightforward to identify from action parsing and environment feedback, without additional models or annotations. Since WebShop and ALFWorld provide only terminal rewards, removing these penalties would cause many intermediate comparisons to become ties, substantially reducing usable preference supervision.

Q4. Notation and typo

We thank the reviewer for the careful reading. We have standardized the indexing in Equations (1) and (3) to i=0,...,N_x−1, including the root comparison, and corrected “RIDGE” in Sec. 3.1.

Thanks for the additional clarifications and experiments. My main concerns have been largely addressed. I'd like to raise the overall score to 3.

We sincerely appreciate the your encouraging feedback and willingness to raise the score. Thank you for recognizing our additional experiments and clarifications. Your thoughtful comments have been valuable in helping us improve the rigor and presentation of our work.

Reviewer_2dNCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

BRIDGE trains a small agent from a stronger one using only the teacher’s text and environment feedback. For each task, it pairs a teacher trajectory from a fixed pool with a new student trajectory. At up to two repeated states, one model continues from the other’s history. Each comparison stops once both branches reach the same state again. The branch with the higher return (reward plus penalties) is preferred, no matter which model produced it, and the student is trained online with DPO. The teacher is a GiGPO-trained Qwen3-8B and the student is Qwen3-4B. BRIDGE scores 65.63/95.62/65.46 on WebShop/ALFWorld/ScienceWorld (Table 1). SFT, the best baseline, scores 58.18/89.78/62.28.

Critical strengths
  1. Labeling by return is a real source of gain. BRIDGE beats SFT by 7.45/5.84/3.18 points (Table 1). If the teacher branch always wins (Teacher+, Table 2), scores drop to 58.53/87.23/61.52. That is below SFT on ALFWorld and ScienceWorld. On ALFWorld, if the seen and unseen splits are pooled (274 games), this is about 262 vs 239 solved games. That gap is probably larger than noise, even with one seed.

  2. Only teacher text is needed. With a DeepSeek-R1-Distill-Llama-8B student, logit-based OPD and TCOD cannot be used directly. BRIDGE still beats SFT there on ALFWorld (97.78 vs 90.88) and ScienceWorld (65.50 vs 61.86). On WebShop the gain is only 0.77.

Critical weaknesses
  1. The first contribution overlaps with two uncited papers. ETO[1] uses the same three benchmarks. It also picks the higher-reward side of a teacher-student pair and drops ties. IPR[2] branches from expert prefixes for step-level DPO. The new parts add little on top. The root-only variant (K=0), which is roughly an online ETO, already scores 65.92/93.06/63.22 (Table 2). K>0 adds −0.29/+2.56/+2.24, from one seed with checkpoints picked on the test set.

  2. The reward, not the teacher, may be doing the work. Only BRIDGE learns from reward-labeled student rollouts, over at least 4,800 online tasks. SFT, OPD and TCOD see reward only indirectly, through the 0.7 filter and the trained teacher. No RL baseline, such as GRPO or GiGPO, is run on the 4B student with the same reward. Table 3 is consistent with a teacher that adds little. On WebShop, a much weaker teacher pool (average 91.26 to 55.60) moves BRIDGE only from 65.63 to 65.39.

[1] Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. Trial and Error: Exploration-Based Trajectory Optimization of LLM Agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), pages 7584–7600, 2024. [2] Weimin Xiong, Yifan Song, Xiutian Zhao, Wenhao Wu, Xun Wang, Ke Wang, Cheng Li, Wei Peng, and Sujian Li. Watch Every Step! LLM Agent Learning via Iterative Step-level Process Refinement. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024), pages 1556–1572, 2024.

Questions for authors
  1. Does BRIDGE start from the base student or from the SFT model, and what is the reference policy?

  2. What are the penalty values, and do they apply in ScienceWorld? Sec. 3.1 and Sec. 4.1 describe them differently.

3 comments

We sincerely thank the reviewer for the careful assessment and valuable suggestions, which have helped us improve the paper.

W1. Novelty relative to ETO and IPR

We appreciate the reviewer for highlighting these relevant works. We have now discussed both ETO and IPR in the Related work section of the revised manuscript.

We respectfully clarify three distinctions. First, BRIDGE constructs bidirectional cross-policy comparisons under both expert- and learner-induced histories, rather than comparing complete trajectories (ETO) or exploring expert prefixes (IPR). Second, BRIDGE localizes comparisons to divergent segments bounded by environment-state reconvergence, without requiring Monte Carlo step-reward estimation as in IPR. Third, unlike ETO's staged exploration and optimization, BRIDGE continuously collects preferences from the evolving student with bounded policy lag, without requiring SFT initialization.

We agree that the strong K=0 results demonstrate the importance of root-level preference learning. Nevertheless, K=2 further improves ALFWorld and ScienceWorld by +2.56 and +2.24 points, respectively. Additional score curves on WebShop also suggest more favorable training dynamics with K>0.

W2. Reward supervision versus teacher contribution

We sincerely appreciate this important observation. To address the concern, we have additionally evaluated GRPO on the same Qwen3-4B student, using the same environment rewards and step-violation penalties. Following GiGPO's training configuration, GRPO uses 150 rollout batches, 16 tasks per batch, and 8 trajectories per task.

GRPO achieves 57.96/83.94/52.83 on WebShop/ALFWorld/ScienceWorld, whereas BRIDGE achieves 65.63/95.62/65.46, outperforming GRPO by +7.67/+11.68/+12.63 points, respectively. These results suggest that BRIDGE's improvements cannot be explained merely by access to reward supervision.

We also appreciate the reviewer's interpretation of Table 3. We agree that robustness to weaker teachers does not establish that teacher guidance is indispensable. Our intention was to evaluate sensitivity to teacher quality, and we will clarify this distinction in the revised manuscript.

Q1. Student initialization and reference policy

BRIDGE initializes directly from the original Qwen3-4B checkpoint, without SFT warm-starting. The DPO reference policy is a frozen copy of this initial checkpoint throughout training.

Q2. Penalty values and applicability

We thank the reviewer for identifying this inconsistency. We have clarified the penalty values in Sec. 4.1 of the revised manuscript. Each violation type incurs a penalty of 0.01, with penalties accumulating when multiple violations occur at the same step. These penalties apply to all three environments, including ScienceWorld, and are used only for preference construction. The reported benchmark evaluation scores remain unaffected.

Thanks for the reply. The GRPO baseline resolves my main concern. Q1 and Q2 are also answered very clearly. I raised the score to 3.

We sincerely thanks for the positive feedback and for raising the score. We are glad that the additional GRPO baseline and clarifications have addressed your main concerns. We truly appreciate your constructive suggestions throughout the review process, which have helped us strengthen the paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We sincerely thank the reviewers for their constructive feedback. In response, we expanded related-work discussions, added new baselines and controlled ablations, and clarified methodological and implementation details. We summarize the main concerns and responses below.

1. Novelty, Related Work, and Effectiveness of Proposed Designs (2dNC W1; 13Yr W1/W2/Q1; Bca9 W2/Q2)

We expanded Related Work to discuss ETO, IPR, Agentic-DPO, and HPL, clarifying BRIDGE's distinctions: online bidirectional teacher–student comparisons across both history sources, preference labels based on realized quality, and reconvergence-bounded local comparisons.

We implemented ETO on all three benchmarks. BRIDGE achieves 65.63/95.62/65.46 on WebShop/ALFWorld/ScienceWorld, versus 59.45/87.96/63.40 for ETO (+6.18/+7.66/+2.06).

To isolate the proposed mechanisms, we added three ablations:

VariantWebShopALFWorldScienceWorld
Teacher-only histories65.1193.8064.22
Fully random branching65.7293.0663.98
Without reconvergence65.2394.1662.82
BRIDGE65.6395.6265.46

Increasing K from 0 to 2 additionally improves ALFWorld and ScienceWorld by +2.56/+2.24. The smaller gains on WebShop are consistent with its much shorter episodes (3.5 steps on average, versus 13.1/15.6). Together, these experiments support the effectiveness of the overall framework and its constituent designs, especially on longer-horizon tasks.

2. Reward Supervision and the Role of Student-Preferred Behavior (2dNC W2; Bca9 W1)

To assess whether BRIDGE's gains merely reflect reward access, we trained GRPO with the same Qwen3-4B student and environment rewards. BRIDGE exceeds GRPO by +7.67/+11.68/+12.63 points across the three benchmarks.

We also reported student-win fractions among non-tied comparisons: they increase from roughly 10% to 60–70% on WebShop and 0% to ~70% on ALFWorld, eventually exceeding teacher-win rates in both. ScienceWorld shows an upward but more variable trend. These findings, alongside the Teacher⁺ ablation, support quality-based labeling rather than automatically preferring teacher outputs.

3. Reconvergence and State Equality (13Yr W3/Q2)

We clarified that state equality uses observations, admissible actions, and environment fingerprints, excluding history and remaining step budgets. We do not claim identical future behavior after reconvergence; it is a practical boundary for local preference attribution. The no-reconvergence ablation provides empirical support for its utility, without establishing exact future-return equivalence.

4. Training Efficiency (Bca9 Q1)

We added score-versus-wall-clock-time curves for BRIDGE, OPD, and GRPO. Despite additional teacher calls, BRIDGE surpasses OPD and GRPO quickly on WebShop, exceeds OPD at comparable training time on ALFWorld, and achieves higher scores on ScienceWorld. We acknowledge that matched wall-clock time does not imply identical rollout budgets.

5. Implementation Clarifications (2dNC Q1/Q2; 13Yr Q3/Q4)

We clarified that BRIDGE starts from the base Qwen3-4B without SFT warm-starting, with a frozen initial checkpoint as the DPO reference. We specified four violations (malformed actions, unparseable reasoning, loops, and invalid actions), each penalized by 0.01 for preference construction across all environments; benchmark evaluation scores are unaffected. We also corrected the equation indexing and a typo in Section 3.1.

Overall, these additions strengthen the comparisons with prior work, substantiate the key design choices, and improve the paper's clarity and reproducibility.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.