- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces BRIDGE, a black-box agent distillation method that learns from both teacher- and student-induced trajectories. Instead of treating teacher behavior as the default target, BRIDGE compares teacher and student continuations using realized trajectory quality and uses the better one as the preferred response for DPO training. It further constructs local cross-policy comparisons and uses state reconvergence to localize preference supervision. Experiments show consistent improvements over prior distillation baselines and across different student model families.
Critical strengths
- The paper identifies a useful limitation of teacher-guided distillation: teacher behavior is not always the better target, especially on student-induced trajectories. BRIDGE addresses this by choosing the better continuation based on its actual trajectory quality, rather than whether it comes from the teacher or the student.
- The experimental results are consistent across settings, and the ablations support the main design choices. In particular, forcing teacher-generated behavior to always be preferred leads to clear performance drops, which directly supports the paper’s central motivation.
Critical weaknesses
- The paper does not report how often the student continuation actually outperforms the teacher. This is important because the main motivation of BRIDGE relies on such cases being common enough to matter. The Teacher+ ablation supports this indirectly, but a direct breakdown would make the central claim more convincing.
- The baseline comparison is somewhat limited. The experiments do not include a black-box distillation baseline or a preference-learning baseline; methods such as Black-Box OPD and Agentic-DPO would be natural comparisons.
Questions for authors
- How does the training cost of BRIDGE compare with the baselines? If BRIDGE uses more teacher calls or environment rollouts, how does it perform under the same training or rollout budget?
- How important is the proposed recurrent-state selection rule? The current method prioritizes shared states, lower-scoring circles, and more revisits. It would be useful to see whether these choices actually outperform random or simpler selection strategies.
3 comments
Thank you for the detailed response and additional experiments. They help address my concerns, though some limitations remain. I will maintain my rating of 3.
We thank you for the thoughtful follow-up and for acknowledging the additional experiments and clarifications. We appreciate your recognition of the improvements and your constructive feedback has been valuable in helping us strengthen the manuscript.
We sincerely thank the reviewer for the thoughtful feedback and valuable suggestions. We have conducted additional experiments and analyses to address these concerns.
W1. Frequency of student-preferred continuations
We appreciate this important suggestion. We have provided the student-win and teacher-win proportions among non-tied comparisons throughout training (figure).
Notably, the student-win rate increases from approximately 10% to 60–70% on WebShop and from nearly 0% to around 70% on ALFWorld, eventually surpassing teacher-win in both environments. ScienceWorld also exhibits a growing, albeit fluctuating, student-win rate. These findings complement the Teacher⁺ ablation, where always preferring teacher continuations substantially degrades performance, further supporting the importance of quality-based preference labeling.
W2. Black-box distillation and preference-learning baselines
We thank the reviewer for suggesting these comparisons. Black-Box OPD (GAD) was evaluated primarily on conversational tasks rather than interactive agent environments. Agentic-DPO uses a substantially different benchmark setting, and adapting its Policy-Preserving Augmentation (PPA) to our environments would require nontrivial schema-preserving trajectory transformations.
We have instead implemented ETO, a closely related black-box preference-learning baseline on the same three benchmarks. ETO achieves 59.45/87.96/63.40 on WebShop/ALFWorld/ScienceWorld, compared with 65.63/95.62/65.46 for BRIDGE. These improvements provide additional evidence for the effectiveness of BRIDGE's preference construction.
Q1. Training cost and budget efficiency
We appreciate this practical concern. We have provided evaluation score versus wall-clock training time curves for OPD, GRPO, and BRIDGE across all three environments (figure). All three methods use online training.
Despite additional teacher calls and environment interactions, BRIDGE demonstrates favorable training efficiency. It rapidly surpasses OPD and GRPO on WebShop, achieves higher performance than OPD within comparable training time on ALFWorld, and consistently reaches higher scores on ScienceWorld. These results demonstrate that the performance gains are not simply attributable to longer training, although equal wall-clock time does not imply identical rollout budgets.
Q2. Recurrent-state selection
We thank the reviewer for this suggestion. We conducted a Fully Random branching ablation, achieving 65.72/93.06/63.98, compared with 65.63/95.62/65.46 for BRIDGE. The proposed selection strategy improves ALFWorld and ScienceWorld by 2.56 and 1.48 points, respectively.
WebShop averages only 3.5 interaction steps, compared with 13.1/15.6 for ALFWorld/ScienceWorld, offering fewer opportunities for informative recurrent states. This explains why random selection performs comparably on WebShop, while guided selection is more beneficial in longer-horizon environments.