- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes UCOB, a framework that learns to use and update agentic skills through credit-aware bidirectional self-distillation. The same language model generates skill-conditioned and no-skill rollouts, which are grouped by task and anchor state. The method selects a high-return record to supervise the opposite context, rather than always treating skill-conditioned behavior as the teacher. Rollout experience also supports task-level and state-level skill generation, utility-based retrieval, and reflection self-training. Experiments cover three model scales across ALFWorld, WebShop, and Search-QA, with additional analyses of continual learning and computational costs.
Critical strengths
-
UCOB considers that retrieved skills may help in some states but mislead the policy in others. Its credit-aware bidirectional self-distillation uses local return evidence to select whether the skill-conditioned or no-skill view provides supervision.
-
The authors conduct ablation studies on anchor-state grouping, mixed rollouts, and reflection self-training, and compare CBSD with fixed-direction distillation. These experiments help assess the contributions of local credit assignment and bidirectional supervision.
-
The experiments cover multiple model scales, evaluation with and without skills, and continual adaptation across environments. The training-cost analysis uses the same backbone and hardware for the compared methods, allowing task performance to be assessed under comparable wall-clock budgets.
Critical weaknesses
-
UCOB relies on skill-conditioned and no-skill rollouts reaching matching anchor states. In larger state spaces or tasks with more complex histories, such matches may become sparse, reducing the available distillation samples. The experiments do not sufficiently examine matching coverage and reliability in these settings.
-
Although the 1.7B model shows substantial gains on interactive tasks, improvements on larger backbones are relatively modest. At 7B, success-rate gains over the strongest reported baselines are 0.8 and 3.1 percentage points on ALFWorld and WebShop, respectively. Search-QA averages remain below the best baseline at all three scales. Multiple-run uncertainty estimates are needed to assess the consistency of the smaller gains.
-
UCOB uses multiple skills together, but improved performance does not mean that every retrieved skill was helpful. An ineffective skill may receive positive credit simply because it accompanies useful skills. Since these scores guide retrieval and reflection self-training, further evidence is needed to establish whether they reliably reflect individual skill contributions.
Questions for authors
-
Could the authors evaluate UCOB with deliberately incorrect or irrelevant skills? This would help determine whether CBSD selectively rejects misleading guidance while retaining useful skills.
-
Would UCOB remain effective when tasks from different environments are mixed during training? How well would the learned skills and skill writer generalize to an unseen environment?
6 comments
Response (Part 2)
Comment 4 (Q1): What happens with deliberately incorrect or irrelevant skills?
Thank you for suggesting this stress test. We deliberately retrieve low-similarity skills during training and compare UCOB with Fixed-direction: fixed skill-to-no-skill distillation with return-gap rejection disabled. Both use the same retrieval rule while evolving their own banks; the token-confidence mechanism is retained.
We then evaluate under low-relevance retrieval, no skills, and normal retrieval:
| Environment | Method | Low-relevance skills | No skills | Default retrieval |
|---|---|---|---|---|
| ALFWorld | UCOB | 83.6 | 85.9 | 86.7 |
| ALFWorld | Fixed-direction | 52.3 | 54.7 | 53.1 |
| WebShop | UCOB | 65.6 | 73.4 | 71.8 |
| WebShop | Fixed-direction | 62.5 | 69.5 | 68.7 |
Values are success rates (%). CBSD supports robust learning under low-relevance guidance, with gains retained when retrieval is disabled. The comparison assesses its adaptive routing and return-gap rejection together, and the no-skill results support internalization during training. This intervention addresses irrelevant skills; semantically relevant but deliberately false instructions require a separate test. See Appendix F.2, Figure 13 and Table 4.
Comment 5 (Q2): Mixed-environment training and transfer to an unseen environment.
We appreciate the suggestion to examine broader training settings. We evaluate joint and sequential multi-environment learning, then discuss skill applicability across environments.
Mixed training. We train both UCOB and SDAR jointly on ALFWorld and WebShop, matching the backbone, task sampling, rollout budget, and training updates:
| Setting | Configuration |
|---|---|
| Policy and budget | Qwen3-1.7B; 300 updates |
| Tasks per update | Eight ALFWorld and eight WebShop tasks |
| Rollouts | Eight per task |
| Skill bank | Both environments, with source tags and environment-conditioned retrieval |
Final evaluation success rates (%) are:
| Method | ALFWorld | WebShop |
|---|---|---|
| UCOB | 95.3 | 73.4 |
| SDAR | 84.4 | 56.3 |
UCOB improves success by 10.9 and 17.1 points under joint training. With the same cumulative ALFWorld task and rollout count as 150 single-environment updates, ALFWorld success increases from 89.1% to 95.3%, suggesting a benefit from joint training with additional WebShop experience. See Section 4.4, Figure 8, and Appendix F.7.
Sequential adaptation. As complementary evidence, the existing ALFWorld → WebShop → Search-QA experiment in Section 4.4, Figure 7 reports:
| Method | Final three-environment average (%) |
|---|---|
| UCOB | 68.4 |
| SFT | 57.0 |
| SDAR | 51.8 |
UCOB achieves stronger final aggregate performance after sequential training. Figure 7 additionally tracks performance on previously trained environments throughout the sequence.
Cross-environment skill applicability. ALFWorld and WebShop involve different task structures and action spaces, so their procedural skills are environment-specific. Accordingly, mixed training shares the policy while retrieving skills within each environment. Our low-relevance and no-skill evaluations further show that UCOB retains its advantage when retrieved guidance is poorly matched or unavailable (Appendix F.2, Table 4), supporting robust policy learning beyond reliance on applicable external skills.
We thank you again for these constructive suggestions. They helped us provide more direct evidence on anchor availability and robustness under irrelevant guidance, clarify shared skill credit, and extend the evaluation toward mixed-environment learning. We would be grateful if you could reconsider your assessment in light of this additional evidence.
Additional results on mixed-environment training
Dear Reviewer jNJv,
Following our earlier response, we would like to share additional experiments addressing your suggestion to evaluate mixed-environment training. We have now completed Skill1, GRPO, and GiGPO comparisons alongside UCOB and SDAR. All methods use Qwen3-1.7B and 300 training updates, with equal numbers of ALFWorld and WebShop tasks in each update and the same task and rollout budget.
Final evaluation success rates (%) are:
| Method | ALFWorld | WebShop |
|---|---|---|
| UCOB | 95.3 | 73.4 |
| SDAR | 84.4 | 56.3 |
| Skill1 | 61.7 | 28.1 |
| GRPO | 59.4 | 60.2 |
| GiGPO | 86.7 | 67.2 |
UCOB achieves the highest final success rate on both environments, exceeding the strongest baseline, GiGPO, by approximately 8.6 and 6.3 percentage points. These additional comparisons strengthen the evidence for UCOB's effectiveness when learning jointly across environments. We have updated Figure 8 in the revised manuscript to include all five methods, with experimental settings in Appendix F.7.
Thank you again for this helpful suggestion. We would welcome your thoughts on these results and any remaining questions, and would be grateful if you could reconsider your assessment in light of the additional evidence.
Response (Part 1)
Thank you for recognizing the motivation for bidirectional supervision and for the concrete suggestions on its reliability and broader applicability. They helped us strengthen the evaluation with anchor-coverage, low-relevance guidance, and retrieval experiments.
Comment 1 (W1): Anchor matching may become sparse as trajectories diverge.
Thank you for raising the availability of local comparisons as trajectories diverge. We measure cross-view anchor coverage over complete 150-step Qwen3-1.7B runs: the fraction of valid records in task-anchor groups containing both views, before teacher selection or gap filtering. We also report coverage excluding each rollout's first valid record.
Values are percentages from pooled counts within each training stage. Non-initial coverage retains the original groups and excludes initial records from both counts. Local comparisons remain widely available beyond common starting points. See Appendix F.3, Table 5.
As a complementary reliability check, we freeze the checkpoint and skill bank and select anchors from 50 ALFWorld tasks independently of rollout outcomes. From each restored state, eight continuations per view select a direction and 16 fresh continuations per view validate it. Maximum-record routing retains a +0.214 average held-out return advantage over its 33 non-tied selections. This tests teacher selection under controlled starting states, complementing coverage under the observable training representation. Full results are in Appendix F.4, Table 6.
Comment 2 (W2): Larger-model gains need multiple-run uncertainty; Search-QA is below the best baseline.
Thank you for requesting repeated-run evidence. We repeated UCOB training twice for Qwen2.5-7B and Qwen3-1.7B on ALFWorld and WebShop. Success rates (%) are shown below; Run 0 is the original main-table run.
The three-run means differ from the original results by 0.3–2.6 percentage points. UCOB's three-run mean remains above the reported baseline results in all four settings. See Appendix F.8, Table 7.
We also agree that Search-QA should be described as competitive: its average accuracy is within 0.8/0.9/0.2 points of the best listed result at 7B/3B/1.7B, respectively. See Section 4.2, Table 2.
Comment 3 (W3): An ineffective skill may receive positive credit from useful companions.
Thank you for highlighting this credit-assignment issue. We have clarified the utility definition and added a reduced-bundle training ablation.
Shared outcome credit. Each utility accumulates usage-associated feedback through an EMA across interactions. A successful rollout can therefore assign positive credit to an ineffective companion skill. Subsequent updates reflect each skill's usage history; isolating its individual contribution requires skill-specific removal or replacement. Appendix C clarifies this interpretation.
Reduced-bundle training. We additionally test Top-1 per pool, retrieving at most one task skill and one state skill during training on ALFWorld and WebShop. This variant has lower training success, particularly on WebShop. The reduced-bundle ablation supports the value of complementary skills, while utilities summarize each skill's usage-associated outcomes across training. See Appendix F.1, Figure 12.