acceptodds
Under review as a conference paper at ICLR 2027

UCOB: Learning to Utilize and Evolve Agentic Skills via Credit-Aware On-Policy Bidirectional Self-Distillation

Abstract

Skills can improve agentic reinforcement learning by reusing past experience as textual guidance, but retrieved skills are not always reliable: they may help in one state while misleading in another. This makes the common privileged-teacher assumption fragile, namely that a skill-conditioned prompt can be treated as a fixed teacher for the no-skill prompt. We propose UCOB, a framework that first learns when to utilize agentic skills via credit-aware on-policy bidirectional self-distillation, then evolves them through utility-aware skill updates and reflection self-training. UCOB constructs two on-policy context views from skill-conditioned and no-skill prompts, compares return-to-go among rollouts sharing the same task and anchor state, and uses the higher-return view as the local teacher. The resulting local credit signal internalizes useful skill-conditioned behavior, corrects misleading skill usage, and further updates the dual-granularity skill memory, informs utility-aware retrieval, and supports reflection self-training. Experiments on ALFWorld, WebShop, and Search-QA show that UCOB outperforms skill-free RL baselines, skill-augmented methods, and self-distillation baselines across model scales, achieving up to 23.5 and 18.0 absolute success-rate gains over SOTA baselines on ALFWorld and WebShop, respectively. Ablations and analyses further validate its core mechanisms, continual adaptation across environments, and modest training overhead.

Primary area:
reinforcement learning
Keywords:
agentic reinforcement learning, skill learning, on-policy self-distillation, credit assignment, skill memory, continual learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_jNJvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UCOB, a framework that learns to use and update agentic skills through credit-aware bidirectional self-distillation. The same language model generates skill-conditioned and no-skill rollouts, which are grouped by task and anchor state. The method selects a high-return record to supervise the opposite context, rather than always treating skill-conditioned behavior as the teacher. Rollout experience also supports task-level and state-level skill generation, utility-based retrieval, and reflection self-training. Experiments cover three model scales across ALFWorld, WebShop, and Search-QA, with additional analyses of continual learning and computational costs.

Critical strengths
  1. UCOB considers that retrieved skills may help in some states but mislead the policy in others. Its credit-aware bidirectional self-distillation uses local return evidence to select whether the skill-conditioned or no-skill view provides supervision.

  2. The authors conduct ablation studies on anchor-state grouping, mixed rollouts, and reflection self-training, and compare CBSD with fixed-direction distillation. These experiments help assess the contributions of local credit assignment and bidirectional supervision.

  3. The experiments cover multiple model scales, evaluation with and without skills, and continual adaptation across environments. The training-cost analysis uses the same backbone and hardware for the compared methods, allowing task performance to be assessed under comparable wall-clock budgets.

Critical weaknesses
  1. UCOB relies on skill-conditioned and no-skill rollouts reaching matching anchor states. In larger state spaces or tasks with more complex histories, such matches may become sparse, reducing the available distillation samples. The experiments do not sufficiently examine matching coverage and reliability in these settings.

  2. Although the 1.7B model shows substantial gains on interactive tasks, improvements on larger backbones are relatively modest. At 7B, success-rate gains over the strongest reported baselines are 0.8 and 3.1 percentage points on ALFWorld and WebShop, respectively. Search-QA averages remain below the best baseline at all three scales. Multiple-run uncertainty estimates are needed to assess the consistency of the smaller gains.

  3. UCOB uses multiple skills together, but improved performance does not mean that every retrieved skill was helpful. An ineffective skill may receive positive credit simply because it accompanies useful skills. Since these scores guide retrieval and reflection self-training, further evidence is needed to establish whether they reliably reflect individual skill contributions.

Questions for authors
  1. Could the authors evaluate UCOB with deliberately incorrect or irrelevant skills? This would help determine whether CBSD selectively rejects misleading guidance while retaining useful skills.

  2. Would UCOB remain effective when tasks from different environments are mixed during training? How well would the learned skills and skill writer generalize to an unseen environment?

6 comments
AuthorsOpenReview ↗

Response (Part 1)

Thank you for recognizing the motivation for bidirectional supervision and for the concrete suggestions on its reliability and broader applicability. They helped us strengthen the evaluation with anchor-coverage, low-relevance guidance, and retrieval experiments.


Comment 1 (W1): Anchor matching may become sparse as trajectories diverge.

Thank you for raising the availability of local comparisons as trajectories diverge. We measure cross-view anchor coverage over complete 150-step Qwen3-1.7B runs: the fraction of valid records in task-anchor groups containing both views, before teacher selection or gap filtering. We also report coverage excluding each rollout's first valid record.

EnvironmentRecordsSteps 1–50Steps 51–100Steps 101–150
ALFWorldAll86.591.693.0
ALFWorldNon-initial86.291.392.5
WebShopAll67.777.787.7
WebShopNon-initial64.274.285.1

Values are percentages from pooled counts within each training stage. Non-initial coverage retains the original groups and excludes initial records from both counts. Local comparisons remain widely available beyond common starting points. See Appendix F.3, Table 5.

As a complementary reliability check, we freeze the checkpoint and skill bank and select anchors from 50 ALFWorld tasks independently of rollout outcomes. From each restored state, eight continuations per view select a direction and 16 fresh continuations per view validate it. Maximum-record routing retains a +0.214 average held-out return advantage over its 33 non-tied selections. This tests teacher selection under controlled starting states, complementing coverage under the observable training representation. Full results are in Appendix F.4, Table 6.


Comment 2 (W2): Larger-model gains need multiple-run uncertainty; Search-QA is below the best baseline.

Thank you for requesting repeated-run evidence. We repeated UCOB training twice for Qwen2.5-7B and Qwen3-1.7B on ALFWorld and WebShop. Success rates (%) are shown below; Run 0 is the original main-table run.

BackboneEnvironmentRun 0Run 1Run 2Avg.
Qwen2.5-7BALFWorld93.089.895.392.7
Qwen2.5-7BWebShop85.983.680.583.3
Qwen3-1.7BALFWorld89.189.090.689.6
Qwen3-1.7BWebShop79.776.678.178.1

The three-run means differ from the original results by 0.3–2.6 percentage points. UCOB's three-run mean remains above the reported baseline results in all four settings. See Appendix F.8, Table 7.

We also agree that Search-QA should be described as competitive: its average accuracy is within 0.8/0.9/0.2 points of the best listed result at 7B/3B/1.7B, respectively. See Section 4.2, Table 2.


Comment 3 (W3): An ineffective skill may receive positive credit from useful companions.

Thank you for highlighting this credit-assignment issue. We have clarified the utility definition and added a reduced-bundle training ablation.

Shared outcome credit. Each utility accumulates usage-associated feedback through an EMA across interactions. A successful rollout can therefore assign positive credit to an ineffective companion skill. Subsequent updates reflect each skill's usage history; isolating its individual contribution requires skill-specific removal or replacement. Appendix C clarifies this interpretation.

Reduced-bundle training. We additionally test Top-1 per pool, retrieving at most one task skill and one state skill during training on ALFWorld and WebShop. This variant has lower training success, particularly on WebShop. The reduced-bundle ablation supports the value of complementary skills, while utilities summarize each skill's usage-associated outcomes across training. See Appendix F.1, Figure 12.

AuthorsOpenReview ↗

Response (Part 2)

Comment 4 (Q1): What happens with deliberately incorrect or irrelevant skills?

Thank you for suggesting this stress test. We deliberately retrieve low-similarity skills during training and compare UCOB with Fixed-direction: fixed skill-to-no-skill distillation with return-gap rejection disabled. Both use the same retrieval rule while evolving their own banks; the token-confidence mechanism is retained.

We then evaluate under low-relevance retrieval, no skills, and normal retrieval:

EnvironmentMethodLow-relevance skillsNo skillsDefault retrieval
ALFWorldUCOB83.685.986.7
ALFWorldFixed-direction52.354.753.1
WebShopUCOB65.673.471.8
WebShopFixed-direction62.569.568.7

Values are success rates (%). CBSD supports robust learning under low-relevance guidance, with gains retained when retrieval is disabled. The comparison assesses its adaptive routing and return-gap rejection together, and the no-skill results support internalization during training. This intervention addresses irrelevant skills; semantically relevant but deliberately false instructions require a separate test. See Appendix F.2, Figure 13 and Table 4.


Comment 5 (Q2): Mixed-environment training and transfer to an unseen environment.

We appreciate the suggestion to examine broader training settings. We evaluate joint and sequential multi-environment learning, then discuss skill applicability across environments.

Mixed training. We train both UCOB and SDAR jointly on ALFWorld and WebShop, matching the backbone, task sampling, rollout budget, and training updates:

SettingConfiguration
Policy and budgetQwen3-1.7B; 300 updates
Tasks per updateEight ALFWorld and eight WebShop tasks
RolloutsEight per task
Skill bankBoth environments, with source tags and environment-conditioned retrieval

Final evaluation success rates (%) are:

MethodALFWorldWebShop
UCOB95.373.4
SDAR84.456.3

UCOB improves success by 10.9 and 17.1 points under joint training. With the same cumulative ALFWorld task and rollout count as 150 single-environment updates, ALFWorld success increases from 89.1% to 95.3%, suggesting a benefit from joint training with additional WebShop experience. See Section 4.4, Figure 8, and Appendix F.7.

Sequential adaptation. As complementary evidence, the existing ALFWorld → WebShop → Search-QA experiment in Section 4.4, Figure 7 reports:

MethodFinal three-environment average (%)
UCOB68.4
SFT57.0
SDAR51.8

UCOB achieves stronger final aggregate performance after sequential training. Figure 7 additionally tracks performance on previously trained environments throughout the sequence.

Cross-environment skill applicability. ALFWorld and WebShop involve different task structures and action spaces, so their procedural skills are environment-specific. Accordingly, mixed training shares the policy while retrieving skills within each environment. Our low-relevance and no-skill evaluations further show that UCOB retains its advantage when retrieved guidance is poorly matched or unavailable (Appendix F.2, Table 4), supporting robust policy learning beyond reliance on applicable external skills.


We thank you again for these constructive suggestions. They helped us provide more direct evidence on anchor availability and robustness under irrelevant guidance, clarify shared skill credit, and extend the evaluation toward mixed-environment learning. We would be grateful if you could reconsider your assessment in light of this additional evidence.

AuthorsOpenReview ↗

Additional results on mixed-environment training

Dear Reviewer jNJv,

Following our earlier response, we would like to share additional experiments addressing your suggestion to evaluate mixed-environment training. We have now completed Skill1, GRPO, and GiGPO comparisons alongside UCOB and SDAR. All methods use Qwen3-1.7B and 300 training updates, with equal numbers of ALFWorld and WebShop tasks in each update and the same task and rollout budget.

Final evaluation success rates (%) are:

MethodALFWorldWebShop
UCOB95.373.4
SDAR84.456.3
Skill161.728.1
GRPO59.460.2
GiGPO86.767.2

UCOB achieves the highest final success rate on both environments, exceeding the strongest baseline, GiGPO, by approximately 8.6 and 6.3 percentage points. These additional comparisons strengthen the evidence for UCOB's effectiveness when learning jointly across environments. We have updated Figure 8 in the revised manuscript to include all five methods, with experimental settings in Appendix F.7.

Thank you again for this helpful suggestion. We would welcome your thoughts on these results and any remaining questions, and would be grateful if you could reconsider your assessment in light of the additional evidence.

Reviewer_nkmcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

UCOB studies how an RL agent should learn from retrieved skills that can help in some states and mislead in others. It samples skill-conditioned and no-skill trajectories, groups records by task and anchor state, and uses the highest-return record to select the direction of a weighted, top-K token distillation loss. The framework also maintains task/state skill memories and trains a reflection writer using skill utilities. Experiments cover three model sizes on ALFWorld, WebShop, and Search-QA, with component ablations, skill-free evaluation, training-cost measurements, and sequential adaptation.

Critical strengths
  1. The useful contribution is making the direction of cross-context supervision depend on local outcomes. D2Skill already uses paired rollouts, dual-granularity memory, and utility-based retrieval; SkillC uses skill/no-skill gaps for credit assignment and curriculum control. UCOB extends these ideas by combining GiGPO-style state grouping with bidirectional distillation. This is a meaningful methodological extension: the skill-conditioned view can receive corrective supervision from the unassisted view, as well as teach it.

  2. The experimental effort is substantial and includes relevant controls. Table 2 reports 89.1% ALFWorld and 79.7% WebShop success for Qwen3-1.7B, versus 79.7% and 45.3% for GiGPO. Figure 4 tests local versus trajectory-level comparison and fixed versus adaptive distillation directions; Table 3 examines frozen memory and reflection backends. These comparisons support the central design more directly than benchmark gains alone. Figure 6 and Appendix F extend the evidence to skill-free use, wall-clock performance, and retention under one environment sequence.

Critical weaknesses
  1. The interpretation of the local credit signal is stronger than its validation. Section 2 compares branch-mean returns, but Eq. (2) selects the single highest-return record and distills its branch distribution. A winning sample does not establish that this distribution has higher expected value, particularly with sparse rewards and small groups. The environment-specific anchor mapping is also unspecified; matching anchors need not match histories or continuation policies. Appendix H assumes a common continuation and an idealized distribution interpolation, so it does not establish improvement for the implemented estimator and truncated token loss. A focused comparison with branch-mean routing, including group counts and uncertainty, would strengthen the claim that CBSD identifies locally misleading skills.

  2. The evaluation protocol needs clearer reporting. Appendix E says skills are retrieved at evaluation, but Table 2 does not label inference conditions; Figure 6 provides only an average across two environments for Qwen3-1.7B. This leaves the extent of skill-free internalization unclear across models. The large gains for the 1.7B model are encouraging, but the 0.8-point ALFWorld lead at 7B is difficult to assess without variability.

  3. The memory analysis supports a useful evolving bank more convincingly than accurate credit for individual skills. Equation (6) credits skills through episode success relative to no-skill rollouts, although several skills can be retrieved together. Figures 5/10 then assess usefulness using the same utilities that drive retrieval. These measurements cannot separate a skill's contribution from its companions. Table 3 supports memory evolution overall; a matched relevance-only retrieval ablation would more directly test the utility feedback mechanism. The individual-skill interpretation should remain qualified until then.

Questions for authors
  1. What exactly defines an anchor in each environment, and how does maximum-record routing compare with branch-mean routing?

  2. How much does utility-aware retrieval improve over relevance-only retrieval with the same evolving bank and budget?

2 comments
AuthorsOpenReview ↗

Response (Part 1)

Thank you for recognizing the adaptive cross-context supervision and for your detailed suggestions on local credit and retrieval. They helped us sharpen the connection between the diagnostic study and the training rule. We have added routing, coverage, and matched-retrieval experiments.


Comment 1 (W1/Q1): Branch-mean diagnostics differ from maximum-record routing; how reliable is the local comparison?

Thank you for highlighting this distinction. The observation study compares branch means to diagnose local view quality. In training, the highest-return record determines both the teacher view and the response used as the distillation reference.

Sensitivity to direction selection. We separate these two choices in a branch-mean variant: select the teacher view by its mean return within each task-anchor group, then use its highest-return record as the target. Gap weighting and pair filters remain unchanged.

Routing ruleALFWorld success (%)WebShop success (%)
Original maximum-record89.179.7
Branch-mean85.278.1

The evaluation gaps are modest, with comparable final training success and episode length in Figure 4. These results support CBSD under both direction-selection rules. The implementation and formula are provided in Appendix F.4, supplementing Section 4.3, Figure 4.

Independent teacher validation. We freeze a Qwen3-1.7B checkpoint and skill bank and select anchors from 50 distinct ALFWorld tasks independently of rollout outcomes. At each replay-restored state, eight continuations per view select a direction and 16 fresh continuations per view validate it. Starting histories and retrieved anchor skills are fixed, and continuations retain their assigned view.

Teacher selectionStates with a teacherBetter / equal / worse on fresh samplesMean held-out return gap
Maximum-record33/5016 / 10 / 7+0.214
Branch-mean48/5020 / 16 / 12+0.082

The gap compares the selected teacher's mean return with the opposite view on fresh samples; better/equal/worse counts its sign. Rows use each rule's non-tied states, so their subsets differ. The positive average gap supports a local value signal that persists under independent sampling. See Appendix F.4, Table 6.

Anchor matching and availability. Records are grouped by exact equality of task ID and anchor text: the current textual observation in ALFWorld, the formatted current-page observation in WebShop, and the current environment-returned text in Search-QA. Interaction histories remain in the prompts and are not part of the grouping key. Section 3.2 and Appendix C describe this construction. We measure the fraction of valid records in groups containing both views, before routing or gap filtering:

EnvironmentRecordsSteps 1–50Steps 51–100Steps 101–150
ALFWorldAll86.591.693.0
ALFWorldNon-initial86.291.392.5
WebShopAll67.777.787.7
WebShopNon-initial64.274.285.1

Values are coverage percentages from pooled counts within each training stage. Non-initial rows exclude each rollout's first valid record while retaining the original groups. Cross-view comparisons remain widely available beyond common starting points. These statistics measure comparison opportunities under the training representation; the independent diagnostic above tests teacher selection from restored environment states. See Appendix F.3, Table 5.

Connection to theory. Under a sufficient task-anchor representation and common continuation, Appendix H relates expected return gaps to advantage differences. An idealized response-level update toward the higher-value view yields local improvement for sufficiently small updates under the stated occupancy-shift bound. Its token-loss analysis characterizes weighted imitation on the retained support. Theory motivates positive-advantage supervision; the experiments assess empirical direction selection and its training effectiveness, including validation under branch-conditioned continuations.

AuthorsOpenReview ↗

Response (Part 2)

Comment 2 (W2): Inference conditions and skill-free internalization should be explicit across models.

Thank you for suggesting a clearer comparison. We evaluate policies trained with full UCOB with retrieval disabled at inference, alongside standard skill-enabled evaluation in Table 2 and Section 4.2:

BackboneInferenceALFWorld success (%)WebShop success (%)Search-QA average (%)
Qwen2.5-7BWith skills93.085.948.2
Qwen2.5-7BWithout skills87.577.347.1
Qwen2.5-3BWith skills92.278.143.6
Qwen2.5-3BWithout skills91.471.943.2
Qwen3-1.7BWith skills89.179.742.1
Qwen3-1.7BWithout skills85.971.142.1

Search-QA uses the seven-dataset macro-average. Qwen3-1.7B without skills exceeds skill-enabled Skill1 by 20.3/9.4 points on ALFWorld/WebShop, supporting the internalization of useful behavior during training.

Baseline inference conditions. The revised text also specifies the compared methods' standard settings:

InferenceMethods
With skillsSkillRL, D2Skill, SAPO, Skill1
Without skillsSkillC, RLSD, SDAR, Skill-SD, StepOPSD

Table 2 identifies copied baseline results in its caption.

Variability. Thank you for encouraging this check. We repeated UCOB training twice for Qwen2.5-7B and Qwen3-1.7B on ALFWorld and WebShop. Success rates (%) are shown below; Run 0 is the original main-table run.

BackboneEnvironmentRun 0Run 1Run 2Avg.
Qwen2.5-7BALFWorld93.089.895.392.7
Qwen2.5-7BWebShop85.983.680.583.3
Qwen3-1.7BALFWorld89.189.090.689.6
Qwen3-1.7BWebShop79.776.678.178.1

The three-run means differ from the original results by 0.3–2.6 percentage points. UCOB's three-run mean remains above the reported baseline results in all four settings. See Appendix F.8, Table 7. Search-QA remains described as competitive.


Comment 3 (W3/Q2): Does utility-aware ranking help with the same skill bank and budget?

Thank you for proposing this control. We hold the trained Qwen3-1.7B checkpoint, skill bank, evaluation tasks, and retrieval budget fixed and change only the ranking:

Inference-time rankingALFWorld success (%)WebShop success (%)
Default: relevance + utility + exploration89.179.7
Relevance only90.675.0

Default ranking adds 4.7 points on WebShop, while relevance-only ranking leads by 1.5 points on ALFWorld. Relevance provides strong guidance, and the combined utility and exploration terms yield an additional benefit on WebShop. This comparison isolates their combined inference-time contribution. Appendix F.1 reports it separately from the training-time ablation in Figure 12, where policies and memories evolve separately.

Interpretation of utility. Co-retrieved skills can share outcome evidence, so utility measures usage-associated feedback. The useful-skill ratio is the fraction of skills whose utility exceeds its initial value. Individual causal attribution requires skill-specific interventions. Appendix C clarifies this interpretation; the matched comparison assesses the practical value of the combined ranking terms.


We appreciate these suggestions. The additions separate the aggregation rule, independent routing reliability, and inference-time retrieval effects, giving more direct evidence for the adaptive-supervision mechanism.

Reviewer_WRnMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

UCOB uses local returns to choose the teacher direction between skill-conditioned and no-skill rollouts. The same experience updates the skill bank. On Qwen3-1.7B, gains over Skill1 are 23.5 points on ALFWorld and 18.0 on WebShop. Search-QA remains below the best listed baseline at each scale.

Critical strengths
  • The fixed-direction controls perform worse even when both views receive on-policy rollouts. This gives a concrete reason to treat teacher selection as part of credit assignment and supports a methodological contribution to skill-based agent training.
  • The small-model gains over rerun Skill1 give the method practical weight. The CBSD ablation strengthens the link between those gains and the proposed supervision, which is important for assessing the contribution of a system with several interacting components.
Critical weaknesses
  • Eq. 2 selects the highest-return record, so continuation noise may decide the teacher. Anchor matching also leaves state comparability unclear. Appendix H's idealized analysis does not validate the implemented token KL; independent resampling is needed to check whether the chosen teacher has higher expected return.
  • The larger-model gains of 0.8–3.1 points are difficult to judge without repeated runs. Table 2 mixes copied and rerun baselines, and WebShop uses only 128 tasks. This limits confidence in the smaller gains and their stability.
Questions for authors
  • How much of the Table 2 gain remains with skill retrieval disabled, separately for each benchmark?
  • In the utility update of Eq. 6, do co-retrieved skills receive the same credit? How is a harmful skill distinguished from helpful skills in the same bundle?
4 comments
AuthorsOpenReview ↗

Response (Part 1)

Thank you for the thoughtful suggestions. We added independent teacher validation, routing and reduced-bundle ablations, anchor coverage, and skill-free evaluation across all three backbones.


Comment 1 (W1): Could the maximum-return teacher be a lucky sample? How do anchor matching and the theory support the implementation?

Thank you for distinguishing high observed return from a reliable teacher direction. We address this through independent validation, a routing ablation, and clarified anchor matching and theory.

Teacher selection on independent samples. We freeze a Qwen3-1.7B checkpoint and skill bank and select one anchor from each of 50 distinct ALFWorld tasks independently of rollout outcomes. At each replay-restored state, eight continuations per view select a direction; 16 fresh continuations per view validate it, totaling 2,400 continuations. Both views share the starting history, anchor skills are fixed, and continuations retain their assigned view.

SelectionStates with a teacherBetter/equal/worseMean held-out gap
Maximum-record33/5016/10/7+0.214
Branch-mean48/5020/16/12+0.082

The held-out gap is the selected teacher's mean return minus the opposite view's; better/equal/worse counts its sign. Rows use each rule's non-tied selection states, so their subsets differ. Maximum-record routing retains a positive average advantage on fresh samples, supporting a repeatable local value signal. See Appendix F.4, Table 6.

Sensitivity to the routing rule. To test an alternative direction-selection rule, we select the teacher view by its mean return within each task-anchor group, then use its highest-return record as the distillation target.

RoutingALFWorld success (%)WebShop success (%)
Maximum-record (UCOB)89.179.7
Branch-mean85.278.1

The evaluation gaps are modest, with comparable final training success and episode length in Figure 4. These results support CBSD under both direction-selection rules. See Section 4.3 and Appendix F.4.

Anchor coverage beyond the initial state. Training groups records by exact equality of task ID and current anchor text, with environment-specific definitions in Appendix C. Histories remain in the prompts and are not part of the grouping key. We measure the fraction of valid records in groups containing both views, before teacher selection or gap filtering. Non-initial counts exclude each rollout's first valid record while retaining the original groups.

EnvironmentRecordsSteps 1–5051–100101–150
ALFWorldAll86.5%91.6%93.0%
ALFWorldNon-initial86.2%91.3%92.5%
WebShopAll67.7%77.7%87.7%
WebShopNon-initial64.2%74.2%85.1%

Cross-view comparisons remain widely available beyond common starting points. Coverage measures training-time opportunities; restored-state validation tests teacher selection under controlled starting states. See Appendices C and F.3, Table 5.

Connection to theory. Under a sufficient task-anchor representation and common continuation, Appendix H identifies the expected return gap with an advantage difference. An idealized response-level update toward the higher-value view increases the local advantage surrogate and, under the stated occupancy-shift bound, improves policy return for sufficiently small updates. The token-loss analysis characterizes weighted imitation on retained support. Theory explains the value of positive-advantage supervision; independent resampling tests empirical teacher selection under branch-conditioned continuations. Routing and coverage experiments provide complementary evidence of effectiveness and availability.


Comment 2 (W2): Are the smaller larger-model gains stable, given mixed baseline sources and only 128 WebShop evaluation tasks?

Thank you for raising the smaller performance margins. Some baselines lacked public implementations at the time, so we use reported results where needed, following common benchmarking practice and identifying sources in Table 2. We also reproduced Skill1, one of the latest related methods at the time. Our evaluation follows SkillRL (NeurIPS 2026) and D2Skill (EMNLP 2026), including the 128-task WebShop set, as detailed in Appendix D.

To assess training variability, we repeated UCOB training twice for Qwen2.5-7B and Qwen3-1.7B on ALFWorld and WebShop. The table reports success rates (%) across three runs; Run 0 is the original main-table result.

BackboneEnvironmentRun 0Run 1Run 2Avg.
Qwen2.5-7BALFWorld93.089.895.392.7
Qwen2.5-7BWebShop85.983.680.583.3
Qwen3-1.7BALFWorld89.189.090.689.6
Qwen3-1.7BWebShop79.776.678.178.1

The three-run means differ from the original results by 0.3–2.6 percentage points. UCOB's three-run mean remains above the reported baseline results in all four settings. See Appendix F.8, Table 7.

AuthorsOpenReview ↗

Response (Part 2)

Comment 3 (Q1): How much performance remains without retrieval on each benchmark?

Thank you for suggesting this evaluation, which helps clarify how much learned behavior remains available without retrieved skills.

Evaluation without skills. We disable retrieval at inference for policies trained with full UCOB and report all three backbones and benchmarks alongside skill-enabled evaluation.

BackboneInferenceALFWorld success (%)WebShop success (%)Search-QA average (%)
Qwen2.5-7BWith skills93.085.948.2
Qwen2.5-7BWithout skills87.577.347.1
Qwen2.5-3BWith skills92.278.143.6
Qwen2.5-3BWithout skills91.471.943.2
Qwen3-1.7BWith skills89.179.742.1
Qwen3-1.7BWithout skills85.971.142.1

Search-QA uses the unweighted mean over seven datasets. Skill-enabled evaluation generally achieves higher scores, while retrieval-disabled policies retain strong performance. To put this retained capability in context, we compare Qwen3-1.7B without skills against Skill1 and SDAR:

ComparatorComparator inferenceALFWorld: comparator → UCOB without skillsWebShop: comparator → UCOB without skills
Skill1With skills65.6 → 85.9 (+20.3)61.7 → 71.1 (+9.4)
SDARWithout skills53.9 → 85.9 (+32.0)58.6 → 71.1 (+12.5)

Values are success rates (%), with gains in percentage points. UCOB without skills exceeds both baselines, including skill-enabled Skill1, supporting the internalization of useful behavior. See Section 4.2, “Performance without skills,” and Table 2.


Comment 4 (Q2): Do co-retrieved skills share credit, and how is a harmful companion distinguished?

Thank you for highlighting the distinction between feedback on a retrieved bundle and credit for an individual skill. We have clarified the utility definition and added a reduced-bundle training ablation.

Co-retrieved skills can share outcome credit. Each utility accumulates usage-associated feedback through an EMA across interactions. A successful rollout can therefore assign positive credit to an ineffective companion skill. Subsequent updates reflect each skill's usage history; isolating its individual contribution requires skill-specific removal or replacement. Appendix C clarifies this interpretation.

Reduced-bundle training. As a complementary test, we reduce co-retrieval to Top-1 per pool: at most one task skill and one state skill during training on ALFWorld and WebShop. This variant has lower training success, particularly on WebShop. The reduced-bundle ablation supports the value of complementary skills, while utilities summarize each skill's usage-associated outcomes across training. See Appendix F.1, Figure 12.


We appreciate the opportunity to strengthen these aspects of the paper. Your suggestions have helped us clarify teacher-selection reliability, performance without skills, and the interpretation of shared skill credit.

Thanks for the follow-up. The fresh rollouts address my main concern about teacher selection. The repeated training runs also help put the smaller gains in perspective. The retrieval results and the clarification on shared skill credit answer my remaining questions. I support acceptance on the strength of the method and its gains on the smaller model.

Thank you for your positive feedback and for recognizing the value of our work; your comments will help strengthen our paper. Should you have any further questions, please feel free to discuss them with us during the rebuttal period.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Meta Response

Dear Area Chair and Reviewers,

Thank you for your constructive feedback. We have revised the manuscript in response to your comments, with additions and substantive changes highlighted in red in the revised PDF. The main updates are summarized below; detailed results and responses are provided in our individual replies.

  1. Teacher-selection reliability. We added branch-mean routing and independent teacher validation on fresh continuations, providing complementary evidence for the local supervision signal (Section 4.3; Appendix F.4).

  2. Anchor matching. We clarified the environment-specific anchor definitions and reported cross-view coverage throughout training, including coverage excluding initial records (Appendices C and F.3).

  3. Performance without skills. We added retrieval-disabled evaluation across all three backbones and benchmarks and clarified baseline inference conditions (Section 4.2, Table 2).

  4. Skill memory and retrieval. We added Top-1-per-pool and relevance-only training ablations, compared retrieval rankings with the checkpoint and skill bank held fixed, and clarified the interpretation of shared skill credit (Appendices C and F.1).

  5. Robustness to irrelevant guidance. We added low-relevance training and evaluation under low-relevance, no-skill, and default retrieval conditions, showing a consistent advantage over fixed-direction distillation (Appendix F.2).

  6. Repeated runs and mixed training. We report three runs for two backbones on ALFWorld and WebShop, with mean success remaining above the reported baselines. We also added joint ALFWorld–WebShop training, where UCOB outperforms SDAR on both environments (Appendix F.8; Section 4.4 and Appendix F.7).

We welcome further questions and discussion and would be happy to clarify any remaining concerns. If the revised evidence addresses your concerns, we would be grateful if you could consider a more favorable assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.