acceptodds
Under review as a conference paper at ICLR 2027

MDSKILL: Learning to Internalize Skill Memory through Multi-View Distillation for Efficient Dynamic Agent Self-Evolution

Abstract

Skill memories encode past experiences as textual prompts to improve reinforcement learning (RL) for self-evolving agents. Complementing RL, On-Policy Self-Distillation(OPSD) provides dense, fine-grained supervision to internalize knowledge from skill-guided trajectories and accelerate policy learning. However, self-generated memories often contain noisy supervision and functionally entangled, redundant, or conflicting knowledge, making direct distillation difficult. Under limited training budgets, these issues can hinder knowledge internalization and even degrade performance. To address this problem, we propose MDSKILL, a multi-view distillation approach for skill memory internalization. MDSKILL organizes skill memories into three complementary functional views—state grounding, action planning, and execution control—and selects high-quality skills using dynamic metrics to construct view-specific and integrated reasoning branches. OPSD then transfers and consolidates complementary experiential knowledge across these branches into a shared policy, enabling efficient skill internalization and continuous agent self-evolution. We evaluate MDSKILL on three agent benchmarks against RL, skill-augmented, and self-distillation baselines that do not rely on stronger external teacher models. Without additional computational overhead, MDSKILL improves upon SOTA performance by up to 7.1% and 8.1% on ALFWorld and WebShop. Extensive ablations and evolution analyses further validate its effectiveness in knowledge internalization and policy improvement. The code is available at the https://anonymous.4open.science/r/MDSKILL-CF53.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Self-evolving agents, Agentic reinforcement learning, On-policy distillation, Skill memory, Knowledge internalization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qni7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MDSKILL aims to move an agent's self-written skills into its weights during RL, instead of only retrieving them into the prompt. It starts from the view that self-generated skills are noisy and mix different functions, which makes distillation from a skill-conditioned teacher unreliable. Its contributions are threefold. First, retrieval uses Thompson sampling over Beta-distributed "Reliability" and "Advantage" scores. Second, skills are kept in separate banks for grounding, planning and control (the multi-view part). Third, distillation is gated and runs in forward, reverse and cross-view directions. MDSKILL reports the highest success rates on ALFWorld and WebShop at 1.7B, 3B and 7B, and is roughly on par on Search-QA.

Critical strengths

S1. The small-scale gains are large. With Qwen3-1.7B, MDSKILL beats UCOB by +7.1 on ALFWorld and +8.1 on WebShop (Table 1), against a broad pool of 13 baselines. If this holds up, it is a useful recipe for small agents.

S2. Section 3 documents a real failure mode. Skill-conditioned OPSD is sensitive to distillation direction, and reverse beats forward (Figs. 2–3). The skill-level diagnostics in Fig. 6 are a reasonable way to ask whether memories are being internalized.

Critical weaknesses

W1. Nothing isolates the multi-view decomposition as the cause. Swap in relevance-only retrieval and ALFWorld drops from 93.8 to 73.4 (Table 3), below skill-free GiGPO (80.5), even though the multi-view machinery is still there. With λMD=0\lambda_{MD}=0 the result is basically GiGPO (81.3 vs. 80.5). The skill-view advantage stays at or below zero for the whole run (Fig. 6a). The key ablation, "w/o multi-view rollout", is shown only as a single noisy training curve. So I suspect the gain comes from the retrieval scoring plus extra distillation signal, not from the grounding/planning/control split. I'd want final numbers for that ablation, UCOB plus your retrieval metrics, and a non-functional three-way split as a control.

W2. The margins are within noise, and some claims are overstated. The ALFWorld rates fit a 128-episode set (e.g. 120/128). That would make the 3B and 7B margins over UCOB 3–4 episodes, from single runs with no seeds or variance. The gate type and λMD\lambda_{MD} also look tuned per scale. Search-QA is not best at any scale, yet the conclusion claims SOTA on three benchmarks. "No additional computational overhead" ignores the extra reflection, verification, retrieval and teacher re-scoring. Three seeds for MDSKILL, GiGPO and UCOB, plus a cost table, would settle much of this.

Questions for authors
  1. [Clarification] Is Table 1 evaluated with or without skill retrieval at test time? This decides whether it measures internalization or augmentation.
  2. [Clarification] What is the "w/o multi-view rollout" configuration, and what are its final success rates? If it is within 2 points of the full model, W1 gets much stronger.
  3. [Clarification] Does UCOB's retrieval use relevance only? Have you tried your Thompson-sampled retrieval on UCOB without multi-view banks?
  4. [Clarification] How many seeds sit behind Table 1, were GiGPO and UCOB re-run under your protocol, and on which split were the gate and λMD\lambda_{MD} chosen?
  5. [Suggestion] A control with three non-functional skill banks would directly test whether the decomposition matters.
7 comments

Thank you for taking the time to review our paper and provide detailed comments and suggestions. We appreciate the opportunity to clarify our work and improve the manuscript.


Summary. Your main concerns are whether the improvements genuinely arise from functional multi-view decomposition and whether the reported gains reliably demonstrate knowledge internalization. Following your suggestions, we have added controlled ablations separating retrieval, view decomposition, and distillation, a random-partition control, and three-seed results. We also clarify skill-free evaluation and provide distributional evidence explaining the benefits of multi-view distillation. The expanded ablations are presented in Section 5.3 of the revised main text.


(W1, Q2, Q3, Q5)1. Does the multi-view design contribute to the improvements?

(1) Does multi-view decomposition help after controlling retrieval and distillation settings?

The w/o Multi-View Rollout variant retrieves from the entire mixed skill bank without functional-view-specific selection. It retains the same skill count, rollout budget, and gating rules, achieving 89.1%/76.6%, compared with 93.8%/82.8% for full MDSKILL.

To further separate decomposition from additional distillation objectives, we compare w/o Multi-View Rollout with w/o Cross-View Distillation: both retain only forward and reverse distillation, but the latter retains functional views. The functional multi-view variant achieves 92.2%/81.3%, improving over the mixed-bank control by 3.1/4.7 percentage points. This supports a contribution from functional decomposition itself, rather than simply adding another distillation objective.

We also expanded the distillation ablations and conducted the retrieval-replacement experiments you suggested. The key results below use Qwen3-1.7B:

ConfigurationALFWorldWebShop
Multi-view distillation design
w/o Distillation82.071.1
w/o Multi-View Rollout89.176.6
w/o Forward Distillation93.078.9
w/o Cross-View Distillation92.281.3
Separating contributions beyond UCOB
UCOB86.774.2
UCOB + Dynamic Retrieval88.375.8
UCOB + Multi-view Design91.479.7
MDSKILL + UCOB Retrieval92.281.3
MDSKILL93.882.8

Within UCOB's configuration, dynamic retrieval improves success by 1.6/1.6 percentage points, whereas the multi-view design improves it by 4.7/5.5 points. Within MDSKILL, replacing dynamic retrieval with UCOB retrieval decreases success by 1.6/1.5 points, while removing multi-view rollout decreases it by 4.7/6.2 points. These within-framework comparisons support contributions from both components, with larger performance differences associated with the multi-view design in this configuration. Each framework retains its own training hyperparameters, so attribution is based on within-framework comparisons.

The additional distillation ablations support complementary contributions: removing forward or cross-view distillation decreases success by 0.8/3.9 or 1.6/1.4 points, respectively. Section 5.3 of the revised main text provides the detailed configurations and complete component ablations.

Figure 6b provides complementary distributional evidence. Compared with w/o Multi-View Rollout, MDSKILL achieves higher token overlap and a smaller, more stable absolute entropy gap. These observations are consistent with functional disentanglement improving teacher–student distributional alignment and enabling more stable and effective knowledge transfer. The controlled ablations establish performance benefits, while Figure 6b helps explain these benefits through the distillation dynamics.

(2) Why does relevance-only retrieval degrade performance despite retaining the multi-view mechanism?

Multi-view organization does not replace skill-quality control. Relevance-only retrieval removes value filtering, allowing relevant but incorrect, redundant, or inapplicable skills to enter each branch. Its degradation shows that multi-view organization alone cannot ensure reliable supervision from a noisy bank; it does not show that decomposition is ineffective when skill quality is controlled.

Moreover, UCOB does not use relevance-only retrieval: it combines utility estimates with a count-based UCB bonus. The deterministic w/o Thompson Sampling variant is also not equivalent to UCOB. Consequently, neither the relevance-only degradation nor a direct comparison between 88.3% and UCOB's 86.7% isolates the multi-view contribution. The added retrieval-replacement controls address this attribution directly.

(3) Why does setting λMD=0\lambda_{MD}=0 produce results close to GiGPO?

Setting λMD=0\lambda_{MD}=0 removes the distillation objective, largely reducing the method to GiGPO-based RL-only training. RL-only still includes skill-conditioned rollouts; it simply does not apply distillation. The purpose of view decomposition is to improve OPD supervision for more efficient and stable knowledge internalization, rather than to obtain larger RL gains merely by reorganizing rollouts. The result is consistent with this motivation.

(4) Why can the skill-view branch still underperform the skill-free branch?

Self-generated skill banks contain substantial noise, whose negative effects can outweigh the benefits of useful guidance. The skill-view branch therefore often trails the skill-free branch during the training dynamics we observe. Skill-view advantage measures the difference between two contexts of the same policy, not improvement over the initial policy.

The relevant evidence is the gap between the branches and how they evolve together, rather than whether the skill-view branch consistently leads. In Figure 6a, full MDSKILL exhibits a smaller negative gap than the ablated variants and does not remain substantially negative. Figure 7a further shows that the functional branches improve together. These observations indicate that harmful guidance is mitigated while useful experience remains available for internalization through distillation.

A healthy agent self-evolution process is one in which the skill-view branch does not substantially lag behind, the branches improve together, and mutual learning and distillation promote efficient knowledge internalization. The value of skill guidance should therefore be assessed jointly through the branch gap, training dynamics, and final skill-free performance, rather than persistent skill-view superiority alone.

(5) Does functional decomposition outperform an arbitrary three-way partition?

Following your suggestion, we added a random-partition control with matched skill collections, bank sizes, retrieval counts, and training budgets. The mixed-bank variant does not partition skills and retains only forward and reverse distillation. The random-partition variant randomly assigns skills to three banks and retains cross-view distillation, using the same distillation procedure as the functional multi-view method.

Skill organizationALFWorldWebShop
Mixed bank, no decomposition89.176.6
Random three-way partition87.575.0
Functional multi-view banks93.882.8

Random partitioning does not improve over the mixed bank, falling behind by 1.6 points on both tasks. With the same distillation procedure, functional partitioning outperforms random partitioning by 6.3/7.8 points, showing that arbitrary partitioning alone does not explain the gains.

The random three-way partition retains cross-view distillation but does not organize skills by decision function. Knowledge transfer across these branches can therefore remain affected by functional entanglement and conflicting supervision, making distillation unstable and degrading performance. This result supports our central idea: the benefits of multi-view distillation depend on meaningful functional disentanglement, rather than arbitrary partitioning of the skill bank.

(Q1) 2. Does the final evaluation measure internalization?

Thank you for identifying the need to clarify this setting. All final evaluations in Table 1 disable skill retrieval and use the skill-free policy. They therefore measure internalized policy capabilities rather than test-time retrieval augmentation. We have clarified this setting in the evaluation protocol and table caption.

Figures 6a and 7a show both branches to diagnose the role of skill guidance during training. Table 1 instead tests whether acquired capabilities remain available after external skills are removed. Temporary skill-view underperformance during training therefore does not contradict improvements in final skill-free performance.

The skill-view branch ultimately serves to improve knowledge internalization into the skill-free policy; its value should not be judged solely by whether it consistently outperforms that policy.


(W2, Q4)3. Reliability, parameter selection, and computational cost

Three-seed results and baseline reruns. We agree that a single run cannot establish the stability of smaller margins. GiGPO and UCOB were re-run under our experimental protocol, and we have added three-seed results for GiGPO, UCOB, and MDSKILL across all three backbones, reporting means and sample standard deviations:

BackboneMethodALFWorld (mean ± std)WebShop (mean ± std)
Qwen3-1.7BGiGPO80.23 ± 1.2245.83 ± 1.67
Qwen3-1.7BUCOB86.97 ± 1.2274.90 ± 2.31
Qwen3-1.7BMDSKILL93.77 ± 0.7582.30 ± 1.61
Qwen2.5-3B-InstructGiGPO82.80 ± 1.3970.83 ± 1.67
Qwen2.5-3B-InstructUCOB91.93 ± 1.2276.53 ± 1.33
Qwen2.5-3B-InstructMDSKILL95.83 ± 0.9280.47 ± 1.55
Qwen2.5-7B-InstructGiGPO90.17 ± 0.9372.43 ± 1.98
Qwen2.5-7B-InstructUCOB93.77 ± 1.5581.00 ± 1.61
Qwen2.5-7B-InstructMDSKILL96.37 ± 0.4684.90 ± 1.61

MDSKILL achieves higher mean performance than GiGPO and UCOB on both tasks across all three backbones. Its mean gains over UCOB are 6.80/7.40, 3.90/3.93, and 2.60/3.90 percentage points for 1.7B, 3B, and 7B, respectively. These results make the evidence for improvement more convincing than reliance on a single run. We do not equate mean ± standard deviation with a statistical significance test.

Gate configuration and distillation weight. Only Qwen3-1.7B uses the strict gate; Qwen2.5-3B and Qwen2.5-7B use the relaxed gate. The motivation and corresponding comparisons are discussed in Appendix D. The strict gate provides more conservative filtering for less reliable skill supervision, whereas the relaxed gate retains more useful supervision when skill guidance is more reliable.

The distillation coefficient is not selected separately for each backbone: all models use λMD=0.2\lambda_{\mathrm{MD}}=0.2. The coefficient ablation is provided in Appendix C.

Scope of performance claims. The abstract explicitly limits the SOTA improvements to ALFWorld and WebShop, while Main Results describes Search-QA performance as “competitive,” not best. However, the broader wording in the conclusion could be ambiguous, and we have revised it to match the abstract and main results.

Computational cost. The multi-view design allocates branches within the existing rollout budget without increasing environment interactions. Skill generation, including reflection and category verification, accounts for approximately 2% of per-step training time, whereas policy rollout and policy update account for approximately 90%. Our design does not increase the computational budget of these two dominant stages.


Thank you again for these suggestions. The retrieval-replacement, matched-distillation, and random-partition controls provide more direct evidence for functional decomposition. The distributional analysis helps explain the benefits of multi-view distillation, while the baseline reruns and three-seed results make the performance improvements more convincing.

Reviewer_bkv4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper internalizes self-generated textual skill memories into LLM agents via on-policy self-distillation. MDSKILL combines three components. Value-aware Thompson-sampling retrieval selects skills. Grounding/planning/control skill banks drive separate rollout branches. A knowledge-gain gate routes forward, reverse, or cross-view token-level KL between successful and failed trajectories. On ALFWorld and WebShop it achieves the best success rates across three Qwen backbones (+7.1/+8.1pp over UCOB on Qwen3-1.7B); on Search-QA it is on par with baselines.

Critical strengths
  1. It is consistently best on ALFWorld/WebShop success across all three backbones, with large gains on Qwen3-1.7B. Both the skill-view and skill-free branches improve, which suggests real internalization.
  2. The distillation-direction analysis (Sec. 3) is informative. The outcome-gated forward/reverse/cross-view routing is clean and easy to reproduce.
Critical weaknesses
  1. The gains are not attributed to multi-view distillation. In Table 3 (1.7B, ALFWorld), relevance-only retrieval reaches 73.4 and w/o Thompson sampling reaches 88.3, against 86.7 for UCOB. Most of the margin over UCOB therefore comes from retrieval scoring, which is orthogonal to the paper’s core claim. “w/o Multi-View Rollout” is undefined and has no matched mixed-bank control. Sec. 3 finds forward distillation harmful, yet the final method relies on it. Under the strict gate, forward distillation fires only when na0=0n^0_a=0, so the gate alone may explain this reversal.
  2. Most gains are within noise. All results are single runs on about 128 episodes, so the SE is roughly 2–2.6pp. The 7B ALFWorld gain (+2.3pp) and the 7B WebShop margin over SDAR (+1.6pp) fall within this range. Gate type and λMD\lambda_{\text{MD}} are chosen per backbone with no stated protocol, and the supporting comparisons are on the evaluation benchmarks.
Questions for authors
  1. What happens if UCOB is given the same retrieval scoring? This would separate the retrieval and multi-view contributions and is the result most likely to change my score. (W1)
  2. Can you add a mixed-bank control with matched skill count, and a “no multi-view + gated forward” variant? (W1)
  3. Can you report mean ± std over at least 3 seeds, and describe how gate type and λMD\lambda_{\text{MD}} were selected? (W2)
8 comments

Thank you for taking the time to review our paper and provide detailed comments and suggestions. We appreciate the opportunity to clarify our work and improve the manuscript.

Summary. Your main concerns are whether the performance gains genuinely arise from multi-view distillation and whether these gains are reliable. Following your suggestions, we have added more informative ablations to separate the contributions of retrieval, view decomposition, and distillation. We also clarify what the relevance-only and w/o Thompson Sampling results establish, and provide three-seed results across all three backbones. The expanded component ablations and contribution-separation experiments are now presented in Section 5.3 (Ablation Studies) of the main text.


(W1-1, Q1) 1. Attributing gains to retrieval versus the multi-view design

(a) Direct controls separating retrieval, view decomposition, and distillation.

We agree that the original retrieval-component ablations in Table 3 alone cannot separate these contributions. We therefore followed your suggestion by giving UCOB the same dynamic retrieval, adding complementary replacement controls, and expanding the multi-view distillation ablations. Section 5.3 of the revised main text presents these component ablations and contribution-separation experiments; the key results below use Qwen3-1.7B.

ConfigurationALFWorldWebShop
Multi-view distillation design
w/o Distillation82.071.1
w/o Multi-View Rollout89.176.6
w/o Forward Distillation93.078.9
w/o Cross-View Distillation92.281.3
Separating contributions beyond UCOB
UCOB86.774.2
UCOB + Dynamic Retrieval88.375.8
UCOB + Multi-view Design91.479.7
MDSKILL + UCOB Retrieval92.281.3
MDSKILL93.882.8

Within UCOB's configuration, dynamic retrieval improves success by 1.6/1.6 percentage points, whereas the multi-view design improves it by 4.7/5.5 points. Within MDSKILL, replacing dynamic retrieval with UCOB retrieval decreases success by 1.6/1.5 points, while removing multi-view rollout decreases it by 4.7/6.2 points. These within-framework comparisons support contributions from both components, with larger performance differences associated with the multi-view design in this configuration. Each framework retains its own training hyperparameters, so attribution is based on within-framework comparisons.

The w/o Multi-View Rollout variant retrieves from the entire mixed skill bank without functional partitioning and retains forward and reverse distillation. To separate view decomposition from the addition of cross-view supervision, we compare it with w/o Cross-View Distillation: both retain only forward and reverse distillation, but the latter retains functional views. Functional decomposition improves success from 89.1%/76.6% to 92.2%/81.3%, a gain of 3.1/4.7 points. This provides direct evidence that the gains are not explained solely by adding another distillation objective.

The remaining ablations also support the role of distillation: removing it reduces success to 82.0%/71.1%. Removing forward or cross-view distillation decreases success by 0.8/3.9 or 1.6/1.4 points, respectively. Together, these results support contributions from both functional organization and the complementary distillation objectives.

(b) What the relevance-only comparison measures.

The relevance-only ablation removes value filtering entirely, directly exposing training to noisy, redundant, and inapplicable skills in the bank; it does not replace our retrieval with UCOB retrieval. The w/o Thompson Sampling variant still retains Reliability, Advantage, and temporal decay. The increase from 73.4% to 88.3% therefore supports the importance of value filtering when learning from a noisy skill bank; it does not measure the incremental benefit of our retrieval mechanism over UCOB.

(c) Why removing Thompson sampling does not reproduce UCOB.

Removing Thompson sampling is a component ablation, not an equivalent implementation of UCOB. UCOB combines utility estimates with a count-based bonus for less-used skills, whereas MDSKILL samples from Beta evidence distributions. Both seek to prevent excessive retrieval concentration. We have added an analysis in Appendix F explaining the metrics, Beta evidence modeling, Thompson sampling, temporal decay, and their differences from alternative retrieval strategies. Comparing 88.3% directly with UCOB's 86.7% cannot isolate the multi-view contribution; the controlled ablations in Section 5.3 of the revised main text address this attribution more directly.

(d) Distributional evidence explaining the benefits of multi-view distillation.

Figure 6b in the main text provides complementary distributional evidence. Compared with w/o Multi-View Rollout, MDSKILL achieves higher token overlap and a smaller, more stable absolute entropy gap. Without view decomposition, overlap fails to improve effectively and the entropy gap fluctuates persistently. These observations are consistent with functional disentanglement improving teacher–student distributional alignment and enabling more stable and effective knowledge transfer. The ablations establish performance benefits, while Figure 6b provides a complementary explanation through the distillation dynamics.


(W1-2, Q2) 2. Mixed-bank control and forward-distillation gating

Mixed-bank configuration. Without view decomposition, the three functional skill banks are pooled, and each skill-conditioned branch retrieves from the entire bank without functional-view-specific selection. Using Qwen3-1.7B, we compare the following five configurations under this mixed-bank setting:

ConfigurationALFWorldWebShop
No Distillation80.545.3
Forward Only — Strict Gate84.471.1
Forward Only — Relaxed Gate85.269.5
Reverse Only88.376.5
Bidirectional86.774.7

Forward distillation still transfers useful knowledge. Section 3 does not claim that forward distillation is universally harmful. It explicitly states:

“Although forward distillation underperforms reverse distillation, it substantially outperforms no distillation.”

The added results agree: forward-only distillation under either gate outperforms no distillation, indicating useful knowledge in skill-guided supervision. However, both variants remain below reverse-only, and bidirectional distillation also fails to outperform reverse-only. The negative effect here concerns the lack of additional gains when forward supervision is combined with reverse correction, rather than performance below the no-distillation baseline.

Gating does not resolve the underlying problem of mixed-skill supervision. Strict and relaxed gates affect forward-distillation performance, but neither forward-only variant reaches reverse-only performance in this experiment. Together with the multi-view controls, these results support distinct roles for the two mechanisms: gating provides coarse filtering based on supervision reliability, whereas functional decomposition improves how experiential knowledge is organized for transfer. Changing the gate alone does not separate noisy, redundant, or conflicting knowledge within mixed skills or guarantee complementary forward and reverse supervision.

The advantage of strict gating is also not universal. Appendix D and the additional cross-model comparisons show that relaxed gating outperforms strict gating on the evaluated 3B and 7B models. This is consistent with stronger models providing more reliable skill guidance, for which strict filtering can also exclude useful forward supervision. Gating should therefore be matched to supervision reliability rather than made uniformly conservative.

More fundamentally, trajectory-level success provides only coarse attribution of distillation quality. A successful trajectory does not imply that every decision supplies useful supervision, while rejected trajectories may still contain valuable knowledge. More precise state- or token-level attribution could reduce the rejection of beneficial signals and further improve OPD.

(W2, Q3) 3. Multiple seeds and parameter selection

We agree that single runs are insufficient to establish the stability of smaller margins. We have added three-seed results for GiGPO, UCOB, and MDSKILL across all three backbones, reporting means and sample standard deviations:

BackboneMethodALFWorld (mean ± std)WebShop (mean ± std)
Qwen3-1.7BGiGPO80.23 ± 1.2245.83 ± 1.67
Qwen3-1.7BUCOB86.97 ± 1.2274.90 ± 2.31
Qwen3-1.7BMDSKILL93.77 ± 0.7582.30 ± 1.61
Qwen2.5-3B-InstructGiGPO82.80 ± 1.3970.83 ± 1.67
Qwen2.5-3B-InstructUCOB91.93 ± 1.2276.53 ± 1.33
Qwen2.5-3B-InstructMDSKILL95.83 ± 0.9280.47 ± 1.55
Qwen2.5-7B-InstructGiGPO90.17 ± 0.9372.43 ± 1.98
Qwen2.5-7B-InstructUCOB93.77 ± 1.5581.00 ± 1.61
Qwen2.5-7B-InstructMDSKILL96.37 ± 0.4684.90 ± 1.61

MDSKILL achieves higher means than GiGPO and UCOB on both tasks across all three backbones. Its mean gains over UCOB are 6.80/7.40, 3.90/3.93, and 2.60/3.90 percentage points for 1.7B, 3B, and 7B, respectively. These results make the evidence for improvement more convincing than reliance on the originally reported single run. We do not equate mean ± standard deviation with a statistical significance test.

The gate motivation and training dynamics are discussed in Appendix D and Figure 8: the strict gate provides more conservative filtering for less reliable skill supervision from 1.7B, while the relaxed gate retains more informative pairs for 3B/7B. The comparisons support adapting the gate to supervision reliability rather than treating parameter count as the sole criterion.

Finally, λMD\lambda_{MD} is not selected separately for each backbone: all models use 0.2. Appendix C reports the coefficient ablation, where this setting performs best among the compared values on both tasks.


Thank you again for these suggestions. The added experiments more directly distinguish retrieval, functional decomposition, distillation, and gating. The distributional analysis helps explain the multi-view benefits, while the three-seed results make the evidence for the overall performance improvements more convincing.

Reviewer_VcRQOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes categorizing the skills in the memory bank into distinct subgroups (related to grounding, control and planning) as opposed to merging them under one unified label which potentially hurts teacher -> student distillation. Starting from the evidence-backed analysis showing that forward distillation from teacher to student performs worse then student to teacher (but better than no distillation), they argue a more systemic grouping of skills tends to help.

They then introduce multiple algorithms including dynamic skill metrics, skill grouping, knowledge gates, teacher-student grouping etc. Experimental results on ALFWorld, WebShop and search-qa are used to validate the hypothesis.

Critical strengths
  1. The paper addresses and analyses a critical issue with current skill-based on-policy distillation methods where noisy and untargeted skills often lead to subpar performance. The hypothesis that grouping skills into specific functionality or view-based categories will help the distillation is well-established and a nice insight.

  2. The method also has strong performance on ALFWorld dataset, surpassing several other methods solely relying on skill-based or expert-teacher based OPD.

Critical weaknesses
  1. The paper presents several technical details with little or no insight into why each was chosen, how it was designed, what other alternatives were explored or why each of this fundamental to the presented hypothesis. For instance, sec 4.2 discussed dynamic skill metrics using Beta distribution, but nothing is adequately justified or technically validated. It is not clear what is reliability or advantage, why beta was particularly chosen, what is meant by "half-life" of of an evidence, what other distributions yield similar results etc.

  2. Likewise, Thompson-Sampled retrieval is presented with little or no justification. Most specifically, it is not clear where is the paper is St(m)S_t(m) used an what benefit does it confer?

  3. In Sec 4.3.1, it is not clear how reliable is the classification of skills into these groupings, how robust is the distillation to errenous categorization, why these categories were particularly chosen, what advantage does using only one of them (as opposed to all of them) have etc? Also, authors should analyze the increased training overhead presented by online categorization of skills into views.

Overall, the paper presents many many ideas and formulas and it is ultimately not clear which of these is central to presented idea and which are embellishments presented to support the central idea.

Questions for authors
  1. Is the issue with noisy distillation specific to skill-based distillation? would other forms of OPSD like using labels or other privileged information have similar issues?

  2. How exactly are the views supporting the learning? From the paper, grounding, planning and control views are together used as V\mathcal{V} - so it is not clear how or what each of these in isolation adds to the improved training.

  3. In the dynamic skill metrics, what do reliability or advantage mean? and how are zrelz^{rel} and zadvz^{adv} designed?

  4. The final accuracy model to be very good on ALFWorld, but preforms worse than few other models on rest of the benchmarks. Is there anything specific to ALFWorld which is benefitting the learning? Is the skill-view decomposition easier in ALFWorld leading to improved accuracy?

2 comments

Thank you for your review and suggestions, particularly your questions about the retrieval mechanism, the problem setting, and the paper’s central contribution.

Summary. Your main concerns center on the explanation of skill retrieval, the problem setting, and the identification of the paper’s central contribution. Because retrieval is a supporting mechanism rather than our core contribution, we gave it insufficient explanation in the original manuscript. We have addressed this omission by adding explanations of the dynamic metrics and comparisons with alternative retrieval strategies in Appendix F, alongside controlled ablations separating retrieval from the multi-view design in Section 5.3 (Ablation Studies) of the main text. All components form part of the agent’s self-evolution loop and serve the shared goal of internalizing experiential knowledge, with multi-view distillation as the central mechanism. Below, we clarify the problem setting and central contribution, address your specific questions, and explain where we disagree with your assessment.


(W1, W2, Q3) 1. Dynamic metrics and retrieval

Due to space limitations, Section 4.2 provided a compressed description. We have added the design rationale and mathematical explanation in Appendix F, and corresponding ablations in Section 5.3. The design has three motivations:

  • Reliability and Advantage measure absolute skill-guided success and improvement over the current skill-free policy, respectively.
  • Beta-based Thompson sampling gives newly generated skills retrieval opportunities, avoiding repeated selection of a small, reliable but increasingly redundant subset. St(m)S_t(m) ranks candidates after relevance filtering.
  • Temporal decay discounts historical evidence so retrieval can reflect changes in a skill’s relative advantage as knowledge is internalized.

Thompson sampling is not the only possible choice; temperature sampling and count-based bonuses are alternatives. Appendix F discusses these differences, and Section 5.3 compares against UCOB-style retrieval. Removing Thompson sampling reduces ALFWorld/WebShop success from 93.8%/82.8% to 88.3%/72.7%, supporting the importance of flexible allocation of skill retrieval opportunities.

Action: Added explanations of the dynamic metrics, Beta evidence modeling, Thompson sampling, temporal decay, and alternative retrieval strategies in Appendix F, with component ablations in Section 5.3 of the main text.


(Q1) 2. The source of noisy distillation

The underlying issue is the unreliability of externalized experience. Incorrect generalizations, missing applicability conditions, and conflicting instructions can enter the policy through distillation. Any OPSD method using unreliable privileged information may encounter this problem.

Skill memory is the specific setting studied here: self-generated memories contain substantial noise, but also valuable knowledge. As discussed in the introduction and Section 3, discarding them loses useful experience, whereas directly distilling them risks transferring misleading guidance. We study how to organize, select, and internalize this experience, rather than claiming that noisy distillation is exclusive to skill memories.


(W3, Q2) 3. View construction, complementarity, and overhead

Skills are generated for a specified view using category-specific prompts, rather than generated freely and classified afterward. We then hide the assigned category and ask the model to predict it; only category-consistent skills enter the corresponding bank. Templates and validation procedures are provided in the appendix. This checks category consistency, rather than guaranteeing error-free categorization.

Grounding, planning, and control separate complementary functions to disentangle experiential knowledge and facilitate its internalization through distillation. Each view can be trained independently, but its standalone performance does not fully characterize its value within multi-view distillation. The relevant question is how it contributes to overall knowledge internalization, rather than which branch achieves the highest success rate. Figure 7 presents branch-wise training success rates, with no branch consistently dominating or underperforming across the observed training dynamics.

Compared with UCOB, MDSKILL does not increase the training budget through multi-view decomposition. Skill generation and category verification together account for approximately 2% of each training step’s runtime, whereas policy rollout and policy update account for approximately 90%. Our design retains the existing rollout budget and policy-update procedure without increasing computation in these two dominant stages. The additional runtime associated with skill maintenance is therefore negligible relative to overall training.

(W4) 4. Identification of the central contribution

We disagree with the characterization that the central idea is unclear or indistinguishable from “embellishments.” Multi-view distillation is the central contribution consistently developed in the title, motivation, and method. Dynamic retrieval is a supporting mechanism for managing noisy skill memories. The main text explains why the views are constructed and how rollout and distillation promote knowledge internalization.

Section 5.3 of the revised main text separates these contributions through component ablations and replacement controls. The key results on Qwen3-1.7B are:

ConfigurationALFWorldWebShop
UCOB86.774.2
UCOB + Dynamic Retrieval88.375.8
UCOB + Multi-view Design91.479.7
MDSKILL + UCOB Retrieval92.281.3
MDSKILL w/o Multi-View Rollout89.176.6
MDSKILL93.882.8

Within UCOB's configuration, dynamic retrieval improves success by 1.6/1.6 percentage points, whereas the multi-view design improves it by 4.7/5.5 points. Within MDSKILL, replacing dynamic retrieval with UCOB retrieval decreases success by 1.6/2.3 points, while removing multi-view rollout decreases it by 4.7/6.2 points. Both components contribute, with larger performance differences associated with the multi-view design in this configuration. Each framework retains its own training hyperparameters, so attribution is based on within-framework comparisons.

We accept that the supporting retrieval mechanism needed further explanation, which we have added in Appendix F. This presentation issue does not, however, establish that the paper lacks a clearly defined central contribution. We ask that the assessment directly consider the multi-view mechanism and the experimental evidence isolating its contribution.

Action: Added retrieval-replacement and multi-view controls in Section 5.3 of the main text to distinguish the contributions of the supporting retrieval mechanism and the central multi-view design.

(Q4) 5. Performance differences across benchmarks

We disagree that the performance gains should be attributed to characteristics specific to ALFWorld. MDSKILL substantially improves upon previous state-of-the-art methods on both ALFWorld and WebShop, achieving the best overall success rates on both benchmarks across all three backbones. The consistent ablation conclusions on both benchmarks in Section 5.3 further support the contribution of the multi-view design.

The more limited gains on Search-QA are already explicitly discussed in Main Results:

“Compared with interactive decision-making tasks, Search-QA depends more on retrieval quality, evidence comprehension, and answer synthesis, and less on recurring environment states where reusable skills offer consistent guidance.”

ALFWorld and WebShop contain recurring grounding, planning, and execution-feedback patterns, providing opportunities for skill reuse and complementary knowledge internalization. Search-QA depends more directly on retrieving and integrating question-specific factual evidence; reusable skills cannot replace missing evidence, which limits the potential gains. This task distinction is already discussed in the manuscript and does not justify restricting the method’s effectiveness to ALFWorld.


We have not claimed or established that view decomposition is easier in ALFWorld. The evidence supports consistent effectiveness across two distinct interactive benchmarks.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their careful reading and constructive comments. We appreciate their recognition of the motivation, distillation-direction analysis, and empirical results of our work.


We have provided point-by-point responses and strengthened the revised manuscript in the following areas:

  1. Separating the contributions of retrieval and multi-view distillation. We expanded Section 5.3 with retrieval-replacement controls, matched-distillation comparisons, and a random three-way partition experiment. These experiments distinguish the contributions of retrieval, functional decomposition, and distillation. We also explain how the distributional evidence in Figure 6b supports more stable and effective knowledge transfer.

  2. Explaining dynamic retrieval and alternative designs. We added a detailed analysis in Appendix F, covering Reliability, Advantage, Beta evidence modeling, Thompson sampling, and temporal decay, together with comparisons to alternative retrieval strategies. This clarifies the role of retrieval as a supporting mechanism within the agent’s self-evolution loop.

  3. Strengthening experimental reliability. We re-ran GiGPO and UCOB under our experimental protocol and added three-seed results for GiGPO, UCOB, and MDSKILL across all three backbones on ALFWorld and WebShop, reporting means and sample standard deviations.

  4. Clarifying evaluation and the scope of performance claims. We explicitly state that MDSKILL’s final evaluation uses the skill-free policy without test-time skill retrieval, measuring internalized capabilities. We also revised the conclusion to distinguish state-of-the-art overall success rates on ALFWorld and WebShop from competitive performance on Search-QA.

  5. Clarifying gating, hyperparameters, and computational cost. We supplemented the mixed-bank distillation-direction and gating comparisons, clarified the gate configurations discussed in Appendix D, and emphasized that all backbones use the same distillation coefficient, with its ablation in Appendix C. We also clarified the runtime of skill maintenance and the unchanged rollout and policy-update budgets.

These revisions make the central contribution—functional multi-view distillation for experiential knowledge internalization—more explicit and provide stronger evidence for its effectiveness. We thank the reviewers again for helping us improve the clarity and experimental support of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.