acceptodds
Desk rejected at ICLR 2027

ReAD: Reinforcement-Guided Capability Distillation for Large Language Models

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Capability distillation specializes a smaller language model toward a specified capability using supervision from a larger teacher. Under a fixed distillation-token budget, an important question is how to allocate supervision across capabilities to improve the target most effectively. Our empirical study shows that single-capability distillation produces budget-dependent cross-capability changes and diminishing target gains, motivating allocation strategies that adapt to the student's evolving state. We propose ReAD, a Reinforcement-guided cApability Distillation framework that combines a task-conditioned allocation prior, on-the-fly teacher supervision, and an uncertainty-aware contextual bandit. ReAD updates its allocation using measured capability changes, rewarding task-aligned gains and penalizing regressions on prioritized capabilities. Across eight target capabilities, two teacher–student pairs, and two token budgets, ReAD achieves higher mean on-target scores than the evaluated target-only baselines. Matched-generator comparisons show additional gains over static and greedy allocation schedules, while transfer diagnostics show smaller off-target declines. Our code is available at: https://anonymous.4open.science/r/ReAD-62B7.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Distillation, LLM capabilities, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_keepOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies capability distillation under a fixed token budget and proposes ReAD, a framework that dynamically allocates teacher supervision across interdependent capabilities. ReAD combines task-conditioned capability priorities with an uncertainty-aware contextual bandit, rewarding task-aligned gains while penalizing regressions in prioritized capabilities. Experiments across eight capabilities, two teacher–student pairs, and two token budgets show improved target performance and reduced negative transfer.

Critical strengths
  • The paper is clearly written, with well-organized and informative figures and tables.
  • The evaluation is extensive and carefully controlled, covering multiple capabilities, model pairs, budgets, seeds, matched allocation baselines, and component ablations.
  • The paper introduces a well-motivated formulation of teacher-guided target distillation that allocates part of a fixed budget to auxiliary capabilities when cross-capability transfer is beneficial.
Critical weaknesses
  • Despite the extensive evaluation, the paper does not compare with closely related online skill- and data-mixture methods such as Skill-It, leaving the benefit of ReAD’s specific allocation design relative to prior approaches unclear.
  • ReAD depends on a predefined capability taxonomy, a learned priority vector, and a fixed probe suite, but its robustness to misspecified signals and its scalability to finer or previously unseen capabilities remain unclear.
Questions for authors
  1. Could the authors adapt Skill-It or another online data-mixture method to the same teacher–student setting and compare it with ReAD under matched token and generation budgets?
  2. How sensitive is ReAD to errors in the inferred capability priorities or noisy probe measurements, and how does performance change as the number of candidate capabilities increases?
  3. How consistent are the learned cross-capability relationships and allocation policies across model families and token budgets?
Reviewer_UGDLOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes to adaptively allocate a fixed distillation budget across capabilities such as reasoning, coding using teacher-generated training data. They further use a contextual bandit to adjust the training mixture based on measured student improvements and regressions, to trade-off among the improvement on the target capability and the degradation of other prioritized skills. The idea is interesting but not entirely novel.

Critical strengths
  • Its generally a worthwhile allocation problem. The framework makes cross-capability transfer explicit under a fixed distillation budget.
  • Clear theoretical scope. The conditional UCB argument is sound, and the authors acknowledge its limits.
Critical weaknesses
  • The mathematics is largely routine. Taylor expansions, KKT conditions, and a standard UCB bound provide explanation rather than substantial theoretical novelty.
  • The central justification is missing. The analysis does not connect immediate proxy-reward gains to final target performance; uncertainty calibration is assumed.
  • The paper evaluates several KD variants, but without comparisons to closely related adaptive mixers such as Skill-It or Aioli, the practical advantage of the proposed allocator remains insufficiently established.
Questions for authors
  • Under what conditions do improvements in cumulative proxy reward imply improvements in final target utility?
  • How does ReAD compare with Skill-It or Aioli under the same generator, teacher, and token budget?
Reviewer_YLNkOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to allocate a fixed distillation-token budget across capabilities to improve a specified target capability. Motivated by budget-dependent cross-capability transfer and diminishing returns from target-only training, it proposes ReAD, which combines learned task-conditioned capability priorities, on-the-fly teacher supervision, and a contextual bandit that adaptively selects training mixtures using capability feedback and uncertainty estimates. Evaluation covers eight target capabilities, two teacher–student pairs, and two token budgets. ReAD reports higher mean target scores than target-only baselines and improvements over static and greedy allocation schedules under a matched generator.

Critical strengths
  1. The paper asks whether allocating some supervision to auxiliary capabilities can improve a target capability more efficiently than target-only training. Its observations of budget-dependent transfer and diminishing returns motivate this question and highlight the importance of training allocation beyond distillation objectives. This is consequential because it opens a useful research direction even if the proposed allocator is an initial solution.

  2. ReAD achieves higher mean target scores across eight capabilities, two teacher–student pairs, and two budgets. The matched-generator comparisons against static and greedy schedules provide additional evidence that allocation contributes to the gains. This breadth supports the potential usefulness of the approach beyond a single task or model pair.

Critical weaknesses

Weakness1: ReAD improves performance under matched student-training-token budgets, but introduces additional costs for offline interventions, priority and reward-model training, and online monitoring and allocation updates. The reported gains establish an advantage under a restricted budget definition, but do not yet establish a net advantage over simpler allocation strategies under comparable total resources.

Weakness2: Generated teacher responses are directly used for distillation, but the manuscript does not specify safeguards for detecting or filtering harmful supervision. Since the allocation reward does not explicitly capture safety regressions, capability improvements could coexist with unmonitored safety degradation.

Weakness3: Removing the identifier also changes the allocation scheme, while “W/o Interaction” only removes the additional regression penalty rather than cross-capability interaction modeling. Moreover, reporting only average performance across capabilities obscures whether these components improve the target or preserve non-target capabilities. These ablations therefore provide limited evidence for the specific roles attributed to the modules.

Questions for authors

N/A. Please refer to the Critical Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.