acceptodds
Withdrawn from ICLR 2027

Learning from Hints, Forgetting Less: Reward-Directed Transfer for Continual Post-Training

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Training-time hints can help models solve difficult tasks, creating successful experience from which to learn. In continual post-training, the challenge is to turn these assisted successes into independent capabilities while retaining earlier ones. However, solely imitating hint-assisted success can pull the model toward particular answers or hint-dependent predictions, introducing behavioral changes that may interfere with previously acquired capabilities. We introduce Hint-Gap Weighted Transfer (HiGHT), which combines online reinforcement learning (RL) with selective transfer from hinted rollouts. Its central signal is the *hint gap*: the signed change in a sampled token's probability induced by the hint at the same response prefix. Positive trajectory advantages select reward-supported experience, while the hint gap modulates the direction and strength of each token's transfer contribution. Raw-prompt reinforcement learning continues to optimize task reward without hints. Across five real-world vision-language domains, HiGHT improves mean final accuracy from 49.94% to 55.12% and backward transfer from to percentage points over hinted-policy distillation combined with online RL (Distill + RL). Controlled local updates show better early-stage retention at matched update norms. These results support learning from hint-assisted experience with less forgetting.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5Pw4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies continual post-training with training-time hints. The goal is to transfer capabilities acquired under privileged/hinted prompts to the raw-prompt policy while reducing forgetting.

The paper proposes Hint-Gap Weighted Transfer (HiGHT). For a hinted rollout, a positive trajectory advantage determines whether the trajectory contributes to transfer. For each sampled token, the method compares its probability under the hinted and raw prompts at the same response prefix, and uses the signed gap qt−ptq_t-p_t to weight the raw-prompt token log-likelihood. HiGHT is combined with online RL under raw and hinted prompts.

Experiments on HOCL, MLLM-CL, and POPE show improved new-task learning and retention over the main Distill + RL baseline.

Critical strengths
  1. Important problem.
    The paper addresses a meaningful question: how to convert success obtained with privileged training-time information into capabilities that remain useful without that information.

  2. Simple and intuitive transfer mechanism.
    Comparing hinted and raw probabilities at the same prefix is a clean way to measure how the hint changes the sampled-token prediction. The signed weighting also allows different tokens within the same successful trajectory to be reinforced or suppressed.

Critical weaknesses
  1. The key missing baseline is reward-gated privileged self-distillation.

    This is my main concern. HiGHT already computes both hinted and raw policy probabilities, but only uses the sampled-token scalar difference qt−ptq_t-p_t.

    A natural baseline would keep the same positive-advantage trajectory gate while performing standard privileged-policy distillation, or an analogous PG-style on-policy self-distillation objective.

    The current Distill + RL baseline does not fully isolate whether the gain comes from the proposed hint-gap signal or simply from making distillation reward-selective and sparse. This comparison is important for establishing the novelty and necessity of HiGHT.

  2. The choice of qt−ptq_t-p_t is insufficiently justified.

    It is unclear why raw probability difference is preferable to alternatives such as log-probability difference, probability ratio, or KL-derived signals. Since qt−ptq_t-p_t is sensitive to absolute probability scale, an ablation over alternative discrepancy measures would strengthen the method.

  3. Core ablations are limited.

    The objective combines several design choices: positive-advantage gating, signed token gaps, exponential weighting, asymmetric clipping, and bidirectional transfer. It is currently unclear which components are responsible for the improvement.

  4. The theoretical analysis is modest.

    The gradient bound shows that the transfer gradient vanishes when the hint gap vanishes, which is a useful sanity property. However, it does not explain why HiGHT should forget less than reward-gated distillation or why the proposed weighting should provide a better retention-plasticity tradeoff.

Questions for authors
  1. Can the authors compare HiGHT against reward-gated privileged distillation, under identical rollouts and RL terms?

  2. Can the authors compare against a PG-style privileged self-distillation objective with the same positive trajectory gate?

  3. Why is q−pq-p preferred over log-probability or KL-derived discrepancy measures?

  4. How important are the exponential weighting and signed negative-token updates?

  5. What is the contribution of bidirectional transfer in MLLM-CL?

Reviewer_GfrsOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Hint-Gap Weighted Transfer (HiGHT), which uses successful hint-assisted rollouts to improve a model’s raw-prompt behavior during continual post-training. The method combines online RL on raw and hinted prompts with a transfer objective: positive reward advantages select hinted responses, while the signed difference between hinted and raw token probabilities determines whether each sampled token is encouraged or suppressed in the raw-prompt policy. Experiments cover a controlled hidden-operator task, five sequential vision-language domains, and an integration with POPE. The direction is relevant and the signed token-level transfer rule is potentially useful, but its novelty relative to prior hint-guided RL and the evidence for its specific mechanism need to be established more clearly.

Critical strengths
  1. The paper addresses an important tension in post-training: hints can produce reward-bearing experience, but the deployed model must succeed without them and retain earlier capabilities.
  2. The proposed hint-gap signal offers a concrete way to distinguish which tokens in a successful hinted response became more or less likely because of the hint. Combining it with raw-prompt RL is a coherent design choice.
  3. The evaluation includes continual-learning metrics, five multimodal domains, an integration with an existing guided-exploration setup, and paired local-update diagnostics that attempt to isolate objective effects.
Critical weaknesses
  1. The relationship to Scaf-GRPO is not discussed, despite close overlap in the central motivation and setting. Scaf-GRPO[1] uses training-time scaffolding to recover learning signal on difficult tasks and evaluates performance without hints; HiGHT’s possible distinction is its signed token-level transfer objective and continual-retention focus, but the manuscript neither cites Scaf-GRPO nor establishes the incremental contribution with a direct comparison.
  2. The paper does not isolate whether the signed hint gap is the key ingredient or whether gains come from adding more reward-selected hinted rollouts. In particular, negative gaps actively suppress tokens from positive-reward responses, but there is limited evidence that this signal reliably identifies unhelpful hint-induced behavior; targeted ablations against positive-only, unsigned, and no-gap transfer would strengthen the mechanism claim.
  3. The main MLLM-CL results use one training seed, one fixed domain order, and 1% of each domain’s training split. The paired local-update intervals are conditional on selected checkpoints and do not capture variation across complete continual-training runs or task orders, making the reported improvements in final accuracy and backward transfer difficult to calibrate.
  4. The breadth of evidence is limited relative to the general claims. The controlled HOCL benchmark is near saturation for several baselines, the five-domain study uses a single Qwen3-VL model, and the POPE result covers one hard-math setting; therefore it remains unclear how robust HiGHT is to model scale, hint type, and domain sequence.

[1] Zhang X, Wu S, Zhu Y, et al. Scaf-grpo: Scaffolded group relative policy optimization for enhancing llm reasoning[C]//International Conference on Learning Representations. 2026, 2026: 131946-131974.

Questions for authors
  1. Explain the distinction from Scaf-GRPO and other hint-guided RL methods, and identify which part of HiGHT is technically new beyond using hints to restore reward-bearing rollouts.
  2. Clarify whether the authors can provide ablations that isolate signed versus positive-only or unsigned hint-gap transfer, including sensitivity to the transfer scale and clipping.
  3. Report any additional seeds or task orders for the full MLLM-CL sequence; if unavailable, explain the limits of drawing conclusions from the single reported run and the checkpoint-conditioned local-update analysis.
  4. I wonder how and why hint-based RL works.
Reviewer_dcnyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Hint-Gap Weighted Transfer (HiGHT), a continual post-training framework designed to integrate privileged hint-assisted experience into an unhinted policy while preventing catastrophic forgetting. The proposed approach is evaluated across three settings that show the effectiveness of the proposed method

Critical strengths
  • The application of on-policy distillation like method is well suited for the context of continual learning
  • The paper is clear and easy to follow
  • Experimental results show the improvement consistently
Critical weaknesses
  • it is not clear how the proposed method is different from other self distillation method fundamentally
  • Missing reference on hinted RL: https://openreview.net/forum?id=vXVjbbcuHy
  • how the solution diversity (e.g. pass@k affects when using hint compared to normal RL)
  • the proposed method is mostly compared against traditional CL baselines, authors should also compare to other hybrid (on+off policy) RL baselines
Questions for authors
  • Did the author attach the code somewhere? as I saw in the paper mention the code at different places: (e.g. The implementation snapshot is archived in revision/RL4CL_revised_table2_20260911/code/awhr_may_source_snapshot`; The script scripts/ plot_ci90_diagnostics.py reproduces the figures and summaries; revision/ci90_ figures_20260918/ contains the numerical values and source checksums.)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.