- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies continual post-training with training-time hints. The goal is to transfer capabilities acquired under privileged/hinted prompts to the raw-prompt policy while reducing forgetting.
The paper proposes Hint-Gap Weighted Transfer (HiGHT). For a hinted rollout, a positive trajectory advantage determines whether the trajectory contributes to transfer. For each sampled token, the method compares its probability under the hinted and raw prompts at the same response prefix, and uses the signed gap to weight the raw-prompt token log-likelihood. HiGHT is combined with online RL under raw and hinted prompts.
Experiments on HOCL, MLLM-CL, and POPE show improved new-task learning and retention over the main Distill + RL baseline.
Critical strengths
-
Important problem.
The paper addresses a meaningful question: how to convert success obtained with privileged training-time information into capabilities that remain useful without that information. -
Simple and intuitive transfer mechanism.
Comparing hinted and raw probabilities at the same prefix is a clean way to measure how the hint changes the sampled-token prediction. The signed weighting also allows different tokens within the same successful trajectory to be reinforced or suppressed.
Critical weaknesses
-
The key missing baseline is reward-gated privileged self-distillation.
This is my main concern. HiGHT already computes both hinted and raw policy probabilities, but only uses the sampled-token scalar difference .
A natural baseline would keep the same positive-advantage trajectory gate while performing standard privileged-policy distillation, or an analogous PG-style on-policy self-distillation objective.
The current Distill + RL baseline does not fully isolate whether the gain comes from the proposed hint-gap signal or simply from making distillation reward-selective and sparse. This comparison is important for establishing the novelty and necessity of HiGHT.
-
The choice of is insufficiently justified.
It is unclear why raw probability difference is preferable to alternatives such as log-probability difference, probability ratio, or KL-derived signals. Since is sensitive to absolute probability scale, an ablation over alternative discrepancy measures would strengthen the method.
-
Core ablations are limited.
The objective combines several design choices: positive-advantage gating, signed token gaps, exponential weighting, asymmetric clipping, and bidirectional transfer. It is currently unclear which components are responsible for the improvement.
-
The theoretical analysis is modest.
The gradient bound shows that the transfer gradient vanishes when the hint gap vanishes, which is a useful sanity property. However, it does not explain why HiGHT should forget less than reward-gated distillation or why the proposed weighting should provide a better retention-plasticity tradeoff.
Questions for authors
-
Can the authors compare HiGHT against reward-gated privileged distillation, under identical rollouts and RL terms?
-
Can the authors compare against a PG-style privileged self-distillation objective with the same positive trajectory gate?
-
Why is preferred over log-probability or KL-derived discrepancy measures?
-
How important are the exponential weighting and signed negative-token updates?
-
What is the contribution of bidirectional transfer in MLLM-CL?