- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates the loss of base-model-supported solution routes during reinforcement learning with verifiable rewards (RLVR). The authors introduce CAGE, an optimizer-conditioned cross-context attribution score that estimates how individual token-level RL gradient contributions affect predictive entropy on a fixed set of held-out correct-solution prefixes. This score determines the placement or strength of selective KL regularization through discrete selection (CAGE-D) or continuous weighting (CAGE).
The evaluation includes finite-counterfactual attribution audits, prospective prediction of solution-family retention, construction-matched intervention controls, and transfer experiments. The authors report improvements in Pass@1 and correct-family retention, together with recovery of large- performance toward the base-model range.
Critical strengths
The empirical evaluation is extensive and includes multiple training seeds, comparisons against existing methods, construction-matched intervention controls, and experiments across four model configurations and mathematical reasoning and code tasks. In particular, CAGE-D’s improvements over matched random targeting support the practical value of the proposed ranking beyond selective KL regularization alone. The finite-counterfactual audits also provide useful evidence about the fidelity of the attribution approximation.
Critical weaknesses
1.The connection between risk attribution and the corrective intervention remains insufficiently established.
CAGE identifies token-level RL gradient contributions predicted to decrease entropy on held-out correct prefixes. However, applying KL regularization at those training positions does not necessarily counteract the attributed entropy decrease. Its effect depends on the optimizer-conditioned interaction between the KL gradient and the probe entropy gradient. The paper validates attribution fidelity and reports downstream intervention benefits, but does not directly demonstrate that KL corrects the identified risk. Furthermore, entropy reduction on a correct prefix can indicate increased confidence in a correct continuation rather than damage to reasoning support. These unresolved connections limit the mechanistic justification of the method and make its applicability under different probe distributions difficult to assess.
2.The strongest evidence for ranking-specific efficacy and large- recovery is concentrated in the Qwen settings.
The Llama and code-transfer experiments are valuable, but they do not establish the same complete evidence chain as the primary study. In particular, the incremental benefit over construction-matched random targeting and the recovery of large- coverage are not comparably established outside the primary Qwen setting. Positive transfer of the overall intervention does not by itself show that the proposed risk-ranking mechanism explains the gains across model families and tasks.
3.The evaluation does not sufficiently distinguish preservation of valid reasoning from preservation of answer-correct but flawed trajectories.
Wen et al. (2025), Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs, show that answer-based Pass@ can credit inaccurate or incomplete reasoning and that conclusions about RLVR can change when both reasoning and answer correctness are required through CoT-Pass@. Their findings do not establish that CAGE preserves flawed trajectories, but identify an important alternative explanation that the present evaluation does not resolve.
The mathematical family-retention pipeline also starts from final-answer correctness. Canonicalization and family-separation audits assess grouping consistency, but do not directly verify reasoning validity. Consequently, restoring base-model Pass@ or retaining base-supported families is insufficient to establish preservation of valid reasoning. This concern bears directly on the paper’s motivation and interpretation of “reasoning-support loss.”
Questions for authors
1.Does selective KL counteract the attributed entropy decrease?
At high-risk positions, have the authors measured whether adding KL reduces the predicted entropy decline on the corresponding probes? Please provide the optimizer-conditioned interaction between the KL gradient and the probe entropy gradient, or controlled measurements of actual probe entropy changes with and without KL. Does the intervention also preserve the probability of correct continuations, rather than only affect entropy?
2.How consistently does ranking-specific efficacy transfer beyond Qwen?
In the non-Qwen setting, what are the paired effects and confidence intervals of CAGE-D versus construction-matched random targeting on accuracy and retention? Is corresponding large- evidence available? Please distinguish evidence for the effectiveness of the overall intervention from evidence for the proposed ranking mechanism.
3.Do the retained trajectories contain valid reasoning?
Can the authors provide a reasoning-validity audit or a CoT-Pass@ evaluation following the distinction raised by Wen et al. (2025)? After excluding answer-correct trajectories with invalid or incomplete reasoning, do the main retention and large- conclusions remain? Such evidence would materially affect my assessment of whether CAGE preserves meaningful reasoning support.
Official Comment by Reviewer aqtH
I thank the authors for the additional clarifications. The additional results address my concern about whether the retained solutions contain invalid reasoning and may lead to wrong comparison. However, I am still concerned about the method's mechanism and worry that it may not generalize. Overall, I will keep my rating.