acceptodds
Under review as a conference paper at ICLR 2027

Which Updates Erase What the Base Model Knew? Optimizer-Conditioned Attribution of Reasoning-Support Loss in RLVR

Abstract

Reinforcement learning with verifiable rewards (RLVR) can improve short-horizon accuracy while erasing solution routes already supported by the base model. Entropy decline reveals this contraction but does not identify which updates cause it. We introduce CAGE (Cross-context Attribution of Gradient Effects), an optimizer-conditioned cross-context attribution score that estimates how each current update changes entropy at held-out correct states and selectively regularizes the predicted high-risk tail. Accumulated CAGE exposure predicts subsequent support loss () and remains predictive when recent entropy decline is included on the same rows. CAGE improves macro Pass@1 by +2.06 pp over OPEFO. Under the construction-matched discrete intervention, CAGE-D exceeds matched random targeting by +2.59 pp in macro Pass@1 and +.037 in correct-family mass, and exceeds OPEFO by +0.93 pp in macro Pass@1; uniform KL, source-strength matching, and mean-probe controls recover only parts of the gain. Selective KL recovers most of the high- loss, with CAGE-D above matched random selection in point estimate. It costs GRPO wall-clock and yields positive transfer point estimates on Qwen-7B, Llama-8B, and code, with weaker accuracy evidence under the larger shifts. Together, these results support targeting update-level risk rather than treating entropy preservation alone as sufficient for intervention design.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning with verifiable rewards, LLM reasoning, training data attribution, influence functions, entropy collapse, optimizer-aware attribution, selective KL regularization, mode coverage

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_aqtHOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the loss of base-model-supported solution routes during reinforcement learning with verifiable rewards (RLVR). The authors introduce CAGE, an optimizer-conditioned cross-context attribution score that estimates how individual token-level RL gradient contributions affect predictive entropy on a fixed set of held-out correct-solution prefixes. This score determines the placement or strength of selective KL regularization through discrete selection (CAGE-D) or continuous weighting (CAGE).

The evaluation includes finite-counterfactual attribution audits, prospective prediction of solution-family retention, construction-matched intervention controls, and transfer experiments. The authors report improvements in Pass@1 and correct-family retention, together with recovery of large-kk performance toward the base-model range.

Critical strengths

The empirical evaluation is extensive and includes multiple training seeds, comparisons against existing methods, construction-matched intervention controls, and experiments across four model configurations and mathematical reasoning and code tasks. In particular, CAGE-D’s improvements over matched random targeting support the practical value of the proposed ranking beyond selective KL regularization alone. The finite-counterfactual audits also provide useful evidence about the fidelity of the attribution approximation.

Critical weaknesses

1.The connection between risk attribution and the corrective intervention remains insufficiently established.

CAGE identifies token-level RL gradient contributions predicted to decrease entropy on held-out correct prefixes. However, applying KL regularization at those training positions does not necessarily counteract the attributed entropy decrease. Its effect depends on the optimizer-conditioned interaction between the KL gradient and the probe entropy gradient. The paper validates attribution fidelity and reports downstream intervention benefits, but does not directly demonstrate that KL corrects the identified risk. Furthermore, entropy reduction on a correct prefix can indicate increased confidence in a correct continuation rather than damage to reasoning support. These unresolved connections limit the mechanistic justification of the method and make its applicability under different probe distributions difficult to assess.

2.The strongest evidence for ranking-specific efficacy and large-kk recovery is concentrated in the Qwen settings.

The Llama and code-transfer experiments are valuable, but they do not establish the same complete evidence chain as the primary study. In particular, the incremental benefit over construction-matched random targeting and the recovery of large-kk coverage are not comparably established outside the primary Qwen setting. Positive transfer of the overall intervention does not by itself show that the proposed risk-ranking mechanism explains the gains across model families and tasks.

3.The evaluation does not sufficiently distinguish preservation of valid reasoning from preservation of answer-correct but flawed trajectories.

Wen et al. (2025), Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs, show that answer-based Pass@kk can credit inaccurate or incomplete reasoning and that conclusions about RLVR can change when both reasoning and answer correctness are required through CoT-Pass@kk. Their findings do not establish that CAGE preserves flawed trajectories, but identify an important alternative explanation that the present evaluation does not resolve.

The mathematical family-retention pipeline also starts from final-answer correctness. Canonicalization and family-separation audits assess grouping consistency, but do not directly verify reasoning validity. Consequently, restoring base-model Pass@kk or retaining base-supported families is insufficient to establish preservation of valid reasoning. This concern bears directly on the paper’s motivation and interpretation of “reasoning-support loss.”

Questions for authors

1.Does selective KL counteract the attributed entropy decrease?

At high-risk positions, have the authors measured whether adding KL reduces the predicted entropy decline on the corresponding probes? Please provide the optimizer-conditioned interaction between the KL gradient and the probe entropy gradient, or controlled measurements of actual probe entropy changes with and without KL. Does the intervention also preserve the probability of correct continuations, rather than only affect entropy?

2.How consistently does ranking-specific efficacy transfer beyond Qwen?

In the non-Qwen setting, what are the paired effects and confidence intervals of CAGE-D versus construction-matched random targeting on accuracy and retention? Is corresponding large-kk evidence available? Please distinguish evidence for the effectiveness of the overall intervention from evidence for the proposed ranking mechanism.

3.Do the retained trajectories contain valid reasoning?

Can the authors provide a reasoning-validity audit or a CoT-Pass@kk evaluation following the distinction raised by Wen et al. (2025)? After excluding answer-correct trajectories with invalid or incomplete reasoning, do the main retention and large-kk conclusions remain? Such evidence would materially affect my assessment of whether CAGE preserves meaningful reasoning support.

1 comment
Reviewer_aqtHOpenReview ↗

Official Comment by Reviewer aqtH

I thank the authors for the additional clarifications. The additional results address my concern about whether the retained solutions contain invalid reasoning and may lead to wrong comparison. However, I am still concerned about the method's mechanism and worry that it may not generalize. Overall, I will keep my rating.

Reviewer_vDF6OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies update-level forgetting in reinforcement learning with verifiable rewards. RLVR can improve low-k accuracy while reducing high-k coverage of reasoning routes supported by the base model, and aggregate entropy contraction cannot identify which updates cause this loss. The authors propose CAGE, an optimizer-conditioned, cross-context attribution score that ranks sampled tokens in the current update by their predicted effect on entropy at held-out correct states. The deployed score uses a forward-computable direct-head approximation and adds approximately 5.5% to the reported GRPO training cost. Positive scores indicate entropy sharpening at held-out correct states that CAGE treats as potentially harmful, and the predicted high-risk tail receives selective KL regularization. Validation includes prospective prediction, construction-matched intervention controls, comparisons with published baselines, transfer experiments, and finite-counterfactual fidelity audits.

Critical strengths

CAGE accounts for the realized optimizer state and evaluates the effects of individual token contributions on held-out correct prefixes. Accumulated CAGE exposure predicts subsequent loss of base-supported correct-family retention after controlling for recent entropy decline, with supporting temporal-placebo and seed-sensitivity analyses. This provides a more specific signal of update-level risk than aggregate entropy contraction alone.

The empirical evaluation uses a careful set of controls. Random targeting, source-strength matching, mean-probe controls, anti-targeting, and the geometry-by-intervention experiment help distinguish CAGE’s ranking from generic selective regularization and simpler alternative explanations. In the primary experiment, continuous CAGE improves macro Pass@1 over the strongest published accuracy baseline at modest additional training cost. The authors also report null or unresolved findings, including the code-accuracy result and some mechanism-specific contrasts, rather than presenting all evidence as equally conclusive.

Critical weaknesses

CAGE uses a frozen bank of correct base-model solutions that the reproduced published baselines do not directly use. The mean-probe and construction-matched controls support an incremental benefit from the full CAGE construction given this resource, but they do not isolate the independent value of obtaining the probe resource itself. Sensitivity to changes in the protected probe population also remains untested.

Two aspects of the reported validation require clarification. First, the incremental R^2 values reported in Table 3 and Appendix A.29 are not readily reconcilable from the current presentation. Second, the continuous arm conditions its attribution score on a provisional binary-mask gradient, whereas the final update uses continuous weights. The reported selector-feedback audit explicitly concerns binary selection, so it is unclear whether the finite-counterfactual audit fully captures the difference between the conditioning update and the final continuous update.

Questions for authors

1.The reproduced published baselines do not directly use the frozen probe bank, while the mean-probe controls simplify how that resource is used. Can the authors more directly separate the value of probe access itself from the value of CAGE’s optimizer-conditioned ranking, for example through a simpler same-probe baseline with a matched regularization budget?

2.How were the token-selection threshold for CAGE-D and the continuous weighting cap selected? Were these choices subjected to sensitivity analysis, and how do materially different values affect accuracy, retention, ranking specificity, and computational cost?

3.Could the authors reconcile the incremental R^2 values reported in Table 3 and Appendix A.29? Do the reported values and the full-versus-mean-probe difference use different definitions, samples, aggregation procedures, or regression specifications?

4.Does the finite-counterfactual evaluation in Table 13 evaluate fidelity to the final update using continuous weights? If not, could the authors clarify the difference between attribution conditioned on the provisional binary-mask gradient and the final continuous update?

Reviewer_36NhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how RLVR can improve reasoning accuracy while maintaining the diversity of correct solution paths supported by the base model. They introduce CAGE (Cross-context Attribution of Gradient Effects), which estimates how individual token-level gradient contributions affect predictive entropy at held-out correct reasoning prefixes at each optimizer step. The method uses the optimizer's update geometry and an LM-head approximation to compute attribution scores, which are then used to selectively apply KL regularization to tokens whose updates are predicted to reduce entropy at the held-out correct prefixes. Experiments across mathematical reasoning and coding demonstrate improvements over existing methods such as OPEFO in accuracy and preservation of base-model reasoning support.

Critical strengths
  1. The proposed idea is an interesting attribution-based approach of entropy decline, and it shows improvements in reasoning accuracy for both Pass@1 as well as Pass@k for high k, demonstrating that CAGE regularization provides benefits of improved accuracy while expanding the coverage of solutions. The empirical evaluation is also extensive with several controls (although more clearly communicating each of the experiments and clarifying what they show would be very helpful).
Critical weaknesses
  1. The writing of the paper is not clear and its presentation is difficult to follow. When describing the method, several quantities are defined with little motivation or explanation on how they are/will be used, while several variables are also not defined prior to using them. This makes it difficult to identify the core algorithm and determine what quantities are actually being measured. Some of the ideas presented in the paper are interesting, but the presentation and writing makes it very difficult to parse.

  2. Although the empirical evaluation is extensive, the high-k comparisons against existing methods are somewhat limited. The authors report Pass@64 for several competitive baselines, including OPEFO, STEER, and Clip-Cov, but only provide Pass@128 to 512 comparisons against GRPO and their own selective-KL intervention controls. Since preserving large-k reasoning performance is part of the motivation for the paper, comparisons against these existing methods at larger k would help establish whether CAGE provides additional benefits over other entropy-preservation approaches.

Questions for authors
  1. Could the authors further clarify the relationship between entropy-based attribution and actual loss of correct reasoning support? In particular, could they investigate whether high-scoring updates directly reduce the probability of previously supported correct continuations, rather than just decreasing entropy?

  2. Could the authors extend the high-k evaluation to the other competitive baselines, particularly OPEFO, STEER, and Clip-Cov? Since preserving large-k reasoning performance is part of the main motivation, these comparisons would help determine whether CAGE provides improvements in pass@k for high-k beyond existing entropy-preservation methods.

  3. How much of CAGE's improvement comes specifically from the proposed attribution mechanism versus having access to additional correct reasoning probes? Additionally, how sensitive are the results to the held out prefixes?

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your time and thoughtful reviews. Please read the authors’ response and share your follow-up comments promptly, indicating whether your concerns have been addressed. Please raise any additional questions early enough to give the authors sufficient time to respond.

Thank you.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.