acceptodds
Under review as a conference paper at ICLR 2027

Not All Reasoning Generalizes: A Case Study on Learning Generalizable Reasoning Strategies in RLVR

Abstract

Reinforcement learning with verifiable rewards (RLVR) can enable generalizable improvement for large language model (LLM) reasoning. This paper studies how to incentivize such generalization via reasoning strategies. We study this question with a controllable equation-solving framework: finding multiple inputs for equations, where is a non-bijective string function. This setting makes reasoning strategies observable while supporting a clear definition of easy-to-hard out-of-distribution (OOD) shifts, e.g., more solutions, longer targets, etc. We uncover a subtle RLVR failure: it teaches the model to solve in-distribution tasks with high accuracy and structured step-by-step derivation, yet under OOD shifts, the same model abandons derivation and collapses to ineffective brute-force enumeration, causing sharp accuracy drops. This observation reveals the deeper challenges of RLVR generalization as it is neither incapable base model nor reward hacking, since the model succeeds in-distribution with the correct reasoning strategy. The key to mitigating this failure is an indirect intervention that surprisingly helps: co-training with bijective inversion tasks. Although these bijective tasks are actually not uniformly harder and even further from the non-bijective OOD evaluation, they make derivation more certain and substantially restore the correct strategy and accuracy under OOD shifts. This effect further motivates our theoretical analysis: outcome-only RLVR can shape reasoning processes through implicitly crediting the effective reasoning strategies more, even without process supervision. To conclude, our analysis framework and findings systematically reveal the crucial role and incentivization of reasoning strategies in RLVR generalization under OOD shifts.

Primary area:
foundation or frontier models, including LLMs
Keywords:
RLVR

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cTtrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies reasoning-strategy generalization in RLVR using synthetic string-function inversion tasks. A model trained on non-bijective tasks learns structural derivation on easier inputs but often switches to brute-force enumeration under harder distribution shifts. Mixing bijective inversion tasks increases derivation and improves the controlled OOD results. The paper interprets this through information gained per token and policy-gradient identities. Additional experiments examine task mixtures, hidden-state probes, mathematical transfer, and a small Sokoban extension.

Critical strengths
  1. The paper is clearly organized and visually polished. The controlled task has an executable verifier and explicit distribution shifts, making the reported behavior easy to inspect.

  2. The mixed-training effect is reproduced across three runs on Length Change. The mean score increases from 5.3% to 21.7%, alongside a substantial increase in derivation. The diversity controls and mixture ablations provide useful context for this result.

Critical weaknesses
  1. The theory remains primarily an explanatory framework. Standard relationships between success probability and policy gradients do not establish that bijective tasks improve generalization through higher information gain per token. This quantity is not directly measured, and the binary-reward, large-group abstraction differs from the actual training setup. The experiments also do not sufficiently distinguish strategy collapse as a cause of failure from enumeration as a response to harder computation or failed execution. The resulting guidance for realistic training is limited.

  2. The broader significance is not established beyond a narrow model and task setting. Core evidence comes from Qwen2.5-7B and synthetic inversion tasks. The additional models are discussed as unsuitable operating points rather than reproducing the main finding. Controlled gains are real, but absolute OOD scores remain low, and mathematical transfer is not uniformly positive. The Sokoban experiment changes the task mixture and is not a direct transfer of the trained inversion model. These results do not yet demonstrate a broadly useful training principle.

Questions for authors
  1. Can the proposed information-gain quantity be measured and used to predict which task mixtures improve generalization? How does the theory account for finite rollout groups, partial-credit rewards, and format rewards in the actual experiments?

  2. What evidence distinguishes strategy collapse causing OOD failure from strategy changes caused by execution difficulty? Can matched tasks or interventions on strategy choice separate these explanations?

  3. Does the finding reproduce on other model families or stronger models when task difficulty is adjusted appropriately? Evidence of consistent benefit on realistic training tasks would materially strengthen the significance claim.

1 comment

Dear Reviewer cTtr,

The authors have provided a detailed reply to your review. Could you please take a look, acknowledge the rebuttal, and give your response?

Thank you for serving as a reviewer for ICLR.

Best,

AC

Reviewer_eTi2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the generalization of reasoning strategies under reinforcement learning with verifiable rewards. It introduces a controlled string-function inversion setting in which models must produce multiple valid inputs for a given output. A model trained only on non-bijective inversion achieves high performance in the source regime but increasingly resorts to brute-force enumeration under harder distribution shifts. Mixing in bijective inversion tasks substantially increases derivation usage and improves OOD performance. The paper supports these observations with repeated runs, alternative function families, mixture-ratio experiments, representation probes, and exploratory math and Sokoban evaluations. It proposes a bits-per-token interpretation and policy-gradient analysis to explain how outcome rewards can favor effective reasoning processes.

Critical strengths
  1. The inversion framework combines held-out functions, executable answer verification, and observable differences in solution strategy. MONO and MIXED achieve similar source-regime performance, while their behavior and performance diverge substantially on harder tasks. On Length Change, the three-run mean score increases from 5.3% to 21.7%, and derivation usage increases from 48.3% to 93.3%. This provides meaningful evidence that high source-regime performance does not ensure stable strategy usage under distribution shift.

  2. Bijective co-training is a concrete intervention with a substantial observed benefit. The finding that a higher bijective fraction increases derivation usage while the balanced mixture achieves better accuracy is particularly useful: it prevents equating structured-looking reasoning with successful reasoning. The accompanying error analysis connects this distinction to incomplete calculations and insufficient answer generation. This makes the empirical account more informative than an accuracy-only comparison.

Critical weaknesses
  1. Propositions 1–2 establish that, under the stated assumptions, policy gradients credit increases in policy-conditional success probability, and that an idealized large-group GRPO estimator recovers this direction. They do not establish the connection between the cardinality-based quantity (D=log(|Y|/|Y*|)), information gained by individual reasoning tokens, and the relative success of derivation versus enumeration. In particular, reducing the valid-solution set to a singleton does not by itself establish that the advantage gap between these strategies increases; this requires assumptions about how each strategy’s success probability changes. The paper does not directly estimate these quantities. The additional interventions are consistent with the proposed explanation, but do not establish it. This matters because explaining the mechanism is presented as a central contribution.

  2. The bijective and non-bijective training families differ in more than solution uniqueness. Appendix A.2 shows that the bijective training tasks mainly use position permutations and digit-wise shifts, whereas the non-bijective tasks involve neighboring-digit interactions, folding, and other lossy operations. The intervention also changes the requested answer count. Consequently, the improvement could reflect training on particular inverse operations or simpler single-answer derivations, in addition to the proposed effect of bijectivity. The alternative-function and diversity controls are useful, but matching the number of functions does not isolate these differences, and aggregate training reward is not a matched measure of task difficulty. The evidence therefore supports the effectiveness of this particular auxiliary-task recipe more strongly than the claim that unique-solution structure produces the proposed strategy-selection mechanism.

Questions for authors
  1. Under what additional assumptions does reducing the number of valid solutions increase the advantage gap between derivation and enumeration? Can the authors connect this claim to strategy-conditioned success probabilities or another measurable quantity, and distinguish what follows from Propositions 1–2 from what remains a hypothesis?

  2. Do the authors have a comparison that holds the function primitives and requested answer count more closely fixed. For example, structurally matched reversible/lossy variants, or non-bijective auxiliary tasks requesting only one answer? Such a comparison would materially affect whether the improvement can be attributed to bijectivity rather than the other changes in the auxiliary curriculum.

  3. Can the authors report correctness conditional on the strategy label within each function and OOD setting, ideally using paired prompts across MONO and MIXED? The current aggregate trends leave open how much of the score improvement comes from changing strategy frequency versus improving execution within a strategy. This distinction is especially relevant given the mixture-ratio results.

1 comment

Dear Reviewer eTi2,

The authors have provided a detailed reply to your review. Could you please take a look, acknowledge the rebuttal, and give your response?

Thank you for serving as a reviewer for ICLR.

Best,

AC

Reviewer_3JHmOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether RLVR-learned reasoning strategies generalize to harder tasks. In a controlled string-function inversion benchmark, models recover several inputs per output; train and test functions are disjoint, and OOD tests vary output length, answer count, and input–output length. MONO, trained on non-bijective functions, performs well in-distribution (≈90%) but degrades sharply OOD; MIXED, adding bijective tasks at 1:1, improves OOD accuracy and the derivation share (48% → 93% on Length Change). On Length Change, three runs average 5.3 ± 1.0% for MONO versus 21.7 ± 3.5% for MIXED (Table B.1). The authors also propose a bits-per-token account.

Critical strengths
  • The testbed is well chosen. The inversion functions are executable, so answers can be checked and traces inspected. The MONO–MIXED gap is large and holds across three runs (Table B.1). Appendix B.2 adds controls for an alternative function family and a training-task diversity baseline that does not reproduce the gain. I was skeptical of the synthetic setup, but within this testbed the gain is convincing.

  • An informative distinction between strategy and execution. Section 6.2 shows that the 3:1 bijective mixture uses derivation most often yet achieves lower accuracy than the 1:1 mixture; Appendix E attributes the shortfall to execution errors. A sound strategy does not guarantee correct execution.

Critical weaknesses
  • LOOP may confound strategy selection with execution failure. My main concern is what the label measures. Figure B.1's judge checks LOOP first, so a trace that starts with valid derivation but ends stuck in repetition is labelled LOOP; Appendix B.1 then folds LOOP into ENUM in every main-paper plot. This conflates a switch to guessing with failed execution; since most MONO failures are loops, the merge lowers MONO's derivation share more than MIXED's, widening the measured gap. The 96% agreement with manual labels shows that annotators and the judge apply the same rubric, not that the rubric captures strategy choice. LOOP is also defined partly by repeated attempts, abrupt termination, and budget exhaustion, so the 8k→5k cut can shift LOOP rates—and thus the ENUM share—independently of strategy (Figure 4). Reporting LOOP separately, and what LOOP traces did beforehand, would resolve this from existing traces.

  • The theory does not establish the proposed information-rate mechanism. Proposition 1 is, under its assumptions, an identity between per-token advantage and the success-probability increment Δp, and I accept it. What I cannot follow is the treatment of Δp as information gain: the per-token gain is log₂((p+Δp)/p) ≈ Δp/(p ln 2), not Δp, and the factor 1/(p ln 2) varies with the state's p, so Δp is not a common scale across states or tasks. D = log(|Y|/|Y*|) is task-level and pπ policy-level, with no stated connection between them. Neither Proposition 1 nor Proposition 2's large-group approximation derives Section 5.2's claim that |Y*| = 1 sharpens the derivation–enumeration advantage gap; it is asserted, not proved. That needs assumptions about strategy success probabilities, or direct evidence. The gradient analysis is useful, but the bit-rate explanation remains a hypothesis.

Questions for authors
  • Can the authors separate strategy choice from execution failure? For Length Change and the short-budget run, please report DERIVE/ENUM/LOOP rates, derivation share among non-LOOP traces, and accuracy by label. How often do LOOP traces begin with derivation, and does the 8k→5k cut change LOOP rates independently of strategy?

  • What establishes the bit-rate mechanism? Which assumptions connect Δp to information gain and imply a larger derivation advantage on bijective tasks? If this is intuition rather than a consequence of Propositions 1–2, please say so.

1 comment

Dear Reviewer 3JHm,

The authors have provided a detailed reply to your review. Could you please take a look, acknowledge the rebuttal, and give your response?

Thank you for serving as a reviewer for ICLR.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.