- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies reasoning-strategy generalization in RLVR using synthetic string-function inversion tasks. A model trained on non-bijective tasks learns structural derivation on easier inputs but often switches to brute-force enumeration under harder distribution shifts. Mixing bijective inversion tasks increases derivation and improves the controlled OOD results. The paper interprets this through information gained per token and policy-gradient identities. Additional experiments examine task mixtures, hidden-state probes, mathematical transfer, and a small Sokoban extension.
Critical strengths
-
The paper is clearly organized and visually polished. The controlled task has an executable verifier and explicit distribution shifts, making the reported behavior easy to inspect.
-
The mixed-training effect is reproduced across three runs on Length Change. The mean score increases from 5.3% to 21.7%, alongside a substantial increase in derivation. The diversity controls and mixture ablations provide useful context for this result.
Critical weaknesses
-
The theory remains primarily an explanatory framework. Standard relationships between success probability and policy gradients do not establish that bijective tasks improve generalization through higher information gain per token. This quantity is not directly measured, and the binary-reward, large-group abstraction differs from the actual training setup. The experiments also do not sufficiently distinguish strategy collapse as a cause of failure from enumeration as a response to harder computation or failed execution. The resulting guidance for realistic training is limited.
-
The broader significance is not established beyond a narrow model and task setting. Core evidence comes from Qwen2.5-7B and synthetic inversion tasks. The additional models are discussed as unsuitable operating points rather than reproducing the main finding. Controlled gains are real, but absolute OOD scores remain low, and mathematical transfer is not uniformly positive. The Sokoban experiment changes the task mixture and is not a direct transfer of the trained inversion model. These results do not yet demonstrate a broadly useful training principle.
Questions for authors
-
Can the proposed information-gain quantity be measured and used to predict which task mixtures improve generalization? How does the theory account for finite rollout groups, partial-credit rewards, and format rewards in the actual experiments?
-
What evidence distinguishes strategy collapse causing OOD failure from strategy changes caused by execution difficulty? Can matched tasks or interventions on strategy choice separate these explanations?
-
Does the finding reproduce on other model families or stronger models when task difficulty is adjusted appropriately? Evidence of consistent benefit on realistic training tasks would materially strengthen the significance claim.
Dear Reviewer cTtr,
The authors have provided a detailed reply to your review. Could you please take a look, acknowledge the rebuttal, and give your response?
Thank you for serving as a reviewer for ICLR.
Best,
AC