- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates the limitations of representation-level metrics in analyzing policy changes induced by Reinforcement Learning with Verifiable Rewards (RLVR). The authors first revisit the previously reported capability-narrowing phenomenon and argue that it may stem from inconsistencies in evaluation protocols. They then examine the discrepancy between representation similarity and model behavior, showing that models trained with different post-training objectives can exhibit nearly identical CKA scores despite measurable differences in their output distributions. To explain this phenomenon, the authors introduce the Strategy-Representation Gap and CKA Blind Spot theorems, arguing that CKA is relatively insensitive to small representation perturbations compared with linear probing. They further propose NSARS, a metric combining CKA with linear probing accuracy, to detect representation differences that CKA may overlook. Additional experiments involving weight interpolation, layer-wise interventions, spectral analysis, and comparisons between full-parameter training and LoRA are presented to investigate the location and characteristics of these changes.
Critical strengths
The paper raises a potentially important concern about representation similarity metrics in RLVR analysis. The observation that different post-trained models can exhibit nearly identical CKA scores while producing measurably different output distributions is interesting. It highlights the need to distinguish global representation similarity from functionally relevant changes in model behavior.
Critical weaknesses
I'm sorry to say, this paper suffers from substantial presentation issues that significantly reduce its readability. For example, Table 2 has a cluttered layout that makes the results difficult to interpret, while several equations suffer from inconsistent notation and formatting. The writing also contains excessive parenthetical expressions, enumerations, and densely packed claims. Moreover, the logical connections between different experiments, theoretical analyses, and central conclusions are not always clearly established. These issues make the paper unnecessarily difficult to follow and weaken its overall clarity.
In addition, this paper makes several strong claims that are not fully supported by the presented evidence. For example, the authors argue that the previously reported RLVR capability-narrowing phenomenon is a measurement artifact. However, the main re-evaluation supporting this conclusion is conducted on a single model with four training seeds and a relatively limited evaluation set. Although the results suggest that the phenomenon may be sensitive to evaluation protocols, they are insufficient to establish that capability narrowing is generally an artifact of incorrect evaluation. Similarly, some of the paper's broader conclusions regarding representation invariance and the mechanisms underlying RLVR appear to extend beyond the scope of the experiments.
Furthermore, the theoretical analysis relies on several strong assumptions whose validity and general applicability are not sufficiently justified. For instance, the Strategy-Representation Gap theorem assumes a particular form of gradient attenuation under KL regularization, which appears closely related to the conclusion the theorem aims to establish. The conditions required for some of the asymptotic claims in the CKA Blind Spot theorem also deserve further clarification. Overall, both the experimental evidence and theoretical arguments appear insufficient to support the strength and generality of the paper's central claims.
Questions for authors
-
could the authors improve the formatting of figures, tables, and equations, reduce excessive parenthetical expressions and enumerations, and clarify the connections between the main findings?
-
could the authors provide stronger evidence that RLVR capability narrowing is caused by evaluation errors? How generalizable are the results from a single-model, four-seed evaluation to other models and tasks?