- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether reinforcement learning (RL)-induced behavioral changes can be anticipated from a model's responses before training. The authors propose Behavioral Response Spectroscopy (BRS), which estimates reward-conditioned behavioral directions through covariance between response features and reward advantages. These directions are compared with actual post-RL behavioral changes across ten language models and four tasks. The experiments show positive directional alignment and demonstrate that subspaces constructed from held-out reward probes capture more behavioral displacement than PCA in the original feature space. Further analyses reveal that much of this transferable structure is associated with response length, while the relative advantages of BRS vary across behavioral representations and length-adjustment settings.
Critical strengths
S1. Interesting research question. The paper investigates whether behavioral changes beyond the optimized reward can be anticipated before RL training. This is a meaningful problem for understanding the broader effects of reward optimization and assessing potential unintended behavioral changes.
S2. Simple and interpretable framework. BRS provides a straightforward way to characterize reward-associated behavioral directions using pre-RL responses. The covariance formulation is intuitive, and the connection to local distributional reweighting offers a clear theoretical motivation.
S3. Extensive empirical analysis. The paper evaluates ten models across reasoning, code generation, and summarization tasks. The held-out-reward protocol, disjoint evaluation prompts, matched baselines, and additional length-adjustment and representation analyses provide a systematic examination of the proposed approach.
Critical weaknesses
W1. Limited additional value beyond simpler length-based baselines. In Table 2, the reward-free Length+PCA baseline matches or exceeds BRS in both model cohorts (0.611/0.738 versus 0.507/0.732). The single length-associated direction also captures a substantial fraction of behavioral displacement. Moreover, after removing length-related variation, the BRS advantage over PCA becomes negative for the Qwen cohort and approximately zero for the other cohort in the surface-feature representation (Table 12). These results suggest that much of the predictive signal can be explained by length-associated behavior, making the additional value of reward conditioning unclear.
W2. Limited evidence for transfer across genuinely distinct rewards. The held-out-reward experiments operate on a set of closely related probes, many of which share lexical patterns or feature definitions with the behavioral measurements. For example, lexical diversity and final-answer format are directly represented in both the reward probes and retained evaluation features. Consequently, the observed transfer may largely reflect correlations within the predefined feature space rather than generalizable structure across independent objectives. The use of predominantly rule-based training rewards further limits the evidence for broader reward settings. Reward-family-level holdouts or independently defined rewards would provide stronger validation.
W3. Sensitivity to behavioral representations and evaluation choices. The main reasoning subspace evaluation uses only seven behavioral coordinates, several of which are naturally associated with response length. In correctness-training endpoint evaluations, log length is retained and the surface features are compared in their original units, potentially affecting directional alignment. More importantly, the relative BRS–PCA gains change substantially across surface and embedding-based representations, length adjustments, and coordinate scaling choices (Tables 4 and 19). Although the paper investigates these sensitivities, they raise concerns about the robustness and generality of the identified behavioral structure. The embedding-based behavioral dimensions also lack independent semantic validation.
W4. Limited methodological novelty and practical predictive validation. The core BRS estimator relies on standard reward–feature covariance, while the theoretical analysis establishes a local distributional-reweighting identity rather than a predictive guarantee for actual parameter updates. The primary empirical evidence is also concentrated on relatively short GRPO training with a fixed LoRA configuration, leaving generalization to longer or full-parameter RL unclear. Furthermore, the main cross-reward metric measures subspace capture without establishing whether BRS can reliably predict the direction or magnitude of individual behavioral changes. More direct feature-level prediction evaluations would better demonstrate its practical utility.
Questions for authors
Several reward probes share definitions or lexical patterns with the evaluated features. How much of the held-out-reward transfer reflects shared structure across objectives rather than correlations induced by the probe design? Would the conclusions hold if all feature-derived probes, or an entire reward family, were excluded when constructing the subspace? This can be computed from the existing pre-RL banks without retraining.