acceptodds
Under review as a conference paper at ICLR 2027

Behavioral Response Spectroscopy: Pre-RL Responses Reveal Shared Structure in RL-Induced Behavioral Change

Abstract

Reinforcement learning (RL) improves a target reward but often changes other properties of a model's responses as well. Can any of these accompanying changes be anticipated before training? We show that variation already present in the starting model contains a measurable signal of later change. Behavioral Response Spectroscopy (BRS) uses multiple answers to the same prompt to measure which response properties tend to accompany higher reward. We then compare these pre-RL directions with the behavioral changes produced by actual RL across ten models and four tasks. Our strongest test asks whether this structure transfers across objectives: we exclude the training reward from the measured directions and use separate prompts for evaluation. On two reasoning tasks, a two-dimensional space built from other reward directions captures 50.7% of post-RL displacement energy across five Qwen2.5 models and 73.2% across five other models, compared with 24.8% and 36.2% for principal components computed from the same responses without reward information. Matched controls reveal a prominent length-associated behavioral mode: multiple response properties co-vary with length, while different rewards can orient this mode toward either shorter or longer answers. This directional regularity is not confined to the original surface features: in an embedding-based behavioral representation, alignment remains positive after pre-RL length adjustment, although the relative advantage of BRS over PCA varies across representations and model cohorts. Overall, pre-RL responses already reveal a transferable component of how RL will reshape behavior.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Language Models, Behavioral Response Spectroscopy, Behavioral Change, Pre-RL Response Variation, Reward--Behavior Covariance, Shared Behavioral Modes, Response-Space Geometry

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zURGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether reinforcement learning (RL)-induced behavioral changes can be anticipated from a model's responses before training. The authors propose Behavioral Response Spectroscopy (BRS), which estimates reward-conditioned behavioral directions through covariance between response features and reward advantages. These directions are compared with actual post-RL behavioral changes across ten language models and four tasks. The experiments show positive directional alignment and demonstrate that subspaces constructed from held-out reward probes capture more behavioral displacement than PCA in the original feature space. Further analyses reveal that much of this transferable structure is associated with response length, while the relative advantages of BRS vary across behavioral representations and length-adjustment settings.

Critical strengths

S1. Interesting research question. The paper investigates whether behavioral changes beyond the optimized reward can be anticipated before RL training. This is a meaningful problem for understanding the broader effects of reward optimization and assessing potential unintended behavioral changes.

S2. Simple and interpretable framework. BRS provides a straightforward way to characterize reward-associated behavioral directions using pre-RL responses. The covariance formulation is intuitive, and the connection to local distributional reweighting offers a clear theoretical motivation.

S3. Extensive empirical analysis. The paper evaluates ten models across reasoning, code generation, and summarization tasks. The held-out-reward protocol, disjoint evaluation prompts, matched baselines, and additional length-adjustment and representation analyses provide a systematic examination of the proposed approach.

Critical weaknesses

W1. Limited additional value beyond simpler length-based baselines. In Table 2, the reward-free Length+PCA baseline matches or exceeds BRS in both model cohorts (0.611/0.738 versus 0.507/0.732). The single length-associated direction also captures a substantial fraction of behavioral displacement. Moreover, after removing length-related variation, the BRS advantage over PCA becomes negative for the Qwen cohort and approximately zero for the other cohort in the surface-feature representation (Table 12). These results suggest that much of the predictive signal can be explained by length-associated behavior, making the additional value of reward conditioning unclear.

W2. Limited evidence for transfer across genuinely distinct rewards. The held-out-reward experiments operate on a set of closely related probes, many of which share lexical patterns or feature definitions with the behavioral measurements. For example, lexical diversity and final-answer format are directly represented in both the reward probes and retained evaluation features. Consequently, the observed transfer may largely reflect correlations within the predefined feature space rather than generalizable structure across independent objectives. The use of predominantly rule-based training rewards further limits the evidence for broader reward settings. Reward-family-level holdouts or independently defined rewards would provide stronger validation.

W3. Sensitivity to behavioral representations and evaluation choices. The main reasoning subspace evaluation uses only seven behavioral coordinates, several of which are naturally associated with response length. In correctness-training endpoint evaluations, log length is retained and the surface features are compared in their original units, potentially affecting directional alignment. More importantly, the relative BRS–PCA gains change substantially across surface and embedding-based representations, length adjustments, and coordinate scaling choices (Tables 4 and 19). Although the paper investigates these sensitivities, they raise concerns about the robustness and generality of the identified behavioral structure. The embedding-based behavioral dimensions also lack independent semantic validation.

W4. Limited methodological novelty and practical predictive validation. The core BRS estimator relies on standard reward–feature covariance, while the theoretical analysis establishes a local distributional-reweighting identity rather than a predictive guarantee for actual parameter updates. The primary empirical evidence is also concentrated on relatively short GRPO training with a fixed LoRA configuration, leaving generalization to longer or full-parameter RL unclear. Furthermore, the main cross-reward metric measures subspace capture without establishing whether BRS can reliably predict the direction or magnitude of individual behavioral changes. More direct feature-level prediction evaluations would better demonstrate its practical utility.

Questions for authors

Several reward probes share definitions or lexical patterns with the evaluated features. How much of the held-out-reward transfer reflects shared structure across objectives rather than correlations induced by the probe design? Would the conclusions hold if all feature-derived probes, or an entire reward family, were excluded when constructing the subspace? This can be computed from the existing pre-RL banks without retraining.

Reviewer_u1Q4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper proposes Behavioral Response Spectroscopy (BRS), a method for anticipating the behavioral changes caused by RL training by sampling responses from the base, pre-RL model. The authors find that 1) features of the response that are correlated with a reward before RL training will be amplified by RL training, and 2) one can capture a substantial amount of behavioral shift caused by RL (on a reward rr) by constructing a subspace built from how the pre-RL model’s responses vary with other, related rewards (excluding reward rr). The paper also finds that response length is a “shared behavioral mode”; that is, it is possible to also use how features correlate with response length (rather than reward) to predict changes after RL.

Critical strengths

Experimental rigor. The sequence of experiments (using the task reward to do BRS in section 3, then using other, non-task rewards for BRS in section 4) has a logical progression and demonstrates that model behavior after RL can be predicted by response variation with respect to rewards other than the one being optimized.

Critical weaknesses

Generalizability of findings. The paper’s primary results (presented in Section 4) are only evaluated on two reasoning tasks (GSM8K, ARC). It would require more evidence to demonstrate that this primary finding is more broadly applicable. Further, the main finding regarding constructing a subspace using rewards other than the RL reward is confounded by the fact that the other rewards are potentially related to the RL reward. For example, correctness may be related to equation density (see Appx. A.5). It is therefore unclear whether these results would generalize to an arbitrary set of rewards. This also raises the question of how one would construct this set of rewards if they were hoping to replicate this BRS result on another task; the authors don’t seem to provide much detail on the motivation/reasoning behind how this set of rewards were picked.

Communication of ideas. This paper was quite hard for me to parse. For one, notation was used without introduction (e.g., sr,is_{r, i}, J^log(1+L)\hat{J}_{log(1+L)}) or not thoroughly explained (MM as the “off-target mask” – I needed to consult a LLM to realize from context that this referred to masking out the target reward feature). Certain sentences were very hard to understand, such as “We first ask whether the direction associated with a reward before training aligns with later change outside its objective” (first sentence of 3.1). What does “direction” refer to? “Later change outside its objective” – which objective, and whose? Overall, this paper felt unnecessarily terminology- and jargon-heavy.

Questions for authors

None

Reviewer_ixh4OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies whether behavioral changes caused by reinforcement learning can be predicted before training. It proposes Behavioral Response Spectroscopy (BRS), which extracts reward-conditioned directions from sampled pre-RL responses based on reward–behavior covariance. The authors evaluate these directions by comparing them with behavioral changes after RL training across multiple language models, tasks, and reward settings. Extensive experiments and analyses are provided, including PCA comparisons, reward permutation controls, and alternative behavioral representations. The main finding is that some pre-RL behavioral variations, especially length-associated patterns, correlate with later RL-induced behavioral changes.

Critical strengths
  1. The paper studies an important and timely question: whether behavioral changes caused by RL can be anticipated from the pre-training behavior distribution. The large-scale evaluation across multiple models, tasks, and reward settings provides useful empirical evidence.
  2. The authors conduct extensive analyses with multiple controls, including PCA baselines, reward permutation tests, and alternative representations, which makes the empirical investigation relatively comprehensive.
Critical weaknesses
  1. The proposed BRS does not introduce a new optimization method or representation learning mechanism. It mainly computes reward-conditioned covariance directions in predefined behavioral feature spaces, making the claimed generality highly dependent on feature selection.
  2. The main conclusions are not robust across representations. Much of the reported transfer is explained by length-associated behavioral patterns, and after length adjustment the advantage over PCA becomes inconsistent across cohorts. This raises concerns that BRS may capture known response statistics rather than a general mechanism of RL-induced behavioral change.
  3. The evaluation does not clearly separate prediction of future RL behavioral changes from post-hoc reward–behavior correlation analysis. Since the measured directions are derived from reward-related behavioral descriptors, stronger tests with unseen objectives or independently learned representations are needed to establish predictive capability.
  4. The visual presentation is below the expected standard for a top-tier conference. Several figures contain excessive annotations, dense layouts, and unclear visual hierarchy, which makes the technical contribution harder to understand.
Questions for authors
  1. How sensitive is BRS to the choice of behavioral representation? Have the authors tested automatically learned representations or alternative feature spaces beyond the manually designed features and MiniLM anchors?
  2. Why is PCA the primary comparison baseline? Could the authors compare against stronger supervised or reward-conditioned representation learning methods?
  3. Since length-associated behavior explains a large fraction of the observed transfer, what evidence demonstrates that BRS captures information beyond response length and other simple behavioral statistics?
  4. Can the authors clarify what practical use case is enabled by BRS? Is the goal prediction, auditing, or controlling RL-induced behavioral changes?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.