acceptodds
Under review as a conference paper at ICLR 2027

Modulating Language Model Reinforcement Learning with Internal Dynamics

Abstract

Reinforcement learning (RL) has become a key approach for improving large language models (LLMs), using feedback from human preferences, verifiable rewards, model signals, or environment interactions. Existing methods mainly determine whether a behavior should be reinforced or suppressed, but largely ignore how strongly the model should learn from it. We find that token representations differ substantially in how they evolve across layers: some stabilize early, while others continue to change until the final layers. These internal dynamics do not indicate whether a decision is good or bad; instead, they provide a natural signal for modulating the strength of policy updates. Based on this observation, we introduce Internal Dynamics Modulation (IDM), a lightweight operator that uses cross-layer representation dynamics to modulate the magnitude of existing policy advantages while preserving their sign. IDM is lightweight and sign-preserving, and can be inserted after an existing advantage estimator without changing the underlying reward source or policy optimization framework. Across different RL scenarios, IDM consistently improves the corresponding baselines, with larger gains on harder and longer-horizon settings. At larger model scales, internal dynamics also identify critical positions more reliably than entropy or log-probability. These results suggest that internal dynamics provide a general modulation signal complementary to the source of reinforcement feedback, offering a simple way to modulate the strength of policy updates without changing their sign.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning, large language models, policy optimization, internal representations

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ChuROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Internal Dynamics Modulation (IDM) for RLVR and RLIF for LLMs. The authors define a Depth-Transition Score (DTS) based on cosine distances between intermediate and final-layer hidden states, with greater weight on deeper layers. DTS is used to reweight sequence-level or token-level advantages while preserving their signs. The authors evaluate IDM with GRPO and Intuitor on Qwen2.5-3B and 7B, training on the MATH dataset and evaluating on 6 benchmarks. IDM consistently improves the reported baselines, with sequence-level weighting generally performing better.

Critical strengths
  • IDM improves GRPO and Ituitor on both sizes of the model and across all 6 benchmarks in 4 categories. Even though IDM is trained on the MATH dataset, the gains carry over to other types of tasks.
  • IDM is a comparatively simpler method and does not introduce additional trainable parameters. The comparisons with entropy, length, random weights, and shuffled DTS help justify the choice of signal.
Critical weaknesses
  • IDM scales advantages by f(DTS)=λ+η⋅DTSf(\mathrm{DTS})=\lambda+\eta\cdot\mathrm{DTS}, while keeping the KL coefficient β\beta fixed. Therefore, it is unclear whether the improvements stem entirely from DTS-based weighting or partly from a shift in the balance between the policy objective and KL regularization.
  • The paper introduces DTS as a measure of how hidden representations change across layers. The main argument is that this information can help determine the strength of policy updates. However, the additional TTRL and agentic experiments use token surprisal instead of DTS. Those experiments do not establish that the proposed internal dynamics signal works in those settings. This limits the evidence for the paper’s claim of generality.
Questions for authors
  • How is DTS shuffled in Figure 4? Is it shuffled across responses or within each response? Could the authors clarify the shuffle procedure and whether it preserves the weight distribution?
  • The reported results are averaged over three runs. Are these independent training seeds or repeated evaluations of the same trained model? Could the authors report standard deviations for all results in Tables 2 and 3?
  • In Figure 5, the mean DTS and upper percentiles (>80) seem to increase relative to the baseline. What does the stability during training mean? How does the analysis distinguish a stable signal from a gradual systematic increase?
Reviewer_7wYxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Internal Dynamics Modulation (IDM), which uses cross-layer hidden-state changes to reweight policy advantages. The method is evaluated on Qwen2.5-3B/7B under RLVR and RLIF, showing consistent improvements across six benchmarks. The paper also provides analyses of the proposed DTS signal and comparisons with alternative weighting strategies.

Critical strengths
  1. The method is simple and practical. It can be applied to existing RL algorithms without introducing additional trainable parameters or changing the reward function.

  2. The experimental results are encouraging, with consistent improvements across model sizes, reward settings, and benchmarks. The analyses of DTS and comparisons with alternative weighting strategies further support the effectiveness of the proposed approach.

Critical weaknesses
  1. The effect of advantage scaling could be better understood. Since the modulation changes the overall magnitude of advantages, it would be useful to compare against a constant-scaling baseline or unit-mean-normalized DTS weights.

  2. The additional test-time and agentic RL experiments use token surprisal rather than DTS. While these results demonstrate the flexibility of the reweighting framework, they provide less direct evidence for the generality of internal-dynamics-based modulation.

Questions for authors
  1. Have the authors tried normalizing DTS weights to unit mean? Would the improvements remain similar?

  2. How does surprisal-based modulation compare with DTS-based modulation in the main RLVR/RLIF settings?

  3. Could the authors report the standard deviations of the main results across the three runs?

Reviewer_SGvnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Internal Dynamics Modulation (IDM), a method that modulates existing policy advantages using a signal derived from the evolution of token representations across transformer layers.

The authors define the Depth-Transition Score (DTS) using the cosine distances between a token's residual-stream representations at intermediate layers and its final-layer representation, averaged with exponentially increasing depth weights (γ=2\gamma=2). The resulting score is transformed into a positive modulation weight,

\qquad \lambda=\eta=0.5,$$ which is multiplied by the original advantage. This preserves the sign of each advantage while modifying its magnitude. The paper considers both a sequence-level variant, which uses the mean DTS across a response, and a token-level variant. The method is applied on top of GRPO for reinforcement learning with verifiable rewards (RLVR) and Intuitor for reinforcement learning from internal feedback (RLIF). The experiments use Qwen2.5-3B and Qwen2.5-7B, trained for one epoch on the MATH training set and evaluated on MATH-500, GSM8K, LiveCodeBench (LCB), CRUXEval, MMLU-Pro, and AlpacaEval. The authors report averages over three runs. On MATH-500, the reported improvements are 3.4/4.5 percentage points under RLVR and 4.1/5.0 percentage points under RLIF for the 3B/7B models respectively.
Critical strengths
  1. The authors decouple the direction and magnitude of the advantage. From an engineering perspective, this approach is valuable (requiring only one additional forward pass, incurring an overhead of approximately 20%, and has no new parameters).
  2. The authors conduct more rigorous self-validation than is typical for similar papers: they did four types of control comparisons (random, shuffled DTS, sequence length, and entropy), monitor the stability of DTS quantiles during training, compare soft versus hard aggregation, and validate their method across two baselines RLVR and RLIF.
Critical weaknesses
  1. The most important baseline is missing: constant weighting. Based on the paper's own reported statistics, the mean DTS across training tokens is approximately 0.09, the 95th percentile is approximately 0.26, and the maximum is around 0.4 (Figure 5). the modulation weight:

f(DTS)=0.5+0.5⋅DTSf(\mathrm{DTS})=0.5+0.5\cdot\mathrm{DTS}

falls between approximately 0.5 and 0.6 for the vast majority of tokens, reaching only around 0.7 at the extremes.

This raises a concern about the mechanism underlying the reported improvements. The first-order effect of IDM is to multiply nearly all advantages by approximately 0.55, while the DTS-dependent variation contributes only a secondary effect of roughly 10–20% in relative magnitude. Under the paper's setting where all algorithms share the same hyperparameters, this global scaling effectively reduces the relative weight of the policy-gradient term compared with the KL regularization term by approximately a factor of 1.8. In particular, the effective KL coefficient changes from

\quad\text{to approximately}\quad \beta_{\mathrm{eff}} =\frac{0.005}{0.55} \approx 0.009.$$ The scaling may also change the interaction between gradient magnitudes and Adam's $\epsilon$ parameter. However, the GRPO baseline is not tuned under this effective scaling. none of the four ablations in Figure 4 includes the most basic control: $$f(\mathrm{DTS})\equiv \mathbb{E}[f(\mathrm{DTS})].$$ A related concern is that the paper does not report the actual distribution of the sequence-level weights $f(\mathrm{DTS}_i)$. If these weights vary only between approximately 0.5 and 0.6 across responses, shuffling DTS values would change the relative advantage magnitudes by at most roughly 20%. Yet the paper reports that shuffled DTS changes the improvement from approximately +2.29 points to a negative gain. 2. The claimed interpretation of IDM is inconsistent with its likely operating mechanism, and the cross-regime experiments use a different signal: The paper repeatedly emphasizes that internal dynamics are not a reward signal and that large changes across transformer layers do not necessarily imply correctness (Sections 1 and 4). However, the main empirical motivation in Section 3.2 is precisely that correct responses have significantly higher mean DTS, with Cohen's $d=0.72$. These claims are difficult to tell without a more careful distinction between DTS as a direct reward and DTS as a reward-correlated proxy. If sequence-level DTS is positively correlated with reward, then the modulation weights $f(\mathrm{DTS}_i)$ will also tend to correlate with the signs of the corresponding advantages $A_i$. So IDM-Seq will tend to assign larger weights to positive advantages and smaller weights to negative advantages. In particular, even when the original group-normalized advantages satisf y $\sum_i A_i=0,$ the weighted advantages will generally satisfy $\sum_i A_i f(\mathrm{DTS}_i)\neq 0.$. This is not reward-independent modulation of advantage magnitudes. Rather, it introduces asymmetric advantage reweighting, which is closely related to existing approaches that treat positive and negative samples differently, including clip-higher mechanisms and positive/negative sample reweighting methods such as DAPO and the PSR/NSR line of work. These closely related approaches are not discussed or cited. The paper needs to distinguish whether the improvements come from internal representation dynamics specifically or from an implicit preference for high-reward trajectories. 3. Another concern is that Table 1 suggests DTS may largely function as an operator detector: operators constitute 65% of high-DTS tokens, compared with only 11% in the reference distribution. Under this interpretation, correct responses have higher DTS may simply be a compositional effect: correct mathematical solutions may contain more operators or other computational tokens. The paper does not include a token-type-controlled weighting baseline that could separate this explanation from the proposed internal-dynamics mechanism. 4. Finally, the abstract claims 'larger gains on harder and longer-horizon settings' but the paper does not provide difficulty varies analyses to show that improvements increase with task difficulty.
Questions for authors

1.Consider add a constant-weight baseline defined by f(DTS)≡E[f(DTS)],f(\mathrm{DTS}) \equiv \mathbb{E} [f(\mathrm{DTS})], for both the sequence-level and token-level variants. Additionally, please report results for the GRPO baseline after adjusting either the KL coefficient β\beta or the learning rate to account for the approximately 0.55 scaling of the policy-gradient term.

  1. Please report the empirical distributions of the sequence-level weights f(DTSi)f(\mathrm{DTS}_i) and token-level weights f(DTS(t))f(\mathrm{DTS}(t)) during training, preferably using quantiles or histograms. Also, why does Figure 2 show DTS means in the range of approximately 0.3–0.6, while Figure 5 reports a mean of approximately 0.09? Please clarify whether these figures use different definitions, normalization procedures, or aggregation levels.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.