- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Internal Dynamics Modulation (IDM) for RLVR and RLIF for LLMs. The authors define a Depth-Transition Score (DTS) based on cosine distances between intermediate and final-layer hidden states, with greater weight on deeper layers. DTS is used to reweight sequence-level or token-level advantages while preserving their signs. The authors evaluate IDM with GRPO and Intuitor on Qwen2.5-3B and 7B, training on the MATH dataset and evaluating on 6 benchmarks. IDM consistently improves the reported baselines, with sequence-level weighting generally performing better.
Critical strengths
- IDM improves GRPO and Ituitor on both sizes of the model and across all 6 benchmarks in 4 categories. Even though IDM is trained on the MATH dataset, the gains carry over to other types of tasks.
- IDM is a comparatively simpler method and does not introduce additional trainable parameters. The comparisons with entropy, length, random weights, and shuffled DTS help justify the choice of signal.
Critical weaknesses
- IDM scales advantages by , while keeping the KL coefficient fixed. Therefore, it is unclear whether the improvements stem entirely from DTS-based weighting or partly from a shift in the balance between the policy objective and KL regularization.
- The paper introduces DTS as a measure of how hidden representations change across layers. The main argument is that this information can help determine the strength of policy updates. However, the additional TTRL and agentic experiments use token surprisal instead of DTS. Those experiments do not establish that the proposed internal dynamics signal works in those settings. This limits the evidence for the paper’s claim of generality.
Questions for authors
- How is DTS shuffled in Figure 4? Is it shuffled across responses or within each response? Could the authors clarify the shuffle procedure and whether it preserves the weight distribution?
- The reported results are averaged over three runs. Are these independent training seeds or repeated evaluations of the same trained model? Could the authors report standard deviations for all results in Tables 2 and 3?
- In Figure 5, the mean DTS and upper percentiles (>80) seem to increase relative to the baseline. What does the stability during training mean? How does the analysis distinguish a stable signal from a gradual systematic increase?