- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies the RLVR training of LLMs with few layers instead of full parameter tuning. It firstly shows that training with some single middle layer can match the performance of full parameter tuning. Such phenomenon can be observed in math and coding tasks. Furthermore, the authors study how to exploit this observation to enable better RL training from three perspectives: (1) increasing the learning rate of the top 5 or top 10 layers that contribute the most to RL training; (2) training only the top 5 or top 10 layers that contribute the most to RL training; (3) training with the middle layers. All 3 methods yield improvements compared to the standard full parameter training, though the improvement vary across model sizes.
Critical strengths
- This paper provides a promising way for efficient RL training. Previous works [1] have shown that RL training only change a few parameters and training on these sparse parameters can match the performance of full parameter tuning. However, identifying these parameters require full parameter training. This paper empirically shows that training with the middle layers can achieve the performance of full parameter training, giving practitioners an easy-to-use default. The majority-voting analysis also offers an interesting perspective on the complementary behaviors learned by different layers.
- It also demonstrates that the trained model via one/few-layer RL training match the performance of full-parameter training in terms of both in-distribution (ID) tasks and out-of-distribution tasks. Accompanied with results across model families, datasets and algorithms, the empirical results provide sound support of their findings.
[1] Mukherjee. et. al. Reinforcement Learning Finetunes Small Subnetworks in Large Language Models. 2025.
Critical weaknesses
- While the paper provides empirical evidence to show that RL gains are mainly from the middle layers, but gives little discussion of why middle layers are the most effective. Can the authors provide some intuitions on this observation?
- The presentation of the middle layers in section 4.3 needs to be highlighted. This heuristic approach gives guidelines to the practitioners on how we can conduct RL training more efficiently and should be highlighted as one of the main takeaways.
Questions for authors
- The observation that training a single layer can match or even exceed full-parameter RL is interesting. Could the authors share more of their intuition on why joint training of all layers might not fully exploit the high-contribution layers?
- Since layer-selective training updates only a fraction of the layers, could the authors briefly report the memory or wall-clock time in RL training compared with full-parameter RL? This would be useful for practice.
5 comments
Rebuttal for Reviewer TBiZ (2/3)
W1/Q1: Why are middle layers most effective, and why does joint training not fully exploit them?
Our intuition is that middle layers are where the reasoning behavior that RL reshapes is formed, an intuition grounded in prior interpretability findings and consistent with our results; joint training does not exploit them because existing methods, lacking knowledge of this structure, train all layers uniformly.
(a) Why middle layers
Prior work. Layer heterogeneity has been studied for inference through layer ablation and interpretability [1, 2], and for supervised fine-tuning, e.g., LISA (Appendix C). In particular, [2] finds that the layers critical for mathematical reasoning are determined during pretraining and remain unchanged after post-training, including RLVR [2, §3.1, Fig. 1], and leaves updating only such layers as future work [2, §5]. However, which layers can absorb RL improvement, and why, has not been studied. To our knowledge, our work is the first to characterize this for RL post-training.
Our intuition. It builds on two lines of prior evidence. First, RL with verifiable rewards mainly re-weights reasoning paths that already exist in the base model rather than creating new ones [3, §4.1]. Second, interpretability studies find that transformer layers play depth-dependent roles [1, §4 and Table 1]:
- Shallow layers integrate local, token-level context and act as an extension of the embedding; intervening on the first layer is catastrophic [1, §3.1 and §4.1; Fig. 2a], ablating it breaks output fluency [2, §3.1], and these layers stabilize early in pretraining [4, §4.3].
- Middle layers build abstract, task-relevant features: probing accuracy for contextual semantics peaks at intermediate layers [1, §4.2; Fig. 4], intermediate representations outperform final-layer ones on downstream tasks [4, §4.1.2; Fig. 1], and model behavior can be effectively edited there, e.g., factual associations are mediated by mid-layer MLPs [5, §2.2, Fig. 2; §3].
- Deep layers convert these features into next-token predictions and sharpen the output distribution [1, §4.3–4.4; Figs. 5–6].
Combining the two, we hypothesize that re-weighting reasoning paths is most effectively expressed in the middle layers, where abstract reasoning features are formed and can still be composed by subsequent layers, rather than in shallow layers, whose token-level features underlie all downstream computation, or in deep layers, which mainly shape the output distribution.
This intuition is consistent with our results: the ranking of layer contributions is determined by the model itself, as it is unchanged under a 3× higher learning rate (Appendix D.7, Table 12) and stable across datasets and tasks (ρ = 0.76 / 0.59; §3.3, Fig. 3); high-contribution layers also improve out-of-domain capabilities (§3.2, Fig. 2), indicating changes to general reasoning behavior.
(b) Why joint training does not fully exploit them
Without knowledge of the layer-wise structure of RL adaptation, existing methods train all layers uniformly. Before our work, how RL improvement is distributed across layers had not been studied. Existing RL post-training methods therefore apply the same optimizer and the same learning rate to every layer; e.g., DAPO trains all parameters with AdamW at a single constant learning rate [6, §4.1], as do our full-parameter baselines (Appendix D.2). As a result, full-parameter RL changes all layers by nearly uniform magnitudes (≈0.5–0.8; Appendix B.2), regardless of how much each layer contributes.
Our systematic study reveals this layer structure, showing that uniform training is suboptimal and making better training possible. Layers differ dramatically in their capacity to absorb RL improvement (§3). Within joint training, boosting the learning rate of the high-contribution layers improves performance, while boosting the low-contribution layers degrades it (§4.1), indicating that updates to low-contribution layers act as perturbations rather than useful adaptation. This is also why a single well-chosen layer can exceed full-parameter training (𝒞 > 1).
We turn this structure into efficient and practical training methods, and the observation opens new directions for future research. Our three strategies all outperform full-parameter RL across model scales:
- Contribution-guided layer-wise learning rates (§4.1): e.g., 53.70 vs. 50.82 on Qwen3-1.7B-Base with the top-10 layers boosted
- Training only high-contribution layers (§4.2): e.g., 69.11 vs. 66.43 on Qwen3-8B-Base with the top-10 layers
- A profiling-free middle-layer heuristic (§4.3): e.g., 68.19 vs. 66.43 on Qwen3-8B-Base, while training 24% faster (see Q2)
More broadly, these results establish component-aware optimization as a promising direction for RL post-training, and we expect more principled layer-aware methods, such as adaptive per-layer learning rates, to yield further gains.
Rebuttal for Reviewer TBiZ (3/3)
W2: Highlighting the middle-layer heuristic
We agree that the profiling-free middle-layer heuristic is the most practical takeaway of our work. It requires no per-layer profiling, and it outperforms full-parameter RL on all three Qwen3 models (51.35 vs. 50.82 on 1.7B, 64.80 vs. 62.97 on 4B, and 68.19 vs. 66.43 on 8B). The new cost measurements below further show that it is also faster and uses less memory than full-parameter RL. In the final version, we will highlight it as a main takeaway in the Abstract and Introduction.
Q2: Memory and wall-clock time
Thank you for your question. To directly answer it, we measure wall-clock training time and peak memory on Qwen3-8B-Base under identical settings, and compare our method against both full-parameter RL (8.19B trainable parameters, 42× that of single-layer training and 8.5× that of Middle-5) and LoRA (r=64 and r=96, with 174.6M and 261.9M trainable parameters, comparable to single-layer training; and r=384, with 1047.5M trainable parameters, exceeding Middle-5). The results show that our method is more efficient than both full-parameter RL and LoRA while achieving better performance, demonstrating its practical value.
| Method | Trainable params | Time (134 steps) | Peak mem / GPU | Math Avg | 𝒞_math |
|---|---|---|---|---|---|
| Base | – | – | – | 58.0 | 0.00 |
| Full | 8.19B (100%) | 5h59min | 70.7% | 66.5 | 1.00 |
| LoRA (r=64) | 174.6M (2.13%) | 5h24min | 63.5% | 66.0 | 0.94 |
| LoRA (r=96) | 261.9M (3.20%) | 5h29min | 63.4% | 66.1 | 0.95 |
| LoRA (r=384) | 1047.5M (12.79%) | 5h37min | 65.2% | 66.3 | 0.98 |
| Layer 16 | 192.9M (2.36%) | 4h01min | 63.4% | 67.1 | 1.07 |
| Middle-5 (L15–19) | 964.7M (11.78%) | 4h33min | 64.0% | 68.2† | 1.20 |
Qwen3-8B-Base trained with GRPO on NuminaMath-CoT for 134 steps on [8]×H200 (141 GB) under identical settings. For LoRA, we use a learning rate of 1e-4, selected by sweeping over commonly used LoRA learning rates {5e-5, 1e-4, 2e-4}; full-parameter, Layer 16, and Middle-5 training use the learning rate tuned for the full-parameter baseline (5e-6). Peak memory is reported as a percentage of per-GPU memory. Math Avg is the average over MATH500, GSM8K, OlympiadBench, and AMC. †Mean over 3 independent training runs (§4.3).
Under the same number of training steps:
- vs. full-parameter RL: Middle-5 reduces wall-clock time by 24% and Layer 16 by 33%; both use less peak memory (64.0% and 63.4% vs. 70.7%) and achieve higher accuracy (68.2 and 67.1 vs. 66.5).
- vs. LoRA: at a comparable parameter count, Layer 16 trains 26–27% faster than LoRA (r=64/96) with higher accuracy (67.1 vs. 66.0–66.1) and the same peak memory. With fewer trainable parameters than LoRA (r=384) (964.7M vs. 1047.5M), Middle-5 trains 19% faster, uses less memory (64.0% vs. 65.2%), and is 1.9 points more accurate (68.2 vs. 66.3).
Together with §4.3, this makes Middle-5 a simple default for practitioners: it requires no profiling, and it is faster, more memory-efficient, and more accurate than full-parameter RL.
We thank the reviewer again for the constructive feedback, which has helped us strengthen both the practical message and the discussion of the paper.
References
[1] Lad, Lee, Gurnee, Tegmark. The Remarkable Robustness of LLMs: Stages of Inference? arXiv:2406.19384v3, 2025.
[2] Nepal et al. Layer Importance for Mathematical Reasoning is Forged in Pre-Training and Invariant after Post-Training. arXiv:2506.22638, 2025.
[3] Yue et al. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? NeurIPS 2025.
[4] Skean et al. Layer by Layer: Uncovering Hidden Representations in Language Models. ICML 2025.
[5] Meng, Bau, Andonian, Belinkov. Locating and Editing Factual Associations in GPT. NeurIPS 2022.
[6] Yu et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476, 2025.
Rebuttal for Reviewer TBiZ (1/3)
We sincerely thank the reviewer for the positive and insightful review. We are encouraged that the reviewer found our work "provides a promising way for efficient RL training," and that, whereas prior work requires full-parameter training to identify the sparse parameters that RL updates, our results show that training the middle layers can match full-parameter training, "giving practitioners an easy-to-use default." We also appreciate the recognition that layer-trained models match full-parameter training "in terms of both in-distribution (ID) tasks and out-of-distribution tasks," that the majority-voting analysis "offers an interesting perspective on the complementary behaviors learned by different layers," and that, with results "across model families, datasets and algorithms, the empirical results provide sound support of their findings."
We address the reviewer's questions below. In summary: