acceptodds
Under review as a conference paper at ICLR 2027

Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training

Abstract

Reinforcement learning (RL) has become a central component of post-training large language models (LLMs), yet little is understood about how RL adaptation is distributed across transformer layers. Existing approaches typically update all model parameters uniformly, implicitly assuming that every layer contributes similarly to the gains obtained during RL post-training. In this work, we challenge this assumption through a systematic layer-wise study of RL training. Surprisingly, we find that training a single transformer layer can recover most of the gains achieved by full-parameter RL training, and in some cases even surpass it. To quantify this phenomenon, we introduce the quantity layer contribution, which measures the fraction of full RL improvement recovered by training a layer in isolation. Across seven models spanning two model families (Qwen3, Qwen2.5), three RL algorithms (GRPO, GiGPO, Dr. GRPO), and three task domains (mathematical reasoning, code generation, and agentic decision-making), we observe a remarkably stable pattern: RL gains are highly concentrated in a small subset of transformer layers, and in many cases in a single layer. More strikingly, the same structural pattern consistently emerges: high-contribution layers concentrate in the middle of the transformer stack, while layers near the input and output ends contribute substantially less. The resulting layer rankings remain strongly correlated across datasets, tasks, model families, and RL algorithms. Our findings have two important implications. First, they reveal a previously unrecognized structural property of RL post-training: most RL gains are concentrated in a small subset of transformer layers rather than being uniformly distributed throughout the network. Second, they suggest new opportunities for improving RL training. Guided by the above observation, we develop simple layer-aware training strategies that consistently outperform standard full-parameter RL training, while ensembles of layer-specialized models provide additional gains through complementary behaviors. Together, our results provide new insights into how RL modifies large language models and suggest a new perspective for understanding and improving RL post-training.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TBiZOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the RLVR training of LLMs with few layers instead of full parameter tuning. It firstly shows that training with some single middle layer can match the performance of full parameter tuning. Such phenomenon can be observed in math and coding tasks. Furthermore, the authors study how to exploit this observation to enable better RL training from three perspectives: (1) increasing the learning rate of the top 5 or top 10 layers that contribute the most to RL training; (2) training only the top 5 or top 10 layers that contribute the most to RL training; (3) training with the middle layers. All 3 methods yield improvements compared to the standard full parameter training, though the improvement vary across model sizes.

Critical strengths
  • This paper provides a promising way for efficient RL training. Previous works [1] have shown that RL training only change a few parameters and training on these sparse parameters can match the performance of full parameter tuning. However, identifying these parameters require full parameter training. This paper empirically shows that training with the middle layers can achieve the performance of full parameter training, giving practitioners an easy-to-use default. The majority-voting analysis also offers an interesting perspective on the complementary behaviors learned by different layers.
  • It also demonstrates that the trained model via one/few-layer RL training match the performance of full-parameter training in terms of both in-distribution (ID) tasks and out-of-distribution tasks. Accompanied with results across model families, datasets and algorithms, the empirical results provide sound support of their findings.

[1] Mukherjee. et. al. Reinforcement Learning Finetunes Small Subnetworks in Large Language Models. 2025.

Critical weaknesses
  • While the paper provides empirical evidence to show that RL gains are mainly from the middle layers, but gives little discussion of why middle layers are the most effective. Can the authors provide some intuitions on this observation?
  • The presentation of the middle layers in section 4.3 needs to be highlighted. This heuristic approach gives guidelines to the practitioners on how we can conduct RL training more efficiently and should be highlighted as one of the main takeaways.
Questions for authors
  • The observation that training a single layer can match or even exceed full-parameter RL is interesting. Could the authors share more of their intuition on why joint training of all layers might not fully exploit the high-contribution layers?
  • Since layer-selective training updates only a fraction of the layers, could the authors briefly report the memory or wall-clock time in RL training compared with full-parameter RL? This would be useful for practice.
5 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer TBiZ (1/3)

We sincerely thank the reviewer for the positive and insightful review. We are encouraged that the reviewer found our work "provides a promising way for efficient RL training," and that, whereas prior work requires full-parameter training to identify the sparse parameters that RL updates, our results show that training the middle layers can match full-parameter training, "giving practitioners an easy-to-use default." We also appreciate the recognition that layer-trained models match full-parameter training "in terms of both in-distribution (ID) tasks and out-of-distribution tasks," that the majority-voting analysis "offers an interesting perspective on the complementary behaviors learned by different layers," and that, with results "across model families, datasets and algorithms, the empirical results provide sound support of their findings."

We address the reviewer's questions below. In summary:

  • Intuition (W1/Q1): Our intuition is that middle layers are most effective because they are where the reasoning behavior that RL reshapes is formed, an intuition grounded in prior interpretability findings and consistent with our results. Joint training does not exploit them because, without knowledge of this structure, existing methods train all layers uniformly. Our findings open a direction toward component-aware optimization for RL post-training.
  • Highlighting the heuristic (W2): We agree that the profiling-free middle-layer heuristic is a main takeaway and will highlight it in the final version.
  • Cost (Q2): Middle-5 trains 24% faster than full-parameter RL, which updates 8.5× more parameters, with lower peak memory and higher accuracy. It is also 19% faster and 1.9 points more accurate than a LoRA with a bit more trainable parameters (r=384), and a single middle layer, at a parameter count comparable to LoRA, trains 26–27% faster with higher accuracy.
AuthorsOpenReview ↗

Rebuttal for Reviewer TBiZ (2/3)

W1/Q1: Why are middle layers most effective, and why does joint training not fully exploit them?

Our intuition is that middle layers are where the reasoning behavior that RL reshapes is formed, an intuition grounded in prior interpretability findings and consistent with our results; joint training does not exploit them because existing methods, lacking knowledge of this structure, train all layers uniformly.

(a) Why middle layers

Prior work. Layer heterogeneity has been studied for inference through layer ablation and interpretability [1, 2], and for supervised fine-tuning, e.g., LISA (Appendix C). In particular, [2] finds that the layers critical for mathematical reasoning are determined during pretraining and remain unchanged after post-training, including RLVR [2, §3.1, Fig. 1], and leaves updating only such layers as future work [2, §5]. However, which layers can absorb RL improvement, and why, has not been studied. To our knowledge, our work is the first to characterize this for RL post-training.

Our intuition. It builds on two lines of prior evidence. First, RL with verifiable rewards mainly re-weights reasoning paths that already exist in the base model rather than creating new ones [3, §4.1]. Second, interpretability studies find that transformer layers play depth-dependent roles [1, §4 and Table 1]:

  • Shallow layers integrate local, token-level context and act as an extension of the embedding; intervening on the first layer is catastrophic [1, §3.1 and §4.1; Fig. 2a], ablating it breaks output fluency [2, §3.1], and these layers stabilize early in pretraining [4, §4.3].
  • Middle layers build abstract, task-relevant features: probing accuracy for contextual semantics peaks at intermediate layers [1, §4.2; Fig. 4], intermediate representations outperform final-layer ones on downstream tasks [4, §4.1.2; Fig. 1], and model behavior can be effectively edited there, e.g., factual associations are mediated by mid-layer MLPs [5, §2.2, Fig. 2; §3].
  • Deep layers convert these features into next-token predictions and sharpen the output distribution [1, §4.3–4.4; Figs. 5–6].

Combining the two, we hypothesize that re-weighting reasoning paths is most effectively expressed in the middle layers, where abstract reasoning features are formed and can still be composed by subsequent layers, rather than in shallow layers, whose token-level features underlie all downstream computation, or in deep layers, which mainly shape the output distribution.

This intuition is consistent with our results: the ranking of layer contributions is determined by the model itself, as it is unchanged under a 3× higher learning rate (Appendix D.7, Table 12) and stable across datasets and tasks (ρ = 0.76 / 0.59; §3.3, Fig. 3); high-contribution layers also improve out-of-domain capabilities (§3.2, Fig. 2), indicating changes to general reasoning behavior.

(b) Why joint training does not fully exploit them

Without knowledge of the layer-wise structure of RL adaptation, existing methods train all layers uniformly. Before our work, how RL improvement is distributed across layers had not been studied. Existing RL post-training methods therefore apply the same optimizer and the same learning rate to every layer; e.g., DAPO trains all parameters with AdamW at a single constant learning rate [6, §4.1], as do our full-parameter baselines (Appendix D.2). As a result, full-parameter RL changes all layers by nearly uniform magnitudes (≈0.5–0.8; Appendix B.2), regardless of how much each layer contributes.

Our systematic study reveals this layer structure, showing that uniform training is suboptimal and making better training possible. Layers differ dramatically in their capacity to absorb RL improvement (§3). Within joint training, boosting the learning rate of the high-contribution layers improves performance, while boosting the low-contribution layers degrades it (§4.1), indicating that updates to low-contribution layers act as perturbations rather than useful adaptation. This is also why a single well-chosen layer can exceed full-parameter training (𝒞 > 1).

We turn this structure into efficient and practical training methods, and the observation opens new directions for future research. Our three strategies all outperform full-parameter RL across model scales:

  • Contribution-guided layer-wise learning rates (§4.1): e.g., 53.70 vs. 50.82 on Qwen3-1.7B-Base with the top-10 layers boosted
  • Training only high-contribution layers (§4.2): e.g., 69.11 vs. 66.43 on Qwen3-8B-Base with the top-10 layers
  • A profiling-free middle-layer heuristic (§4.3): e.g., 68.19 vs. 66.43 on Qwen3-8B-Base, while training 24% faster (see Q2)

More broadly, these results establish component-aware optimization as a promising direction for RL post-training, and we expect more principled layer-aware methods, such as adaptive per-layer learning rates, to yield further gains.

AuthorsOpenReview ↗

Rebuttal for Reviewer TBiZ (3/3)

W2: Highlighting the middle-layer heuristic

We agree that the profiling-free middle-layer heuristic is the most practical takeaway of our work. It requires no per-layer profiling, and it outperforms full-parameter RL on all three Qwen3 models (51.35 vs. 50.82 on 1.7B, 64.80 vs. 62.97 on 4B, and 68.19 vs. 66.43 on 8B). The new cost measurements below further show that it is also faster and uses less memory than full-parameter RL. In the final version, we will highlight it as a main takeaway in the Abstract and Introduction.


Q2: Memory and wall-clock time

Thank you for your question. To directly answer it, we measure wall-clock training time and peak memory on Qwen3-8B-Base under identical settings, and compare our method against both full-parameter RL (8.19B trainable parameters, 42× that of single-layer training and 8.5× that of Middle-5) and LoRA (r=64 and r=96, with 174.6M and 261.9M trainable parameters, comparable to single-layer training; and r=384, with 1047.5M trainable parameters, exceeding Middle-5). The results show that our method is more efficient than both full-parameter RL and LoRA while achieving better performance, demonstrating its practical value.

MethodTrainable paramsTime (134 steps)Peak mem / GPUMath Avg𝒞_math
Base–––58.00.00
Full8.19B (100%)5h59min70.7%66.51.00
LoRA (r=64)174.6M (2.13%)5h24min63.5%66.00.94
LoRA (r=96)261.9M (3.20%)5h29min63.4%66.10.95
LoRA (r=384)1047.5M (12.79%)5h37min65.2%66.30.98
Layer 16192.9M (2.36%)4h01min63.4%67.11.07
Middle-5 (L15–19)964.7M (11.78%)4h33min64.0%68.2†1.20

Qwen3-8B-Base trained with GRPO on NuminaMath-CoT for 134 steps on [8]×H200 (141 GB) under identical settings. For LoRA, we use a learning rate of 1e-4, selected by sweeping over commonly used LoRA learning rates {5e-5, 1e-4, 2e-4}; full-parameter, Layer 16, and Middle-5 training use the learning rate tuned for the full-parameter baseline (5e-6). Peak memory is reported as a percentage of per-GPU memory. Math Avg is the average over MATH500, GSM8K, OlympiadBench, and AMC. †Mean over 3 independent training runs (§4.3).

Under the same number of training steps:

  • vs. full-parameter RL: Middle-5 reduces wall-clock time by 24% and Layer 16 by 33%; both use less peak memory (64.0% and 63.4% vs. 70.7%) and achieve higher accuracy (68.2 and 67.1 vs. 66.5).
  • vs. LoRA: at a comparable parameter count, Layer 16 trains 26–27% faster than LoRA (r=64/96) with higher accuracy (67.1 vs. 66.0–66.1) and the same peak memory. With fewer trainable parameters than LoRA (r=384) (964.7M vs. 1047.5M), Middle-5 trains 19% faster, uses less memory (64.0% vs. 65.2%), and is 1.9 points more accurate (68.2 vs. 66.3).

Together with §4.3, this makes Middle-5 a simple default for practitioners: it requires no profiling, and it is faster, more memory-efficient, and more accurate than full-parameter RL.


We thank the reviewer again for the constructive feedback, which has helped us strengthen both the practical message and the discussion of the paper.

References

[1] Lad, Lee, Gurnee, Tegmark. The Remarkable Robustness of LLMs: Stages of Inference? arXiv:2406.19384v3, 2025.

[2] Nepal et al. Layer Importance for Mathematical Reasoning is Forged in Pre-Training and Invariant after Post-Training. arXiv:2506.22638, 2025.

[3] Yue et al. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? NeurIPS 2025.

[4] Skean et al. Layer by Layer: Uncovering Hidden Representations in Language Models. ICML 2025.

[5] Meng, Bau, Andonian, Belinkov. Locating and Editing Factual Associations in GPT. NeurIPS 2022.

[6] Yu et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476, 2025.

Reviewer_NPnzOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how individual Transformer layers can adapt during RL post-training. The method trains one complete layer and freezes all other parameters. The authors define “layer contribution” as the fraction of the full-parameter RL gain recovered by training that layer alone. They evaluate this approach across several Qwen-based models, three RL algorithms, and math, code, and interactive tasks. Some single-layer runs match or exceed full-parameter training, and middle layers often perform well. The authors also propose training strategies based on layer rankings and a fixed middle-layer strategy that requires no layer search.

Critical strengths
  1. The layer-by-layer experiments provide useful empirical observations. The authors study single-layer adaptation across several model sizes and RL settings. They also compare layer rankings across datasets and tasks. Some individual layers recover most of the gain from full-parameter training. These results show that the choice of trainable layer can have a substantial effect on adaptation.

  2. The paper turns these observations into a usable training strategy. The middle-five-layer strategy in Section 4.3 requires no layer search and exceeds the corresponding full-parameter baseline on all three Qwen3 models. For example, the average math score for the 8B model increases from 66.43 to 68.19. This result partly addresses the cost of layer selection.

Critical weaknesses

1. The practical cost–performance benefit is not sufficiently established. Updating one layer does not mean computing only one layer. Rollouts and forward passes still use the full model. During training, gradients must also pass through the frozen layers after the selected layer. Freezing parameters can reduce some computation and optimizer-state memory. However, the paper does not directly compare total training time, peak GPU memory, or computational cost. The reduction in trainable parameters alone is therefore not enough to establish a practical efficiency benefit.

In addition, finding the best single layer or selecting layers by contribution ranking requires multiple independent training runs. The fixed middle-five-layer strategy avoids this search. However, it still lacks a cost–performance comparison with full-parameter training and parameter-efficient methods such as LoRA. The gains over full-parameter training in the main single-layer experiments are also limited. It is therefore unclear under which resource constraints a practitioner should prefer this approach.

2. The “layer contribution” measure provides limited support for the paper’s broader interpretation. This measure shows how much performance gain a layer can recover when trained independently while all other parameters remain frozen. It does not directly measure that layer’s actual contribution during full-parameter training. Restricting updates to one layer may lead to a different adaptation process. The frozen layers also remain part of the model’s computation.

The results therefore support the claim that training certain layers alone can be sufficient to obtain similar performance. They do not directly establish that the gains from full-parameter RL training are concentrated in those layers. Cross-task ranking correlations and parameter-change analyses provide useful clues, but they do not fully connect these two claims. This gap limits the paper’s explanation of how RL post-training works.

Questions for authors
  1. After including layer-selection costs where applicable, how do single-layer training and the fixed middle-five-layer strategy compare with full-parameter training and methods such as LoRA in total training time, peak GPU memory, and final performance? These comparisons would help establish the practical value of the approach.

  2. Should “layer contribution” be read as a layer’s ability to adapt independently, or as its actual contribution during full-parameter training? Please distinguish these interpretations and identify the direct evidence for the latter.

11 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer NPnz (1/3)

We thank the reviewer for the thoughtful and constructive review. We are glad the reviewer found that our layer-by-layer experiments "provide useful empirical observations" across "several model sizes and RL settings," that "some individual layers recover most of the gain from full-parameter training," and that "the choice of trainable layer can have a substantial effect on adaptation." We also appreciate the recognition that the paper "turns these observations into a usable training strategy": the middle-five-layer strategy "requires no layer search and exceeds the corresponding full-parameter baseline on all three Qwen3 models."

The reviewer's two main points, practical cost–performance and the interpretation of layer contribution, are both addressed below with new results and evidence from the paper. In summary:

  • Practical value (W1/Q1): We add a direct wall-clock and memory comparison on Qwen3-8B-Base against full-parameter RL and LoRA (r=64/96/384, with a swept learning rate). Middle-5 trains 24% faster than full-parameter RL and 16–19% faster than LoRA, and achieves higher accuracy than both, even against a LoRA with more trainable parameters (r=384: 68.2 vs. 66.3). At a parameter count comparable to LoRA (r=64/96), a single middle layer (Layer 16) trains 26–27% faster with higher accuracy and comparable memory.
  • Interpretation (W2/Q2): We agree that 𝒞(k) by itself measures standalone adaptation capacity. Beyond 𝒞(k), a controlled experiment within full-parameter training (§4.1) shows that the same layers play a disproportionate role in full-parameter RL: simply increasing the learning rate of these high-contribution layers improves performance, whereas doing so for the low-contribution layers degrades it. We will make this interpretation explicit in the revised version.
AuthorsOpenReview ↗

Rebuttal for Reviewer NPnz (2/3)

W1/Q1: Practical cost–performance benefit

Thank you for your question. To directly answer it, we measure wall-clock training time and peak memory under identical settings, and compare our method against both full-parameter RL (8.19B trainable parameters, 42× that of single-layer training and 8.5× that of Middle-5) and LoRA (r=64 and r=96, with 174.6M and 261.9M trainable parameters, comparable to single-layer training; and r=384, with 1047.5M trainable parameters, exceeding Middle-5). The results show that our method is more efficient than both full-parameter RL and LoRA while achieving better performance, demonstrating its practical value.

MethodTrainable paramsTime (134 steps)Peak mem / GPUMath Avg𝒞_math
Base–––58.00.00
Full8.19B (100%)5h59min70.7%66.51.00
LoRA (r=64)174.6M (2.13%)5h24min63.5%66.00.94
LoRA (r=96)261.9M (3.20%)5h29min63.4%66.10.95
LoRA (r=384)1047.5M (12.79%)5h37min65.2%66.30.98
Layer 16192.9M (2.36%)4h01min63.4%67.11.07
Middle-5 (L15–19)964.7M (11.78%)4h33min64.0%68.2†1.20

Table R1. Qwen3-8B-Base trained with GRPO on NuminaMath-CoT for 134 steps on 8×H200 under identical settings. For LoRA, we use a learning rate of 1e-4, selected by sweeping over commonly used LoRA learning rates {5e-5, 1e-4, 2e-4}; full-parameter, Layer 16, and Middle-5 training use the learning rate tuned for the full-parameter baseline (5e-6). Peak memory is reported as a percentage of per-GPU memory. Math Avg is the average over MATH500, GSM8K, OlympiadBench, and AMC. †Mean over 3 independent training runs (§4.3).

Our method is 24–33% faster than full-parameter RL and 16–28% faster than LoRA, with higher accuracy than both. Under the same number of training steps, Layer 16 reduces wall-clock time by 33% relative to full-parameter RL and 26–28% relative to LoRA, and Middle-5 by 24% and 16–19%, respectively. Both also achieve higher math accuracy than full-parameter RL and LoRA (67.1 and 68.2 vs. 66.5 and 66.0–66.3). Even with more trainable parameters than Middle-5 (1047.5M vs. 964.7M), LoRA (r=384) is 1.9 points less accurate, 19% slower, and uses more memory (65.2% vs. 64.0%); likewise, a LoRA with more trainable parameters than Layer 16 (r=96: 261.9M vs. 192.9M) still underperforms it, indicating that which layers are adapted matters more than adapter capacity. Both methods use peak memory comparable to or lower than LoRA (63.4–64.0% vs. 63.4–65.2%) and lower than full-parameter RL (70.7%).

The layer-selection cost can be avoided, and layer rankings can be reused across datasets and tasks. Middle-5 (§4.3) selects layers purely by position and requires no additional training runs. Identifying the best single layer does require additional runs, but this cost is incurred only once per model: layer rankings are strongly correlated across datasets (Spearman ρ = 0.76) and even across tasks (ρ = 0.59) (§3.3), so a layer selected on one dataset can be reused for other datasets and tasks.

The gains are substantial and consistent across model scales. The single-layer experiments in §3 show that training only ~2% of the parameters is sufficient to recover the full RL gain, with several layers even exceeding full-parameter RL. Building on this finding, all three layer-aware strategies in §4 outperform full-parameter RL at every model scale:

ModelFull-param RLBoost B10 (§4.1)Only B5 (§4.2)Middle-5 (§4.3, no profiling)
Qwen3-1.7B-Base50.82 ± 0.4053.70 ± 0.4051.53 ± 0.2451.35 ± 0.28
Qwen3-4B-Base62.97 ± 0.7864.42 ± 0.1365.87 ± 0.7064.80 ± 0.30
Qwen3-8B-Base66.43 ± 0.4067.42 ± 0.4068.97 ± 0.3168.19 ± 0.62

All values are mean ± std over 3 independent training runs.

The gains are substantial relative to the total RL improvement: up to +2.88 on 1.7B (Boost B10, 43% of the total RL gain), +2.90 on 4B (Only B5, 27%), and +2.68 on 8B (Only B10: 69.11 ± 0.10, 32%). Even the profiling-free Middle-5 heuristic improves over full-parameter RL at all three scales, with larger gains on larger models (+0.53 on 1.7B vs. +1.83 and +1.76 on 4B and 8B), while also training 24% faster on Qwen3-8B-Base (Table R1).

Our approach is simple to use in practice, and the layer structure we observe opens room for further algorithm design. Our results point to a simple default for practitioners: train only the middle five layers. Compared with full-parameter RL, it is faster, uses less memory, and achieves higher accuracy; compared with LoRA, it is faster and more accurate at comparable or lower memory. Moreover, the strong and stable layer structure suggests that RL algorithms with layer-specific treatment, such as layer-wise learning rates or optimizers, are a promising direction. We will include Table R1 in the appendix of the revised paper.

AuthorsOpenReview ↗

Rebuttal for Reviewer NPnz (3/3)

W2/Q2: Interpretation of "layer contribution"

Thank you for the question. Below, we distinguish the two interpretations and provide evidence that high-contribution layers also play an important role in full-parameter RL.

𝒞(k) measures a layer's standalone adaptation capacity, consistent with how the paper defines and uses it. We agree with this distinction, and this is exactly what our experiments are designed to measure. By itself, 𝒞(k) measures how much of the full RL gain can be recovered by training a layer alone, rather than directly measuring the layer's share of the gain within full-parameter training. This matches how §2.2 motivates the metric ("isolating each layer's ability to absorb RL-induced improvement") and how our Conclusion states the finding ("layers differ dramatically in their capacity to capture RL-induced improvement"). We will make it explicit throughout the revised paper.

A controlled experiment within full-parameter training shows that high-contribution layers also play a disproportionate role in full-parameter RL. Beyond 𝒞(k), the paper also provides evidence on the second interpretation through a controlled experiment within full-parameter training (§4.1). Keeping all layers trainable, we increase the learning rate of only the top-k layers ranked by 𝒞(k). This improves over standard full-parameter RL on all three Qwen3 models (Boost B10 column in the table under W1), whereas applying the same increase to the lowest-ranked layers (Boost Wk) degrades performance. Since all layers are trained jointly in both conditions, this asymmetry shows that the layers identified by standalone training also play a disproportionate role during full-parameter RL. We will add this evidence to the revised version, and we thank the reviewer for prompting this clarification.


In conclusion, we hope the new cost–performance comparison and this clarification address the reviewer's concerns. If so, we would be grateful if the reviewer would consider updating the score, and we are happy to discuss any remaining questions.

Reviewer_SfffOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates layer-wise heterogeneity in reinforcement learning post-training of large language models. Starting from the same initial model, the authors independently update individual transformer layers while freezing all other parameters and compare the resulting improvements with full-parameter RL. They define “layer contribution” as the fraction of the full-parameter improvement recovered by single-layer training.

The experiments cover seven Qwen-related models, three GRPO-related algorithms, and mathematical, coding, and interactive agent tasks. In the studied settings, some single-layer models match full-parameter training, and layers in the middle region are generally more effective. Layer rankings within a model are positively correlated across training datasets and tasks. The authors further explore contribution-guided learning-rate adjustment, selective training of high-ranked layers, and a profiling-free middle-layer heuristic, obtaining improved mathematical performance on three Qwen3 models. Additional experiments examine prediction complementarity and parameter-change magnitudes.

The paper is primarily an empirical study of RL adaptation within restricted parameter subspaces rather than a new RL optimization algorithm.

Critical strengths
  1. A clear and consequential empirical finding supported by substantial experimentation.

The observation that single-layer training can approach full-parameter RL is noteworthy. Full layer scans on three Qwen3 models and a Qwen2.5 math model, together with additional settings, make the evidence more substantial than a demonstration on a few hand-picked layers. The finding is directly relevant to parameter-efficient post-training and understanding adaptation capacity.

  1. Translation of the observation into testable training strategies with useful controls.

The authors evaluate learning-rate boosts for high-ranked layers, selective training, and a middle-five-layer heuristic rather than stopping at descriptive rankings. Contrasts with low-ranked layers strengthen the evidence. The profiling-free heuristic is particularly interesting because it avoids the expensive preliminary search required by exhaustive layer profiling.

Critical weaknesses

I apologize for submitting incorrect review comments earlier. Please refer to the corrected comments below.

1. Lack of mechanistic explanation. The paper clearly characterizes the contribution of middle layers but provides little explanation of why they are particularly important. Without a testable mechanistic hypothesis, it remains unclear whether this finding generalizes to larger models, MoE architectures, or multimodal models.

2. Missing PEFT baselines and optimization analysis. The paper lacks direct comparisons with established PEFT methods, making it difficult to assess the relative advantages of middle-layer training. Moreover, the observation that single-layer training can outperform full-parameter training (C > 1) suggests potential optimization interference, which deserves further investigation.

Questions for authors

1.Mechanism. The paper establishes what happens but not why. Have the authors tested any mechanistic hypothesis—e.g., whether the high contribution of middle layers correlates with measurable properties of pretrained weights such as effective rank, representational plasticity, or gradient covariance structure? Even preliminary correlational evidence on one or two models would materially affect how broadly the finding can be trusted.

2.Meaning of C > 1. Single-layer training sometimes outperforms full-parameter RL (C > 1), which strongly suggests inter-layer interference or gradient dilution during joint training. Have the authors analyzed gradient norms or update-direction conflicts across layers under full-parameter training?

3.Comparison with PEFT. “Training only a few layers” is essentially a PEFT strategy, yet the paper does not compare against LoRA/DoRA or LISA. Can the authors add a comparison between “training only the middle k layers” and LoRA with matched parameter count under the same setting? This is essential for judging the practical value of the proposed strategies.

6 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer Sfff (1/4)

We thank the reviewer for the detailed review and the thoughtful questions. We are encouraged that the reviewer found our work to present "a clear and consequential empirical finding supported by substantial experimentation," that the observation that single-layer training can approach full-parameter RL "is noteworthy," and that our full layer scans "make the evidence more substantial than a demonstration on a few hand-picked layers." We also appreciate the recognition that the paper translates the observation "into testable training strategies with useful controls," that "contrasts with low-ranked layers strengthen the evidence," and that the profiling-free heuristic "is particularly interesting because it avoids the expensive preliminary search."

Regarding the Critical Weaknesses section. We respectfully note that this section appears to refer to a different submission: it discusses SPEGen, per-label threshold calibration, solvent F1 and ratio MAE, cartridge/solvent/ratio prediction, and pollutant-level aggregation, none of which appear in our paper. We would be grateful if the reviewer could double-check this section and, if appropriate, reconsider the rating in light of our actual submission. Below, we answer each of the reviewer's three questions, which are directly relevant to our work.

In summary of our responses:

  • Mechanism (Q1): Across Qwen3-1.7B/4B/8B-Base, layer contribution is predicted by the effective rank of each layer's output representations: layers with a lower effective rank contribute more under RL (ρ = −0.66 / −0.60 / −0.53, all p < 0.001). Other weight and inference-time analyses do not recover 𝒞, which makes our direct RL measurements essential.
  • Meaning of 𝒞 > 1 (Q2): At the initial policy, the GRPO gradient is concentrated in high-contribution layers and is more consistent across prompts there, yet full-parameter RL updates all layers by nearly the same amount. Joint training therefore dilutes its update across layers with weak gradient signal, which explains why a single well-chosen layer can exceed it.
  • Comparison with PEFT (Q3): Training only the middle layers outperforms LoRA at matched parameter counts: Middle-5 beats LoRA with more trainable parameters (r=384) by 1.9 points while training 19% faster, and a single middle layer beats LoRA with r=64/96 while training 26–27% faster.
AuthorsOpenReview ↗

Rebuttal for Reviewer Sfff (2/4)

Q1: Mechanism

Thank you for the question. Across all three Qwen3 models, layer contribution is predicted by the effective rank of each layer's output representations: layers whose representations have a lower effective rank contribute more. This shows that the middle-layer structure is rooted in the pretrained model itself, which supports the generality of our finding.

Background. Our intuition is that middle layers are where the reasoning behavior that RL reshapes is formed: middle layers are where abstract, task-relevant features are built [1, §4.2], and RL with verifiable rewards mainly re-weights existing reasoning paths [2, §4.1]. The paper already rules out one simple explanation, namely that high-contribution layers are those whose parameters change the most (Appendix B.2, Fig. 9), and shows that contribution is determined by the model itself: it is unchanged under a 3× higher learning rate and stable across datasets and tasks (Appendix D.7; §3.3).

New correlational analysis: layers whose output representations have a lower effective rank contribute more. Following the reviewer's suggestion, we test whether 𝒞 is predicted by measurable properties of the pretrained model, using the base models of Qwen3-1.7B/4B/8B (i.e., the initial policies of RL). We run each base model on 128 training prompts from NuminaMath-CoT, formatted with the same chat template as in RL training and truncated to 512 tokens. For each layer ℓ, we collect the layer's output hidden states and randomly sample 32 token positions per prompt, giving a matrix H_ℓ ∈ ℝ^{4096×d}, where d is the hidden size. After centering H_ℓ, we compute its singular values σ_1, …, σ_r, normalize them to p_i = σ_i / Σ_j σ_j, and take the entropy-based effective rank [3]:

erank(H_ℓ) = exp(−Σ_i p_i log p_i).

The effective rank ranges from 1, when all representations lie along a single direction, to r, when the variance is spread evenly across all r directions. A lower value therefore means that the layer's representations are concentrated in fewer dominant directions. We then compute the Spearman correlation between erank(H_ℓ) and 𝒞(ℓ) across all layers of each model.

Qwen3-1.7B-Base (28 layers)Qwen3-4B-Base (36 layers)Qwen3-8B-Base (36 layers)
Spearman ρ with 𝒞−0.66 (p < 0.001)−0.60 (p < 0.001)−0.53 (p < 0.001)

On all three models, layers whose representations occupy a lower-dimensional subspace have higher contributions. Our interpretation is that a layer whose representations are concentrated in a few dominant directions leaves many low-energy directions free, so RL can reshape its behavior in those directions without disrupting the features that subsequent layers rely on.

Our systematic RL measurements directly reveal the layer structure of RL adaptation, including what weight and inference-time analyses cannot capture. We want to emphasize that our work directly measures layer contribution and makes the effective-rank analysis above possible. The relation was discovered and validated against the 𝒞 measured by our layer-by-layer RL training.

  • Weight and inference-time analyses cannot explain 𝒞. The spectral norm, stable rank, and mean effective rank of the weight matrices show no consistent relation with 𝒞; middle layers are the most robust to deletion at inference time [1, §3.1, Fig. 2a], yet they contribute the most under RL; and a linear probe for response correctness (Qwen3-1.7B-Base) is most accurate in the late layers, which contribute the least (ρ = −0.58 with 𝒞).
  • Our work reveals the phenomena that matter for RL training. A single layer can exceed full-parameter RL (𝒞 > 1), the layer ranking is stable across datasets, tasks, and RL algorithms, and this structure translates into training strategies that outperform full-parameter RL (§4).

References

[1] Lad, Lee, Gurnee, Tegmark. The Remarkable Robustness of LLMs: Stages of Inference? arXiv:2406.19384v3, 2025.

[2] Yue et al. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? NeurIPS 2025.

[3] Roy and Vetterli. The Effective Rank: A Measure of Effective Dimensionality. EUSIPCO 2007.

AuthorsOpenReview ↗

Rebuttal for Reviewer Sfff (3/4)

Q2: Meaning of 𝒞 > 1

Thank you for the question. To directly answer it, we analyze the per-layer GRPO gradient of the initial policy on Qwen3-1.7B-Base and Qwen3-8B-Base. The RL gradient is concentrated in high-contribution layers and is more consistent across prompts there, yet full-parameter RL updates all layers by nearly the same amount. Joint training therefore spends much of its update on layers with weak and inconsistent gradient signal, which is the dilution the reviewer describes and explains why a single well-chosen layer can exceed full-parameter RL (𝒞 > 1).

Background. Existing RL post-training methods apply the same optimizer and learning rate to every layer, and full-parameter RL changes all layers by nearly uniform magnitudes regardless of their contribution (Appendix B.2, Fig. 9). Within joint training, boosting the learning rate of the high-contribution layers improves performance on all three Qwen3 models, while boosting the lowest-ranked layers degrades it (§4.1).

New gradient analysis: the RL gradient is concentrated in high-contribution layers.

How we compute the gradients. Starting from the base model (the initial policy of RL), we sample 4 responses for each of 512 training prompts with the same chat template, temperature, and maximum response length (3,072 tokens) as in RL training, and score them with the training reward. For each prompt group with mixed rewards (220 groups on Qwen3-1.7B-Base and 235 on Qwen3-8B-Base), we compute the GRPO policy gradient with group-normalized advantages and token-mean loss; at the first step the importance ratio equals 1, so this is exactly the gradient that full-parameter RL follows at initialization. For each layer ℓ and group i, this gives a gradient g_{i,ℓ}. We report two statistics per layer:

  • Gradient norm: the average of ‖g_{i,ℓ}‖ over groups. All layers have the same shape, so norms are directly comparable across layers.
  • Gradient agreement: the average pairwise cosine similarity between g_{i,ℓ} and g_{j,ℓ} across groups, which measures how consistently different prompts push the layer in the same direction.

We then compute the Spearman correlation between each statistic and 𝒞 across layers.

Qwen3-1.7B-Base (28 layers)Qwen3-8B-Base (36 layers)
Gradient norm vs. 𝒞ρ = 0.65 (p < 0.001)ρ = 0.49 (p = 0.003)
Gradient agreement vs. 𝒞ρ = 0.48 (p = 0.009)ρ = 0.69 (p < 0.001)
Gradient norm, middle layers0.45–0.52 (Layers 11–17)0.35–0.40 (Layers 14–22)
Gradient norm, late low-contribution layers0.25–0.33 (Layers 21–26)0.20–0.27 (Layers 25–33)
  • The gradient signal follows layer contribution. On both models, layers with higher contribution receive larger and more consistent gradients. The late low-contribution layers receive about half the gradient norm of the middle layers.
  • The update does not. Full-parameter RL moves every layer by nearly the same amount (Appendix B.2), as Adam with a single learning rate normalizes per-coordinate step sizes. Low-contribution layers thus receive full-size updates driven by a weak and inconsistent gradient signal.
  • This provide explanations for 𝒞 > 1. Full-parameter RL dilutes its update across all layers, including those whose gradient carries little consistent signal, whereas training only a high-contribution layer concentrates the entire update where the signal is strongest. This is also why the layer-aware strategies in §4 outperform full-parameter RL.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.