- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes a self-distillation method that constructs a stronger teacher by modifying the execution order of a model's internal layers, repeating selected layers to increase computation without changing model parameters. Specifically, the method first uses a small set of labeled examples to identify which layers to repeat for improved accuracy. The resulting looped model then serves as the teacher, while the student follows the original layer execution order and is trained through reverse KL distillation. Experiments show that the proposed method achieves approximately 3 percentage points of improvement over the CRISP baseline on Qwen3-1.7B GSM8K.
Critical strengths
-
The paper explores an interesting direction by constructing stronger self-teachers through layer looping and distilling the benefits into standard models.
-
The experimental results show relatively consistent improvements across multiple models and reasoning benchmarks.
Critical weaknesses
-
The paper lacks sufficiently compute-matched baseline comparisons. Layer search evaluates many candidate execution paths, and LoOPSD often requires multiple distillation rounds. Although the paper includes multi-round baselines, the additional search computation is not adequately accounted for. Could simpler approaches, such as sampling multiple reasoning trajectories and using majority voting or best-of-N selection for self-training, achieve comparable improvements under the same computational budget?
-
The experiments use only one training seed, and some improvements over strong baselines are relatively modest (around 3–4 percentage points). Without repeated training runs, it is difficult to determine whether these gains are robust or partly due to randomness. Additional seeds and evaluations across different training configurations would strengthen the empirical evidence.
-
The paper provides limited explanation of why repeating particular layers improves reasoning. Although the layer-search results demonstrate that execution order matters, the underlying mechanism remains unclear. More controlled ablations would be helpful, such as comparing repeated blocks with concentrating the same amount of additional computation in a single layer, or testing alternative repetition patterns under matched computational budgets.
Questions for authors
-
Could the authors provide a deeper theoretical explanation of why repeating specific layers improves reasoning performance?
-
Could the authors report results with more random seeds and error bars, and provide compute-matched comparisons with simpler alternatives?
4 comments
Author Response to Reviewer h3s1 (Part 2/2)
W3 & Q1. Why the repeated-layer structure matters
The paper provides limited explanation of why repeating particular layers improves reasoning. Although the layer-search results demonstrate that execution order matters, the underlying mechanism remains unclear. More controlled ablations would be helpful, such as comparing repeated blocks with concentrating the same amount of additional computation in a single layer, or testing alternative repetition patterns under matched computational budgets.
Could the authors provide a deeper theoretical explanation of why repeating specific layers improves reasoning performance?
Structure Ablation. Repeating a useful contiguous block gives a stronger teacher than placing the same additional computation in one layer or splitting it across blocks. On Qwen3-4B/GSM8K-128, we compare four execution structures:
- Standard path: Execute the original 36 layers without repetition.
- Single-layer repetition: Execute one layer six additional times.
- Split-block repetition: Repeat two separate three-layer blocks once each.
- Contiguous-block repetition: Repeat one contiguous six-layer block once.
We use the same route-search budget for all three repeated-execution structures. All three repetition alternatives add six layer executions, resulting in 42 executed layers.
Table H4. Qwen3-4B/GSM8K-128: alternative execution structures.
| Teacher execution structure | Executed layers | Teacher accuracy (%) |
|---|---|---|
| Standard path | 36 | 27.22 |
| Six additional executions of one layer | 42 | 10.08 |
| Two separate three-layer blocks, repeated once | 42 | 28.81 |
| One contiguous six-layer block, repeated once | 42 | 33.74 |
As shown in Table H4, the selected contiguous block outperforms the tested split-block alternative by 4.93%, while single-layer repetition even reduces accuracy below the standard path. Simply adding more layer executions does not guarantee improvement; selecting an effective execution route is essential.
Mechanistic Interpretation. Prior studies suggest that pretrained Transformer middle layers exhibit representational similarity and tolerate certain changes in execution order [1, 2]. Westerhoff et al. [3] further associate the effectiveness of layer repetition with boundaries between different representational regimes.
These findings suggest that a suitable repeated block may further refine answer-relevant representations while remaining compatible with subsequent layers. In contrast, unsuitable repetition may disrupt subsequent computation. This provides a possible explanation for why some pretrained layer compositions benefit from repeated execution while others do not.
Answer Readout. To further examine this interpretation, we compare the standard path and selected loop with answer readout:
Table H5. Qwen3-4B/ProofWriter: answer readout at identical model weights and contexts.
| Execution path | Mean accuracy, layers 25–35 (%) | Final output accuracy (%) |
|---|---|---|
| Standard 36-layer path | 55.09 | 59.80 |
| Selected 42-layer loop | 60.43 | 66.00 |
As shown in Table H5, the selected loop improves intermediate-layer answer readout by 5.34% and final output accuracy by 6.20% without changing model weights. Together with Figure 6 and Appendix Figure 11, these results support our interpretation that useful repeated execution can make answer-relevant information more accessible.
References
[1] Sun et al. Transformer Layers as Painters. AAAI 2025.
[2] Lad et al. The Remarkable Robustness of LLMs: Stages of Inference? NeurIPS 2025.
[3] Westerhoff et al. Programs-of-Layers in LLMs through the Lens of Cortical Areas. arXiv preprint, 2026.
Gentle Reminder: Rebuttal and Updated Paper Available
Dear Reviewer h3s1,
This is a gentle reminder that we have submitted our rebuttal and updated the paper accordingly. If you have a moment, we would greatly appreciate it if you could take a look and share any further comments during the discussion phase. We are very happy to clarify any points or provide additional information as needed.
Thank you very much for your time and for engaging with our submission.
Best regards,
Authors of Submission 4558
Eagerly await your valuable feedback|Just One Day Left in the Discussion
Dear Reviewer h3s1,
Thank you again for your thoughtful review. With the discussion deadline approaching, we wanted to briefly follow up on the three points you raised.
- W1. Compute-matched simpler baselines. We compare LoOPSD with majority-vote and best-of-N self-training at the same total compute budget. LoOPSD reaches 78.14% accuracy, outperforming both baselines by more than 33%.
- W2. Robustness across seeds and training settings. LoOPSD† outperforms the reported CRISP results on all five benchmarks across all three training seeds. It also maintains its advantage on GSM8K when we vary the distillation objective, learning rate, and student sampling configuration.
- W3. Why the repetition pattern matters. With six extra layer executions in each case, repeating a contiguous block yields higher accuracy than repeating a single layer or two separate blocks. We provide a detailed explanation in our rebuttal of why repeating a well-chosen block may improve reasoning.
If you have a moment before the deadline, we’d welcome any thoughts on the new results or any points you’d like us to clarify. Thank you again for your time and feedback.
Best regards,
Authors of Submission 4558
Author Response to Reviewer h3s1 (Part 1/2)
We thank the reviewer for the suggestions on compute-matched alternatives, reproducibility, and the mechanism of repeated execution. We answer each concern directly.
W1 & Q2. Compute-matched majority-vote and best-of-N self-training
LoOPSD outperforms majority-vote and best-of-N self-training at the same total compute budget. We compare LoOPSD with two trajectory-selection self-training baselines in Table H1:
Table H1. Qwen3-1.7B/GSM8K-400: trajectory-selection baselines at equal total FLOPs.
As shown in Table H1, LoOPSD outperforms both baselines by over 33%, with all methods receiving the same 135.2 PFLOPs. These results demonstrate that spending equivalent computation on sampling and selecting standard-path trajectories does not yield comparable improvements.
W2 & Q2. Robustness across random seeds and training configurations
Multiple Seeds. LoOPSD† consistently outperforms CRISP with low variability across three training seeds. We report the mean accuracy and standard deviation of LoOPSD† alongside the originally reported CRISP results.
Table H2. Qwen3-1.7B: comparison with CRISP across five benchmarks.
As shown in Table H2, even the lowest-performing LoOPSD† seed outperforms CRISP on all five benchmarks, including GSM8K and SVAMP, where the improvements are relatively modest. The low variability across seeds further supports the consistency of these gains.
Different Training Configurations. LoOPSD† maintains consistent performance across different training settings. To further evaluate the robustness of our results, we vary the distillation objective, learning rate, and student sampling configuration in Table H3.
Table H3. Qwen3-1.7B/GSM8K-400: robustness across training configurations.
For LoOPSD† variants, bold configuration entries indicate changes from the paper setting; dashes indicate unchanged settings.
As shown in Table H3, LoOPSD† achieves 79.53–81.27% accuracy across the tested configurations, consistently exceeding the original CRISP result of 76.65%. Together with Table H2, these results demonstrate that LoOPSD† maintains its gains across both training seeds and configurations.