- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(−1)
Read review
Paper summary
This paper propose MOSO (Mixture of Segment Operators), which casts depth reduction as learning a routing schedule over a frozen pretrained model. Its actions are segment operators that skip, run or repeat a short group of consecutive layers; this family contains static pruning, layer skipping and looped computation as special cases, and, under explicit linear-block conditions, the authors show that a segment repeat can realize maps that no layer-by-layer action sequence can.
Critical strengths
-
The empirical results are still reasonably strong despite the compute-comparison caveat.
Although I have concerns about whether the comparisons are strictly FLOP-matched, MOSO still shows consistently competitive and often clearly stronger results across many settings. -
The method is evaluated across multiple LLM backbones, which supports its generality.
A notable strength of the paper is that MOSO is tested on several model families, including LLaMA, Qwen, and Mistral, rather than on a single backbone.
Critical weaknesses
-
The fairness of the comparison in Table 1 is unclear from a compute perspective.
Although the methods are compared under the same nominal layer-removal budget, MOSO is allowed not only to skip layers but also to repeat certain layers/segments. As a result, the actual FLOPs can differ substantially across methods even within the same “loose” or “tight” setting. Therefore, matching by the number of skipped layers or by nominal budget does not necessarily constitute a fair apples-to-apples comparison. -
The empirical comparison is missing some relevant prior work.
While the paper compares against several dynamic routing baselines, it does not include some closely related methods from the adaptive-depth / recurrent-computation literature, such as [1] MeSH: Memory-as-State-Highways for Recursive Transformers, [2] Generative Recursive Reasoning, [3] Looping Back to Move Forward: Recursive Transformers for Efficient and Flexible Large Multimodal Models, and [4] DND: Boosting Large Language Models with Dynamic Nested Depth.. -
There is a potentially serious causal mismatch in the router design.
The paper states that the router uses a pooled summary of the hidden state, but it does not clearly specify whether this summary is causally constrained. For ar language models, routing decisions at inference time should only depend on the current prefix, since future tokens are unavailable. If, during training or evaluation, the router is allowed to pool over hidden states from the full teacher-forced sequence, then it may implicitly exploit information from future tokens. This would introduce both train–test mismatch and potential future-token leakage. I view this as a serious concern, and the paper should explicitly clarify whether the router is strictly causal / prefix-only, and if not, quantify the impact of this mismatch.(Ref: Mixture-of-recursions: Learning dynamic recursive depths for adaptive token-level computation).
Questions for authors
NA
2 comments
Thanks for the rebuttal. Although I still do not believe the paper currently meets the standard for acceptance—primarily because the authors have only provided promises of future work to address certain concerns rather than concrete revisions—I am raising my score from 1 to 2 to encourage the authors’ further research.
Response to Reviewer guTF
Thank you for the constructive review and for highlighting compute fairness, related work, and causality.
Compute fairness. Our operator cost counts repeated layer executions, and Tables 5–6 report realized FLOPs including additional-component overhead. However, nominal skipped-layer budgets do not ensure identical computation. For example, on Qwen2.5-7B, MoSO achieves 66.6 Avg6 at 0.859 compute, versus ShortGPT’s 60.7 at 0.857. We will make realized compute more visible and add strictly matched comparisons and explicit budget enforcement.
Related work. Appendix A discusses the relationship to recursive computation. We will expand this discussion to address MeSH, Generative Recursive Reasoning, RecursiveVLM, and DND, clarify differences in training requirements and routing granularity, and add applicable baseline comparisons.
Causality. We agree that pooling visibility requires explicit clarification and verification. Equation (4) masks routing steps; this alone does not establish token causality. Although an observed prompt can condition its continuation, future target tokens must not influence earlier predictions. We will specify token visibility and add prefix-only evaluation and causality checks, incorporating any necessary corrections.
We appreciate your feedback and will include the additional validation in the resubmission.