acceptodds
Under review as a conference paper at ICLR 2027

Dynamic Depth Routing of Large Language Models with Mixture of Segment Operators

Abstract

Large language models (LLMs) process every input through a deep stack of layers, which makes inference slow and costly, yet many inputs do not need the full stack. Existing methods reduce this depth either by permanently pruning layers or by learning to skip layers per input, but most decide about one layer at a time, even though the quality of the reduced model depends on the whole schedule of which layers are skipped, run or reused. We propose MOSO (Mixture of Segment Operators), which casts depth reduction as learning a routing schedule over a frozen pretrained model. Its actions are segment operators that skip, run or repeat a short group of consecutive layers; this family contains static pruning, layer skipping and looped computation as special cases, and, under explicit linear-block conditions, we show that a segment repeat can realize maps that no layer-by-layer action sequence can. A lightweight router scores all candidate segments jointly, so each decision is made in the context of the schedule being built, and the schedule is learned end to end without any search over schedules. Across five LLMs from three model families, MOSO attains the highest average accuracy at each reported operating point, with its lead over the strongest baseline growing at the tighter one, and controlled experiments show that the learned schedule contributes to it.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, dynamic depth, depth routing, adaptive computation, model compression

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_guTFOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(−1)
Read review
Paper summary

This paper propose MOSO (Mixture of Segment Operators), which casts depth reduction as learning a routing schedule over a frozen pretrained model. Its actions are segment operators that skip, run or repeat a short group of consecutive layers; this family contains static pruning, layer skipping and looped computation as special cases, and, under explicit linear-block conditions, the authors show that a segment repeat can realize maps that no layer-by-layer action sequence can.

Critical strengths
  1. The empirical results are still reasonably strong despite the compute-comparison caveat.
    Although I have concerns about whether the comparisons are strictly FLOP-matched, MOSO still shows consistently competitive and often clearly stronger results across many settings.

  2. The method is evaluated across multiple LLM backbones, which supports its generality.
    A notable strength of the paper is that MOSO is tested on several model families, including LLaMA, Qwen, and Mistral, rather than on a single backbone.

Critical weaknesses
  1. The fairness of the comparison in Table 1 is unclear from a compute perspective.
    Although the methods are compared under the same nominal layer-removal budget, MOSO is allowed not only to skip layers but also to repeat certain layers/segments. As a result, the actual FLOPs can differ substantially across methods even within the same “loose” or “tight” setting. Therefore, matching by the number of skipped layers or by nominal budget does not necessarily constitute a fair apples-to-apples comparison.

  2. The empirical comparison is missing some relevant prior work.
    While the paper compares against several dynamic routing baselines, it does not include some closely related methods from the adaptive-depth / recurrent-computation literature, such as [1] MeSH: Memory-as-State-Highways for Recursive Transformers, [2] Generative Recursive Reasoning, [3] Looping Back to Move Forward: Recursive Transformers for Efficient and Flexible Large Multimodal Models, and [4] DND: Boosting Large Language Models with Dynamic Nested Depth..

  3. There is a potentially serious causal mismatch in the router design.
    The paper states that the router uses a pooled summary of the hidden state, but it does not clearly specify whether this summary is causally constrained. For ar language models, routing decisions at inference time should only depend on the current prefix, since future tokens are unavailable. If, during training or evaluation, the router is allowed to pool over hidden states from the full teacher-forced sequence, then it may implicitly exploit information from future tokens. This would introduce both train–test mismatch and potential future-token leakage. I view this as a serious concern, and the paper should explicitly clarify whether the router is strictly causal / prefix-only, and if not, quantify the impact of this mismatch.(Ref: Mixture-of-recursions: Learning dynamic recursive depths for adaptive token-level computation).

Questions for authors

NA

2 comments
AuthorsOpenReview ↗

Response to Reviewer guTF

Thank you for the constructive review and for highlighting compute fairness, related work, and causality.

Compute fairness. Our operator cost counts repeated layer executions, and Tables 5–6 report realized FLOPs including additional-component overhead. However, nominal skipped-layer budgets do not ensure identical computation. For example, on Qwen2.5-7B, MoSO achieves 66.6 Avg6 at 0.859 compute, versus ShortGPT’s 60.7 at 0.857. We will make realized compute more visible and add strictly matched comparisons and explicit budget enforcement.

Related work. Appendix A discusses the relationship to recursive computation. We will expand this discussion to address MeSH, Generative Recursive Reasoning, RecursiveVLM, and DND, clarify differences in training requirements and routing granularity, and add applicable baseline comparisons.

Causality. We agree that pooling visibility requires explicit clarification and verification. Equation (4) masks routing steps; this alone does not establish token causality. Although an observed prompt can condition its continuation, future target tokens must not influence earlier predictions. We will specify token visibility and add prefix-only evaluation and causality checks, incorporating any necessary corrections.

We appreciate your feedback and will include the additional validation in the resubmission.

Thanks for the rebuttal. Although I still do not believe the paper currently meets the standard for acceptance—primarily because the authors have only provided promises of future work to address certain concerns rather than concrete revisions—I am raising my score from 1 to 2 to encourage the authors’ further research.

Reviewer_n3udOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a router that traverses a frozen LLM and chooses whether to skip, execute or repeat the next one or two layers. The router ist rained through a soft operator mixture suing language-modeling, distillation, compute and balancing losses; inference uses hard sequential decisions. The paper reports improved average multiple-choice accuracy at two compute levels across five backbones, suppotred by structural expressivity and perturbation results.

Critical strengths

The problem is relevant and direct training without search-generated route labels is an appealing direction. The evaluation spans several model families, reports realized compute and includes schedule-substitution controls and latency measurements.

Critical weaknesses
  1. Missing ablations: The paper never compares segment-level operations to per-layer routing under an equal training and compute budget. The segment repetition tests use separately trained routers (some using more compute than the dense model), and ShortGPT slightly outperforms MoSO under matched training.
  2. Unclear results: The authors claim only the router is trained, yet the frozen dense model's accuracy changes when routing is turned off. This discrepancy needs explanation.
  3. Limited novelty and altered baselines: PoLar already routes over multi-layer segments. Furthermore, modified baselines like Dr. LLM and Router-Tuning (Appendix D.1) make the true gains hard to assess.
  4. Weak theoretical value: The proofs show expressivity differences under strict assumptions, adding math complexity without explaining why the method works better in practice or backing the empirical claims.
  5. Incomplete evaluation: Testing only covers multiple-choice benchmarks and ignores actual text generation, which is a major gap for an LLM inference efficiency method. Latency tests also use shared schedules instead of realistic per-input routing.
Questions for authors
  1. Contribution of segment routing: How does MoSO compare to a single-layer router (skip/execute/repeat) under the same training setup and compute budget? Also, how often are segment repeats used in the main experiments?
  2. Unexpected accuracy changes: If the backbone and LM head are frozen, why does turning off routing after training increase LLaMA-2-7B accuracy from 64.8 to 66.0 (Appendix E.1)?
  3. Novelty: How does MoSO differ from PoLar beyond its training setup, and can you directly compare them experimentally?
  4. Baseline fairness: How do the changes to Dr. LLM and Router-Tuning affect their results? Please compare against the original implementations at matched compute budgets, expanding on the Dr. LLM analysis in Appendix E.3.
  5. Generation evaluation: Can you test MoSO on text generation and measure end-to-end latency with per-input routing? Also, how do you handle routing decisions and KV caches during autoregressive generation?
  6. Training and inference mismatch: Training uses a soft mixture of operators, while inference picks just one. How big is the resulting performance drop, and does it grow under tighter compute budgets?
2 comments
AuthorsOpenReview ↗

Response to Reviewer n3ud

Thank you for the detailed review and helpful suggestions.

Experimental controls. You are correct that full execution with frozen weights should reproduce the dense model; we will audit the Appendix E.1 discrepancy. Figure 5 shows comparable performance with ShortGPT’s layer choices on LLaMA-3.2-3B and gains from learned schedules on Qwen. We will clarify this scope and add matched-training, matched-compute single-layer and no-O(2,2) ablations, a learned static control, and operator frequencies from the main checkpoints.

Novelty and baselines. MoSO’s intended distinction from PoLar is learning without search-generated route labels and conditioning decisions on evolving execution states. Appendix E.3 already includes public-code/default-settings Dr.LLM comparisons: on LLaMA-3.1-8B-Instruct, MoSO obtains 68.5 Avg6 with 28.00 executed layers, versus 63.4 with 28.71. We will clearly distinguish adapted baselines and add direct PoLar and original-implementation comparisons at matched compute.

Generation and latency. Appendix E.5’s approximately 1.3× speedups use shared schedules. We will add generation evaluation and request-specific latency measurements, including scheduling overhead, and specify schedule persistence and KV-cache handling.

Training–inference mismatch and theory. Lemma 6 is a single-step result, so we will measure the complete soft-to-hard performance gap at both budgets. We will also shorten the theoretical exposition and clarify that its conditional results do not guarantee empirical gains.

We appreciate your guidance and will incorporate these additions into the resubmission.

I thank the authors for the constructive response. The clarification of the differences from PoLar and the existing native Dr.LLM comparison partially address my concerns about novelty and baseline fairness.

However, the main issues remain unresolved. In particular, the paper still lacks controlled ablations showing the benefit of segment-level operations, an explanation for the accuracy changes when routing is disabled, and generation-quality and realistic latency evaluations. These are central to establishing the method's contribution and practical relevance.

The proposed additional experiments are appropriate, but since their results are not yet available, I maintain my original assessment.

Reviewer_AsUDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MoSO, a method that learns input-dependent depth scheduling for frozen, pre-trained large language models. The router selects from six operators to skip, execute, or repeat a segment consisting of one layer or two adjacent layers. Training uses a soft hybrid approach that combines language modeling, distillation, and computationally relevant objectives; during inference, operators are selected discretely. The paper provides a unified perspective on existing depth-routing methods and, under the assumption of linear blocks, proves that repeating layer segments offers different expressive capabilities than operating layer by layer. Experiments across five backbone models, six benchmarks, and two budget settings show that accuracy gains are more pronounced under tighter computational budgets.

Critical strengths
  1. The paper is well-written and clearly organized.

  2. The paper presents a well-formulated operator framework, supported by mathematical analysis of the expressivity of segment-level scheduling.

  3. MoSO demonstrates promising accuracy-computation trade-offs across multiple LLM backbones and benchmarks, supported by useful ablation studies and empirical analysis.

Critical weaknesses
  1. Although the paper provides theoretical analysis of the potential advantages of segment-level scheduling, its practical benefits remain unclear due to the lack of targeted ablation studies (see Q4).

  2. Several of the compared baselines are relatively dated, while more recent and closely related methods, such as PoLar, are missing from the experimental comparisons. This makes it difficult to fully assess MoSO's advantages over strong baselines (see Q2).

Questions for authors

Q1. The paper states that only the router is trained while the pretrained weights remain frozen. Why does the routing-off control in Appendix E.1, where every layer is executed, improve performance over the original dense model? Shouldn't they be mathematically equivalent?

Q2. MoSO shares substantial similarities with PoLar [1] in its attention-based predictor and segment-level scheduling, but the paper provides no experimental comparison with PoLar. Could the authors include a direct comparison to substantiate MoSO’s claimed advantages?

Q3. Is MoSO’s compute budget a hard constraint or a soft target, and how is it enforced? In some loose-budget settings, MoSO uses more computation than some baselines (0.859 vs 0.815). Does its performance advantage persist when actual computation is strictly matched, e.g., by applying a stricter execution limit or truncation to meet the same budget?

Q4. Is there empirical evidence that segment-level repetition (ABAB) improves performance over layer-wise repetition (AABB), e.g., an ablation removing O(2,2)

[1] Ziyue Li et al. Skip a Layer or Loop It? Learning Program-of-Layers in LLMs. ICML 2026.

2 comments
AuthorsOpenReview ↗

Response to Reviewer AsUD

Thank you for the thoughtful review and constructive suggestions.

Q1. Routing-off control. You are correct that executing every layer once with frozen backbone and LM-head parameters should reproduce the dense computation. We will audit the discrepancy in Appendix E.1 under identical inputs and evaluation settings and document any necessary corrections.

Q2. PoLar. Section 2 and Appendix A acknowledge PoLar’s segment routing. Our intended distinction is learning without search-generated route labels, with decisions conditioned on intermediate execution states and remaining budget. We will clarify this positioning and add a direct comparison.

Q3. Compute budget. The training objective uses a soft compute penalty. Tables 5–6 report realized computation, but operating points are not strictly matched across all methods. Some comparisons are already close: on Qwen2.5-7B, MoSO achieves 66.6 Avg6 at 0.859 compute, versus ShortGPT’s 60.7 at 0.857. We will add stricter matched-compute comparisons and explicit budget enforcement.

Q4. Segment repetition. Proposition 3 establishes conditional expressivity, while practical benefit requires further evidence. We will add matched-training ablations restricting routing to single-layer actions or removing O(2,2), and report operator usage from the main checkpoints.

We appreciate your feedback and will include these additional checks and experiments in the resubmission of this work.

Thank you for taking the time to respond to my questions. I appreciate the care you have taken in addressing my comments. I encourage you to continue refining this work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.