acceptodds
Under review as a conference paper at ICLR 2027

X-MoD: Practical Scaling Laws for Sparse-Depth Routing Beyond Mixture-of-Depths

Abstract

Mixture-of-Depths (MoD) enables conditional computation across Transformer depth by routing only a subset of tokens through selected layers, but its original one-sparse–one-dense alternation tightly couples total capacity to active capacity and limits sparse-depth scaling. We introduce X-MoD, a scalable sparse-depth architecture that decouples token sparsity from anchor stride, allowing total parameter count to grow while keeping active-equivalent capacity nearly fixed. To make deep sparse routing trainable, X-MoD combines dense anchors with variance-scaled layer-wise gating and depth-wise token balancing. To make this regime analyzable and usable, we formulate sparse-depth routing as a conditional architecture-design problem: given compute, context length, and active-equivalent backbone size, how should the routing configuration be chosen? We develop a practical scaling-law framework by fitting X-MoD relative to FLOP-matched dense baselines, yielding an interpretable law that decomposes performance into sparse-capacity gain, sparse-context correction, and anchor-stride interaction. The law predicts validation loss across routing configurations and reveals how context length, model scale, and anchor stride shape sparse-depth performance. We validate the architecture and law through pretraining sweeps, held-out scaling-law prediction, ablations, downstream evaluations, and comparisons with Dense, MoD, and representative MoE baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, Conditional computation, Mixture-of-Depths, Scaling laws

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_yPNZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

X-MoD extends MoD (Mixture-of-Depths) by enabling A (anchor stride) x K (token sparsity ratio) sparse layers between dense layers and only 1/K tokens are routed to the next layers. So, the total parameters can grow far beyond the size of active-equivalent dense or MoE models. Three components are used for stable training of these deep networks: (1) layer-wise gating with variance, (2) depth-wise token balancing, and (3) several successive dense layers before the sparse-dense blocks. This paper also fit a scaling law that shows how K and A affect the validation loss gap between X-MoD and a matched-FLOPs dense baseline.

Critical strengths
  1. This paper explores the depth direction of conditional computation, which is less studied than width-wise MoE, and identifies a clear limit of MoD (total-to-active ratio below 2).
  2. Table 1 shows that total parameters can scale substantially (e.g., 936M active to 8.24B total) while active-equivalent size stays fixed.
Critical weaknesses
  1. Generalization of the law is unclear. It is fitted on 109 observations with one dataset, one optimizer, and models up to 1.65B, with the sweeps never crossed. Whether it holds at larger scales, where routing dynamics may differ, is an open question.
  2. Section 3 is hard to follow. It blends design choices, heuristic assumptions, and a complex parameterization into one formula, with heavy notation and abstract terms (capacity reward, sparse-context correction). A short high-level narrative before the algebra would help. Several symbols are not defined in the main text: u, w, z, K_hat, A_hat, and tau and the update rule in Eq. 8.
  3. Fig. 2 is hard to read on its own. The fixed settings for each sweep (e.g., N_act, A, L) are only in Appendix B.4 and should appear in the caption.
  4. Reproducibility. The anonymous repo in footnote 1 returned "file not found" when I checked.
Questions for authors
  1. Table 10 shows near-complete coverage (99.99% for A4K8) with very low consecutive-layer overlap, so tokens appear to take turns being updated. Is this still conditional computation, or just capacity spread evenly? Does routing correlate with token importance?
  2. Is the router called on every sparse layer? If so, how much overhead it incurs?
  3. X-MoD uses 161–298 layers. Do the per-layer overhead and sequential depth offset the FLOP savings in wall-clock terms, especially for decoding? Report decoding latency at several batch sizes might help further evaluate the value of this work.
9 comments
AuthorsOpenReview ↗

Summary of Our Response

We sincerely thank the reviewer for the thoughtful assessment and for recognizing the potential of sparse-depth capacity scaling. Below, we summarize our responses and the corresponding manuscript clarifications.

For transparency, AI tools assisted with language polishing and presentation; the authors are responsible for the technical content, experimental results, and conclusions.

1. Generalization (W1). We summarize grouped holdout and 64k/1.65B out-of-range results, and will extend evaluation to broader training settings and larger models in future work.

2. Presentation and code access (W2–W4). We have clarified notation and added fixed sweep settings to Fig. 2, with changes highlighted in blue. Repository access has been restored, as explained in our general Official Comment.

3. Conditional routing and scoring cost (Q1–Q2). We explain why broad coverage remains compatible with input-dependent routing and show that scoring FLOPs are small relative to attention and FFN computation.

4. Realized efficiency (Q3). Existing end-to-end measurements show net training and validation-forward speedups with routing, token movement and sequential-layer overhead included.

We appreciate the reviewer's time and will be happy to clarify any remaining questions during the discussion period.

AuthorsOpenReview ↗

Detailed Response 1

W1: Generalization of the scaling law

Thank you for your thoughtful comments on the generality of the law. We agree that evaluating a wider range of training settings and larger models would be valuable. We appreciate this suggestion and would like to provide some further context on the current validation.

The fitting experiments cover context lengths from 2k to 32k and active-equivalent sizes from 165M to 936M, with 109 independent training runs contributing one validation-loss observation each at the target compute budget. To examine transfer beyond this range, we reserve the 64k and 1.65B settings for out-of-range testing and evaluate them with the fitted coefficients frozen. The corresponding protocol is described in Section 4.1 and Appendix B.4–B.5.

Our aim in organizing the sweeps was to study the interactions of K with context length, active-equivalent size and anchor stride under controlled conditions. We therefore vary K jointly with each of L, active-equivalent size and A. As a complementary check, Fig. 3(b–d) evaluates predictions for complete context-length, anchor-stride or model-scale groups withheld during fitting.

For ease of reference, we summarize the prediction errors and configuration choices for the two out-of-range settings in Fig. 3(a) below:

Out-of-range settingMAERMSELowest predicted-loss candidateLowest measured-loss candidate
64k context0.0088860.009446A1K22A1K22
1.65B active-equivalent size0.0069700.007614A1K16A1K16

In both settings, the candidate with the lowest predicted loss also has the lowest measured loss among those evaluated. We find this agreement, together with the grouped holdout results, encouraging for the law's use in sparse-depth configuration selection. We also appreciate the reviewer's emphasis on examining how these findings extend to broader conditions. Building on the present results, we will conduct more extensive experiments across training settings and evaluate larger active-equivalent models in future work, to further assess the generality of these scaling relationships.

AuthorsOpenReview ↗

Detailed Response 2

W2: Intuition and notation

Thank you for the helpful presentation suggestion. The intuition behind the law can be summarized as follows:

At fixed training compute and active-equivalent size, increasing sparsity expands total parameter capacity but reduces the context available to each sparse layer. The law captures the balance between these two effects. Anchor stride controls how the active-equivalent budget is allocated between dense anchors and sparse refinements, with saturating benefits as the sparse share increases. These effects are modeled through the loss difference relative to a matched dense reference.

We have also made corresponding clarifications in the revised manuscript: Section 3.2 now specifies the roles of u and w, the text immediately following Eq. (14) defines all three fitted weights u, w and z, and the sentence preceding Eq. (16) explicitly identifies the continuous optimum at A = 1. All these changes are highlighted in blue. The relevant notation is explained below:

  • uu, ww and zz are the fitted weights of the capacity, sparse-context and anchor-stride terms, respectively. Their fitted values are reported in Appendix C.1, Table 8.
  • The starred values of K^\hat K and A^\hat A in Fig. 2 identify the lowest measured loss in each evaluated sweep. In particular, A^=5\hat A=5 denotes the best tested stride. In Eq. (16), K^\hat K denotes the approximate continuous optimum derived at A = 1.

Regarding τ\tau in Eq. (8), it controls the strength of the token-choice bias. The bias bib_i starts at zero at the beginning of each sparse interval, increases by one whenever token i is selected, remains unchanged when it is skipped, and is reset at the next dense anchor. Subtracting τbi\tau b_i from the current routing logit discourages repeated selection while preserving input-dependent routing.

\quad

W3: Fixed settings in Fig. 2

Thank you for this helpful suggestion. We have added the following concise description to the Fig. 2 caption, consistent with Appendix B.4, and highlighted it in blue:

Fixed settings: (a,b) Nact=556MN_{\mathrm{act}}=556\mathrm{M}, A = 1; (c) L = 32k, A = 1; (d) Nact=165MN_{\mathrm{act}}=165\mathrm{M}, L = 32k.

The detailed sweep grids and compute-matching protocol remain in Appendix B.4.

\quad

W4: Code availability

Access through the original anonymous link has been restored. We have provided a detailed explanation in our general Official Comment shared with all reviewers and the Area Chair, titled “Clarification of code availability and restoration of access”. We sincerely apologize for the inconvenience caused by the interruption and would be grateful if you could revisit the repository.

Reviewer_9qYzOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces X-MoD, a sparse-depth Transformer that increases total capacity while keeping average active capacity controlled. It places multiple sparse layers between dense anchors and uses token balancing and residual scaling to stabilize training. It also fits an empirical law for loss relative to a compute-matched dense baseline. Evaluation includes pretraining sweeps, held-out prediction, downstream tasks, ablations, and throughput measurements.

Critical strengths
  • The proposed architecture improves over dense and alternating sparse-depth baselines under matched training compute, with measured throughput gains over the tested expert models.
  • The experiments cover multiple model sizes, downstream tasks, and component ablations, providing useful evidence for the effectiveness of sparse-depth expansion.
Critical weaknesses
  • The practical value of the scaling law is unclear. The paper presents the law as a way to choose model configurations, but the main models are selected through grid search. Predicting loss accurately does not necessarily mean choosing the best configuration. Showing that the law finds good configurations with fewer experiments would better support its purpose.

  • The ablation experiments need more detail. The paper removes dense anchors or initial dense layers while claiming to keep active capacity unchanged, but does not explain what replaces those layers. These changes could also affect performance. I could not access the code to check, so the authors should provide the exact model configurations.

Questions for authors
  1. On the held-out experiments, does the scaling law choose a configuration that performs nearly as well as the best one tested?
  2. Why does the full scaling law fit the data worse than the simpler version? Does the simpler version also predict held-out results better?
  3. In the gated-residual ablation, do you change which parts of the layer are gated, remove the learned scaling factor, or both?
7 comments
AuthorsOpenReview ↗

Summary of Our Response

We sincerely thank the reviewer for the careful and constructive assessment. We have addressed the questions with concrete configuration-selection examples, clearer ablation definitions, and a direct comparison of the full and reduced scaling laws.

For transparency, AI tools assisted with language polishing and presentation; the authors are responsible for the technical content, experimental results, and conclusions.

Below, we summarize our responses. Detailed explanations and results are provided in the corresponding responses and the revised appendix, where newly added text is highlighted in blue.

1. Practical configuration selection (W1/Q1). We explain how the law ranks feasible configurations under practical constraints and provide examples from the pilot grid and both out-of-range settings. The predicted choices match the best measured candidates under the corresponding constraints.

2. Ablation definitions and code access (W2/Q3). We clarify the sparse-layer replacements in the dense-anchor and dense-prefix ablations, and that the residual-scaling ablation removes only the learnable coefficient while retaining router-score gating. Repository access has been restored, with details in our shared Official Comment.

3. Full versus reduced scaling law (Q2). We explain the residual formulation and compare the two versions using their fitted coefficients. The reduced law achieves lower combined in-range fitting RMSE and lower pooled MAE and RMSE over the nine out-of-range configurations.

We greatly appreciate the reviewer's time and questions. We hope the added evidence and explanations address the concerns, and we will be happy to clarify any remaining points promptly during the discussion period.

AuthorsOpenReview ↗

Detailed Response 1

W1/Q1: Practical use of the scaling law and configuration selection

Thank you for highlighting the distinction between loss prediction and configuration selection. The purpose of the pilot grid search is to establish a fair comparison between tuned X-MoD and MoE configurations. We search both architectures and select the lowest-loss tested settings under the matching constraints, addressing the concern that favorable X-MoD configurations might otherwise be compared with weak MoE baselines.

Training computational cost is primarily determined by total training FLOPs, inference computational cost by FLOPs per processed token, and model-capacity constraints at deployment by active and total parameter counts. Accordingly, given a context length LL, a required active-equivalent parameter count NactN_{\mathrm{act}} and a total-parameter budget NN, we first identify feasible (K,A)(K,A) pairs using the architecture's parameter-count relation. We then substitute each candidate into the fitted law and select the one with the lowest predicted loss. This allows candidate selection without training every configuration.

We have added a concrete selection example in Appendix C.2, Table 9, covering all 24 X-MoD configurations in the existing 165M, 32k pilot grid (Fig. 4). The law achieves MAE = 0.00624, RMSE = 0.00746, R² = 0.9823, and Spearman rank correlation = 0.9835. After filtering candidates by the two MoE total-parameter budgets of 0.6964B and 1.2258B, selecting the lowest predicted loss yields A4K8 and A3K16, respectively. Both match the best measured feasible configuration. This agreement holds both when matching the corresponding MoE sparsity and when allowing K and A to vary jointly.

For Q1, we additionally report configuration selection for the two out-of-range settings in Fig. 3(a): a 64k context length and 1.65B active-equivalent parameters. Selecting the lowest predicted loss yields A1K22 and A1K16, respectively. Both are the lowest-loss measured configurations among the tested candidates, as shown below.

64k context length

ConfigurationPredicted lossMeasured lossPredicted rankMeasured rank
A1K162.6214532.61685954
A1K182.6208372.61300643
A1K202.6205762.60864422
A1K222.6205652.60737811
A1K242.6207332.62762035

1.65B active-equivalent parameters

ConfigurationPredicted lossMeasured lossPredicted rankMeasured rank
A1K142.5055562.51600044
A1K162.5051532.50200011
A1K182.5051922.51000022
A1K202.5055252.51500033

Together with the pilot-grid example, these results illustrate how the law can guide configuration selection both within and beyond the fitted context-length and model-scale ranges.

AuthorsOpenReview ↗

Detailed Response 2

W2: Dense-anchor and dense-prefix replacements

Thank you for asking us to clarify the ablation configurations. As stated in Appendix B.6 (“Component removals”) of the submitted manuscript, these experiments use X-MoD A1K16 at 936M active-equivalent parameters, a 32k context length, and a training budget of 1.5×10201.5 \times 10^{20} FLOPs.

When removing dense anchors or the dense prefix, each affected dense layer is replaced by 16 sparse layers with K=16K=16. The configurations are summarized below. All three retain an active-equivalent depth of 28 and approximately 936M active-equivalent parameters.

ConfigurationDense prefix layersDense anchor layersSparse layersTotal layers
Full X-MoD A1K16412192208
w/o dense anchors40384388
w/o dense prefix012256268

In the dense-anchor ablation, token-choice bias is still reset at the original interval boundaries. Although both ablations have more total parameters, the full model achieves lower validation loss (Table 3), supporting the contribution of the dense components beyond total parameter capacity alone. The added training and validation curves in Appendix C.6, Fig. 8 complement these final-checkpoint results.

The repository is now accessible through the original anonymous link. We apologize for the interruption and have explained the issue and restoration of access in our general Official Comment, “Clarification of code availability and restoration of access”, which is visible to all reviewers.

Q3: The gated-residual ablation

Only the learnable residual scaling coefficient is removed, equivalent to fixing it to one. The router-score gating is retained and continues to apply to both the attention and FFN residuals. The routing mechanism and the scope of gating are unchanged. Thus, “w/o gated residual scaling” specifically tests the contribution of the learnable scale while retaining whole-layer router-score gating.

Reviewer_MUhqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a modified Mixture of Depths architecture (X-MoD) that scales MoD beyond the simple alternating sparse and dense layers, in an MoE-like manner. X-MoD places AKAK sparse layers between each dense layer, which separates the token sparsity KK from the stride AA. The total parameter count can then grow approximately linearly in KK while the active parameter count stays nearly fixed. To make training feasible, the authors add layer-wise gating with variance scaling, a depth-wise token-choice bias, and a dense prefix. The authors also propose a scaling law for X-MoD, fitted as a residual over a FLOP-matched dense baseline. The architecture and the law are evaluated with pretraining sweeps on FineWeb-Edu. Experiments on held-out and out-of-range loss prediction and comparisons against dense, MoD and MoE baselines are presented at three scales (556M, 936M and 1.65B active params).

The paper proposes an interesting and simple architecture and empirically shows how it scales and how it compares with MoEs. While this model architecture could be potentially useful, some of the design decisions are not fully explained or analyzed.

Critical strengths
  1. The proposed X-MoD architecture enables scaling MoD, which was previously limited by the alternating sparse–dense structure. The authors propose to stack multiple sparse layers between dense blocks.

  2. The scaling law is validated on out-of-range settings (L = 64k and N_act = 1.65B) that were specified before fitting, as well as with leave-one-group-out splits, rather than only by in-sample fit.

  3. The mechanism controls in Table 2 (Routed-FFN and Selected-Q/full-KV) and the routing diagnostics in Table 10 are useful. In particular, they show that the token-choice bias is essential: without it, routing collapses onto a fixed subset of tokens, and the model becomes worse than the dense baseline.

Critical weaknesses
  1. Training stability at depth. A central claim is that the proposed stabilizers (gated variance scaling, token-choice bias, dense prefix) make deep sparse stacks trainable (Section 2.3). The evidence is final validation losses from single runs. It would help to have training curves, gradient-norm or residual-norm statistics or similar analysis to support this.

  2. It is unclear how useful X-MoD is compared to MoE. The comparison in Table 1 is made at L = 32k, where attention makes up most of the counted FLOPs (roughly 70–90%, based on the per-token FLOPs in Table 6). X-MoD's FLOPs saving comes almost entirely from sparse layers attending over fewer tokens. So at matched training FLOPs, X-MoD sees about twice as many training tokens as MoE (Table 7). The paper's own Routed-FFN control, which matches token exposure, is worse than MoE 8:64. This suggests the gain may come from extra tokens rather than from depth routing using parameters better. A comparison at matched tokens, or at a shorter context (e.g. 4k) where attention does not dominate, is needed to separate these.

  3. The scaling law does not entirely answer the question it poses. The goal is to pick KK and AA given compute, context length and model size (Eq. 9), but the law only gives a clear answer for KK, roughly growing with L\sqrt{L} and independent of model size. The anchor-stride term has no optimum, and the best tested value sits at the edge of the grid. Compute CC does not appear on the right-hand side of Eq. 14, so the law cannot say how X-MoD's advantage changes with training length. The paper also does not use the law to choose its own main configurations, which come from a pilot grid.

Questions for authors
  1. How does X-MoD compare to MoE at matched training tokens, or at matched FLOPs with a shorter context length (e.g. 4k)?
  2. Can the authors provide training curves and gradient-norm statistics for the largest models, and repeat the stability ablations on A3K16?
  3. Were any X-MoD runs unstable or divergent during development, and if so, under which settings?
  4. I would be curious to know how the scaling law predicts performance for model sizes in the range of 4-8B, do the authors have any insight on this?
7 comments
AuthorsOpenReview ↗

Summary of Our Response

We sincerely thank the reviewer for the careful and constructive assessment. We have addressed the questions with additional training-dynamics evidence, a shorter-context MoE comparison, and a clearer explanation of how the scaling law supports configuration selection under practical constraints.

For transparency, AI tools assisted with language polishing and presentation; the authors are responsible for the technical content, experimental results, and conclusions.

Below, we summarize our responses. Detailed explanations and results are provided in the corresponding responses and the revised appendix, where newly added text is highlighted in blue.

1. Training dynamics and convergence (W1/Q2/Q3). We added training and validation curves for Tables 1–3, including the largest models (Appendix C.6, Figs. 6–8). No non-convergence or divergence was observed for X-MoD within the evaluated range.

2. A shorter-context comparison with MoE (W2/Q1). The added 936M, 4k comparison shows lower loss for X-MoD with slightly fewer total parameters and 21% lower per-token training FLOPs at matched training compute (Appendix Fig. 9).

3. Compute dependence and practical use of the scaling law (W3/Q4). We clarify compute dependence through the matched Dense reference and how the law ranks feasible (K,A)(K,A) configurations under context, active-capacity and total-parameter constraints, including the 4–8B case.

We greatly appreciate the reviewer's time and questions. We hope the added evidence and explanations address the concerns, and we will be happy to clarify any remaining points promptly during the discussion period.

AuthorsOpenReview ↗

Detailed Response 1

W1/Q2: Training dynamics and stability

Thank you for suggesting that we provide evidence of the training dynamics. We have added training and validation loss curves in Appendix C.6:

  • Figure 6: all main comparisons in Table 1 at 556M, 936M and 1.65B active-equivalent parameters, including A3K16 at the largest scale.
  • Figure 7: the 936M mechanism controls in Table 2, covering A4K8 and A3K16.
  • Figure 8: the 936M A1K16 component ablations in Table 3.

The largest-model trajectories show decreasing losses without visible divergence over the training budget. In the component ablations, removing dense anchors or token-choice bias produces persistently higher validation loss, while residual scaling and the dense prefix provide smaller improvements. These curves complement the final-checkpoint results by showing the optimization trajectories directly.

In the final published version, we will include the complete set of training and validation curves for the experiments reported in the paper in the appendix for easy reference.

AuthorsOpenReview ↗

Detailed Response 2

W2/Q1: Compute-matched comparison and token exposure

Thank you for highlighting the role of training-token exposure. We agree that processing more tokens within the same training budget can contribute to X-MoD's gains. This is enabled by its lower per-token computation.

Training computational cost is primarily determined by total training FLOPs, inference computational cost by FLOPs per processed token, and model-capacity constraints at deployment by active and total parameter counts. Accordingly, we match training FLOPs and active-equivalent model size while maintaining comparable total parameter counts. Since C=DϕtrainC=D\phi_{\mathrm{train}}, these fixed configurations have different token budgets when their per-token costs differ; matching token counts instead would change their training-compute budgets.

To address the shorter-context setting, we conduct an additional comparison at L=4kL=4\mathrm{k} and Nact=936MN_{\mathrm{act}}=936\mathrm{M}, retaining the main-experiment training budget of C=1.5×1020C=1.5\times10^{20} FLOPs.

We choose the configurations in two steps. First, our scaling analysis motivates adapting sparsity to context length, and the 4k sweep in Fig. 2(a) achieves its lowest measured loss at K=6K=6. We therefore select K=6K=6. Second, we pair MoE 8:48 with two shared experts with X-MoD A3K6, matching nominal sparsity while keeping total capacity close. We use MoE 8:48 rather than 8:64 because the latter would introduce a substantially larger total-parameter budget; choosing A=3A=3 gives X-MoD a comparable parameter count.

ModelTotal parametersRelative training FLOPs/tokenValidation loss
MoE 8:48 s23.42B1.002.469
X-MoD A3K63.27B0.792.452

Relative FLOPs/token are normalized to the corresponding dense reference. At the matched training budget, X-MoD processes approximately 1.27×1.27\times as many training tokens as MoE.

X-MoD achieves lower validation loss in this shorter-context setting with slightly fewer total parameters and 21% lower per-token training FLOPs. Training and validation curves are provided in Appendix C.6, Fig. 9.

We acknowledge that X-MoD is constrained by the context visible to its sparse layers and the saturating gains from increasing AA. At fixed active-equivalent size, our KK-sweeps show that expanding total capacity through greater token sparsity initially lowers loss, but excessive sparsity eventually raises it as sparse-layer context shrinks. Increasing AA provides diminishing returns. We therefore do not claim that X-MoD outperforms MoE across all total-parameter budgets.

At L=32kL=32\mathrm{k}, Table 1 demonstrates competitive or better model quality against the selected MoE 8:64 and 8:128 baselines. The 936M systems benchmark additionally shows 1.73×1.73\times and 1.42×1.42\times higher validation-forward throughput for the corresponding X-MoD configurations. Thus, even when additional token exposure contributes to the quality gains, these results demonstrate effective use of a matched training budget together with practical forward-efficiency benefits. They suggest that sparse-depth routing could offer an effective alternative to sparse-width routing for model-capacity expansion in suitable settings.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary for the Area Chairs

We sincerely thank all PCs, SACs, ACs, and Reviewers for their time and effort in evaluating our paper and for their constructive feedback. Below is a brief summary of the revisions and reviewer feedback.

  • Reviewer MUhq: Added training curves and shorter-context comparisons addressed concerns about training stability and X-MoD's usefulness relative to MoE. The reviewer confirmed that all points raised in the review had been addressed.
  • Reviewer 9qYz: Configuration-selection analyses, clearer ablation definitions, restored code access, and full/reduced-law comparisons addressed concerns about practical utility and reproducibility. The reviewer confirmed that the original concerns were substantially addressed. We have since added the separately requested attention-gating ablation.
  • Reviewer yPNZ: Decoding experiments and routing-overhead analysis addressed practical-efficiency concerns, as explicitly acknowledged by the reviewer. We subsequently added experiments at fixed model size with different training budgets to investigate the remaining question about compute dependence.

Our work contributes a scalable sparse-depth architecture and practical configuration guidance, now supported by broader experimental evidence.

All newly added manuscript content is highlighted in blue. Detailed responses to each reviewer's comments are provided in our individual replies.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.