- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
X-MoD extends MoD (Mixture-of-Depths) by enabling A (anchor stride) x K (token sparsity ratio) sparse layers between dense layers and only 1/K tokens are routed to the next layers. So, the total parameters can grow far beyond the size of active-equivalent dense or MoE models. Three components are used for stable training of these deep networks: (1) layer-wise gating with variance, (2) depth-wise token balancing, and (3) several successive dense layers before the sparse-dense blocks. This paper also fit a scaling law that shows how K and A affect the validation loss gap between X-MoD and a matched-FLOPs dense baseline.
Critical strengths
- This paper explores the depth direction of conditional computation, which is less studied than width-wise MoE, and identifies a clear limit of MoD (total-to-active ratio below 2).
- Table 1 shows that total parameters can scale substantially (e.g., 936M active to 8.24B total) while active-equivalent size stays fixed.
Critical weaknesses
- Generalization of the law is unclear. It is fitted on 109 observations with one dataset, one optimizer, and models up to 1.65B, with the sweeps never crossed. Whether it holds at larger scales, where routing dynamics may differ, is an open question.
- Section 3 is hard to follow. It blends design choices, heuristic assumptions, and a complex parameterization into one formula, with heavy notation and abstract terms (capacity reward, sparse-context correction). A short high-level narrative before the algebra would help. Several symbols are not defined in the main text: u, w, z, K_hat, A_hat, and tau and the update rule in Eq. 8.
- Fig. 2 is hard to read on its own. The fixed settings for each sweep (e.g., N_act, A, L) are only in Appendix B.4 and should appear in the caption.
- Reproducibility. The anonymous repo in footnote 1 returned "file not found" when I checked.
Questions for authors
- Table 10 shows near-complete coverage (99.99% for A4K8) with very low consecutive-layer overlap, so tokens appear to take turns being updated. Is this still conditional computation, or just capacity spread evenly? Does routing correlate with token importance?
- Is the router called on every sparse layer? If so, how much overhead it incurs?
- X-MoD uses 161–298 layers. Do the per-layer overhead and sequential depth offset the FLOP savings in wall-clock terms, especially for decoding? Report decoding latency at several batch sizes might help further evaluate the value of this work.
9 comments
Detailed Response 1
W1: Generalization of the scaling law
Thank you for your thoughtful comments on the generality of the law. We agree that evaluating a wider range of training settings and larger models would be valuable. We appreciate this suggestion and would like to provide some further context on the current validation.
The fitting experiments cover context lengths from 2k to 32k and active-equivalent sizes from 165M to 936M, with 109 independent training runs contributing one validation-loss observation each at the target compute budget. To examine transfer beyond this range, we reserve the 64k and 1.65B settings for out-of-range testing and evaluate them with the fitted coefficients frozen. The corresponding protocol is described in Section 4.1 and Appendix B.4–B.5.
Our aim in organizing the sweeps was to study the interactions of K with context length, active-equivalent size and anchor stride under controlled conditions. We therefore vary K jointly with each of L, active-equivalent size and A. As a complementary check, Fig. 3(b–d) evaluates predictions for complete context-length, anchor-stride or model-scale groups withheld during fitting.
For ease of reference, we summarize the prediction errors and configuration choices for the two out-of-range settings in Fig. 3(a) below:
| Out-of-range setting | MAE | RMSE | Lowest predicted-loss candidate | Lowest measured-loss candidate |
|---|---|---|---|---|
| 64k context | 0.008886 | 0.009446 | A1K22 | A1K22 |
| 1.65B active-equivalent size | 0.006970 | 0.007614 | A1K16 | A1K16 |
In both settings, the candidate with the lowest predicted loss also has the lowest measured loss among those evaluated. We find this agreement, together with the grouped holdout results, encouraging for the law's use in sparse-depth configuration selection. We also appreciate the reviewer's emphasis on examining how these findings extend to broader conditions. Building on the present results, we will conduct more extensive experiments across training settings and evaluate larger active-equivalent models in future work, to further assess the generality of these scaling relationships.
Detailed Response 2
W2: Intuition and notation
Thank you for the helpful presentation suggestion. The intuition behind the law can be summarized as follows:
At fixed training compute and active-equivalent size, increasing sparsity expands total parameter capacity but reduces the context available to each sparse layer. The law captures the balance between these two effects. Anchor stride controls how the active-equivalent budget is allocated between dense anchors and sparse refinements, with saturating benefits as the sparse share increases. These effects are modeled through the loss difference relative to a matched dense reference.
We have also made corresponding clarifications in the revised manuscript: Section 3.2 now specifies the roles of u and w, the text immediately following Eq. (14) defines all three fitted weights u, w and z, and the sentence preceding Eq. (16) explicitly identifies the continuous optimum at A = 1. All these changes are highlighted in blue. The relevant notation is explained below:
- , and are the fitted weights of the capacity, sparse-context and anchor-stride terms, respectively. Their fitted values are reported in Appendix C.1, Table 8.
- The starred values of and in Fig. 2 identify the lowest measured loss in each evaluated sweep. In particular, denotes the best tested stride. In Eq. (16), denotes the approximate continuous optimum derived at A = 1.
Regarding in Eq. (8), it controls the strength of the token-choice bias. The bias starts at zero at the beginning of each sparse interval, increases by one whenever token i is selected, remains unchanged when it is skipped, and is reset at the next dense anchor. Subtracting from the current routing logit discourages repeated selection while preserving input-dependent routing.
W3: Fixed settings in Fig. 2
Thank you for this helpful suggestion. We have added the following concise description to the Fig. 2 caption, consistent with Appendix B.4, and highlighted it in blue:
Fixed settings: (a,b) , A = 1; (c) L = 32k, A = 1; (d) , L = 32k.
The detailed sweep grids and compute-matching protocol remain in Appendix B.4.
W4: Code availability
Access through the original anonymous link has been restored. We have provided a detailed explanation in our general Official Comment shared with all reviewers and the Area Chair, titled “Clarification of code availability and restoration of access”. We sincerely apologize for the inconvenience caused by the interruption and would be grateful if you could revisit the repository.
Summary of Our Response
We sincerely thank the reviewer for the thoughtful assessment and for recognizing the potential of sparse-depth capacity scaling. Below, we summarize our responses and the corresponding manuscript clarifications.
For transparency, AI tools assisted with language polishing and presentation; the authors are responsible for the technical content, experimental results, and conclusions.
1. Generalization (W1). We summarize grouped holdout and 64k/1.65B out-of-range results, and will extend evaluation to broader training settings and larger models in future work.
2. Presentation and code access (W2–W4). We have clarified notation and added fixed sweep settings to Fig. 2, with changes highlighted in blue. Repository access has been restored, as explained in our general Official Comment.
3. Conditional routing and scoring cost (Q1–Q2). We explain why broad coverage remains compatible with input-dependent routing and show that scoring FLOPs are small relative to attention and FFN computation.
4. Realized efficiency (Q3). Existing end-to-end measurements show net training and validation-forward speedups with routing, token movement and sequential-layer overhead included.
We appreciate the reviewer's time and will be happy to clarify any remaining questions during the discussion period.