- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes CARVING, a method for jointly learning the computational topology and parameters of standard and looped Transformers under an explicit compute budget.
The method begins with an overconnected Transformer, introducing learnable hard-concrete gates for individual attention and MLP sublayers as well as connections between nonadjacent layers. The gates are optimized jointly with the model parameters using a Lagrangian objective that balances language-modeling loss against expected computational cost. After the search phase, inactive components are removed and the resulting sparse computation graph is compiled into a smaller Transformer with a fixed, nonuniform topology.
The authors evaluate CARVING on Fully Looped Transformers ranging from 140M to 1.3B parameters and standard Transformers up to 2.7B parameters. They compare the resulting models against full-depth Transformers, shallower models with similar inference costs, and post-hoc structured pruning baselines.
The main results demonstrate that CARVING can improve validation bits per byte (BPB) while substantially reducing inference FLOPs, KV-cache requirements, and decoding costs. At the 1.3B looped scale, the carved model reduces validation BPB from 0.6716 to 0.6609 while achieving approximately 3.5× higher decoding throughput. At the 2.7B standard scale, BPB decreases from 0.7072 to 0.7008 with approximately 2.1× higher throughput.
Additional analyses investigate the contribution of learned inter-layer connections, compute-optimal scaling, consistency of learned topologies across model sizes, and architectural specialization on synthetic reasoning tasks.
The paper's central contribution is demonstrating that jointly optimizing sublayer selection and inter-layer connectivity during training can produce nonuniform Transformer architectures with favorable predictive-quality and inference-efficiency tradeoffs.
Critical strengths
- A meaningful extension of structured architecture optimization, supported by strong empirical improvements in validation quality and inference efficiency.
The principal strength is the combination of a flexible topology search space with practical, measurable improvements in the resulting Transformer architectures.
Unlike conventional structured pruning methods that remove components from already pretrained models, CARVING learns which sublayers and inter-layer connections to retain during training. It also differs from architecture-search methods that learn a small repeated cell, allowing different computational structures at different depths.
Although the hard-concrete gates and Lagrangian optimization are established techniques, their application to joint sublayer and inter-layer topology optimization in standard and looped language models is a meaningful contribution.
The empirical improvements are substantial.
In Table 1, CARVING reduces inference FLOPs by approximately 47% and 44% at the 1.3B looped and 2.7B standard scales, respectively, while achieving lower validation BPB than the corresponding deep models.
The improvements also extend beyond theoretical FLOP reductions. The carved architectures require substantially smaller KV caches and achieve approximately 3.5× and 2.1× higher measured decoding throughput on an H100 GPU.
The comparison with shallower Transformers is particularly important. At similar inference costs and approximately matched retraining FLOPs, the carved models achieve better validation quality, suggesting that the gains are not simply explained by reducing model depth or training a smaller model on more tokens.
These results are consequential because they demonstrate a practical improvement in the language-modeling loss versus inference-efficiency tradeoff, which directly supports the paper's central objective.
- Broad experimental evaluation and informative ablations that support the importance of learning nonuniform computation graphs.
The paper provides a relatively comprehensive empirical evaluation across model scales, Transformer families, training budgets, and architectural components.
The sublayer-only ablation is especially relevant. By disabling inter-layer connection gates while retaining sublayer selection, the authors investigate whether the additional connectivity search contributes beyond conventional structured layer removal.
For standard Transformers, the complete method achieves larger FLOP reductions and better validation performance than the sublayer-only configurations. This supports the importance of searching over connectivity rather than solely deciding which attention or MLP modules to remove.
The learned topologies also exhibit consistent structural patterns, including retention of early attention layers, reduced attention usage at greater depths, and relatively greater preservation of MLP computation.
The experiments on synthetic tasks provide complementary evidence of architectural specialization. In modular addition, the method identifies an attention-only topology matching the minimum-cost successful architecture found through exhaustive sublayer-subset evaluation under the stated accuracy criterion.
The paper further reports compute-optimal scaling experiments, post-hoc pruning comparisons, actual decoding throughput, and detailed downstream evaluations.
Taken together, these analyses substantially strengthen the contribution beyond a single performance comparison and suggest that nonuniform architectures can be a useful design dimension for Transformer efficiency.
Critical weaknesses
- The experimental protocols do not fully disentangle the value of the learned topology from additional search compute and search-stage weight optimization.
My main methodological concern is that the strongest results use different training protocols for looped and standard Transformers, complicating the interpretation of the improvements as consequences of the learned architecture itself.
For looped Transformers, the discovered topology is retrained from scratch, which provides useful evidence of architectural transferability. However, Appendix A.5 explicitly states that topology-search computation, amounting to approximately 15% of the reference training budget in the headline setting, is additional to the matched retraining FLOPs.
Consequently, the headline looped comparison does not strictly match total end-to-end training computation, including architecture search.
For standard Transformers, the search cost is included in the training budget, but the strongest compressed models continue training from the search-stage weights.
Importantly, Appendix B.3 shows that the 1.3B standard Transformer retrained from scratch at 0.56× inference FLOPs reaches 0.828920 BPB, compared with 0.819331 for the deep reference. Its performance is nearly identical to the shallower baseline, which reaches 0.828956.
The more favorable results of the aggressively compressed standard models therefore cannot be attributed to the discovered topology alone. They also depend on the parameter initialization and optimization trajectory inherited from the search.
The authors acknowledge this distinction, but it is consequential for interpreting the paper's central claim that learning topology provides better architectures than conventional Transformer designs.
In addition, the post-hoc pruning comparison does not fully resolve this issue. The pruning baselines use one-shot importance estimates and relatively limited post-pruning recovery, while CARVING optimizes its structure throughout an earlier training phase. Their substantially worse performance may partly reflect the difficulty of recovering from aggressive post-hoc deletion rather than the intrinsic superiority of the selected topology.
A stronger experimental design would match the complete search-plus-training budget for looped models and more clearly separate topology quality from optimization-path advantages in standard models.
This would help establish whether CARVING discovers intrinsically superior architectures or primarily benefits from the interaction between structural optimization and weight training.
- Improvements in validation language-modeling loss do not consistently translate into better downstream capabilities, limiting the practical significance of the claimed quality-efficiency improvements.
Despite the strong improvements in validation BPB, the carved models consistently underperform their larger deep baselines on aggregate downstream evaluations.
In Table 1, the 1.3B looped carved model achieves CORE-22 of 44.59 compared with 47.24 for the deep baseline. At the 2.7B standard scale, CARVING obtains 42.54 compared with 45.44.
The gaps of approximately 2.65 and 2.90 points are substantial relative to the reported validation-loss improvements.
Moreover, the carved 2.7B model slightly underperforms the shallower baseline on CORE-22 (42.54 versus 42.83), despite achieving better validation BPB.
This suggests that the optimized next-token objective may favor removing components that contribute relatively little to average language-modeling loss but remain important for downstream reasoning, contextual integration, or few-shot generalization.
The authors acknowledge this limitation in Section 7 and suggest that fewer retained attention sublayers may contribute to the observed degradation. However, this explanation is not directly validated.
The pass@k results at equal inference FLOPs are encouraging, particularly for the 2.7B standard model, but they answer a different question: how much task performance can be obtained by allocating a fixed inference budget across potentially more generated samples. They do not establish that the carved model has superior per-sample predictive capabilities.
This distinction is important because the paper motivates CARVING as improving the quality-efficiency frontier of Transformer architectures. The evidence convincingly supports an improvement in validation BPB and throughput, but it does not establish a uniformly better frontier for downstream capabilities.
A clearer characterization of which abilities are lost, and whether the topology search can preserve them, would substantially strengthen the practical significance of the work.
Questions for authors
- End-to-end compute matching and the independent contribution of the learned topology (Critical Weakness 1).
Can the authors provide comparisons that include topology-search FLOPs in the total training budget for the headline looped Transformer results?
In particular, does CARVING retain its validation-quality advantage when the deep, shallower, and carved models are compared under exactly the same total search-plus-training compute?
For standard Transformers, can the authors clarify how much of the headline gain comes from the learned topology itself versus continuing from the search-stage weights?
The difference between the scratch-retrained and search-weight-continued standard models in Appendix B.3 suggests that this distinction is particularly important at aggressive compression ratios.
A compute-matched experiment comparing scratch retraining, search-weight continuation, and appropriately initialized shallower baselines would materially strengthen the causal attribution of the reported gains.
- Downstream capability degradation and the role of retained attention layers (Critical Weakness 2).
Can the authors provide a more detailed analysis of why the carved models achieve lower validation BPB but systematically worse CORE-22 scores than their deep baselines?
In particular, is the degradation concentrated in tasks requiring long-range dependencies, few-shot reasoning, or other capabilities that may rely disproportionately on attention computation?
A targeted ablation restoring selected attention layers, while maintaining a comparable inference budget through compensating removals elsewhere, could help establish whether the learned attention allocation is responsible for these losses.
It would also be valuable to clarify whether the observed capability degradation persists across different context lengths and evaluation regimes.
These results would materially affect my assessment of whether CARVING improves practical model efficiency beyond the language-modeling objective used during architecture search.
- Comparison with simpler nonuniform architectures and stronger pruning procedures (Critical Weakness 1).
How much of CARVING's improvement requires the complete learned topology, rather than simpler hand-designed or heuristic alternatives inspired by the observed structural patterns?
For example, could the authors compare against a nonuniform architecture that retains more attention in early layers, progressively reduces attention with depth, and adds a small number of fixed long-range residual connections, while matching the carved model's inference FLOPs?
Such a comparison would help distinguish the benefit of optimization-based topology discovery from the benefit of using a nonuniform architecture in general.
Additionally, would iterative structured pruning with importance-score recomputation and a better-tuned recovery schedule substantially narrow the gap to CARVING?
These comparisons would help determine whether the learned graph itself is the key contribution or whether much of the performance gain can be recovered through simpler architectural design and optimization strategies.
3 comments
- RE: Downstream degradation. Under our new method, Carved models now beat both the full-depth and shallower-depth baselines on the downstream tasks. See the attached tables part of the general comment. Specifically, averaged CORE-22 relative to the full-depth model is now +0.8 at looped 1.3B and +1.0 at standard 2.7B, whereas it was −2.65 and −2.90 in the submission. On the 13 stable CORE tasks (those, as defined by Heineman et al, 2025, whose scores increase with model size and with a signal-to-noise ratio of at least 1.5), the gain is significant (i.e. +3.5 points at looped 1.3B, 95% interval 2.5 to 4.5). One likely reason fits the hypothesis mentioned in the review, where the previous submission’s fixed FLOP target kept a third to a half of the attention sublayers, while our new revised method kept 69–90%. We indeed perform a knockout analysis (remove one sublayer at a time from the final trained model and measure the rise in validation loss) to validate the criticality of the sublayers Carving retains, and find that the median kept sublayer of the carved 2.7B model raises the loss *more than five times as much* as removing the median sublayer of the full-depth baseline. In standard Transformers this ratio grows with scale, from about 2.4 times at 140M to 5.5 times at 2.7B.
- RE: Which abilities are still weaker. Notably, if we keep inference-cost equal for pass@k, Carved models outperform baselines on TriviaQA, LAMBDA, GSM8K, and SQuAD for both pairs, and on MATH-500 at standard 2.7B. At 16 samples each for pass@k, the carved 2.7B model only trails on TriviaQA and LAMBADA, that is, knowledge recall and long-context prediction (and by 0.1 point on SQuAD). We also find that after one and two epochs instruction tuning, carved models keep a significant CORE-22 lead on the 13 stable tasks.
- RE: Simpler nonuniform architectures. The new method learns a different topology pattern compared with the previous version, namely the standard models remove one early block of layers ending about a third of the way through the depth (which is the opposite of the late-layer redundancy reported in pruning literature). As mentioned above, the kept sublayers are also more critical to the Carved model, where removing the median kept sublayer of the carved 2.7B model costs more than five times as much loss as removing the median sublayer of the full-depth model. We also trained a fixed nonuniform topology from scratch at looped 370M without Carving by taking the removals of our carved 140M, 770M and 1.3B looped models, mapped by relative depth. It reaches 0.793 bpb against 0.788 (carved) and 0.808 (full-depth). Model Carving’s found topology, thus, is a better product of optimization-based topology discovery. We also tested hand-designed, MLP-heavy controls for the symbolic tasks in the submission, which perform worse than carved models.
Dear Reviewer aKrE,
Thank you again for your review. As the discussion period ends on 18 November, we would be grateful to know whether our improvements consolidating the carving process, adding stronger baselines, and improving downstream performance address your main concerns. We are happy to answer any further questions.
We would like to thank the reviewer for their careful review and feedback! The review’s two critical weaknesses are the two main things we’ve changed. In short, compute is now matched end to end as there is only one training run for Model Carving (no separate search vs. retrain phase), carved models beat both Shallower-depth and Full-depth on CORE-22 (see shared table), and the pruning baselines are stronger.