acceptodds
Under review as a conference paper at ICLR 2027

Model Carving: Optimizing the Topology of Standard and Looped Transformers

Abstract

Transformer architectures are largely fixed: the same block repeats across layers, and the connections between them, for both standard and looped transformers, are prescribed in advance rather than learned. We ask: can a Transformer's topology, learned jointly with its weights, beat a hand-designed one on the quality-efficiency frontier? In this paper, we introduce Carving, a method to learn a model's topology during training. Carving places a gate on each edge of a Transformer's computation graph, trains the gates jointly with the weights, and compiles the remaining graph into a sparser, carved model. On both standard and looped Transformers, carved models beat the larger models they were carved from, reaching lower validation loss with half the inference FLOPs, one-third and one-half the KV cache, and and faster decoding throughput at the 1.3B and 2.7B scales, respectively. Carved models also beat training-FLOP-matched shallower Transformers with the same inference cost in validation quality and throughput. The architectures learned share similar features across model sizes, scales, and seeds, removing 35% of the MLP sublayers and 53% of the attention sublayers on average. These results suggest that standard, homogeneously repeated blocks are not always the most efficient design. Carved models also outperform larger models pruned post-hoc to their same inference cost on the quality-efficiency frontier, showing that the learned topology both differs from and beats standard post-hoc compression.

Primary area:
foundation or frontier models, including LLMs
Keywords:
neural architecture search, learned Transformer topology, looped Transformers, structured pruning, KV cache reduction, efficient inference, large language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_aKrEOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CARVING, a method for jointly learning the computational topology and parameters of standard and looped Transformers under an explicit compute budget.

The method begins with an overconnected Transformer, introducing learnable hard-concrete gates for individual attention and MLP sublayers as well as connections between nonadjacent layers. The gates are optimized jointly with the model parameters using a Lagrangian objective that balances language-modeling loss against expected computational cost. After the search phase, inactive components are removed and the resulting sparse computation graph is compiled into a smaller Transformer with a fixed, nonuniform topology.

The authors evaluate CARVING on Fully Looped Transformers ranging from 140M to 1.3B parameters and standard Transformers up to 2.7B parameters. They compare the resulting models against full-depth Transformers, shallower models with similar inference costs, and post-hoc structured pruning baselines.

The main results demonstrate that CARVING can improve validation bits per byte (BPB) while substantially reducing inference FLOPs, KV-cache requirements, and decoding costs. At the 1.3B looped scale, the carved model reduces validation BPB from 0.6716 to 0.6609 while achieving approximately 3.5× higher decoding throughput. At the 2.7B standard scale, BPB decreases from 0.7072 to 0.7008 with approximately 2.1× higher throughput.

Additional analyses investigate the contribution of learned inter-layer connections, compute-optimal scaling, consistency of learned topologies across model sizes, and architectural specialization on synthetic reasoning tasks.

The paper's central contribution is demonstrating that jointly optimizing sublayer selection and inter-layer connectivity during training can produce nonuniform Transformer architectures with favorable predictive-quality and inference-efficiency tradeoffs.

Critical strengths
  1. A meaningful extension of structured architecture optimization, supported by strong empirical improvements in validation quality and inference efficiency.

The principal strength is the combination of a flexible topology search space with practical, measurable improvements in the resulting Transformer architectures.

Unlike conventional structured pruning methods that remove components from already pretrained models, CARVING learns which sublayers and inter-layer connections to retain during training. It also differs from architecture-search methods that learn a small repeated cell, allowing different computational structures at different depths.

Although the hard-concrete gates and Lagrangian optimization are established techniques, their application to joint sublayer and inter-layer topology optimization in standard and looped language models is a meaningful contribution.

The empirical improvements are substantial.

In Table 1, CARVING reduces inference FLOPs by approximately 47% and 44% at the 1.3B looped and 2.7B standard scales, respectively, while achieving lower validation BPB than the corresponding deep models.

The improvements also extend beyond theoretical FLOP reductions. The carved architectures require substantially smaller KV caches and achieve approximately 3.5× and 2.1× higher measured decoding throughput on an H100 GPU.

The comparison with shallower Transformers is particularly important. At similar inference costs and approximately matched retraining FLOPs, the carved models achieve better validation quality, suggesting that the gains are not simply explained by reducing model depth or training a smaller model on more tokens.

These results are consequential because they demonstrate a practical improvement in the language-modeling loss versus inference-efficiency tradeoff, which directly supports the paper's central objective.

  1. Broad experimental evaluation and informative ablations that support the importance of learning nonuniform computation graphs.

The paper provides a relatively comprehensive empirical evaluation across model scales, Transformer families, training budgets, and architectural components.

The sublayer-only ablation is especially relevant. By disabling inter-layer connection gates while retaining sublayer selection, the authors investigate whether the additional connectivity search contributes beyond conventional structured layer removal.

For standard Transformers, the complete method achieves larger FLOP reductions and better validation performance than the sublayer-only configurations. This supports the importance of searching over connectivity rather than solely deciding which attention or MLP modules to remove.

The learned topologies also exhibit consistent structural patterns, including retention of early attention layers, reduced attention usage at greater depths, and relatively greater preservation of MLP computation.

The experiments on synthetic tasks provide complementary evidence of architectural specialization. In modular addition, the method identifies an attention-only topology matching the minimum-cost successful architecture found through exhaustive sublayer-subset evaluation under the stated accuracy criterion.

The paper further reports compute-optimal scaling experiments, post-hoc pruning comparisons, actual decoding throughput, and detailed downstream evaluations.

Taken together, these analyses substantially strengthen the contribution beyond a single performance comparison and suggest that nonuniform architectures can be a useful design dimension for Transformer efficiency.

Critical weaknesses
  1. The experimental protocols do not fully disentangle the value of the learned topology from additional search compute and search-stage weight optimization.

My main methodological concern is that the strongest results use different training protocols for looped and standard Transformers, complicating the interpretation of the improvements as consequences of the learned architecture itself.

For looped Transformers, the discovered topology is retrained from scratch, which provides useful evidence of architectural transferability. However, Appendix A.5 explicitly states that topology-search computation, amounting to approximately 15% of the reference training budget in the headline setting, is additional to the matched retraining FLOPs.

Consequently, the headline looped comparison does not strictly match total end-to-end training computation, including architecture search.

For standard Transformers, the search cost is included in the training budget, but the strongest compressed models continue training from the search-stage weights.

Importantly, Appendix B.3 shows that the 1.3B standard Transformer retrained from scratch at 0.56× inference FLOPs reaches 0.828920 BPB, compared with 0.819331 for the deep reference. Its performance is nearly identical to the shallower baseline, which reaches 0.828956.

The more favorable results of the aggressively compressed standard models therefore cannot be attributed to the discovered topology alone. They also depend on the parameter initialization and optimization trajectory inherited from the search.

The authors acknowledge this distinction, but it is consequential for interpreting the paper's central claim that learning topology provides better architectures than conventional Transformer designs.

In addition, the post-hoc pruning comparison does not fully resolve this issue. The pruning baselines use one-shot importance estimates and relatively limited post-pruning recovery, while CARVING optimizes its structure throughout an earlier training phase. Their substantially worse performance may partly reflect the difficulty of recovering from aggressive post-hoc deletion rather than the intrinsic superiority of the selected topology.

A stronger experimental design would match the complete search-plus-training budget for looped models and more clearly separate topology quality from optimization-path advantages in standard models.

This would help establish whether CARVING discovers intrinsically superior architectures or primarily benefits from the interaction between structural optimization and weight training.

  1. Improvements in validation language-modeling loss do not consistently translate into better downstream capabilities, limiting the practical significance of the claimed quality-efficiency improvements.

Despite the strong improvements in validation BPB, the carved models consistently underperform their larger deep baselines on aggregate downstream evaluations.

In Table 1, the 1.3B looped carved model achieves CORE-22 of 44.59 compared with 47.24 for the deep baseline. At the 2.7B standard scale, CARVING obtains 42.54 compared with 45.44.

The gaps of approximately 2.65 and 2.90 points are substantial relative to the reported validation-loss improvements.

Moreover, the carved 2.7B model slightly underperforms the shallower baseline on CORE-22 (42.54 versus 42.83), despite achieving better validation BPB.

This suggests that the optimized next-token objective may favor removing components that contribute relatively little to average language-modeling loss but remain important for downstream reasoning, contextual integration, or few-shot generalization.

The authors acknowledge this limitation in Section 7 and suggest that fewer retained attention sublayers may contribute to the observed degradation. However, this explanation is not directly validated.

The pass@k results at equal inference FLOPs are encouraging, particularly for the 2.7B standard model, but they answer a different question: how much task performance can be obtained by allocating a fixed inference budget across potentially more generated samples. They do not establish that the carved model has superior per-sample predictive capabilities.

This distinction is important because the paper motivates CARVING as improving the quality-efficiency frontier of Transformer architectures. The evidence convincingly supports an improvement in validation BPB and throughput, but it does not establish a uniformly better frontier for downstream capabilities.

A clearer characterization of which abilities are lost, and whether the topology search can preserve them, would substantially strengthen the practical significance of the work.

Questions for authors
  1. End-to-end compute matching and the independent contribution of the learned topology (Critical Weakness 1).

Can the authors provide comparisons that include topology-search FLOPs in the total training budget for the headline looped Transformer results?

In particular, does CARVING retain its validation-quality advantage when the deep, shallower, and carved models are compared under exactly the same total search-plus-training compute?

For standard Transformers, can the authors clarify how much of the headline gain comes from the learned topology itself versus continuing from the search-stage weights?

The difference between the scratch-retrained and search-weight-continued standard models in Appendix B.3 suggests that this distinction is particularly important at aggressive compression ratios.

A compute-matched experiment comparing scratch retraining, search-weight continuation, and appropriately initialized shallower baselines would materially strengthen the causal attribution of the reported gains.

  1. Downstream capability degradation and the role of retained attention layers (Critical Weakness 2).

Can the authors provide a more detailed analysis of why the carved models achieve lower validation BPB but systematically worse CORE-22 scores than their deep baselines?

In particular, is the degradation concentrated in tasks requiring long-range dependencies, few-shot reasoning, or other capabilities that may rely disproportionately on attention computation?

A targeted ablation restoring selected attention layers, while maintaining a comparable inference budget through compensating removals elsewhere, could help establish whether the learned attention allocation is responsible for these losses.

It would also be valuable to clarify whether the observed capability degradation persists across different context lengths and evaluation regimes.

These results would materially affect my assessment of whether CARVING improves practical model efficiency beyond the language-modeling objective used during architecture search.

  1. Comparison with simpler nonuniform architectures and stronger pruning procedures (Critical Weakness 1).

How much of CARVING's improvement requires the complete learned topology, rather than simpler hand-designed or heuristic alternatives inspired by the observed structural patterns?

For example, could the authors compare against a nonuniform architecture that retains more attention in early layers, progressively reduces attention with depth, and adds a small number of fixed long-range residual connections, while matching the carved model's inference FLOPs?

Such a comparison would help distinguish the benefit of optimization-based topology discovery from the benefit of using a nonuniform architecture in general.

Additionally, would iterative structured pruning with importance-score recomputation and a better-tuned recovery schedule substantially narrow the gap to CARVING?

These comparisons would help determine whether the learned graph itself is the key contribution or whether much of the performance gain can be recovered through simpler architectural design and optimization strategies.

3 comments

We would like to thank the reviewer for their careful review and feedback! The review’s two critical weaknesses are the two main things we’ve changed. In short, compute is now matched end to end as there is only one training run for Model Carving (no separate search vs. retrain phase), carved models beat both Shallower-depth and Full-depth on CORE-22 (see shared table), and the pruning baselines are stronger.

  • RE: End-to-end compute matching. We have updated the method so Carving runs as one training run, and every FLOP is charged to the same budget as the full-depth and shallower-depth models. Under this, carved models reach lower bpb across all comparisons. Further, given this compute-matched setting, we calculate that the full-depth looped model would need 1.71× the training FLOPs to reach the loss of the carved looped 1.3B model’s loss.
  • RE: Topology vs. optimization. We agree our submission did not separate the two. We make the main claim now about Carving’s single run, where, at equal training FLOPs, it beats training the full-depth model, training a shallower-depth model, distilling a student, and pruning the full-depth model. See the general comment for the results of separating the overparameterized Transformer from the gate removals. We also tested the optimization, where at 140M and 370M, we train the final carved architecture from scratch at equal training FLOPs and this recovers all of the gains from the carved looped Transformer and carved standard Transformer.
  • RE: Stronger pruning baselines. We replace the one-shot baselines with Sheared-LLaMA-style (learned masks), Minitron-style (width pruning with distillation) and ShortGPT-style (layer removal) full pipelines at standard 1.3B, with each trained to the carved run's total FLOPs. The carved model leads all three on bpb and by 3.1-3.7 CORE-22 points. It also leads by 1.9 points a Minitron-style model that we gave 1.21× the training FLOPs, and distilled students at all six sizes we distilled using the conventional KL logit distillation. We also ran the suggested ShortGPT-style removal, with importance recomputed before each of seven removals and retraining after each. Trained to the same total FLOPs, the iterative structured pruning still loses to the carved model by 0.026 bpb and 3.1 CORE-22 points (95% interval 2.0 to 4.2).
  • RE: Downstream degradation. Under our new method, Carved models now beat both the full-depth and shallower-depth baselines on the downstream tasks. See the attached tables part of the general comment. Specifically, averaged CORE-22 relative to the full-depth model is now +0.8 at looped 1.3B and +1.0 at standard 2.7B, whereas it was −2.65 and −2.90 in the submission. On the 13 stable CORE tasks (those, as defined by Heineman et al, 2025, whose scores increase with model size and with a signal-to-noise ratio of at least 1.5), the gain is significant (i.e. +3.5 points at looped 1.3B, 95% interval 2.5 to 4.5). One likely reason fits the hypothesis mentioned in the review, where the previous submission’s fixed FLOP target kept a third to a half of the attention sublayers, while our new revised method kept 69–90%. We indeed perform a knockout analysis (remove one sublayer at a time from the final trained model and measure the rise in validation loss) to validate the criticality of the sublayers Carving retains, and find that the median kept sublayer of the carved 2.7B model raises the loss *more than five times as much* as removing the median sublayer of the full-depth baseline. In standard Transformers this ratio grows with scale, from about 2.4 times at 140M to 5.5 times at 2.7B.
  • RE: Which abilities are still weaker. Notably, if we keep inference-cost equal for pass@k, Carved models outperform baselines on TriviaQA, LAMBDA, GSM8K, and SQuAD for both pairs, and on MATH-500 at standard 2.7B. At 16 samples each for pass@k, the carved 2.7B model only trails on TriviaQA and LAMBADA, that is, knowledge recall and long-context prediction (and by 0.1 point on SQuAD). We also find that after one and two epochs instruction tuning, carved models keep a significant CORE-22 lead on the 13 stable tasks.
  • RE: Simpler nonuniform architectures. The new method learns a different topology pattern compared with the previous version, namely the standard models remove one early block of layers ending about a third of the way through the depth (which is the opposite of the late-layer redundancy reported in pruning literature). As mentioned above, the kept sublayers are also more critical to the Carved model, where removing the median kept sublayer of the carved 2.7B model costs more than five times as much loss as removing the median sublayer of the full-depth model. We also trained a fixed nonuniform topology from scratch at looped 370M without Carving by taking the removals of our carved 140M, 770M and 1.3B looped models, mapped by relative depth. It reaches 0.793 bpb against 0.788 (carved) and 0.808 (full-depth). Model Carving’s found topology, thus, is a better product of optimization-based topology discovery. We also tested hand-designed, MLP-heavy controls for the symbolic tasks in the submission, which perform worse than carved models.

Dear Reviewer aKrE,

Thank you again for your review. As the discussion period ends on 18 November, we would be grateful to know whether our improvements consolidating the carving process, adding stronger baselines, and improving downstream performance address your main concerns. We are happy to answer any further questions.

Reviewer_Bg6kOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript introduces CARVING, a technique that places gates and trains the gates on each edge of a Transformer block in LLMs. The authors present the design of CARVING in standard transformer and looped transformer networks. CARVING consumes the first 15% of the complete training to train the gates. Then CARVING retrains the carved model from scratch or continues from the search weights. The experiments examine the effectiveness of CARVING using one 1.3B looped transformer model and a 2.7B standard transformer model. The authors conclude that CARVING can find smaller models and achieve 50% of the inference FLOPs and 3.5X decode throughput.

Critical strengths
  1. The idea of CARVING is interesting. If successful, it can be adopted as a practical approach for modern LLM architecture search.
Critical weaknesses
  1. The description of CARVING is inconsistent. In the Introduction and Figure 1, the authors state "retraining the carved models from scratch as well as a single continuous training procedure." and "Looped models retrain from scratch with search additional; standard models continue from search weights with search included." While in Section 2.3, the authors state that "Then, we retrain carved looped Transformers from scratch to test the topology alone. For the standard Transformers, the carved topology retrained from scratch..."

  2. In the comparison with base models (1.3B looped Transformer and 2.7B standard Transformer) and shallower models, Table 1 only shows Val. BPB, average of CORE-22, Gen-9, and pass@K for Gen-9. There is not sufficient evidence to support the conslusion "The learned topology alone matches the deep model from a fresh initialization in both families."

    • LLM evaluation needs both validation metrics and downstream tasks. Please show the per-task CORE-22 and Gen-9 results, then justify the conclusion. Validation loss alone is not sufficient. Please refer to papers [1, 2].
    • The average of CORE-22 shows that the Shallower model is better than CARVING for the 2.7B standard transformer. And CARVING is 0.38 higher than Shallower in the 1.3B looped Transformer. Please justify the aforementioned claim.
    • The choice of BPB is also less convincing. Why not directly use validation loss or perplexity, since the validation dataset has the same number of tokens and number of bytes?
  3. Please explain the loss curve drop in Figure 3.

  4. Please justify the choice of tokens per parameter. Why use 100 for a looped transformer and 40 for a standard transformer? Why not follow the Chinchilla law?

  5. The choice of 15% budget is arbitrary. Please justify this selection and how it can generalize to larger models.

  6. The manuscript misses two references that are closely related to CARVING.

[1] Liu, Hong, Sang Michael Xie, Zhiyuan Li, and Tengyu Ma. "Same pre-training loss, better downstream: Implicit bias matters for language models." In International Conference on Machine Learning, pp. 22188-22214. PMLR, 2023. [2] Lourie, Nicholas, Michael Y. Hu, and Kyunghyun Cho. "Scaling Laws Are Unreliable for Downstream Tasks: A Reality Check." In EMNLP (Findings), pp. 16167-16180. 2025.

Questions for authors

Please address my concerns in Critical Weaknesses.

3 comments

We would like to thank the reviewer for their detailed list and feedback! In short, our new training process (gates train in the same run as the weights, no search phase anymore) means every model now follows the exact same protocol, minus the Carving method of course. Carved models now beat the shallower-depth models on CORE-22 at both pairs that were flagged in the review, and both families share one training FLOP budget.

  • RE: Inconsistent description of retraining. We now use a single training run for Carving with no search phase and no retraining.
  • RE: Evidence that "the learned topology alone matches the deep model". We agree and drop that claim, as you were right that retraining from scratch matched the dense baseline at 0.76× inference FLOPs but not at 0.56×. With the revised method, we beat both the full-depth and shallower baselines at isoFLOP and isoToken. This holds even when we retrained the standard and looped carved Transformers from our new method at 140M.
  • RE: Shallower beats CARVING on CORE-22. With the revised method, we also now show that Model Carving produces models that beat the full-depth, shallow-depth, and DenseFormer baselines on CORE-22, pass@k on five generative tasks (GSM8K, SQuAD, MATH-500, LAMBADA, TriviaQA), and chat evaluations after instruction tuning. (and, validation loss, as before.) The general comment gives the main table and the requested per-task breakdown.
  • RE: Why bpb. We used bpb for ease of comparison with publicly available models and other tokenizers. In addition, we’ve rescored publicly released models with our own evaluation harness, and found that our new Carved models have higher downstream performance than released models trained with significantly more compute. The standard 1.3B -> 940M carved model outscores OPT-1.3B and Pythia-1.4B, whose estimated training costs are 2.8 and 4.6 times its own, with fewer inference FLOPs than either, and the 2.7B -> 1.7B carved model outscores the larger Delphi-3.4B.
  • RE: The loss drop in Figure 3. The loss drop is the learning-rate cooldown, as we use the WSD schedule with a constant LR for the first 80% of training and linear decay to zero over the last 20%. We will mark where it starts in the figure.
  • RE: Tokens per parameter, and Chinchilla. Every full-depth model now trains for 50 tokens per parameter in both families. Chinchilla optimum is a useful baseline which we begin with, but we aim to show Carving’s gains hold both isoFLOP and isoToken up to Chinchilla optimum scales through overtraining as is done much in practice now. Further, Chinchilla scaling does not apply to looped Transformers, so we trained our own curves. To this end, we re-fit compute-optimal curves at multiple training budgets for both families, and show that, much like the existing scaling plot (following Hoffmann et al., 2022), the Carved optimum has lower loss at every budget, and this gap grows even larger with compute, from 0.009 to 0.016 bpb in the looped family and from 0.005 to 0.014 in the standard family.
  • RE: The 15% search budget. We have removed this search budget, and the model’s dynamic pressure dictates the sublayer removals. Now, most removals happen in the first 20% of training as an outcome, and we keep the same fixed settings at every size from 140M to 2.7B.
  • RE: DenseFormer and LayerDrop. We now cite DenseFormer and compare against it (see general comment #2) and specify that we use DenseFormer-style connections. We also cite LayerDrop, and compare how it drops whole layers at random during training for subnetworks to be later extracted vs. Model Carving which learns which sublayers to remove.

The revision did address my concern. However, this revision made significant changes to the method. This change does not align with the reviewer guideline "New experiments should not significantly change the content of the submission. Rather, they should be limited in scope and serve to more thoroughly validate existing results from the submission." In accordance with this policy, I will keep my rating.

Thank you again for your thoughtful feedback and for noting that our revisions have strengthened the paper and all your concerns have been resolved.

We respectfully disagree that addressing your comments have significantly altered the paper in a way that precludes a score increase.

The new experiments consolidated the method from performing two runs into one continuous run with no change to anything else that we had originally set up. We do not believe this constitutes a significant change to the method or the paper including, how the gates are set-up, the main idea, baselines, or any other portion of the paper. It definitely improves the efficiency of the method, without altering the substance of this work.

Given that your concerns have directly been resolved, we kindly request that you might reconsider updating your score.

Reviewer_YGEEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether learning a Transformer’s topology jointly with its weights can improve the trade-off between model quality and computational efficiency. It proposes CARVING, which starts from an overconnected Transformer and learns hard-concrete gates over sublayers and inter-layer connections under a FLOP budget, then compiles the retained components into a smaller model. Experiments on standard and looped Transformers show lower validation loss than larger dense models, with roughly half the inference FLOPs and up to 3.5× higher decoding throughput, although downstream accuracy remains somewhat lower. Experiments on synthetic tasks further show that CARVING can discover compact, task-specific architectures.

Critical strengths
  • This paper proposes carving a Transformer model through hard gating, starting from an overconnected network. The results show validation performance comparable to or better than the dense model, with substantially reduced inference cost.
Critical weaknesses
  • Some experimental comparisons and practical efficiency measurements are missing, and the discussion of related work should be expanded. These gaps make it harder to assess the contribution of carving and its practical benefits. Please see the questions below.
Questions for authors
  1. Since this paper uses an overconnected network as the starting point for pruning, the authors should discuss related work such as Attention Residuals [1]. Could the authors also include the overconnected model pretrained without carving as a baseline?
  2. Appendix B.5 describes the inference-speed evaluation, and “Decode tok/s” in Table 1 reports throughput at the largest tested batch size that fits each model. What is the per-token decoding latency (TPOT) at a batch size of 1? How do the models compare at the same batch size?
  3. Since FLOP reductions may not translate directly into training-time reductions due to sparsity, could the authors report the actual wall-clock training time, including topology search?

[1] Kimi Team et al. Attention Residuals. arXiv:2603.15031, 2026.

3 comments

We would like to thank the reviewer for their assessment and requests.

  • RE: Attention Residuals (ARs): ARs and Model Carving indeed both address redundancy in Transformers. However, Model Carving simply uses *static* scalars for its connections / gates, whereas Attention Residuals compute a soft-max over depth *for each token* in each layer. We note that these need not be mutually exclusive, and will mention ARs in the Related Works!
  • RE: Time per output token (TPOT) at batch 1, and equal batch sizes. We time both models at equal batch size: at batch 1, carved models have 17–29% lower TPOT, and at the largest batch both fit they decode 1.21–1.45× faster. See the table below for per-token decode latency at batch 1, full-depth→carved (in ms/token on one H100).
Model1,024 Context (ms)4,096 Context (ms)8,192 Context (ms)
Looped 1.3B14.8 → 12.215.3 → 12.216.2 → 12.7
Standard 1.3B3.20 → 2.573.40 → 2.713.63 → 2.87
Standard 2.7B5.21 → 3.905.69 → 4.096.36 → 4.49

RE: Wall-clock training time. We no longer run a separate search phase, so the gates train in the same run as the weights. Our savings also don’t depend on sparse kernels as we do recompilation in the background. With regards to wall-clock training time, our full one run of Carving (as there is no longer a search phase) cost 2.14 for Carved models against 1.67 seconds for full-depth Transformers due to the connection mixing (forward, we compute these additional weighted mixes; backwards, we compute gradient of every mixing weight i.e. the sum over all tokens of each source's activations times the incoming gradient). However, we are improving this by fusing the mixing into the model's own kernels, which would bring down the carved update time to exactly the time of the full-depth update.

Thank you for the additional experiments and for answering my questions. These results largely address my concerns, and I will maintain my rating.

Dear Reviewer YGEE,

Thank you for your very helpful feedback. We wanted to gently request that our resolutions and responses be reconsidered for a potential score improvement. Our additional experiments cost several hundred additional GPU hours, and seem to have addressed the original concerns. We are happy to provide any further clarification or perform any runs in the remaining time if questions remain. Thank you!

Comments on the paper

2 comments

We would like to thank all three reviewers. Since the first submission, we’ve changed how Model Carving runs in large part to address many of the valid concerns raised here!

Updates:

  • Single training run: Changed from two -> one training run for all models (removing the separate search and retraining stage). Comparisons are still done at equal training FLOPs. For clarity, we now call the submission's “deep” model the “full-depth” model and the “shallower” model the “shallower-depth” model.
  • Dynamic pruning pressure: We address the difficulty of finding an appropriate Carving strength by replacing the PI controller and 15% search budget with a gate pressure that *smoothly, and dynamically adjusts* based on an estimate of how much loss that sublayer is worth if we used its compute as extra training tokens. We predict this with well-established scaling and power laws, which lets us keep one set of fixed settings for every size.
  • Models: We train these new models for the FLOPs it takes to train the full-depth model to 50 tokens per parameter, and at scales from 140M–2.7B for standard Transformers and 140M–1.3B for looped Transformers.

These changes, notably the dynamic pruning pressure, place the compute-quality tradeoff directly within training itself instead of external through a penalty coefficient, target sparsity, etc., and allow carved models to settle at the balance between a sublayer’s compute cost and its prediction contribution.

Expanded comparisons: We’ve extended baselines, evaluations, and controls. We find Model Carving beats equal-inference-cost baseline models that are (1) distilled students (2) pruned in Sheared-LLaMA-, Minitron-, and ShortGPT-style, and Model Carving’s gains hold through generative pass@k, downstream CORE evals, instruction tuning, and decode throughput against the full-depth models (equal and max batches).

Specific, notable updates to the results: Carved models now reach lower validation bpb and higher CORE-22 than the full-depth and shallower models at both isoFLOP (compute-matched) and isoToken. Also, we ran a second training seed which repeats all of the performance gains.

We report smaller inference savings (up to 34% vs. previously 50%) and decode increase (1.45x faster at equal batches) because a sublayer is now removed only when its compute is worth more as training tokens which remedies the over-Carving problem that resulted in carved models underperforming on downstream evaluations previously. We also trained for 50 tokens per parameter instead of the submission’s 100 (looped) and 40 (standard). Larger runs are within our immediate future work.

Main table (all models train with the same FLOPs):

ModelInference GFLOPs/token ↓Decode tokens/s ↑BPB ↓Perplexity ↓CORE-22 ↑
Looped 1.3B
Full-depth10.674710.7049.0435.2
Shallower8.056330.7109.2133.1
Carved (ours)8.17573 (1.22x)0.6838.4736.0
Standard 1.3B
Full-depth2.822,1250.7299.8029.8
Shallower2.063,0040.73910.0929.3
Distilled2.063,0050.74910.4129.7
Carved (ours)2.032,818 (1.33×)0.7279.7231.2
Standard 2.7B
Full-depth5.621,0490.6768.2836.0
Shallower3.771,6070.6898.6334.4
Carved (ours)3.741,517 (1.45×)0.6758.2737.0

Ablation on where the gain comes from:
Model Carving has two differences from a standard Transformer. It starts from an overparameterized network, with gated, normalized sublayer writes and learned connections, and then removes sublayers during training. To separate the effect of gate removal vs. learned connections, we train the same overparameterized model with zero pressure on the gates (at the same training FLOPs). At 370M, this model indeed comes near the carved model’s performance (it matches carved standard Transformers, and 0.005 bpb worse than carved looped Transformers). However, Model Carving achieves this performance while saving inference FLOPs for free– 21% (standard) and 22% (looped) less inference FLOPs than dense models. Therefore, Model Carving achieves the performance of an overparameterized model at the efficiency of a smaller model.

TaskLooped 1.3BStandard 2.7B
CORE-22 tasks (centered accuracy*). full-depth / shallower / carved
HellaSwag (0-shot)49.0 / 48.6 / 54.152.0 / 49.9 / 51.9
Jeopardy39.5 / 32.5 / 40.039.3 / 37.6 / 40.5
BIG-bench QA Wikidata62.9 / 62.6 / 64.163.9 / 63.7 / 64.8
ARC-Easy67.1 / 64.1 / 67.265.8 / 63.7 / 64.0
ARC-Challenge29.2 / 24.5 / 32.029.1 / 26.7 / 29.2
COPA46.0 / 48.0 / 58.050.0 / 56.0 / 54.0
CommonsenseQA6.7 / 3.4 / 4.00.4 / -1.7 / 7.1
PIQA50.5 / 50.6 / 52.151.9 / 52.4 / 52.3
OpenBookQA21.1 / 20.3 / 26.727.7 / 26.1 / 22.1
LAMBADA55.8 / 53.7 / 58.555.3 / 52.9 / 54.4
HellaSwag (10-shot)52.4 / 50.9 / 57.053.8 / 51.7 / 54.4
Winograd53.1 / 56.0 / 62.659.0 / 56.8 / 64.8
WinoGrande23.1 / 20.9 / 29.627.2 / 23.8 / 30.2
BIG-bench Dyck languages11.9 / 14.6 / 14.316.8 / 14.9 / 14.6
AGIEval LSAT-AR13.0 / 6.5 / 2.79.2 / 2.7 / 3.8
BIG-bench CS algorithms44.2 / 36.8 / 40.042.4 / 36.8 / 42.3
BIG-bench operators27.6 / 25.2 / 28.629.5 / 30.0 / 29.0
BIG-bench repeat copy logic9.4 / 6.2 / 3.16.2 / 0.0 / 6.2
SQuAD51.5 / 50.2 / 54.350.8 / 48.6 / 50.9
CoQA35.0 / 35.0 / 37.837.7 / 35.4 / 38.5
BoolQ6.9 / 0.7 / -12.35.9 / 11.0 / 21.4
BIG-bench language identification18.0 / 17.8 / 18.018.1 / 17.7 / 17.2
CORE-22, mean of the 22 tasks35.2 / 33.1 / 36.036.0 / 34.4 / 37.0
CORE-22, mean of the 13 stable tasks45.4 / 43.8 / 48.947.2 / 45.3 / 47.5
Generative tasks (pass@k, %, within the full-depth model’s inference cost)
GSM8K17.5 / 18.4 / 24.118.1 / 19.7 / 25.2
MATH-5005.9 / 11.0 / 8.98.8 / 11.3 / 12.6
TriviaQA35.2 / 31.8 / 39.634.5 / 36.2 / 36.4
SQuAD57.6 / 58.6 / 62.257.1 / 54.5 / 60.2
LAMBADA69.7 / 70.2 / 74.969.6 / 69.2 / 72.4

*centered accuracy: 0 is chance, 100 is perfect, below chance is negative

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.