acceptodds
Withdrawn from ICLR 2027

TwIST: Rigging the Lottery in Transformers via Zero-Shot Subnetworks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We introduce , a novel distributed system for efficient training and deployment of Transformer-based models. unifies two important paradigms: distributed training and pruning, to induce an inference time property that allows us to randomly sample competitive subnetworks on the fly. trains randomly generated subnetworks in parallel, periodically aggregating and resampling, yielding high-performance subnets that require no fine-tuning or search heuristics. This enables robust, zero-cost pruning at deployment, achieving perplexity scores close to state-of-the-art post-training methods while bypassing their post-training overhead (e.g., calibration, Hessian inversion). Crucially, even when post-training baselines operate on a stronger dense checkpoint than , they still lose at aggressive sparsity, often by orders of magnitude, evidence that designing training around subnet structure produces better sparse models than searching for them after the fact. We validate this across three architectures (GPT-2 and GPT-2 XL, with multi-head attention and Qwen3-0.6B with grouped-query attention) over a wide range of text-generation tasks. 's advantage emerges under aggressive pruning (e.g., 50%+ sparsity), where it significantly outperforms baselines. For example, on GPT-2/WikiText-103 it achieves 23.14 PPL while the closest baseline follows at 31.64. At the most aggressive Qwen3-0.6B pruning setting, reduces perplexity by approximately \(2.5\)–\(4.6\times\) relative to the strongest evaluated baseline, as those baselines collapse entirely. Furthermore, as a structured pruning method, produces smaller, dense matrices, translating to tangible inference speedups and memory savings on inference server deployments (e.g., CPUs) that lack sparse computation support. We provide the complete implementation https://anonymous.4open.science/r/twist2-21E6here.

Primary area:
optimization
Keywords:
Distributed training, Pruning, efficient training and deployment, compression

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WdNMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper mainly focuses on fine tuning and compression of large LLM models. Unlike existing pruning and compression methods, which usually work on a target model size and may not be easily adaptable when a different size is needed, the paper proposes a federated learning approach that aims to replace the current fine-tuning step and supply usable subnetworks of various sizes with no additional online cost. To achieve this, at each training phase, the full model is randomly partitioned, distributed, and trained such that each subnetwork should still be able to make predictions. The parallelization is a key contribution of the work and the engineering does lead to at times rather significant memory and time gains. The numerical experiments and ablation studies are abundant, and there appears to be some evidence to support the proposed scheme. However, some questions remain unclear.

Critical strengths

The paper focuses on what I think is a very relevant problem, model compression remains a important topic and balancing between performance and memory/inference time on edge devices is the key challenge, the zero-shot pitch of the paper does make an appealing effort.

While Im not fully convinced that the proposed approach is optimal in achieving their objective the engineering is in my opinion the strongest point of the work, by parallelizing and enabling training over multiple potentially weak devices, its what makes the pipeline partially justifiable as its arguably more costly than existing methods.

The paper does in my opinion a decent job of providing a good amount of experiments and comparisons, as well as being fairly honest about the issues with extracting subnetworks at sizes different from those used during training, showing the limitations of the method.

Critical weaknesses

One of the findings of modern LLM diagnostics is that a pretrained model often has different functionalities in different blocks, attention heads or layers. I think even some early work on GPT-2 models exposed this. Existing importance-based and reconstruction-based pruning methods such as sparseGPT at least attempt to preserve aspects of this structure, whereas the proposed method essentially forces the pretrained blocks to adapt to different combinations. This in my opinion raises serious questions about both the adaptation required during training and the reliability of the subnetworks. I think this is a major question that the proposed method while partially acknowledging unequal component importance, ultimately fails to adequately address or convince the reader about, and with all the claims on zero-shot prediction, how much can i trust those results with an arbitrary combination? and to what extent?

I notice that almost all reported results show only perplexity, and I think it is not sufficient for truly showing the effectiveness of the resulting models. PPL measures how well the model predicts reference tokens, but not necessarily the factual correctness or consistency of its generated outputs. A model can achieve good perplexity while still being of limited use in practice. I think the paper needs to show more to convince the reader that these subnetworks are meaningfully functional.

Questions for authors

Under the written formulation, I can see the training across different worker being extremely uneven, a certain combination of subnetwork can blowup or have magnitudes high loos than others, do you observe that? how do you resolve that? reading from the equations everything is treated equally but if 15 minibatch training does not resolve a loss blowup then averaging that into the full model seems like a numerical nightmare, and reading from the tables numerical instability certainly can happen.

There is considerable literature in federated learning showing that weight averaging can be unreliable when workers drift toward different local objectives and does not necessarily improve the global model, and I think many people in the field have opted for gradient averaging or other methods. Why is simple weight averaging chosen here, particularly when workers train different subnetworks? Is there any justification for this decision beyond the reported ablation?

According to the paper the first 2 and final 2 layers are always fixed to avoid model breaking down, and only middle sections are being partitioned and shuffled. Is there any attempt in trying to understand what happens to the middle portion during training? do these middle layers actually develop into different modules or gradually become similar to each other? without additional diagnostics I can see both happening given the problem formulation.

As model size increases and the number of partitioned attention heads and layers grows, the number of subnetwork combinations can grow exponentially, at some point even with cleaver parallelization the computation requirement in training can grow much higher than any of the existing methods, There is also a need to generalize to combinations never seen during training. The paper currently evaluates relatively small models and is relatively tame in designing the partitions, and I do not expect this to be fully resolved given limited resources, but what would your answer be based on your experience? modern stoa LLMs are at trillion parameter level and will the proposed method be scalable at all?

Reviewer_oQczOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TwIST, a distributed training approach that trains randomly sampled structured Transformer subnetworks across workers and periodically aggregates them into a shared parent model. Attention heads/KV groups and FFN blocks are partitioned during training, with the goal of making randomly extracted subnetworks usable directly at deployment without additional calibration, search, or fine-tuning.

Experiments on GPT-2, GPT-2 XL, and Qwen3-0.6B show that TwIST is competitive with post-training pruning methods at some moderate sparsity levels and can substantially outperform them under aggressive pruning. The paper also reports reduced training memory/communication and up to 1.49× training and CPU inference speedups.

Critical strengths
  1. TwIST provides a simple and practically appealing way to obtain multiple structured subnetworks from one trained parent model without deployment-time search, calibration, or fine-tuning. The structured removal also produces smaller dense matrices rather than relying on sparse hardware support.

  2. The improvement at aggressive sparsity is substantial rather than marginal. For example, at several 1/4–1/8-width settings, conventional post-training baselines degrade severely while TwIST remains usable. The paper also goes beyond perplexity and reports training memory, communication, and measured CPU inference benefits.

Critical weaknesses
  1. Most experimental baselines are post-training pruning methods such as SparseGPT, Wanda, Z-Pruner, and Block Prune. However, TwIST changes the training procedure specifically to make many subnetworks deployable. Training-aware elastic methods are therefore arguably the more direct comparison. In particular, LLaMaFlex is discussed in the related work as a close method for zero-shot subnetwork extraction, but is not experimentally compared. A comparison with methods such as LLaMaFlex/MatFormer/Flextron, or a clearer explanation of why such comparison is not feasible, would make the empirical claim considerably stronger.

  2. The practical advantage is less clear at moderate compression, while the strongest gains occur at rather aggressive sparsity. For example, on Qwen3-0.6B with both attention and FFN pruning at 4/8, SparseGPT/Wanda outperform TwIST on all three reported datasets; TwIST becomes stronger mainly at 2/8 and especially 1/8, where absolute perplexity has already degraded substantially. The evaluation is also limited to models up to 1.5B and primarily perplexity, without downstream language-model tasks. This makes it difficult to determine whether TwIST provides a better accuracy-efficiency Pareto point in the operating regime most relevant to modern LLM deployment.

Questions for authors
  1. Could the authors provide an experimental comparison with a training-aware elastic model such as LLaMaFlex, MatFormer, or Flextron? Since LLaMaFlex is already identified as closely related, understanding how TwIST compares at matched model size or compute would materially affect my assessment.

  2. Can the authors report the actual parameter/FLOP reduction and measured latency corresponding to the main quality results, rather than only the block ratio κ\kappa? In particular, I would like to better understand whether the large gains at 2/8 and 1/8 correspond to practically attractive quality-efficiency operating points.

  3. The main quality experiments use Masked TwIST, while the claimed training memory/communication benefits require physical subnetworks. Were the system results in Table 4/Figure 6 obtained with True TwIST or another implementation, and has the quality equivalence between Masked and True TwIST been empirically verified?

  4. Are the results in Tables 1, 3, and 7 based on multiple independent training seeds and multiple randomly extracted subnetworks? Since random subnet extraction is central to the method, reporting the number of samples and variance would help establish the robustness of the reported gains.

Reviewer_CerBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TwIST extends independent subnetwork training (IST) to Transformers. Attention (KV-group) and FFN blocks are randomly partitioned across workers, trained locally, and periodically aggregated, with Nfull/Nsub\sqrt{N_{\text{full}}/N_{\text{sub}}} activation rescaling. The goal is a parent model from which random structured subnets can be extracted without calibration, search, or fine-tuning. Experiments fine-tune GPT-2, GPT-2 XL, and Qwen3-0.6B on WikiText-103, C4, and TinyStories, comparing against DDP followed by post-training pruning (SparseGPT, Wanda, Z-Pruner, Block Prune). The paper also reports training and CPU-inference efficiency gains.

Critical strengths
  1. Extraction is essentially free and yields smaller dense matrices, so the speedups are real without sparse kernels.
  2. At aggressive structured sparsity, subnets degrade gracefully with low variance across random draws (Figs. 4, 7), while the evaluated baselines collapse.
Critical weaknesses
  1. Confounded comparison. TwIST gets 3 epochs of training aware of the subnets, while the baselines are pruned with no recovery fine-tuning, and some are unstructured. Missing baselines:

    • training-time elastic methods (LayerDrop, RaPTr, MatFormer, LLaMaFlex);
    • structured PTP methods (LLM-Pruner, SliceGPT);
    • DDP + pruning + an equal fine-tuning budget.

    Novelty over IST plus structured dropout is therefore unclear.

  2. "Single checkpoint" claim unsupported.

    • Headline results use a separate parent per κ\kappa.
    • The truly single-checkpoint column degrades sharply at high sparsity (21,122 PPL on Qwen3 C4 at 1/81/8).
    • The dense parent loses ~3 PPL (15.96 → 19.06).
    • At moderate sparsity TwIST often loses, e.g., GPT-2 C4 both 6/12: 56.03 vs 36.55.
Questions for authors
  1. Table 3 (ffn): Z-Pruner and Block Prune report identical values in three rows, and Z-Pruner gives ~75k PPL at 50% sparsity where Wanda gives 13.26. Please verify these implementations.
  2. Do the baselines exclude the same edge layers and use the same rescaling?
  3. How does TwIST compare with DDP + structured pruning + equal fine-tuning, and with MatFormer/LLaMaFlex?
  4. 3.24 tokens/s for GPT-2 124M on a 12-core Xeon seems far too slow. What is measured?
  5. Please report variance over seeds and subnet draws, and at least one downstream task beyond perplexity.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.