acceptodds
Withdrawn from ICLR 2027

Train Once, Rewire Later: Execution-graph Polymorphic Transformers

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

16 views

Abstract

Transformer checkpoints are tied to the *execution graph* used during training. Compilers can optimize how a fixed graph is executed, but changing the graph itself typically requires training a new model. Even a minimal rewrite exposes this dependence: executing attention and the feed-forward layer in parallel raises the language-modeling loss of a 120M transformer from to bits per byte (BPB), despite preserving the same parameters and operators. We call this sensitivity to layerwise execution changes *execution-graph rigidity* and show that it is learned rather than architectural. We introduce *Execution-graph Polymorphic Transformers* (EPTs), which instead train a single set of shared weights across a reference execution graph and alternative layerwise programs, with a consistency objective that aligns their predictions across executions. For the controlled sequential-to-parallel rewrite, the all-parallel program costs at most BPB from 1B to 7B; after 10B training tokens, downstream accuracy differs by at most percentage points across four tasks. We then extend the same objective to six modes per layer: sequential, parallel, reverse, attention only, feed-forward network only, and skip. Measurement-guided search on one 1B checkpoint finds different programs for Blackwell, H100, and L40S, reaching , , and prefill speedup. Programs selected under a BPB search-window budget cost – BPB on disjoint validation contexts. EPTs therefore turn the execution graph from a fixed training choice into a budgeted compiler decision, allowing one training run to support different hardware deployment strategies.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Transformer architectures, Execution graphs, Large language models, Efficient inference, Hardware-aware compilation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mXvWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper aims to allow a model, once trained, to dynamically adjust per-layer execution strategies (e.g., running Attention and FFN in parallel, reordering them, or skipping computations) according to varying hardware targets. The way is to share weights across execution modes during training and constrain their outputs to maintain consistency. Consequently, a single 1B model achieves approximately 1.57–1.71× prefill speedups across three GPU architectures. The core idea is interesting, and experiments are transparently reported, but the complete solution remains limited to short training schedules and narrow test scenarios.

Critical strengths
  1. Practical relevance: Enabling a single weight set to adapt to diverse hardware reduces the cost of retraining models for each deployment setting.

  2. Clear comparison between sequential and parallel execution: Preserving identical parameters and operators allows direct observation of model sensitivity to execution dependencies and whether training mitigates it.

  3. Transparent reporting: The authors distinguish between baseline model quality loss and additional degradation from altered execution modes, while disclosing training overheads, search errors, and negative ablation results.

Critical weaknesses
  1. Complete method unverified on fully trained models: The core six-mode experiments use a 1B model trained on only 98M tokens. While 7B model results and 10B-token runs are provided, they only evaluate sequential vs. parallel modes, leaving the full six-mode framework unvalidated under long training schedules.

  2. Speedups stem primarily from compute reduction, not graph rewrites: In Table 9, limiting execution to sequential, parallel, and reversed modes yields a modest 1.003× speedup. Adding layer-pruning and FFN-skipping increases this to 1.602× (versus 1.669× for the full method), showing that gains mainly come from removing computations rather than reordering.

  3. "Only ~0.05 BPB loss" omits full costs: This metric is relative to the EPT baseline itself. Before execution modifications, the 1B EPT model already performs worse by 0.0367 BPB compared to a standard model under equal data and token budgets. Furthermore, training takes ~2.29× longer per data batch.

  4. Narrow scope of speedup benchmarks: Primary evaluation is limited to batch size 8, prompt length 1,024, and no KV cache during prefill. Where the performance gain comes from, e.g., autoregressive decoding, variable sequence lengths, or varied batch sizes, remains unverified.

  5. Lack of downstream evaluation for real acceleration configs: Downstream task evaluations are only conducted on two-mode variants. The final hardware-optimized configurations alter layers and branches significantly, making validation-set BPB an insufficient proxy for downstream QA, reasoning, or generation capabilities.

Questions for authors
  1. Can you clarify where the performance gain actually comes from?
Reviewer_LMVzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Execution-graph Polymorphic Transformers (EPTs). These models share one set of weights across different execution programs. The model can run attention and the feed-forward network (FFN) in sequence, in parallel, or in reverse order. It can also run only one component or skip the layer.

Training uses a sequential reference program and an alternative program. Both learn to predict the next token. An added loss encourages their predictions to agree. After training, a search selects programs based on measured speed and an allowed loss in prediction quality.

The paper first studies two modes: sequential and parallel execution. It tests models up to 7B parameters in short training runs. It also trains a 430M model on 10B tokens and evaluates both modes on four downstream tasks. The six-mode deployment study uses a 1B model trained on 98M tokens. It reports 1.57–1.71x speedups across three GPUs during prefill, the processing of an input prompt. These programs increase loss by about 0.05 bits per byte (BPB) compared with the same model’s sequential program. BPB measures prediction loss per input byte. Lower values are better.

Critical strengths

The controlled evidence that models can learn to tolerate changes in execution. The sequential-to-parallel experiment changes the connection between attention and the FFN without removing either component. In matched 120M runs, the consistency loss increases prediction agreement from 81.25% to 90.53%. In the three-mode experiment, a compute-matched comparison reduces the average program cost from 0.0328 to about 0.016 BPB. The 10B-token experiment adds evidence that the two-mode result persists with longer training. Together, these results support more than an observation that some parts of a frozen model are redundant.

The use of direct measurements and clear reporting. Programs selected from one model produce measured speedups on three GPUs. The paper separates predicted quality costs from measured costs. It reports that the selected programs slightly exceed the quality budget on separate validation data. It also reports that predictor-guided search does not beat plain measurement-guided search in four tested seeds.

Critical weaknesses

The experiments do not isolate the speed benefit from changing dependencies. Table 9 compares all six modes, {S,P,R,A,F,K}, with {S,A,K}. Here, S means sequential, P parallel, R reverse, A attention-only, F FFN-only, and K skip. The comparison reports 1.669x versus 1.602x speedup. However, it removes FFN-only execution as well as parallel and reverse execution. The difference therefore does not show how much the dependency changes contribute. The family that allows only sequential, parallel, and reverse execution reaches just 1.003x speedup. A comparison between {S,A,F,K} and the full family is needed to separate the benefit of dependency changes from component removal. This matters because earlier methods already support cheaper execution through removal.

The limited evidence for the six-mode deployment setting. The main six-mode 1B model uses only 98M training tokens. The longer training and downstream task results apply to the two-mode setting. Also, the six-mode reference model already trails its same-token sequential specialist by 0.0367 BPB. The logged six-mode training setup takes 2.29x more time per unique token, although the setups also differ in implementation and batch size. The comparison with a smaller dense model shows a speed-quality trade-off, not a clear advantage. The authors disclose these limits, but the results do not yet establish whether six-mode flexibility is worth its training cost and quality loss.

Questions for authors

Clarification: For the matched six-mode runs with and without consistency loss, what are the absolute reference BPB and average program BPB? A smaller gap between programs does not necessarily mean better prediction quality.

Targeted experiment: How does {S,A,F,K} compare with all six modes under the same search budget and repeated random seeds? This comparison directly tests the added value of parallel and reverse execution while retaining all removal choices.

Evidence request: Do you have training-duration results or downstream task results for the selected six-mode programs? More two-mode results do not resolve this question. A complete production-serving evaluation is not required to address this concern.

Reviewer_H3gHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper trains a composable model in which the connectivity between operators can be chosen at runtime via a search, as opposed to requiring retraining for each specific connectivity. This is validated on LLMs of five different sizes.

Critical strengths

The paper trains a set of LLM operators whose layer connectivity can be rewired at deployment time, and it reports only a small loss in quality.

Critical weaknesses
  1. Prior methods (Leveraging the true depth of LLMs, TMLR 2026) modify pretrained models’ execution graphs without retraining from scratch. The paper does not establish what multi-program training adds at matched quality and compute budgets. All experiments train from scratch on ClimbMix with a small vocabulary, leaving adaptation of existing pretrained checkpoints untested.
  2. The speedup seems to be mostly coming from skipping layers and dropping the FFN, which is closer to pruning/LayerDrop than to rewiring. It is unclear what is the advantage of rewiring specifically. Consistency provides only modest improvements, and the controls’ missing reference scores prevent determining whether lower relative program costs reflect better absolute quality.
  3. The motivation for hardware-specific rewiring isn’t really backed up by experiment. Showing the search produce three different wiring for three difference architecture is not sufficient. A the H100-picked program on Blackwell might not be necessarily bad either.
  4. Selected programs incur roughly 0.05 BPB beyond an existing 0.0367 BPB specialist gap. The single smaller model comparison trades a marginal speed gain for worse quality, while six-mode training takes 2.29× more time per token at the reported configurations. Full quality latency frontiers against smaller dense and elastic models are missing.
  5. The 1B six-mode and 3B/7B binary checkpoints receive only 98M tokens. The longer training experiment uses the 430M binary family and reports no hardware speedup. The accelerated six-mode programs lack downstream and free-running generation tests.
  6. Predictor-ranked local search fails to outperform plain local search across four seeds, leaving the full pipeline’s advantage over simpler search unestablished. Searches require 185–303 evaluations per GPU without reporting total search time or variability across repeated runs. The selected programs also slightly exceed the quality budget on held-out contexts, motivating uncertainty estimates and more conservative acceptance criteria.
Questions for authors

Please address the weakness with additional experiments or explanations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.