acceptodds
Under review as a conference paper at ICLR 2027

LLMForge: Multi-Backend Hardware-Aware Neural Architecture Search with Infinite-Head Attention for Edge Language Models

Abstract

Efficient language model deployment requires architectures that balance model quality with the energy and latency costs of the target hardware. We present LLMForge, a hardware-aware neural architecture search framework for pretrained language models. Its Infinite-Head Attention parameterization decouples query-head count from head dimensions and allows separate query/key and value widths. Together with variable key–value grouping and MLP widths, this defines a flexible search space for capacity allocation within and across layers. To evaluate candidates efficiently, LLMForge converts each pretrained model into an elastic supernet and jointly trains sampled subnetworks with shared weights, avoiding separate candidate training during search. Four interchangeable backends provide GPU measurements, accelerator modeling, joint model–chip optimization, and cost prediction from smartwatch measurements. Their cost evaluations guide multi-objective search alongside supernet validation loss, allowing the selected architectures to adapt to both the hardware backend and the optimization objectives. We evaluate the framework using five pretrained models spanning 135M-4B parameters. Across eight model–accelerator configurations, the search reduces modeled energy per token by a median of 8.4% relative to uniform scaling at matched supernet validation loss.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
neural architecture search, hardware-aware NAS, multi-objective optimization, energy-efficient inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wLw6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LLMForge presents a hardware-aware architecture-search framework for pretrained language models. Its Infinite-Head Attention (IHA) search space independently varies the query-head count, key–value group count, query/key dimension, value dimension, and MLP width. The authors elastify five pretrained models into shared-weight supernets using coordinated slicing, learned group alignment, and a single supernet-training run per base model. Forge-DSE then uses NSGA-II to search quality–cost trade-offs with four backend classes: GPU profiling, analytical accelerator modeling, joint model–chip co-design, and a smartwatch cost predictor. Across eight model–accelerator configurations, the paper reports a median 8.4% reduction in modeled energy per token relative to uniform scaling at matched supernet validation loss.

Critical strengths

•The paper addresses a practical and relevant problem: selecting language-model architectures under hardware-specific energy and latency constraints rather than relying only on parameter count or FLOPs. •The paper provides a broad empirical study across five pretrained language models and multiple hardware-evaluation backends, examining how hardware characteristics and deployment objectives influence architecture selection. The detailed experimental protocols and reported per-seed results also provide a useful basis for analyzing the behavior and reproducibility of the proposed search framework.

Critical weaknesses

• Novelty is incremental and insufficiently isolated. IHA primarily relaxes constraints among existing attention dimensions; it does not introduce a fundamentally new attention computation. The paper does not convincingly establish that the combination of independently tunable head count, KV grouping, query/key width, and value width is a sufficiently substantial advance over elastic Transformer search, GQA-based design, and prior hardware-aware NAS. Table 1 provides only a high-level comparison, without direct, budget-matched comparisons against the closest methods. • The ablation study is insufficient to isolate the contribution of IHA. The IHA-versus-GQA comparison is limited to a single model and hardware backend, with only a small hypervolume improvement. The paper does not establish whether the benefits of IHA generalize across models and hardware targets or whether the gains primarily come from the search algorithm and heterogeneous capacity allocation.

Questions for authors
  1. What is the key technical novelty of IHA compared with existing elastic Transformer architectures and GQA-based designs? Can the authors provide direct, budget-matched comparisons with the closest existing methods to demonstrate its advantages?
  2. Can the authors provide additional ablations across different models and hardware backends to establish that the benefits of IHA are consistent and attributable to the expanded search space, rather than primarily to NSGA-II or heterogeneous capacity allocation?
  3. How reliable are the hardware cost estimates across the different backends, particularly for the analytical accelerator models and the ring-based simulator? How do potential prediction errors affect the reported energy savings and the ranking of selected architectures?
1 comment

We really appreciate your thoughtful review, especially the push to separate what comes from the expanded attention space versus per-block allocation and NSGA-II, and to test our cost models under prediction error. Working through both made our empirical case much sharper. Please check the green and purple parts in the revised PDF.

How IHA Differs From Prior Designs Under Equal Budgets (W1, Q1)

Rather than a custom kernel, IHA is a weight-sharing search-space parameterization: every candidate runs on standard GQA/MHA kernels while combining per-block elastic head dimension (independent of nhn_h, allowing dqk≠dvd_{qk} \neq d_v) and per-block KV grouping in one pretrained supernet:

MethodHead dimensionKV grouping across depthBuilt from a pretrained LM
Flextron, LLaMaFlex, MinitronFixed×\times✓\checkmark
Puzzle, DHA, QCQAFixed✓\checkmark✓\checkmark
AutoTinyBERT, ElasticViTElastic per head×\times×\times
IHA (LLMForge)Elastic, independent of nhn_h✓\checkmark✓\checkmark

In budget-matched searches (832 NSGA-II evaluations, 3 seeds on one supernet), Flextron-style elasticity (only nhn_h and MLP width searched) saves −0.1% (Gemmini) and −0.7% (rDXE) at 135M vs. 6.6% and 8.6% for IHA, and half of IHA's saving at 360M (5.0% vs. 9.4% Gemmini, 3.8% vs. 8.2% rDXE). Whenever a target penalizes a missing knob, restricted spaces trail IHA by 0.046 to 0.151 held-out hypervolume (HV): locking KV drops 135M rDXE savings from 8.6% to 2.2%, while locking head dimension (GQA-style) drops 135M Gemmini 2,048-decode savings from 7.0% (10.1% for IHA (dqk=dvd_{qk}=d_v)) to 2.6%. Narrow spaces work when aligned with a target's bottleneck (GQA-style on rDXE), whereas combining both freedoms keeps HV regret within 0.0125 across every SmolLM2 target and decode length, and decoupling dqk≠dvd_{qk} \neq d_v in full IHA cuts measured H100 decode energy 4.0% to 8.9% under fused kernels (PyTorch 2.12).

Pricing target hardware cost rather than parameter or FLOP proxies is equally critical. Re-priced on rDXE, proxy searches degrade TPOT 1.7% to 18.4% vs. uniform scaling, while every LLMForge seed beats every proxy seed in 3-objective HV:

Search objective (832 evaluations, 3 seeds)135M rDXE HV360M rDXE HV
Parameter count or FLOPs0.339 to 0.3670.352 to 0.397
LLMForge (rDXE cost model)0.374 to 0.3880.407 to 0.435

Disentangling the Search Space, Per-Block Widths, and NSGA-II (W2, Q2)

Across all five SmolLM2 settings in General Response §1, comparing uniform grids with per-block NSGA-II and random sampling separates all three factors (3-seed medians at matched loss):

Factor isolatedControlled comparisonMatched-loss energy
Elastic head dimension (no search)Uniform GQA-style vs. IHA gridGQA-style uses 5.3% to 10.9% more
Head dimension + KV (no search)Uniform Flextron-style vs. IHA gridFlextron-style uses 6.2% to 24.4% more
Per-block allocationNSGA-II vs. own uniform grid (GQA-style)Search uses 6.4% to 9.7% less
Search optimizerNSGA-II vs. random (IHA vs. uniform)NSGA-II 6.6% to 9.4% less, random 1.5% to 3.9% more

Expanding the space saves 5% to 24% energy on uniform grids before search (largest at 2,048 decode), and NSGA-II adds another 6% to 10%. On Qwen3-0.6B (48x larger space), LLMForge saves 2.9% to 4.6% energy on all four targets at 832 evaluations (beating GQA- and Flextron-style on Gemmini by 0.031 and 0.018 HV), rising after 100 generations (3,232 evaluations) to 8.9% on Gemmini vs. 3.6% to 3.7% for fixed-128 spaces.

How Reliable Are the Hardware Cost Models? (Q3)

We validated cost fidelity via component perturbations, Monte Carlo noise, and device runs:

ScopePerturbation or checkImpact on matched-loss saving
Gemmini, Eyeriss, FLATDRAM, buffer, or MAC energy ±30%\pm 30\%Shifts ≤0.75\le 0.75 pts, 0 sign flips
135M & 360M, all 4 backendsIndependent 5% lognormal error (1,000 draws)Positive in 99.6% to 100% (median 5.7% to 8.7%)
135M & 360M, all 4 backendsIndependent 10% lognormal error (1,000 draws)Positive in 86% to 99% (median 4.2% to 8.0%), rDXE rows 95% & 99%

On Timeloop substrates, 95.5% to 98.6% of decode energy is off-chip byte traffic, making rankings insensitive to unit weights. On a Pixel Watch 5, testing our frozen predictor on 324 newly measured models gives 4.6% median energy error (215 drift-free runs) and keeps measured order on 96.5% (energy), 95.2% (TTFT), and 88.7% (TPOT) of pairs separated by ≥8.4%\ge 8.4\% predicted cost. On device, smartwatch-searched 135M cuts TPOT 7.0% and energy 7.6% vs. uniform in all 3 rounds at lower loss, while ignoring INT8 group size raises TPOT 4.7% to 18.7% at matched loss.

Reviewer_6kbYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LLMForge, a hardware-aware neural architecture search algorithm that searches for efficient architectures within a pretrained LLM. LLMForge starts from a pretrained model and makes it elastic by introducing separate query/key and value widths as searchable dimensions. To keep the representations aligned when key–value groups are merged, the paper proposes learnable rotations before they are pooled. The authors evaluate their technique on five pretrained language models and four hardware backends, and find that it yields models with lower energy per token at the same validation loss than uniformly scaled baselines.

Critical strengths

[S1] The proposed IHA parameterisation, together with the elastification step, allows the pretrained LLM to be converted into a weight-sharing supernet, which removes the need to train each candidate separately. This is much cheaper than retraining models from scratch, especially since LLMs are costly to train. The supernet training and search are also decoupled, which means that the same trained supernet can be used over a variety of hardware-aware searches.

[S2] The framework is extensively evaluated on a diverse set of hardware backends: modeled accelerators, a multi-chip simulator and real hardware measurements.

[S3] The authors have released their research code, which makes the results easier to reproduce.

Critical weaknesses

[W1] While the proposed technique looks promising, the headline results on energy reduction seem rather modest (median of 8.4% energy reduction per token). Because it is also not clear what the actual overhead is of the supernet training (the authors only mention that it took 2500 steps of 41M tokens), it is hard to justify the cost of supernet training for the obtained gains. Additionally, the work closely relates to Flextron (Cai et al., ICML 2024), where a more or less similar workflow is suggested and the current submission only differs in the variable query/key and value dimensions. Given the modest performance improvements and the incremental nature of the contribution, the overall impact of the paper appears limited.

[W2] The ablation on the different search spaces in Section 3.1 highly biased towards the search space with IHA. Limiting the search space for GQA naturally reduces the diversity of solutions that can be found, affecting overall hypervolume of the pareto front. A comparison as done in Figure 5 would make more sense in that regard.

Questions for authors

[Q1] Related to [W1]: what is the wall clock time for the supernet training for each of the 5 base models?

[Q2] Figure 5 only shows the evaluated architectures sampled by LLMForge and its respective base model. Where are the uniform scaling baselines situated on the graphs? Also, it is somewhat weird to put the headline results for the comparison against uniform scaling baselines (Table 9) in the Appendix, while this should be the whole selling point in this section. Moving the restricted-search-space ablation to the appendix and Table 9 to the main text would make more sense.

[Q3] Could the authors specify how the median saving at matched loss against the uniform scaling is calculated in Table 9 and clarify this in the text? Additionally, what is the performance of the models for both LLMForge and the uniform scaling baselines at matched energy and latency levels?

[Q4] The architectures in the ranking study (Appendix C) are fine-tuned starting from the supernet weights, which may favour agreement with the supernet scores. Does the ranking still hold when the architectures are instead initialised directly from the original pretrained checkpoint? What will be the overall difference in performance?

1 comment

Thank you for reading our paper closely and for your suggestions on evaluation and layout. Your point on hypervolume favoring wider spaces was spot-on, and your questions on supernet cost, matched-cost quality, and initialization sharpened our analysis. Please check the orange and purple parts in the revised PDF.

Practical Gains and Supernet Training Cost (W1, Q1)

  • Why 8.4% reflects a large quality-cost shift: Raw slices rank candidates before weight recovery, so practical 60% to 80% energy targets sit above the 40% loss bound (≤1.4×\le 1.4\times full-width loss), where LLMForge saves 8.3% median energy at matched loss and lowers loss at 188/189 uniform points by 0.251 nats median. After 8,000 standalone steps from the checkpoint, a 65%-energy Gemmini 135M pick recovers from 1.60x slice loss to 1.26x pretrained loss (inside the 40% bound) and beats its uniform partner by 0.301 nats. Over the full uniform front on all four targets, matched-energy loss drops 0.14 to 0.17 nats (135M) and 0.24 to 0.28 nats (360M).
  • Advance over Flextron: At equal 832-evaluation budgets on one supernet, Flextron-style elasticity (only nhn_h and MLP width searched, head dimension and KV fixed) saves −0.1% (Gemmini) and −0.7% (rDXE) at 135M vs. 6.6% and 8.6% for IHA, and half of IHA's saving at 360M and Qwen3-0.6B (Gemmini, 100 generations: 3.7% vs. 8.9%). Combining per-block head dimension and KV grouping (plus dqk≠dvd_{qk} \neq d_v, cutting measured H100 decode energy 4.0% to 8.9% under fused kernels) delivers these gains at 549x lower supernet token cost.
  • Supernet wall-clock hours and token share (Q1): Each supernet trains once on one H100 for 2,500 steps of 65,536 tokens at length 4,096 (163.8M tokens, correcting the 41M typo at 1,024):
Base model135M360MQwen3-0.6BQwen3-1.7BQwen3-4BFlextron
H100 hours4.4 h6.4 h9.9 h12.2 h27.7 hunreported
% pretrain tokens0.0082%0.0041%0.00046%0.00046%0.00046%4.5% to 8.2% (89.9B)

Wall times scale from 100 timed steps, and a full 135M rebuild took 4 h 25 min (within 0.5%) and reproduced rankings on 2,653 shapes at Spearman 0.9999 (~60.6 GPU-hours total, amortized over 15 to 45 min searches).

Fair Search-Space Comparisons Beyond Hypervolume (W2)

  1. Matched-loss energy: GQA-style (fixed head dimension, elastic KV) matches IHA on rDXE where KV dominates, while on Gemmini it trails IHA by 2.1 to 4.4 pts on SmolLM2 (2.6% vs. 7.0% at 2,048 decode, 10.1% for IHA (dqk=dvd_{qk}=d_v)) and 5.3 pts on Qwen3-0.6B at 100 generations, as fixing head dimension costs GQA's uniform grid 6.8% to 10.9% more energy before search.
  2. Dedicated GQA-only supernet: A separate 135M supernet trained on the GQA-style space yields no better shapes: scored on the shared supernet, its finds tie IHA on rDXE and need 4.8% more Gemmini energy.
  3. Standalone 8,000-step checkpoint training (no supernet scorer): At 65% / 90% Gemmini energy, trained 135M losses from the checkpoint are 3.608 / 2.770 (Uniform), 3.353 / 2.722 (Flextron-style), 3.248 / 2.732 (GQA-only supernet), and 3.264 & 3.307 / 2.726 (IHA: best score & nearest 65%). All searched picks beat uniform (0.255 to 0.360 nats at 65%). Dropping both freedoms in Flextron-style costs 0.089 nats (10.7 SEs) vs. IHA, while GQA-style keeps KV and ties IHA at 32 decode.

Figure Updates and Quality at Matched Cost (Q2, Q3)

  • Q2: Adopted: former Table 9 is Table 1 in §3.1 with each row's uniform-front HV, the new Fig. 6 plots the uniform front against a searched front, and search-space curves moved to App. I (Fig. 8).
  • Q3: For uniform point u=(Lu,Eu)u=(L_u, E_u) and lowest searched energy E∗E^* at loss ≤Lu\le L_u, saving is 1−E∗/Eu1 - E^*/E_u (median over 29 to 37 points/seed, 3-seed median/setting, 8.4% median over 8 settings). Swapping axes (Lu−L∗L_u - L^* at cost ≤Cu\le C_u) gives 3-seed median loss drops (nats) at matched energy / TPOT of 0.14 to 0.17 / 0.15 to 0.17 on 135M Timeloop substrates, 0.24 to 0.28 / 0.24 to 0.32 at 360M, 0.25 / 0.10 on 360M rDXE, and 0.17 / −0.03 on 135M rDXE (exchanging 1.4% to 1.9% TPOT for 8.6% energy).

Checkpoint vs. Supernet Initialization (Q4)

In App. C, all 20 Qwen3-0.6B models were initialized from the published base weights rather than the supernet (now stated explicitly), so its +0.98 Spearman (τb=+0.91\tau_b=+0.91, 147/147 concordant above 0.20 nats) has zero supernet-init bias. Training 135M Gemmini pairs for 8,000 steps from both starts shifts searched losses by <0.01<0.01 nats:

Gemmini energyUniform (ckpt)LLMForge (ckpt)Uniform (supernet)LLMForge (supernet)
65%3.6083.307 (−0.301)3.5683.313 (−0.255)
90%2.7702.726 (−0.045)2.7812.735 (−0.046)

Every lead exceeds 11 paired SEs, and the 0.255-nat supernet-trained lead matches the 0.238-nat slice gap.

Reviewer_cUyNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents LLMForge, a hardware-aware architecture search framework for pretrained language models. It combines a flexible attention search space, shared-weight supernet training, and multiple hardware backends to explore model quality–cost trade-offs. Across eight model–accelerator configurations, it reports a median 8.4% reduction in modeled energy per token over uniform scaling at matched supernet validation loss.

Critical strengths

(+) Flexible and unified deployment-oriented framework. The framework combines pretrained model adaptation with multiple hardware backends, enabling architecture search across GPUs, modeled accelerators, and a smartwatch-based cost predictor. Its consistent energy savings over uniform scaling across eight modeled configurations provide useful evidence that nonuniform capacity allocation can improve quality–cost trade-offs.

Critical weaknesses

(-) Limited overall novelty. Hardware-aware Transformer NAS and NSGA-II are established, while pretrained model elastification, weight slicing, and shared-weight training have precedents such as Flextron. The main additions appear to be greater independence among attention dimensions and the associated parameter adaptation mechanisms. Expanding the search space is relatively straightforward; the more substantive potential novelty lies in coordinated slicing and learned group alignment. However, their necessity and benefits over simpler alternatives are not sufficiently established.

(-) Insufficient ablation of the proposed additions. Figure 6 shows a hypervolume improvement from 0.380 to 0.391, but does not quantify energy or latency savings at matched loss, or loss improvements at matched cost. This comparison covers only SmolLM2-135M on modeled Gemmini hardware and does not isolate individual architectural freedoms. Appendix C provides slicing comparisons before additional training and a ranking-reliability study, but these do not establish the independent benefits of the learned adaptation mechanisms. The overall 8.4% modeled energy saving also cannot be attributed specifically to these additions. Component-wise ablations and direct quality–cost comparisons across more models and real devices are needed.

Questions for authors
  1. Could the authors provide controlled ablations isolating the benefits of independent Q/K and V dimensions, variable KV grouping, and learned group alignment? In particular, beyond the hypervolume improvement in Figure 6, how much energy or latency reduction do these additions provide at matched validation loss, and do these gains persist on another model or hardware target?
  2. The reported median 8.4% energy reduction is based on modeled accelerator costs. Could the authors provide a matched-validation-loss comparison against uniform scaling on the H100 or Pixel Watch, reporting measured energy and latency for the selected architectures? This would help establish whether the gains translate to real deployment.
1 comment

Thank you for a very clear and direct review. You pressed us on two key points: separating each architectural freedom and supernet mechanism rather than treating the 8.4% saving as one bundle, and testing whether modeled gains hold up in measured power and latency on real devices. Both pushed us to run the controlled ablations and hardware runs below. Please check the blue and purple parts in the revised PDF.

Isolating Each Architectural and Supernet Addition (W1, Q1)

Each row removes one addition while keeping the supernet, budget (832 NSGA-II evaluations), and 3 seeds fixed. Savings are energy per token at matched loss vs. uniform scaling:

Addition removedEffect at matched lossWhere tested
Variable KV grouping (Fixed KV)135M rDXE ring saving 8.6% → 2.2%, −0.024 to −0.046 HV in all 5 SmolLM2 settings, every seed pair135M & 360M, Gemmini & rDXE
Elastic head dimension (GQA-style)135M Gemmini, 2,048 decode: 7.0% → 2.6%, as KV traffic rises from 0.3% to 18.3% of decode energy135M & 360M Gemmini (−0.036 HV at 360M), Qwen3-0.6B at 100 gens (3.6% to 3.7% vs. 8.9%)
Decoupled dqk,dvd_{qk}, d_v (IHA (dqk=dvd_{qk}=d_v))Fused H100 kernels: all 21 in-budget Qwen3-1.7B front models set dqk≠dvd_{qk} \neq d_v in 71% of layers, cutting decode energy 4.0% to 8.9% (Q2). Tying dqk=dvd_{qk}=d_v suffices on 25-gen edge runsQwen3-1.7B, H100
Learned group alignment+3.0% / +2.7% energy and TPOT without it (No alignment / Procrustes), +0.060 / +0.049 nats slice loss135M Gemmini
Both freedoms (Flextron-style: nhn_h & MLP searched)−0.1% / −0.7% (Gemmini / rDXE) vs. 6.6% / 8.6% for IHA at 135M, half of IHA's saving at 360M135M & 360M, Gemmini & rDXE

IHA stays within 0.0125 HV of the best space in every SmolLM2 setting, while dropping either freedom loses up to 0.162 HV. Latency follows: on Timeloop substrates (Gemmini, Eyeriss, FLAT), TPOT ranks identically to energy (Spearman 1.000), while on rDXE search lowers both at 360M (3.6% TPOT, 8.2% energy) and exchanges 1.4% to 1.9% TPOT for 8.6% energy at 135M.

Group alignment and coordinated slicing vs. simpler baselines. Retraining the 135M supernet with one mechanism changed tests fidelity over 896 shapes (384 uniform + 512 front):

135M supernetMean loss differenceGemmini HV (3 seeds)
Original recipe, rebuiltreference0.338 (0.334, 0.342)
Prefix rotary order+0.045 nats0.336 (0.335, 0.339)
Procrustes alignment+0.049 nats0.321 (0.315, 0.328)
No alignment+0.060 nats0.329 (0.327, 0.331)

Our recipe lowers loss on 73% to 96% of front shapes (5x to 6x the 0.008-nat rebuild noise), and learned alignment lifts every Gemmini seed above every unaligned or Procrustes seed while keeping rank stability (Spearman ≥0.995\ge 0.995).

Quality at Matched Cost and After Standalone Training (W2)

  • Matched energy and latency: Across 135M and 360M on all four backends, searched fronts lower median loss over uniform scaling by 0.14 to 0.17 nats (135M) and 0.24 to 0.28 nats (360M) at matched energy (0.15 to 0.32 nats at matched TPOT on Timeloop), beating 188/189 uniform points by 0.251 nats median at 60% to 80% energy.
  • Optimizer ablation: On 135M Gemmini at matched loss, NSGA-II saves 6.6% energy in IHA and 4.5% in GQA-style, while random sampling uses 3.3% more energy than uniform scaling.
  • Standalone 8,000-step training: Against uniform partners using 0.9% to 1.5% more Gemmini energy, trained 135M picks lead by 0.301 / 0.255 nats at 65% energy (checkpoint / supernet init) and 0.045 / 0.046 nats at 90% (≥11\ge 11 paired SEs on validation and 96 unseen documents).

Measured Energy and Latency on Physical Devices (Q2)

  1. Pixel Watch 5 (INT8, 3 rounds): Within the 40% loss bound on 135M, smartwatch search beats the best uniform shape in all 3 rounds on loss (3.65 vs. 3.73), TPOT (74.5 vs. 80.1 ms, 7.0% lower), and energy (62.2 vs. 67.3 mJ, 7.6% lower), and three further pairs save 0.8% to 7.8% energy within 1.1% TPOT.
  2. Server CPU (4,096 context, equal trained loss): Elastic head dimension cuts decode TPOT over head-dim-64 baselines by 22.2% (135M) and 11.3% (360M) median, or 21.3% and 10.2% on group size 64 (GS64). Unadapted Gemmini picks run 1.20x to 1.35x slower on the watch and server CPU by shifting to GS16, showing why target-specific search matters.
  3. NVIDIA H100 (ZEUS, Qwen3-1.7B): Under PyTorch 2.10, grouped dqk≠dvd_{qk} \neq d_v hit an unfused fallback (4.9x decode energy), which search avoided (12/13 front models at 1.7B and 4B). Under PyTorch 2.12 (0.97x fused cost), re-searching grows the front from 5 to 21 models (all 21 setting dqk≠dvd_{qk} \neq d_v in 71% of layers), cutting decode energy 4.0% to 8.9% across all four compressed staircase points (1.26x to 1.36x loss, 2.0% median over all 7 points) and lowering loss 0.16 to 0.26 nats at matched energy.

Comments on the paper

1 comment

We appreciate the time and care all three reviewers put into our work. The reviews converged on three central questions: which degrees of freedom in Infinite-Head Attention (IHA) drive the gains, how raw slice scores relate to trained quality, and whether savings hold on real hardware through target-specific cost models. In the revised PDF, changes are color-coded by reviewer: blue for Reviewer cUyN, orange for Reviewer 6kbY, green for Reviewer wLw6, and purple for points shared across reviewers.

1. What Each Attention Freedom Contributes

To ablate IHA on one supernet (832 NSGA-II evaluations, 3 seeds), each row locks one or more freedoms: IHA (dqk=dvd_{qk}=d_v) ties Q/K and V widths to test decoupling dqkd_{qk} from dvd_v, Fixed KV locks nkvn_{kv}, GQA-style locks head dimension, and Flextron-style locks both (nhn_h and MLP width searched, Cai et al., ICML 2024). Entries show median energy savings (%) over IHA uniform scaling at matched loss (32 decode unless marked 2,048):

Space135M Gemmini135M Gemmini (2,048)135M rDXE360M Gemmini360M rDXEQwen3-0.6B Gemmini
Flextron-style−0.1−20.0−0.75.03.82.1
Fixed KV4.92.62.28.05.84.5
GQA-style4.52.68.37.19.2−0.7
IHA (dqk=dvd_{qk}=d_v)8.010.18.39.78.23.6
IHA6.67.08.69.48.22.9
  1. Elastic head dimension and KV grouping jointly drive the edge gain. Flextron-style yields no saving at 135M and half of IHA's at 360M. Because the rDXE ring penalizes head and KV structure while Gemmini penalizes head dimension (dropping 135M 2,048-decode savings from 7.0% to 2.6% when locked), narrow spaces fall 0.057 to 0.162 hypervolume (HV) behind when missing a target's active knob, whereas combining both freedoms keeps HV within 0.0125 of the best space across all five SmolLM2 settings.
  2. Decoupling dqkd_{qk} and dvd_v (IHA vs. IHA (dqk=dvd_{qk}=d_v)). Tying dqk=dvd_{qk}=d_v shrinks per-block shapes 4x and suffices on 25-generation edge runs, whereas decoupling dqk≠dvd_{qk} \neq d_v in full IHA yields clear gains under fused GPU kernels and longer searches: on an H100 (PyTorch 2.12), all 21 in-budget Qwen3-1.7B front models set dqk≠dvd_{qk} \neq d_v in 71% of layers, cutting decode energy 4.0% to 8.9%.

On Qwen3-0.6B (4.0×10144.0 \times 10^{14} shapes, 48x SmolLM2), extending Gemmini search from 25 generations (above) to 50 and 100 (3 seeds) gives energy savings (%):

Qwen3-0.6B space50 generations100 generations
Flextron-style3.53.7
Fixed KV7.18.5
GQA-style1.93.6
IHA (dqk=dvd_{qk}=d_v)6.58.3
IHA7.38.9

With 208 shapes per block (vs. 52 in IHA (dqk=dvd_{qk}=d_v) and 4 to 64 in the rest), IHA rises from 2.9% to 8.9% (+6.0 pts), overtaking IHA (dqk=dvd_{qk}=d_v) to lead all spaces at 50 and 100 generations.

2. How Raw Slice Scores Relate to Post-Training Quality

Median matched-loss savings (%, 3 seeds) over uniform scaling span the full front and the 40% loss bound (≤1.4×\le 1.4\times full-width loss):

Backend (135M / 360M)Energy (full front)Latency (full front)Energy (40% loss bound)
Gemmini6.6 / 9.46.6 / 9.82.6 / 2.0
Eyeriss6.5 / 8.76.5 / 8.04.2 / −0.1
FLAT7.2 / 8.77.5 / 9.44.2 / 1.2
rDXE ring8.6 / 8.2−1.9 / 3.61.6 / 3.9

Raw slices rank candidates before weight recovery. Practical 60% to 80% energy targets sit above the 40% bound, where LLMForge saves 8.3% median energy and lowers loss at 188/189 uniform points by 0.251 nats median. Standalone 8,000-step checkpoint training brings a 65%-energy Gemmini 135M pick from 1.60x slice loss to 1.26x pretrained loss (inside the 40% bound), beating its uniform partner by 0.301 nats. Within the 40% bound, LLMForge saves energy on 7/8 SmolLM2 settings and by 2.9% to 8.2% on all four targets at Qwen3-0.6B.

3. Why Target-Specific Cost Models Matter on Real Hardware

Minimizing parameter count or FLOPs works when hardware cost tracks weight bytes (as on Timeloop), yet worsens rDXE decode latency by 1.7% to 18.4% because it ignores stage packing and lane padding. Searching with the rDXE cost model instead beats every proxy seed in 3-objective HV. Likewise, running Gemmini-searched models on a Pixel Watch 5 or server CPU slows decoding by 1.20x to 1.35x because their widths shift the INT8 runtime from fast group-64 kernels down to group size 16. Searching directly with our smartwatch predictor avoids this cliff and cuts measured watch TPOT by 7.0% and energy by 7.6% over uniform scaling at lower loss. Because one supernet serves every target search (15 to 45 min), its one-time cost is amortized: 2,500 steps at length 4,096 (163.8M tokens, 4.4 to 27.7 H100 hours across five models, ≤0.0082%\le 0.0082\% of pretrain tokens, correcting 41M at 1,024).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.