- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
LLMForge presents a hardware-aware architecture-search framework for pretrained language models. Its Infinite-Head Attention (IHA) search space independently varies the query-head count, key–value group count, query/key dimension, value dimension, and MLP width. The authors elastify five pretrained models into shared-weight supernets using coordinated slicing, learned group alignment, and a single supernet-training run per base model. Forge-DSE then uses NSGA-II to search quality–cost trade-offs with four backend classes: GPU profiling, analytical accelerator modeling, joint model–chip co-design, and a smartwatch cost predictor. Across eight model–accelerator configurations, the paper reports a median 8.4% reduction in modeled energy per token relative to uniform scaling at matched supernet validation loss.
Critical strengths
•The paper addresses a practical and relevant problem: selecting language-model architectures under hardware-specific energy and latency constraints rather than relying only on parameter count or FLOPs. •The paper provides a broad empirical study across five pretrained language models and multiple hardware-evaluation backends, examining how hardware characteristics and deployment objectives influence architecture selection. The detailed experimental protocols and reported per-seed results also provide a useful basis for analyzing the behavior and reproducibility of the proposed search framework.
Critical weaknesses
• Novelty is incremental and insufficiently isolated. IHA primarily relaxes constraints among existing attention dimensions; it does not introduce a fundamentally new attention computation. The paper does not convincingly establish that the combination of independently tunable head count, KV grouping, query/key width, and value width is a sufficiently substantial advance over elastic Transformer search, GQA-based design, and prior hardware-aware NAS. Table 1 provides only a high-level comparison, without direct, budget-matched comparisons against the closest methods. • The ablation study is insufficient to isolate the contribution of IHA. The IHA-versus-GQA comparison is limited to a single model and hardware backend, with only a small hypervolume improvement. The paper does not establish whether the benefits of IHA generalize across models and hardware targets or whether the gains primarily come from the search algorithm and heterogeneous capacity allocation.
Questions for authors
- What is the key technical novelty of IHA compared with existing elastic Transformer architectures and GQA-based designs? Can the authors provide direct, budget-matched comparisons with the closest existing methods to demonstrate its advantages?
- Can the authors provide additional ablations across different models and hardware backends to establish that the benefits of IHA are consistent and attributable to the expanded search space, rather than primarily to NSGA-II or heterogeneous capacity allocation?
- How reliable are the hardware cost estimates across the different backends, particularly for the analytical accelerator models and the ring-based simulator? How do potential prediction errors affect the reported energy savings and the ranking of selected architectures?
We really appreciate your thoughtful review, especially the push to separate what comes from the expanded attention space versus per-block allocation and NSGA-II, and to test our cost models under prediction error. Working through both made our empirical case much sharper. Please check the green and purple parts in the revised PDF.
How IHA Differs From Prior Designs Under Equal Budgets (W1, Q1)
Rather than a custom kernel, IHA is a weight-sharing search-space parameterization: every candidate runs on standard GQA/MHA kernels while combining per-block elastic head dimension (independent of nh, allowing dqk=dv) and per-block KV grouping in one pretrained supernet:
In budget-matched searches (832 NSGA-II evaluations, 3 seeds on one supernet), Flextron-style elasticity (only nh and MLP width searched) saves −0.1% (Gemmini) and −0.7% (rDXE) at 135M vs. 6.6% and 8.6% for IHA, and half of IHA's saving at 360M (5.0% vs. 9.4% Gemmini, 3.8% vs. 8.2% rDXE). Whenever a target penalizes a missing knob, restricted spaces trail IHA by 0.046 to 0.151 held-out hypervolume (HV): locking KV drops 135M rDXE savings from 8.6% to 2.2%, while locking head dimension (GQA-style) drops 135M Gemmini 2,048-decode savings from 7.0% (10.1% for IHA (dqk=dv)) to 2.6%. Narrow spaces work when aligned with a target's bottleneck (GQA-style on rDXE), whereas combining both freedoms keeps HV regret within 0.0125 across every SmolLM2 target and decode length, and decoupling dqk=dv in full IHA cuts measured H100 decode energy 4.0% to 8.9% under fused kernels (PyTorch 2.12).
Pricing target hardware cost rather than parameter or FLOP proxies is equally critical. Re-priced on rDXE, proxy searches degrade TPOT 1.7% to 18.4% vs. uniform scaling, while every LLMForge seed beats every proxy seed in 3-objective HV:
Disentangling the Search Space, Per-Block Widths, and NSGA-II (W2, Q2)
Across all five SmolLM2 settings in General Response §1, comparing uniform grids with per-block NSGA-II and random sampling separates all three factors (3-seed medians at matched loss):
Expanding the space saves 5% to 24% energy on uniform grids before search (largest at 2,048 decode), and NSGA-II adds another 6% to 10%. On Qwen3-0.6B (48x larger space), LLMForge saves 2.9% to 4.6% energy on all four targets at 832 evaluations (beating GQA- and Flextron-style on Gemmini by 0.031 and 0.018 HV), rising after 100 generations (3,232 evaluations) to 8.9% on Gemmini vs. 3.6% to 3.7% for fixed-128 spaces.
How Reliable Are the Hardware Cost Models? (Q3)
We validated cost fidelity via component perturbations, Monte Carlo noise, and device runs:
On Timeloop substrates, 95.5% to 98.6% of decode energy is off-chip byte traffic, making rankings insensitive to unit weights. On a Pixel Watch 5, testing our frozen predictor on 324 newly measured models gives 4.6% median energy error (215 drift-free runs) and keeps measured order on 96.5% (energy), 95.2% (TTFT), and 88.7% (TPOT) of pairs separated by ≥8.4% predicted cost. On device, smartwatch-searched 135M cuts TPOT 7.0% and energy 7.6% vs. uniform in all 3 rounds at lower loss, while ignoring INT8 group size raises TPOT 4.7% to 18.7% at matched loss.