acceptodds
Under review as a conference paper at ICLR 2027

One Checkpoint, Many Sizes: Quality and Latency in Multi-Exit LMs

Abstract

Multi-exit language models turn one training run into a configurable family of deployment sizes. We characterize their quality and latency through capacity-controlled comparisons of chained Matryoshka suites with independently trained, token-matched model ladders. At the 200M scale, resizing our initially under-allocated later exits (16–31% below their baselines) closes the observed accuracy gap, with panel-dependent gains. At 3B, we evaluate a branched design that combines earlier junction taps with exit-private neck layers. A version within 0.5% of classic largest-exit capacity gains 0.37–0.84 accuracy points over classic in two paired seeds, with lower byte-level perplexity. In these runs, its seven-task accuracy approaches the independent vanilla endpoints and its byte-PPL improves over both. Timing makes the execution tradeoff explicit: deep exits take 22–47% longer per generated token at batch 1 on one H100 despite similar matmul FLOPs. Together, the results show how capacity allocation and branching can improve suite quality, while measured latency identifies the workload-specific cost of each exit. This integrated view of capacity, quality, and runtime provides a concrete basis for designing and selecting multi-exit model families.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-exit language models, matryoshka models, capacity matching, model families, inference latency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ATyaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines quality–latency trade-offs in multi-exit language models through capacity-controlled comparisons. Building on chained Matryoshka architectures, it evaluates a branched design combining exit-private neck layers with early junction taps, allowing deeper exits to bypass private layers while retaining shared computation.

Experiments at the 200M and 3B scales use independently trained, token-matched models as baselines. Resizing under-capacity exits removes the observed accuracy deficit at 200M. At 3B, the capacity-matched branched variant improves accuracy over the classic architecture by 0.37–0.84 percentage points across two seeds and lowers byte-level perplexity. However, H100 measurements reveal 22%–47% higher batch-1 decoding latency for deeper exits despite comparable FLOPs.

Critical strengths
  1. The capacity-controlled comparisons provide credible evidence for the branched design. At 3B, the matched variant improves seven-task accuracy by 0.37–0.84 percentage points across two paired seeds while keeping its largest-exit capacity within 0.5% of the classic model. The alternative accuracy panel and byte-level perplexity also favor this variant. This is the strongest evidence supporting the architectural contribution, since the gains persist after controlling for the original design’s larger parameter budget.

  2. The latency measurements reveal a substantial cost that the quality comparisons alone would overlook: deeper exits incur 22%–47% higher batch-1 decoding latency despite comparable FLOPs. The pattern persists across three attention backends and randomized process restarts, while higher-batch results show that some exits recover prefill performance. I consider this an important empirical contribution because it establishes where shared-checkpoint models sacrifice runtime performance, even when their capacity and predictive quality approach those of independent models.

Critical weaknesses
  1. The evidence for a reliably beneficial architectural change remains limited. The main capacity-matched comparison includes only two training seeds, with seven-task gains of 0.84 and 0.37 percentage points. At 200M, the necked model’s advantage over the resized classic model also reverses across evaluation panels. Although the authors appropriately acknowledge these limitations, the experiments establish improvements for particular checkpoints more convincingly than a consistent advantage of the proposed design. This limits the strength of the paper’s central architectural contribution.

  2. The description of the capacity-matched variant is inconsistent at a point essential to interpreting the main experiment. Sections 3.1 and 5 describe reducing the largest exit’s private depth, whereas Appendix B specifies changing the middle segment from eight shared plus two private layers to seven shared plus three private layers, leaving the final segment unchanged. The latter changes which layers receive deeper-exit gradients, rather than simply removing largest-exit private capacity. Until the actual configuration is clarified, it is difficult to determine precisely which intervention produced the reported gains or to reproduce the key comparison.

Questions for authors
  1. I found the description of the capacity-matched variant somewhat difficult to reconcile with Appendix B. Section 3.1 describes reducing the largest exit’s private depth, but the appendix instead changes the middle segment from eight shared plus two private layers to seven shared plus three private layers. Which configuration was used in the experiments, and should the main-text description be revised?

  2. The distillation term in Section 3.1 would benefit from a more explicit definition. What loss and temperature are used, and is the largest exit’s teacher distribution detached from the computation graph? In particular, I would like to understand whether the distillation losses also update the teacher branch.

  3. The related work mentions loss reweighting as an existing approach to the largest-exit quality gap. Was this strategy considered for the classic baseline in Section 3.2? A comparison, if available, would help assess how much the branching design adds beyond adjustments to the training objective at the same capacity and token budget.

  4. The two capacity-matched runs in Section 5 both improve over classic, although the accuracy gains are modest. Would it be possible to provide paired evaluation confidence intervals for the seven-task average difference at each seed? These would help establish how sensitive the observed gains are to the evaluation sample. I would also be interested in any additional training seeds already available, since evaluation intervals alone do not capture run-to-run variability.

3 comments
AuthorsOpenReview ↗

Response to ATya

Thank you — your Q1 identified a real inconsistency, and it is the clearest actionable item across all three reviews.

Q1 / W2 (which configuration is the matched variant? — the inconsistency you identified). Appendix B is correct; §3.1 is wrong, exactly as you inferred. The intervention moves one middle-segment layer from the shared body into the middle exit's private neck (8 shared + 2 private → 7 shared + 3 private @2,304); the largest exit's own private segment (6 exclusive @4,352) is unchanged. The capacity effect comes from the early-tap routing: the largest exit traverses the shared body only, so re-routing that layer into exit-1's neck removes it from exit-2's path (36 → 35 layers), bringing exit-2 in-use capacity to +0.45% vs classic (3,036.0M vs 3,022.2M, tensor-exact audit). Exit-1 retains the original necked arm's layer count and widths; exit-0 retains the classic arm's geometry — their trained weights and outputs need not be identical. The revised PDF already replaces the §3.1 sentence with: "the matched variant moves one middle-segment layer from the shared body into the middle exit's private neck, which removes that layer from the largest exit's traversal path and brings its in-use capacity within 0.5% of classic, while leaving the largest exit's own private segment unchanged."

Q2 (distillation term). Full specification: teacher-to-student soft cross-entropy, -Σ_v softmax(z_teacher)_v · log_softmax(z_student)_v, per-token, temperature 1 (no scaling, as in Godey & Artzi's Eq. 2) — up to the teacher's constant entropy this is the forward-KL direction KL(p_teacher ‖ p_student), not reverse KL. The teacher-logit input is detached: a custom autograd Function returns no gradient for the teacher logits, so no gradient flows through the teacher-logit path; the largest exit trains on its own CE only. Student losses still update the shared layers on their paths. Student exits use L_m = (1−α)·ce_m + α·distill with α = 0.3 for all suite runs. This specification is added to §3.1 in the revised PDF (with the KL-direction wording corrected there as well).

Q3 (loss reweighting as the classic-side alternative). We tested it: nine weighted-objective arms plus one unweighted replay control (ten runs) at 200M, seed 1234 (per-exit loss weights, mean-1-normalized; the same mechanism Godey & Artzi study). Best arm: exit-2 seven-task 41.38 vs replay control 41.14 (+0.24 pts; byte-PPL ≤0.3% better). Against the original width-first baseline (41.25), the resized classic and necked arms score 43.48 and 44.04 (+2.23/+2.79). This sweep did not close the observed gap; it does not exclude benefits from other weightings or at 3B — that is why §3.2's baseline is architectural. The sweep table is added to the appendix in the revised PDF.

Q4 / W1 (paired evaluation CIs; additional seeds — the evidence-strength concern). Both halves are now delivered. Paired per-item 95% CIs (log-samples re-evals of the same checkpoints, 10,000-replicate bootstrap joined per doc): 7-task matched−classic exit-2 = +0.84 [+0.13, +1.54] (s1234), +0.37 [−0.33, +1.06] (s3456), −0.09 [−0.82, +0.62] (s2345) — narrower than the unpaired intervals announced earlier. HellaSwag's per-task paired intervals exclude zero in all three seeds (component evidence). All 70 comparison-side checks (including repeated matched-arm checks) reproduce the canonical task metrics at the reported precision. On additional training seeds: the third paired seed (2345), run under the descriptive ±0.7 bands declared in App D of the submitted paper and reported unconditionally: −0.09 — inside the band; across three seeds the benefit is not sign-consistent, and we do not claim these bands quantify training-seed uncertainty (three-seed summary table in App L).

Thanks for the reply. I will keep my score the same.

AuthorsOpenReview ↗

Reply #2 to Reviewer ATya

Thank you for the follow-up. We appreciate your recognition of the capacity-controlled comparisons as “credible evidence for the branched design” — indeed “the strongest evidence supporting the architectural contribution” — and of the latency measurements as “an important empirical contribution.” These capture the two contributions we aim to establish.

We have addressed the concerns raised in your review through the following revisions and additional experiments:

  • Geometry: We corrected the §3.1–App. B inconsistency in both places where the incorrect phrasing appeared.
  • Distillation: §3.1 now specifies teacher-to-student soft cross-entropy at temperature 1, with detached teacher logits.
  • Loss weighting: A new appendix reports the 200M loss-reweighting sweep.
  • Seed variability: We trained a third paired seed during the response period and report its result unconditionally: −0.09, within the pre-declared ±0.7 band in App. D. App. L reports the paired confidence intervals for all three seeds, addressing Q4.

We agree that the architectural gain is modest and not sign-consistent across seeds, and the revised manuscript states this limitation explicitly. We hope the expanded evidence and clearer qualification help you assess the contribution.

For Q4, we also provide the per-task paired results below. Each entry is the matched−classic difference at exit 2, in percentage points, followed by its 95% paired bootstrap confidence interval. We use 10,000 bootstrap replicates, resampling paired evaluation documents within each task, with the same protocol as the seven-task intervals. These intervals quantify evaluation-set uncertainty conditional on each trained model pair; they do not capture variability from retraining with different seeds.

TaskSeed 1234Seed 3456Seed 2345
ARC-Easy+1.01 [−0.38, +2.40]−0.38 [−1.77, +1.01]+0.67 [−0.72, +2.02]
ARC-Challenge+0.51 [−1.54, +2.56]+1.71 [−0.43, +3.84]−0.68 [−2.73, +1.37]
HellaSwag+1.44 [+0.89, +2.01]+1.76 [+1.22, +2.30]+0.89 [+0.36, +1.43]
LAMBADA+1.01 [−0.06, +2.10]+0.47 [−0.62, +1.53]+0.45 [−0.62, +1.51]
OpenBookQA+0.80 [−1.80, +3.40]−0.20 [−2.60, +2.20]−0.60 [−3.40, +2.20]
PIQA−0.11 [−1.41, +1.20]+0.44 [−0.92, +1.80]+0.22 [−1.09, +1.52]
WinoGrande+1.18 [−1.58, +4.10]−1.18 [−4.03, +1.66]−1.58 [−4.50, +1.34]
Seven-task mean+0.84 [+0.13, +1.54]+0.37 [−0.33, +1.06]−0.09 [−0.82, +0.62]

WinoGrande remains a component result, consistent with the paper. ARC-Easy, LAMBADA and WinoGrande use raw accuracy (acc); the remaining tasks use normalized accuracy (acc_norm).

Reviewer_sbAjOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies chained Matryoshka multi-exit language models, following Godey & Artzi (2026): width-increasing segments are joined by junctions, and every exit is a complete language model. These suites are compared against independently trained, token-matched "vanilla" ladders, with explicit accounting of parameter capacity.

  • 200M scale. The authors' initial width-first instantiation under-allocated its later exits (16–31% fewer parameters) and trailed vanilla. Resizing the exits to matched capacity removes the gap.
  • 3B scale (35B tokens). In a "necked" design, each exit gets private tail layers, and deeper segments tap the shared body before the neck.
    • The necked design improves on the classic chain in two paired seeds.
    • A capacity-matched variant, whose largest exit is within 0.5% of classic, gains +0.84 / +0.37 points on the 7-task panel and lowers byte-perplexity by about 0.6%.
    • On the 7-task panel its accuracy approaches, but does not exceed, vanilla.
  • Latency. On one H100 at batch 1, the middle and largest exits decode 22–47% slower than vanilla models with similar matmul FLOPs. The reason is their deeper paths: 35–39 layers vs. 28.

The paper also reports training-FLOP and storage accounting, batch-scaling results, KV-reuse fractions, and an offline answer-deferral analysis.

Critical strengths
  1. Careful capacity accounting. The paper separates three quantities and backs them with boot-time audited counts (Table 6):

    • exit-resident parameters;
    • per-token logical parameters (NE+LH);
    • all-exits-hot storage.

    This is a useful methodological contribution; many-in-one model papers are often loose about it.

  2. Unusually transparent reporting. Readers can judge the strength of the evidence themselves from:

    • per-seed values and paired contrasts;
    • both vanilla seeds;
    • a predeclared WinoGrande target that failed to replicate (App. D);
    • explicit neutral bands.
  3. Measured runtime, not FLOP proxies. Batch-1 timing uses randomized restarts, three attention backends, and checks of cached-decode logits against prefill. Batch scaling and model FLOP utilization (MFU) are also reported.

  4. An informative offline deferral analysis (App. J).

    • An oracle that escalates only the small exit's errors would gain +3–5 points over always running the larger exit.
    • An agreement gate recovers always-large accuracy while the small exit answers 79–91% of items.
Critical weaknesses

W1. The main architectural gain is small relative to seed variation.

  • Gain within the authors' own neutral band. The matched-necked minus classic gain on the primary metric is +0.84 and +0.37 points (mean +0.60). Under the predeclared ±0.7 band (App. D), one seed and the mean fall in the neutral range.
  • Seed noise is as large as the effect. The two vanilla 3B seeds differ by 0.78 points on avg8 (62.91 vs. 63.69), the same size as the claimed effect.
  • Not statistically established. With two seeds per arm, a 0.4–0.8-point difference on seven zero-shot tasks is not established.
  • Perplexity gain is small. The byte-perplexity improvement (~0.6%) is more consistent across seeds but also small.
  • Still below vanilla. The matched largest exit is slightly below both vanilla runs on the 7-task panel (56.01 / 55.96 vs. 56.11 / 56.17).

W2. The 200M "capacity allocation recovers quality" result is largely expected.

  • Self-inflicted deficit. The deficit came from the authors' own under-sized instantiation; the paper notes that Godey & Artzi already target per-exit capacity.
  • Expected outcome. Exits with 16–31% fewer parameters trailing, and then matching once capacity is restored, reads more as a sanity check than as one of three headline findings.
  • Weak evidence. These results are single-seed, several gains are on near-chance tasks, and the sign depends on the evaluation panel. The necked suite beats vanilla on the 7-task panel (44.04 vs. 42.82) but not on avg8 (51.10 vs. 51.23; Table 1).

W3. The resource accounting cannot yet settle the cost/benefit question.

  • Only bounds on training compute. Table 32 gives an upper bound of 950 EF for the suite (which double-counts the shared body) and a lower bound of 590 EF, versus 976 EF for the ladder. The actual saving could therefore be anywhere from ~3% to ~40%. Measured GPU-hours or exact FLOP accounting would resolve this cheaply.
  • Modest storage saving. Checkpoint storage shrinks by about 12% (242.9 vs. 274.8 GB).
  • Deployment scenario not measured. The motivating adaptive-serving scenario (escalation with reuse of shared computation) is evaluated only offline, without conditional latency.
Questions for authors
  1. Is the batch-1 timing run with CUDA graphs or torch.compile, or in eager mode? Can you repeat the decode measurement for the matched suite and vanilla 3B in an optimized serving setup (e.g. CUDA graphs or vLLM)? Does the depth premium persist?
  2. Can you report the actual training cost of the suite (GPU-hours, or exact FLOPs counting the shared body once) instead of the 590–950 EF bounds?
  3. If compute allows, can you add a third seed for classic and matched-necked at 3B, or a multi-seed replication at an intermediate scale? Otherwise, please describe the gain as within noise where appropriate.
  4. Can you separate the necked bundle into tap-only, neck-only and both? For example at 200M with at least 3 seeds.
  5. Under what concrete serving workload does one suite beat training three models? An end-to-end measurement of one escalation policy, with KV reuse over the shared body, would make the deployment argument concrete.
1 comment
AuthorsOpenReview ↗

Response to sbAj

Thank you for the careful reading and for engaging with the capacity accounting. On W3 we replace analysis with measurement; on W1 the additional seed is complete and reported below.

W1 (gain small relative to seed variation). Four points:

  • Primary endpoint first: three paired seeds read +0.84 / +0.37 / −0.09 on the seven-task panel (mean +0.37) — small and not sign-consistent across three seeds; the third is inside the earlier descriptive ±0.7 band (App D, submitted paper), which does not quantify training-seed uncertainty. We keep §5's scope (observed checkpoints).
  • Avg8 shows positive matched−classic exit-2 gains in all three seeds (+0.93/+1.08/+0.43) — a previously defined secondary outcome, not a substitute primary criterion.
  • HellaSwag (highest-n among the seven primary tasks, 10,042 items) has per-task paired intervals excluding zero in all three seeds (+1.44 [+0.89,+2.01]; +1.76 [+1.22,+2.30]; +0.89 [+0.36,+1.43]) — component evidence.
  • byte-PPL improves over classic on all four OOD corpora in every seed (relative changes of four-corpus mean byte-PPL: −0.56%/−0.67%/−0.52%). The earlier WinoGrande target did not replicate (+3.94/−0.07, original necked vs classic); we retain that failure and treat task-level results as components. Full table: App L.

W2 (200M capacity recovery is expected). Partially accepted: for our width-first instantiation the recovery is a sanity check; the revised PDF demotes it to an audit result (allocation audit + panel-independent byte-PPL recovery). Godey & Artzi target per-exit capacity in their construction; our contribution is the audit methodology plus failure-mode documentation.

W3 (resource accounting cannot settle cost/benefit; your Q2). Now measured:

  • Training: 525.9 / 559.9 / 590.1 GPU-h (classic / original necked / matched) vs 632.9 for the ladder, same configs ⇒ 16.9% / 11.5% / 6.8% savings for the stated allocations; exact FLOPs and energy remain unmeasured. At 200M the suite costs more than its ladder (62.7 vs 47.5 GPU-h); the saving is scoped to these large-scale runs.
  • Storage: BF16 weight files 6.94 GB (one checkpoint, three sizes) vs 9.82 GB (three vanilla models): −29.3%.
  • Deployment: eager decode premium 1.51× at batch 1, flat in batch (Q1 below); component timings project 20.8–32.8% below always-large for a small-only decision rule, with Q5's gate and accuracy caveats.

Q1 (CUDA graphs / vLLM; does the depth premium persist?). Table 3 is eager mode (no inference-time CUDA-graph support in the fork); its audited protocol randomizes three attention backends (FA2/cuDNN-fused/eager) across three process restarts with cached-decode-logit checks against prefill. The same-process twin the question targets (flash): batch-1 decode, matched exit-2 47.91 ms/tok vs vanilla 31.73 ms/tok — 1.51×, flat in batch (1.48× at batch 8, 1.51× at batch 32). Table 3's protocol gives 1.45–1.46× for the same pair — separate timing experiments, not evidence that optimized serving preserves the premium. Our torch.compile attempts produced no valid compiled timings before interruption or termination; traces include graph breaks around TransformerEngine operations. We cannot answer whether the premium persists with CUDA graphs or an optimized serving engine. (vLLM would additionally require a Megatron→HF conversion.) Tables in App L.

Q3 (third seed / intermediate-scale replication). The third 3B pair is complete (above); multi-seed intermediate-scale and tap/neck factorial experiments are not in this revision; the claims above are limited to completed experiments.

Q4 (separate tap-only / neck-only / both). We have not isolated tap-only, neck-only, and combined effects in a three-seed factorial experiment; the existing −jcat (junction-gate) and −nodistill ablations address different factors (gating, distillation) at 200M at fixed capacity. We therefore attribute the observations to the bundled redesign and make no isolated causal claims for the private necks.

Q5 (concrete serving workload where one suite beats three models). For seed 1234, matched necked uses 590.1 versus 632.9 GPU-h (6.8% less) and 6.94 versus 9.82 GB of BF16 weight files (29.3% less) than the vanilla ladder; its larger exits carry an eager latency premium. On conditional serving we are deliberately precise: separate eager exit timings give a projected cost C(f) = 6,944.7 + f×11,937.2 ms/item for a hypothetical gate that decides from the small exit alone (excluding gate overhead) — at an illustrative escalation fraction f = 0.187, ≈9,177 ms/item, ~23% below always-large. But the agreement gate requires both exits' predictions and cannot realize these savings (App K: an analytical answer-deferral ceiling), and accuracy at these projected costs has not been demonstrated. We do not claim measured conditional-serving savings.

Reviewer_PyTNOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper looks at the trade-off between quality and latency in multi-exit language models, comparing them against ladders of independently trained models with matched token budgets. At 200M scale, the authors find that the later exits were initially under-allocated, and that resizing them brings accuracy back. At 3B scale, an early-tap architecture with exit-private neck layers does better than the classic chained design; this includes a variant whose largest exit is capacity-matched, evaluated with two paired seeds. The paper also measures inference latency on H100 and accounts for checkpoint storage and analytical training FLOPs. The takeaway is a concrete trade-off: a shared checkpoint can give better quality, yet at low batch sizes it can be considerably slower at inference than standalone models.

Critical strengths
  1. The controls go a long way towards making the multi-exit quality results interpretable. Table 1 shows that the largest exit of the original 200M model has only 0.694 times the capacity of the vanilla model, and that resizing raises seven-task accuracy from 41.25 to 43.48. At 3B, Table 2 adds a matched necked sibling: its largest-exit capacity ratio is 1.011 versus 1.006 for classic, and it improves seven-task accuracy by 0.84 and 0.37 points in the two paired runs. I also appreciate that the authors credit this to the redesign as a whole and do not claim an isolated causal effect of the private necks.
  2. Runtime is actually measured rather than inferred from parameter counts. The batch-1 timing protocol is randomized and covers three attention backends and three process restarts. Table 3 shows that the matched largest exit decodes at 1.45–1.46 times vanilla latency, even though its matmul-FLOP ratio is 0.973. Together with the batch sweep and the clear distinction between model storage and per-token cost, this makes the paper a useful empirical study of what these models cost at deployment.
Critical weaknesses
  1. The architectural gain is small given how little training replication there is, and it depends on which comparison panel one looks at. There are only two paired 3B runs, and most of the 200M and supporting arms have a single seed. On seven-task accuracy, the matched largest exit is still below both vanilla runs; on avg8 the ordering changes, and on byte-PPL it improves. At 200M, necked and resized-classic swap places between seven-task accuracy and avg8. The authors acknowledge these caveats, but they do make me less confident that the architectural change gives a reliable benefit beyond the specific checkpoints observed.
  2. The latency cost is measured directly, but the resource advantage of the whole family is not measured as directly. Table 4 compares resumable model-plus-optimizer directories and a matmul-only bound on training FLOPs for the original necked suite, whereas the closest quality control is a different, matched sibling. The 242.9 vs. 274.8 GB numbers refer to training artifacts, not to what has to be held in memory at inference, and the ≤950 vs. 976 EF accounting does not show savings in wall-clock time or energy. So the end-to-end value of getting the full ladder from one run is still somewhat unclear, especially given the large latency premium at fixed exits. A resource–quality frontier computed consistently for the same configurations would make the main practical conclusion more convincing.
Questions for authors
  1. Could you report measured training GPU-hours (including any teacher/distillation cost) and inference-time weight-only storage, for both the original and the matched necked suites, compared with the independent ladder? Please use the same configurations as in the quality comparisons, and explain how compute in the shared trunk is counted.
  2. Would it be feasible to add one more paired training seed for classic and matched necked? If not, could you quantify how sensitive the recommendation is to the 0.37-point gain observed with the second seed? I would keep evaluation bootstrap intervals separate from training-seed uncertainty here.
  3. Can the claimed deployment frontier be shown with accuracy and measured latency for the same checkpoints and batch sizes, including the high-batch cases? The current fixed-exit results support an empirical trade-off, but the offline agreement rule needs predictions from both exits, so by itself it cannot show savings from conditional compute.
  4. Which conclusions would still hold if only the alternative avg8 panel were used at 200M? I would like to see capacity recovery distinguished from a consistent advantage of necked over resized classic, since this affects how broad the architectural contribution really is.
  5. Missing related references: [1] VisionSelector: End-to-End Learnable Visual Token Compression for Efficient Multimodal LLMs, ACM International Conference on Multimedia, 2026.
1 comment
AuthorsOpenReview ↗

Response to PyNT

Thank you for the positive assessment of the controls and the measured runtime protocol.

W1 (small gain, limited replication, panel-dependence). We agree with your reading, and §5's framing already claims improvements for the observed checkpoints rather than an architecture-level effect. Primary three-seed result: 7-task +0.84 / +0.37 / −0.09 (mean +0.37) — small and not sign-consistent across three seeds (Q2). Secondary: avg8 positive in all three seeds (+0.93/+1.08/+0.43); HellaSwag's per-task paired intervals exclude zero in all three (component evidence); byte-PPL improves on all four corpora in every seed. The 200M panel asymmetry you flag is stated in the revised PDF (Q4); paired CIs in our ATya reply and new App L.

Q1 (measured GPU-hours + weight-only storage, shared-trunk counting). Measured end-to-end from the training logs and checkpoint files of the quality-comparison configurations, including the original necked arm:

ConfigurationJob wall; GPUsGPU-h
Classic suite 3B, 35B tok (s1234)16h26m; 32525.9
Original necked suite 3B (s1234)17h30m; 32559.9
Matched necked suite 3B (s1234)18h26m; 32590.1
Independent ladder: 500M + 1.5B + 3B4h40m;16 + 11h39m;16 + 11h37m;32632.9

Savings vs the ladder: 16.9% (classic) / 11.5% (original necked) / 6.8% (matched). The ladder's 27h56m sums separate job walls under mixed 16/32-GPU allocations — not one elapsed run; we claim GPU-hours, not equal-resource time-to-train. These supplement the analytical FLOP bounds; exact FLOPs and energy remain unmeasured. Counting: the shared trunk is computed once; private necks, exit CEs, and distillation add the rest. The distillation teacher is the suite's own largest exit in the same forward pass — no separate teacher, so the GPU-hours include that work. Weight-only storage: see W2(ii).

W2 (resource advantage not measured as directly; Table 4 configs; artifacts vs inference). Conceded on three counts, fixed by measurement: (i) Table 4 compared the original necked suite's artifacts against the ladder while the closest quality control was the matched sibling — the table above covers all three variants at the quality-comparison configurations, resolving the inconsistency; (ii) the 242.9-vs-274.8 GB figures were training artifacts — the inference-relevant quantity is weight-only storage: 6.94 GB (one checkpoint, three sizes) vs 9.82 GB (−29.3%); the largest matched exit's BF16 weights are estimated from audited parameters at 6.07 GB, versus approximately 6.01 GB for vanilla (runtime memory additionally includes caches and other allocations); (iii) energy is not measured — we claim GPU-hours and job wall-clock only. The resource–quality comparison is assembled in Q3.

Q2 (more paired seeds / sensitivity of the 0.37-point gain). Done: a third paired seed (2345), identical recipe, trained to completion and evaluated. Readout under the ±0.7 bands declared in App D of the submitted paper, reported unconditionally: 7-task −0.09 [−0.82, +0.62]. It is inside the descriptive ±0.7 band, which does not quantify training-seed uncertainty. Paired evaluation intervals are kept separate from training-seed uncertainty, as you suggest (three-seed table: App L; per-seed CIs: our ATya reply).

Q3 (deployment frontier on the same checkpoints). Tables 2–3 report quality and batch-1 timing; our Q1/W2 supply training GPU-hours and BF16 weight storage. App K.1 retains the original-necked artifact/FLOP-bound comparison. We have not added a unified frontier for the matched suite or across all batch sizes; the new batch sweep measures decode only. The honest shape: the smallest exit is free (≈1.0× latency, parity accuracy); the larger exits carry a 22–47% batch-1 latency premium, and the matched largest exit does not exceed the vanilla seven-task endpoints in the first two seeds. The offline agreement rule shows no conditional-compute savings and is kept out of any frontier claim.

Q4 (avg8-only at 200M). Capacity recovery holds on both panels (resized classic reaches vanilla parity: 43.48 vs 42.82 on 7-task; 51.40 vs 51.23 on avg8). The necked-over-resized-classic ordering does not hold on avg8 (51.10 vs 51.40). We state this asymmetry in the revised PDF; byte-PPL — improved by the necked design on all four corpora — is the panel-independent 200M signal. These 200M comparisons are single-seed: allocation sensitivity in this instantiation, not a reliable necked-over-resized advantage. On the 3B claim: the first two seeds improve on both panels; the third is −0.09 on the primary seven-task panel and +0.43 on avg8 — secondary evidence, not replication of the primary effect.

Q5 (missing reference). VisionSelector (Zhu et al., ACM MM 2026, DOI 10.1145/3767308.3835408) is added to related work in the revised PDF — complementary: token-level learnable compression inside a multimodal LLM, vs our exit-level construction of a model family from one checkpoint.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for recognizing the paper's capacity-controlled comparisons (PyTN, sbAj, ATya), its "unusually transparent reporting" (sbAj), and that "runtime is actually measured rather than inferred from parameter counts" (PyTN). The three reviews converge on two concerns — statistical strength of the 3B architectural gain, and the resource/ deployment accounting — and one factual error in our main text. Rather than only argue, we used the response period to produce new measured evidence:

  1. A third paired training seed (2345) for classic and capacity-matched necked at 3B: trained to completion and evaluated. Three-seed readout, reported unconditionally: 7-task +0.84 / +0.37 / −0.09 (mean +0.37) — the gain is small and not sign-consistent across three seeds; the third result is inside the earlier descriptive ±0.7 band (App D of the submitted paper), which does not quantify training-seed uncertainty. The claim stands as stated in §5 — improvements for the observed checkpoints, not a statistically established architecture-level effect. Secondary panels stay positive: avg8 +0.93/+1.08/+0.43; matched byte-PPL better in every seed.
  2. Measured training cost supplements the FLOP bounds (PyTN Q1, sbAj Q2): suite arms 525.9 (classic) / 559.9 (original necked) / 590.1 (matched) GPU-h vs 632.9 GPU-h for the independent 3B ladder, same configs as the quality comparisons — measured savings 16.9% / 11.5% / 6.8% for the stated allocations; exact FLOPs and energy remain unmeasured. Weight-only inference storage (BF16 weight files): 6.94 GB (all three sizes, one checkpoint) vs 9.82 GB for the three vanilla models (−29.3%).
  3. Paired per-item 95% CIs (10,000-replicate bootstrap over log-samples re-evals) replace the conservative unpaired intervals: 7-task matched−classic exit-2 = +0.84 [+0.13, +1.54] (s1234), +0.37 [−0.33, +1.06] (s3456), −0.09 [−0.82, +0.62] (s2345). HellaSwag's per-task paired intervals exclude zero in all three seeds (component evidence).
  4. We confirmed the configuration inconsistency ATya found: §3.1's description of the matched variant is wrong; Appendix B is correct. The corrected sentence and the audited geometry are in our reply to ATya, and the fix is in the revised PDF.
  5. Loss reweighting was tested (ATya Q3): nine weighted-objective arms plus an unweighted replay control at 200M — the best weighted arm moves exit-2 accuracy by ≤ +0.24 pts over the control — an order of magnitude less than the architectural interventions.
  6. Latency: fixed-exit premiums measured; adaptive serving honestly scoped (sbAj Q1, PyTN Q3): batch-1 eager decode premium 1.51× (47.91 vs 31.73 ms/tok), flat in batch (1.48–1.51× at 8/32). Component timings project an escalation policy at 20.8–32.8% below always-large for a small-only decision rule — but the paper's agreement gate executes both exits (App K calls it an analytical deferral ceiling) and accuracy at those projected costs is not demonstrated; see our reply to sbAj. torch.compile attempts produced no valid compiled timings (graph breaks around TransformerEngine ops); we cannot answer whether the premium persists under CUDA graphs.
  7. Revised PDF (v2, 41 pp) uploaded with this response (a pdfdiff will show these changes): (i) §3.1's description of the matched variant corrected per ATya Q1 — every occurrence, including the related sentence later in the main text; (ii) the distillation specification added to §3.1 per ATya Q2 (teacher-to-student soft cross-entropy, temperature 1, detached teacher-logit input); (iii) VisionSelector added to related work and the bibliography per PyTN Q5; (iv) the 200M necked-vs-resized panel asymmetry stated explicitly per PyTN Q4; (v) the 200M finding reframed as an allocation audit per sbAj W2, with the matching intro/summary wording adjusted; (vi) the 200M loss-reweighting sweep table added to the appendix per ATya Q3; (vii) new Appendix L with the discussion-period measurements backing this response (three-seed paired replication, projected escalation cost, batch-scaling premiums) plus cross-references added in App D and App K. Main-text corrections and clarifications as listed; page layout preserved (main text still ends p. 9; within the discussion-phase page limit).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.