acceptodds
Under review as a conference paper at ICLR 2027

Contextual Utility of Quantization Moves in Extreme Low-Bit LLMs

Abstract

Accurate measurements of isolated weight replacements need not predict their joint effect on a quantized model. We study this gap for legal replacements at a fixed bitrate, separating isolated pricing error, composition error and changes in evaluation data. On Llama-3.2-1B and activation-ordered 3B GPTQ bases, option-KL composition residuals are 26× and 42× the sums of absolute midpoint-pricing errors; large residuals remain with exact-singleton prices. Exhaustive Qwen3-0.6B lattices reveal context-dependent effects: six of eight moves in the stratified lattice have statistically resolved ARC-KL sign reversals, excluding an exact monotone additive representation. Pair measurements improve prediction of unqueried combinations on both Qwen and Llama. Their value for selection depends on the criterion: the stratified Qwen front retains a quality-coverage advantage after refitting, whereas the 3B additive and quadratic fronts are nearly equal in quality at a one-standard-error tolerance. Low-order endpoint structure leaves continuous gradient responses undetermined, and shared-midpoint prices can differ from those of the executed subset. Support-matched controls show that restricting the number of replacements curbs the failures of large combinations, while additional endpoint search gives bank-dependent NLL/ARC trade-offs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM quantization, extreme low-bit quantization, post-training quantization, contextual utility, quantization interactions, GPTQ, model compression

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NU34OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how well the effects of individual quantization changes predict their combined impact on low-bit LLMs. The authors distinguish errors in estimating individual changes from errors caused by interactions between changes, and study these effects across several quantized models.

The experiments show that accurate estimates of individual changes can lead to large errors when multiple changes are applied together. The authors further analyze these interactions using discrete state lattices and show that pairwise information can improve predictions of unseen combinations. However, the improvements in prediction do not always translate into better model selection, and the benefits of interaction-aware methods vary across settings.

Critical strengths
  • Clear motivation and well-designed experiments. By separating individual estimation errors from interaction effects, the experiments provide convincing evidence that accurate predictions are not sufficient for reliable combination selection.

  • Sound technical analysis with meaningful empirical findings. In particular, the paper demonstrates contextual sign reversals and shows that incorporating pairwise interactions improves predictions of unseen combinations.

Critical weaknesses
  1. Limited generalizability of the findings. The main interaction experiments focus on small 8-move lattices, with five overlapping moves between the two Qwen settings. While the results demonstrate clear contextual interactions, no statistically significant sign reversals are confirmed on Llama-3.2-3B.

  2. Incomplete comparison with nonlinear additive baselines. The paper shows that contextual reversals cannot be exactly captured by monotone additive models, but such models may still provide useful approximations in practice. Although Table 12 includes a monotone single-index baseline, it does not compare its loss-space coverage or selection regret with those of the quadratic model.

  3. Limited evidence of benefits from additional combination queries. The endpoint beam does not consistently outperform simpler support-limited additive selection. For example, results indicate that cap-four addition achieves better ARC-KL than the beam, while the beam's NLL improvement over cap-two addition is not statistically resolved. These results leave the additional computational cost of combination queries difficult to justify across settings.

Questions for authors
  1. Could the authors test more quantization combinations from the existing data and report how often significant contextual reversals lead to better loss coverage?

  2. Could the authors compare the monotone-index model in Table 12 with the quadratic model using the same quality coverage and selection error metrics?

2 comments
AuthorsOpenReview ↗

Response to Reviewer NU34

Thank you for pointing directly to the two missing quantitative links: tying reversals to downstream decision coverage, and evaluating the monotone single-index baseline with the same decision metrics as the quadratic model.

Q1. Across more existing combinations, how often do resolved reversals translate into better coverage? We enumerated all 3,394 four- to eight-dimensional sub-lattices (faces) of both 256-state Qwen grids (1,697 faces each; Appendix D.1, Table 16), evaluating local base/singleton/pair anchors against unseen higher-support combinations while retaining the parent simultaneous max-tt reversal family. At dimensions 4 and 5, quadratic selection improves coverage over exact addition roughly 3x more often when an ARC reversal is present on Stratified—whereas on Witness (which lacks strong helpful reversals), gains are rare either way:

Parent gridDim.Faces w/ ARC reversal: quad improvesFaces w/o reversal: quad improvesFixed-4 gain (w/ vs. w/o, pp)
Stratified4284 / 621 (45.7%)68 / 499 (13.6%)+5.129 vs. +0.164
Stratified5218 / 372 (58.6%)15 / 76 (19.7%)+7.723 vs. +4.965
Witness424 / 585 (4.1%)16 / 535 (3.0%)+0.328 vs. +0.109
Witness538 / 354 (10.7%)5 / 94 (5.3%)+3.957 vs. +2.914

Note that overlapping faces form a descriptive census (significance applies to the parent reversal tests, not per-face counts, and Appendix D.1 includes negative/zero differences plus 1,154 support-split sensitivity checks). Overall, context reversals identify regions where pair modeling helps, rather than guaranteeing pair gains everywhere.

Q2. Monotone single-index vs. quadratic on quality coverage and selection regret. Table 4 (§4) now puts all five models on identical observations across 2,000 paired bootstrap refits (small-support splits: ≤3\le 3 on Qwen, ≤2\le 2 on Llama):

  • Stratified Qwen (163 unseen states): pair quadratic achieves 99.9% one-SE quality coverage and 0.006 standardized grid regret, vs. 81.0% / 0.060 for exact addition, 56.9% / 0.069 for fixed monotone index, and 55.3% / 0.060 for learned monotone index.
  • Witness Qwen (163 unseen states): exact addition (0.035 regret, 91.7% coverage) outperforms quadratic (0.053 regret, 97.3% coverage) while fixed/learned monotone indices trail on coverage (66.0% / 74.4%) and regret (0.077 / 0.138).
  • Llama 3B standard-INT3 (219 unseen states): all predictors reach ≥99.3%\ge 99.3\% coverage (100.0% quadratic), with 0.0004 regret for quadratic vs. 0.043 exact addition and 0.060 learned monotone.

Two boundaries stay explicit in Appendix D: regret CIs are pointwise, and under a strict equal-KK returned-set control, no pairwise vs. additive coverage difference survives Bonferroni correction across the five tested KK sizes.

Generalizability & value of combination queries. On standard INT3 Llama-3.2-3B, the 8-move replacement lattice still resolves no sign reversals; we separate that from a fixed-edit scale control (applying the identical 4-projection scale update across two standard INT3 contexts), which resolves opposite NLL signs on WikiText (+0.0013+0.0013 vs. −0.0008-0.0008) and C4 (+0.0011+0.0011 vs. −0.0011-0.0011; Appendix L.2). Practically, rather than spending extra queries to beat cap-2 on an expensive beam, PTV redirects complete combination queries toward fast acceptance on cheap proposals—achieving +3.55 [1.68, 5.48] pp normalized ARC-Challenge accuracy over full-price cap-two at 12% of total forward work on 3B (Table 5), while Figure 4 transparently shows the settings (new panels, act-order 1B, Qwen3-1.7B) where task gains vanish.

Reviewer_NU34OpenReview ↗

Response to the rebuttal

Thank you for the detailed rebuttal and additional experiments. The new results help address my concerns regarding contextual reversals. However, the generalizability of these findings of the proposed approach remain somewhat limited, particularly given the inconsistent downstream improvements. I appreciate the authors' efforts but will maintain my original rating.

Reviewer_TRvcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper challenges the assumption that individual moves for quantization of LLMs can be composed in an additive manner. Specifically, the paper evidences the weaknesses of additive composition and proposes alternative mechanisms.Furthermore, it studies the influence of the loss function (e.g., reconstruction, next token NLL) on the quality of the composed moves.

Critical strengths

The analysis of the weaknesses of composing individual moves in an additive fashion is interesting.

The study of alternatives to this additive process is valuable.

The analysis of the influence of the loss function is also interesting.

Critical weaknesses

I must say that I truly struggled to understand the paper and, for the first time in my career, asked the help of an AI to summarize and clarify the paper. In its current state, I believe that the paper is not accessible to non-experts, by which I do not mean people who are knowledgeable in quantization but who are truly experts in the specific topic covered in the paper.

The discussion is also complicated by the fact that many concepts are mixed (loss function, composition strategy,…), which makes it hard for the reader to draw clear conclusions. Ultimately, after reading the paper, I fail to understand what strategy seems the most promising.

I do believe that there is valuable material in this paper, but I feel that, in its current state, it will fail to truly reach the audience.

Questions for authors

See Weaknesses above.

1 comment
AuthorsOpenReview ↗

Response to Reviewer TRvc

Your feedback on readability drove most of the structural rewrite. In the submitted draft, evaluation metrics, approximation diagnostics, and selection algorithms were interleaved across sections without stating clearly which strategy we recommend.

We reorganized the paper into four distinct pieces:

  • One workflow summary (Figure 1, §1): Visually contrasts full additive optimization (top) with propose-then-verify (PTV) (bottom), which is our single recommended training-free strategy: use cheap coarse singleton evaluations only to propose one candidate per support size, screen candidates in random order with complete NLL first, and accept the first feasible candidate that improves complete ARC option-KL (or keep the base if none passes).
  • Separated quantity roles (Table 1, §2): Explicitly fixes WikiText NLL as the feasibility constraint (≤0\le 0), ARC option-KL to a fixed teacher as the selection objective, and held-out ARC-Challenge accuracy strictly as the post-hoc readout (never used during selection).
  • Predictors vs. selectors (§§3–4 vs. §5): Sections 3–4 evaluate how well additive, calibrated, and pairwise predictors model unseen states; Section 5 and Algorithm 1 define and benchmark selectors under matched endpoint work. Dense derivations, earlier beam runs, and full lattice tables moved to Appendices A–P.
  • Clearer takeaway (Table 5, Figure 4, §§5–6): On healthy act-order 3B, PTV raises expected normalized ARC-Challenge accuracy from 35.61% to 39.51% at 12% of cap-two's measurement work (48.6 vs. 394 endpoint equivalents), while panel ablations and Qwen3-1.7B/1B runs show where the rule plateaus or yields null task gains. When gradient training is allowed, 200-step scale calibration (464 s) is kept clearly separated as a training baseline (39.09%).
Reviewer_wMgFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks what happens when you combine several weight-encoding "moves" in a quantized LLM, where a move swaps a group or projection to another encoding at the same bitrate (e.g. GPTQ to RTN). Subsets of nn moves form a Boolean lattice. Eq. 2 splits the error of an additive prediction into pricing error (bad single-move estimates) and composition error (single-move effects not adding up), measured on the same evaluation units. Main findings: (i) an own-midpoint gradient price predicts single-move effects much better than standard gradient or Gauss-Newton proxies (Prop. 1, Tables 2, 8, 9). (ii) On Llama-3.2-1B and 3B GPTQ bases, additively selected sets of 61-134 moves have composition residuals 26x and 42x the pricing error, and this persists with exact single-move prices (Table 1). (iii) On two exhaustive 256-state Qwen3-0.6B lattices, some moves flip sign depending on context, and pairwise models predict unseen combinations better than additive ones. A 3B GPTQ lattice repeats the prediction gain but shows no sign flips. (iv) In selection on larger banks, capping the number of changes removes most of the failure, and an exact-endpoint beam gives bank-dependent trade-offs.

Critical strengths
  1. The same-unit split of pricing vs composition error is clean and convincing. Exact single-move prices leave almost the same residual (+1.268 vs +1.257 ARC-KL at 3B, Table 1), so the failure isn't a gradient approximation issue. The KL ≥ 0 argument (base 0.3695 plus price sum -1.304 is impossible) makes the point with no modeling assumptions. This is the most reusable result in the paper.

  2. The sign-flip test is done properly: simultaneous max-t intervals over all 2,816 contrasts, Bonferroni across cells, and both contexts have to be resolved with opposite signs. The authors also report their nulls (3B lattice, Tables 14, 27, 33), which makes me trust the rest more.

Critical weaknesses
  1. The 26x and 42x numbers come from sets an optimizer picked, and the selection gains are small or limited to a weak base. Those residuals are on the set minimizing summed ARC prices over 224 or 392 options. An optimizer exploiting an additive model will pick moves whose effects overlap, and with a bounded loss you'd expect saturation (many moves claiming the same reducible KL). So this is the worst case, not typical composition error. The denominator is also the near-zero midpoint pricing error, which inflates the ratio. Residuals on random subsets of the same size would show how general this is. On selection, the clearest gain (19% perplexity drop with the 2-change beam, Table 4) is on Llama-3.2-1B GPTQ without act-order, a weak base whose 3B version collapses (App. F.6). On act-order bases the beam is unresolved at 1B (Table 28), -0.003 NLL at 3B, and no better than cap-two addition (Table 33). What survives is basically "change few projections". And full pair information costs 1+n+(n2)1+n+\binom{n}{2} queries (~77k for the 3B bank), so it's not clear any of this helps in practice. This is the main reason for my score.

  2. The structural evidence is narrow and a lot of data is reused. The sign flips come from two 8-move Qwen3-0.6B lattices that share five moves, in a custom 2.5-bit format. The only standard GPTQ lattice shows none (App. D.1). Table 7 shows many comparisons reuse units that were already looked at, and several analyses are post hoc. The authors disclose this, which I appreciate, but with this many banks, metrics and tolerances there's a real forking-paths concern that per-comparison intervals don't cover. I can't tell which conclusions would hold on a new model.

Questions for authors
  1. [Clarification] What is the composition residual (exact single-move prices) for random subsets of the 1B and 3B banks at sizes 8 to 128, compared to the selected sets in Table 1? If random subsets add up nearly fine, the story is "additive selection exploits saturation" rather than "effects don't compose".
  2. [Clarification] How much of the residual does a simple saturating transform of the summed prices absorb (fit on random subsets, test on the selected sets)?
  3. [Clarification] Can you rerun the Table 4 comparisons on fresh units for the no-act-order 1B bank, like you did for 3B in App. F.7?
  4. [Future work] Is there a sub-quadratic way to get pair information for banks with hundreds of moves?
  5. [Clarification] Do the conclusions hold on a stronger base, e.g. act-order GPTQ at 1B or GuidedQuant?
3 comments
AuthorsOpenReview ↗

Response to Reviewer wMgF

Thank you for the concrete experimental questions—especially distinguishing typical combination behavior from optimizer-exploited extremes.

Q1. Random subsets (sizes 8–128) vs. optimizer-selected sets. Your hypothesis is right: additive optimization concentrates composition error rather than every combination failing equally. Across 448 random states on each of two healthy activation-ordered INT3/g128 banks (32 TRAIN / 32 TEST per support; supports {8, 16, 32, 64, 128, 132, 134} on 3B and {8, 16, 32, 45, 47, 64, 96} on 1B), every selected ARC residual exceeds all 32 support-matched TEST controls (exact-singleton prices throughout):

Bank & price sourceMovesSelected ARC residualRandom mean abs. residualRatio
3B, midpoint1321.2570.089614.0x
3B, exact singleton1341.2680.073517.2x
Healthy 1B, exact singleton450.4440.06307.04x
Healthy 1B, coarse endpoint470.3760.05806.48x

Auditing 253 smaller additive proposals (supports 1–64) on the 3B bank against exact-support random controls (3/support) gives mean absolute residual ratios of 10.69x, 16.39x, 14.11x, and 14.35x across supports 1–8, 9–16, 17–32, and 33–64 (Figure 2a, Appendix C.1, Table 12; overlapping proposals in this retrospective census are correlated).

Q2. How much does a simple saturating transform absorb? Substantially: saturation accounts for most of the ARC residual magnitude (Table 3, Appendix C.2). Fitting solely on random TRAIN states and testing on the selected sets:

  • Nonnegative KL floor clipping: removes ~74% of selected ARC error on 3B (1.268 to 0.333; 1.257 to 0.324) and 49–57% on healthy 1B.
  • Exponential / monotone links: remove 83–92% on 3B (0.205–0.214 / 0.098–0.107) and 87–98% on healthy 1B (Table 15).
  • Nearby-price control: on 16 price-matched 3B states, proportional scaling, exponential, and local mean all predict selected ARC-KL within ~0.007 (Table 11).

Still, two decision issues persist (§3): strictly increasing scalar calibration preserving 0 leaves fixed-support additive rankings unchanged (Remark 1); and small NLL residuals break feasibility near the boundary—on healthy 1B exact selection, the NLL residual is only 0.240x the random mean, yet packing predicted NLL near zero (−0.00016-0.00016) yields +0.00502+0.00502 upon execution.

Q3. Fresh-unit rerun on the original non-act-order 1B bank. Regenerating that archived bank from scratch yielded base NLL 3.979 instead of archived 2.946. Rather than mixing mismatched checkpoints, we report its new states separately in Appendices C and E and run new controlled experiments on healthy activation-ordered 1B and 3B GPTQ bases.

Q4 & Q5. Stronger bases (act-order 1B, GuidedQuant), pair costs, and data reuse.

  • Act-order GPTQ 1B and 3B: Act-order 1B confirms the selection-concentrated residual (7.04x) while keeping a null ARC-Challenge readout. On act-order 3B (with policies, output conventions, and primary contrasts locked before the 1,407-item readout), PTV first-improvement achieves 39.51% expected length-normalized accuracy vs. 35.61% base (+3.91 [1.99, 5.86]) and 35.96% full-price cap-two (+3.55 [1.68, 5.48]), recovering 37% of the gap to FP (46.06%) at 12% of cap-two's work (48.6 vs. 394 endpoint equivalents; Table 5). Raw accuracy vs. base is +1.71 [−0.09-0.09, 3.52]; 200-step scale calibration (464 s) reaches 39.09%.
  • GuidedQuant: Not evaluated; left explicitly outside our claims (§6).
  • Sub-quadratic pair costs: Instead of reconstructing (n2)≈77k\binom{n}{2} \approx 77\text{k} pairs, PTV uses cheap singleton proposals and verifies complete NLL/ARC on 2.17 expected combination queries in the 3B pool.
  • Data reuse: §5 and Table 9 separate the locked primary 3B readout (Table 5) from subsequent panel/architecture checks on reused cohorts (+0.43 and +1.01 on new 3B panels with CIs crossing zero, and nulls on 1B and Qwen3-1.7B; Figure 4).

Thanks for the new experiments. Q1 and Q2 are answered, and I agree with the reframing: additive selection concentrates error where the model saturates.

There are two remaining issues however:

  1. Q3: rebuilding the original 1B bank gave base NLL 3.979 vs 2.946 archived. What caused this, and does any current result depend on archived states that can't be regenerated?

  2. The downstream gain doesn't replicate unfortunately. The locked 3B result looks good, but the new 3B panels are within noise, Qwen3-1.7B and 1B are null, raw accuracy is unresolved, and 200-step scale calibration is within 0.4 points. PIQA and HellaSwag don't move.

I'm keeping my score.

AuthorsOpenReview ↗

Follow-up response to Reviewer wMgF

Thank you for confirming Q1 and Q2, and for the reframing, which the revision adopts.

Q3: cause of the 1B baseline gap. We have localized the gap but not identified its root cause. The original candidate tensors of that bank have not been recovered despite a search of our archives. The rebuilt bank (base NLL 3.979) is therefore a different bank, and we now report it as such.

  • Unchanged inputs. The floating-point model and evaluation data are the same: the archived ARC teacher and the current floating-point model agree to KL 1.4×10−91.4\times10^{-9} over all 592 items. The rebuild reused the original 128×2048 calibration tokens.
  • Unchanged code. Between the producing commit and the rebuild, the integer-encoding and scale kernels are unchanged, and we found no change that alters the non-act-order GPTQ computation.
  • Where the gap sits. It concentrates in the first MLP. The 14 projections of the first two layers carry 74% of the absolute change in singleton NLL prices. Replacing a single layer-1 MLP encoding (gate→GPTQ-0.1% or down→RTN) brings the rebuilt state within about 0.03 NLL of its archived counterpart, against 1.03 between the two bases.

Without the original tensors we cannot compare weights or separate numerical-environment from calibration-activation effects. This sensitivity is consistent with the fragility of non-act-order GPTQ that you noted for 3B (Appendix C, "The earlier non-act-order reconstruction").

Dependence on the unrecovered bank. Yes. In the previous version, four main-text items used it. In the uploaded revision:

  • Table 2's 1B rows come from the healthy act-order 1B bank (coarse / exact prices: 47 / 45 changes, composition error 0.376 / 0.444 ARC-KL; zero pricing error with exact prices).
  • The NLL–KL disagreement in §2 is 81/224 (act-order 1B) and 183/392 (act-order 3B).
  • The 1B midpoint comparison in §3 uses the act-order 1B bank (223/224 and 222/224 correct signs, vs. 50 and 179 for the current gradient).
  • The motivating reconstruction example in §1 remains, explicitly labeled as coming from the earlier bank.

All archived-bank results are grouped in a labeled appendix (F.1) as measurements that cannot be re-evaluated. For the banks behind the main conclusions:

  • Act-order 3B: the candidate tensors are retained, and an earlier restoration replayed base and combined states to within 10−610^{-6} NLL.
  • Healthy act-order 1B: all 451 measured states are reconstructed exactly from retained raw evaluation units.

Downstream gains. We agree that the task gain does not replicate stably across settings. The 3B result is a locked readout on retained candidates and per-item records. However, the two new 3B panels are within noise, Qwen3-1.7B and act-order 1B show no gain, raw accuracy is unresolved, PIQA/HellaSwag do not move, and 200-step scale calibration is within 0.4 points.

The revised abstract, introduction, limitations and conclusion now state this scope directly. Propose-then-verify is a forward-only composition procedure in which every accepted state improves the declared selection objective. Whether that becomes a task gain depends on candidate quality and on how well option-KL tracks the task. Stable downstream gains are stated as open rather than claimed.

We keep three things:

  • the algorithm;
  • the locked 3B result with its cost relative to full-price cap-two (12% of its endpoint work);
  • the decision-level finding on the declared objective: full singleton pricing spends 393 endpoint equivalents yet classifies 28 of the 29 truly feasible proposals as infeasible.

The practical guidance now presents short scale calibration as a strong alternative when gradients are available.

Thank you again for the careful reading.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary of Manuscript Revision

We have uploaded a revised 10-page manuscript addressing the questions raised across all three reviews. Keeping our fixed-bitrate move-composition setup and same-data error decomposition intact, we restructured the main text for clarity and added targeted experiments testing each reviewer's hypotheses:

  1. Restructured presentation and single recommended workflow (Reviewer TRvc):

    • Figure 1 (§1) visually contrasts standard additive selection with our recommended training-free procedure, propose-then-verify (PTV), right at the start of the paper.
    • Table 1 (§2) separates the three measured quantities—feasibility constraint (WikiText NLL), selection objective (ARC option-KL), and held-out readout (ARC-Challenge accuracy)—and separates predictors (§§3–4) from selectors (§5, Algorithm 1), moving dense derivations and earlier runs to the appendices.
  2. Random vs. optimizer-selected subsets and saturation (Reviewer wMgF):

    • Across 448 random states on each of two healthy activation-ordered INT3 banks (Llama-3.2-1B and 3B), optimizer-selected ARC composition residuals are 6.5x–17.2x the mean absolute residual of support-matched random controls; each of the four selected states has a residual exceeding all 32 corresponding held-out controls (Figure 2b, Appendix C). A census of 253 smaller additive 3B proposals across supports 1–64 shows the same concentration (10.7x–16.4x; Figure 2a, Table 12).
    • Nonnegative floor clipping removes 49–74% of selected ARC error, and exponential/monotone links fitted solely on random training states absorb 83–98% (Table 3, Table 15). Still, any strictly increasing calibration preserving the zero threshold leaves fixed-support additive choices unchanged (Remark 1), and small NLL residuals push boundary-packed proposals into infeasibility.
  3. Decision value of contextual reversals and monotone baselines (Reviewer NU34):

    • Table 4 (§4) compares exact/fitted addition, fixed- and learned-direction monotone single-index models, and pair-quadratic models on identical small-support observations using both one-SE Pareto coverage and standardized grid regret. Pairwise terms cut regret tenfold on Stratified Qwen (0.060 to 0.006; coverage 99.9% vs. 55–81%), whereas Witness Qwen and equal-KK candidate-size controls (Appendix D) show where pair estimates do not help.
    • Across all 3,394 four- to eight-dimensional sub-lattices of both Qwen grids (Appendix D.1, Table 16), quadratic selection improves coverage roughly 3x more often when a resolved ARC sign reversal is present on Stratified (45.7% vs. 13.6% at dimension 4).
  4. Stronger-base construction and scope boundaries (Reviewers wMgF, NU34):

    • On the healthy activation-ordered INT3 Llama-3.2-3B base (with policies and contrasts locked prior to the 1,407-item held-out ARC-Challenge readout; Table 5, §5), PTV first-improvement raises expected length-normalized accuracy from 35.61% to 39.51% (recovering 37% of the gap to FP 46.06%), beating full-price two-change selection (+3.55 pp) at 12% of its measurement work (48.6 vs. 394 endpoint equivalents), and matching a 200-step scale-calibration reference (39.09%, 464 s).
    • Figure 4 and §5 also report raw-likelihood results and controlled ablations across new coarse panels, tasks (PIQA/HellaSwag), and model banks (Qwen3-1.7B, act-order Llama-3.2-1B), separating the primary 3B readout from subsequent cohort-reuse analyses and delimiting where candidate scarcity or surrogate mismatch limits task transfer.

Point-by-point responses follow separately under each review.

Area_Chair_RAfbOpenReview ↗

Please check author responses

Dear reviewers,

We’re about halfway through the discussion period. This is a friendly reminder to check other's reviews and the author responses. If the authors have answered questions or responded to critiques in reviews, please leave a comment for the authors!

Best, AC

AuthorsOpenReview ↗

Update to the Revised Manuscript

We have uploaded an updated manuscript following Reviewer wMgF’s follow-up. This revision makes three focused changes:

  1. Healthy-bank evidence in the main text. Table 2 now uses the healthy activation-ordered Llama-3.2-1B bank alongside the activation-ordered 3B bank. The 1B loss-disagreement and midpoint-pricing comparisons also use the healthy bank. Measurements from the earlier non-act-order 1B bank are collected and explicitly labeled in Appendix F.1.

  2. Reconstruction diagnostics and provenance. Appendix C documents the original 1B reconstruction discrepancy (base NLL 3.979 versus archived 2.946), the checks on the floating-point model, evaluation data and quantization code, and the concentration of the discrepancy in early MLP projections. It also states that the original candidate tensors have not been recovered.

  3. Task results and practical guidance. The abstract, introduction and conclusion now distinguish the locked 3B result from the subsequent panel, architecture and task checks. We retain the locked improvement from 35.61% to 39.51% expected length-normalized ARC-Challenge accuracy and its measurement cost, while explicitly reporting that this task gain does not repeat stably across settings. The practical guidance also identifies short scale calibration as a strong alternative when gradients are available.

The revision preserves the paper’s central findings on selection-concentrated composition error, saturation, and the decision value of contextual measurements.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.