acceptodds
Under review as a conference paper at ICLR 2027

Fidelity Is Not Selection: Low-Bit Optimizer States Across Codebooks and Bit Widths

Abstract

Quantizing Adam’s persistent moment states reduces optimizer memory, but representations with the same nominal bit width can train differently, making exhaustive comparison expensive. A cheap selector is the error a representation introduces into the next Adam update. However, this score reads a reference state once, whereas training repeatedly applies the codec to states it has already encoded; whether its ranking remains valid for selection is unclear. We test this prospectively by locking each choice before candidate training across codebooks, bit widths, training working points, and an external low-bit optimizer. At a common 130M condition across 12 independent training units, the error retained the same ordering of our three representations, yet its choice was best by point estimate under a uniform map in all 12 units, intermediate under a floating-point map in all 12, and never best under a dynamic-exponent map, where it was worst in 11. In one paired 4+4-bit condition the ordering persisted and the mismatch did not recur. Similar fidelity values also accompanied sharply different stability outcomes at larger working points. In the external optimizer, the error rule beat the published default in all six units at the longer budget but missed the lowest-loss setting in five. Thus, the failure mode is not an unstable fidelity ranking, but a stable one whose training meaning changes across settings.

Primary area:
optimization
Keywords:
optimizer-state quantization, low-bit optimizers, update fidelity, selection validity, AdamW, training stability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Jn2BOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper asks whether the one-step update error of a low-bit Adam state representation can be used to choose among representations before training. It finds that the error ranking stays stable across settings, but the training quality and stability of the chosen representation do not. The authors conclude that a stable fidelity ranking is not a reliable basis for selection.

Critical strengths

The paper addresses a practical question: how to choose among low-bit optimizer state representations without training each one to the full budget.

The experimental protocol is careful. The selected representation is fixed before training, all candidates are trained with the same seeds and data order, and the decision thresholds are set in advance.

The authors limit their claims to what was tested and openly report inconclusive results.

Critical weaknesses

The paper is hard to read. Many terms are used before they are defined, or are never clearly defined, for example "codec transitions", "matched units" and "fidelity ranking". "Matched units" appears in line 53, but "training unit" is only defined in line 111. Terms should be defined before they are used, so the reader can follow the paper on a first read.

The abstract is unclear, starting from the first sentence. In the first sentence, it is not clear what is being compared. In the second sentence, it is not clear what the "selector" selects. The abstract should use simple, direct language.

The paper uses different names for the same concept: "fidelity ranking" (line 26), "error ranking" (Figure 1a) and "error ordering" (lines 76 and 88). "Fidelity ranking" first appears in line 26 without a definition. The authors should choose one term, define it and use it throughout.

The presentation needs more care. For example, the values in the "range" column of Table 1 are not aligned. Overall, the paper feels rushed and does not meet the presentation standard expected at ICLR. It feels like the paper was written by AI, and the writing is not good.

Low-level details such as those in lines 106 to 109 (step counts, batch size, evaluation steps) belong in the appendix. The main body should be interesting to read and focus on the key ideas and insights.

Questions for authors

Almost all experiments use 3 bits, with only one small 4+4-bit test, and the main finding disappears at 4+4 bits. Why do the title and abstract not say this? The title, "Across Codebooks and Bit Widths", suggests a broader study of bit widths than the paper provides.

2 comments

Thank you for the detailed, line-level list of presentation problems and for the question on scope. We made every change you list and added the bit-width tests that the title refers to. Page and line numbers below refer to the revised PDF.

W1–W5. Presentation

  • W1, terms used before they are defined. "Training unit" is defined where it first appears, in the third paragraph of the Introduction (page 2, lines 080–083; a unit fixes the model seed and data order, so every candidate of a unit trains under identical conditions), and again in Section 2 (page 3, lines 126–127); "codec transitions" is removed; the second paragraph of the Introduction (page 1, lines 045–053) defines the one-step update error and the error ranking (lower error is better; the rule selects the lowest-error candidate) before either is used.
  • W2, the abstract. Rewritten (page 1, lines 011–030). The first sentences say what is compared (three 3-bit representations of Adam's moments under three first-moment codebooks), what is selected (the lowest one-step update error on a reference state) and how the test is run (the choice is locked before any candidate trains, then every candidate trains on the same units); the scope (3-bit states, 130M, the boundary tests) is stated in the abstract itself.
  • W3, one concept under several names. "Fidelity ranking", "fidelity ordering", "error ordering", "numerical ranking" and "one-step fidelity" are replaced throughout by one-step update error for the quantity and error ranking for the order it induces; "fidelity" now refers only to the general notion (the title, the Introduction sentence that introduces the measure, and Related Work where a cited paper uses it).
  • W4, Table 1 (page 5, lines 217–243). The range column uses one format in every row (a signed three-decimal range, "−0.095-0.095 to −0.044-0.044"), non-endpoints are marked uniformly, and the table gains boundary rows (4+4 bits on two unit sets, 3+4 and 4+3 bits, 40,000 steps) so that the main text's scope statements can be read off one table.
  • W5, training details in Section 2 (page 3, lines 121–125). The recipe (warmup, schedule, batch, weight decay, kernels, the held-out check) moved to Appendix A.2; Section 2 keeps one sentence with a pointer.

Q1. The title and abstract do not say that the study is almost entirely at 3 bits

Agreed. The revised abstract states that the codec tests compare three 3-bit representations, that under the dynamic-exponent map the repeatable cost of the error's choice against the plain representation appeared with a 3-bit first moment and stayed below the preset threshold with a 4-bit one without the choice becoming the lowest-loss representation in every unit, and that the test at ten times the budget was inconclusive.

The title is unchanged. To test the bit-width axis in "Across Codebooks and Bit Widths" systematically alongside the 3+3-bit and 4+4-bit runs, the revision adds mixed-width tests (3+4 and 4+3 bits on the first set of units) and a second six-unit set at 4+4 bits, run under the protocol of the 3-bit test (six units per condition, selections locked before training, and the same-math controls and AdamW references recorded for these units in the 3-bit tests, which do not depend on the candidates' width). All eighteen units pass the control check, and the one-step error selected pre under every map at every width, as at 3+3 bits.

Dynamic-exponent map, learning rate 7e-4, 4,000 steps (Table 1; Δj\Delta_j = late loss of the selected representation minus that of plain, range over six units; M/Unres./B = units with material loss / unresolved / below the preset cost threshold; m+v = bits of the first and second moment):

Width (m+v)Rank of selected preΔj\Delta_j range (nats)M/Unres./BVerdict
3+3, Set 1 (submission)worst 6+0.020+0.020 to +0.046+0.0466/0/0repeatable cost
3+4, Set 1 (new)middle 1, worst 5+0.015+0.015 to +0.034+0.0346/0/0repeatable cost
4+3, Set 1 (new)best 5, middle 1−0.019-0.019 to −0.009-0.0090/0/6below preset cost
4+4, Set 1 (submission)best 4, middle 2−0.021-0.021 to −0.008-0.0080/0/6below preset cost
4+4, Set 2 (new)best 2, middle 4−0.021-0.021 to −0.009-0.0090/0/6below preset cost

With a 3-bit first moment the selected representation finished above plain in all six units (all above the preset margin) and had the highest loss in five of them. With a 4-bit first moment it finished below plain in all six, so the cost stayed below the threshold, but the rank mismatch did not vanish: at 4+3 post was lowest by 0.003 nats in one unit; at 4+4 post was lowest in four units of the second set (by 0.0003–0.004) and in two of the first (by 0.002). Under the uniform map the selected representation was the lowest-loss one in all six units at both mixed widths and in three of six at 4+4 on the second set (post in the other three, by 0.003–0.005). No run at these widths rose more than 1 nat above its post-warmup minimum.

In the tested conditions the repeatable cost of choosing by the one-step error therefore follows the first-moment width. The revision reports this first-moment bit-width boundary across the tested conditions and presents 4+4 bits and the mixed widths as boundary tests of the 3-bit result in the abstract, in Section 3 (the bit-width paragraph and the boundary rows of Table 1) and in Appendix B.5 (Table 10).

Reviewer_Vu4YOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether a cheap one-step Adam update error can select low-bit optimizer-state representations before training, motivated by the mismatch between scoring candidates on a single fixed reference state and the fact that real training repeatedly re-encodes evolving optimizer states. The paper runs controlled experiments that compare numerical transfer (whether error rankings stay consistent) versus decision transfer (whether those rankings predict training quality and stability). It finds that while one-step error ordering can be relatively stable across many settings, the selected representation’s training rank and stability depend strongly on factors such as the codebook and scaling rule, and similar fidelities can still lead to different instability outcomes. Consequently, one-step fidelity behaves more like a diagnostic than a reliable standalone selector.

Critical strengths

Once the bit width is fixed, low-bit optimizers still involve many other design choices, such as the codebook, scaling rules, and quantization variables. These choices can lead to noticeably different training outcomes. Therefore, an important problem is how to select an appropriate representation of the optimizer states without running full training.

Critical weaknesses
  1. The paper’s experimental conclusions do not provide actionable guidance on how to choose among low-bit optimizer-state representations. Can the paper leverages architectural or optimization characteristics of Transformers, or offer theory-driven insights, to provide more specific recommendations?

  2. The current work mainly offers an empirical study under common quantization settings. It would be stronger if, based on these analyses, the paper could propose improved selection methods or practical guidelines rather than stopping at descriptive results.

  3. The experiments focus on existing standard quantization approaches. It would be valuable to extend the evaluation to other classes of quantization methods, such as vector quantization, to better demonstrate the generality of the findings.

Questions for authors
  1. Since one-step error cannot reliably select among low-bit optimizer-state representations, could the paper propose a new selection method that is stable across settings?

  2. Building on the empirical analysis, can the paper develop a new quantization method that makes one-step error a stable criterion for choosing among low-bit optimizer-state representations?

  3. Can the work leverage architectural or optimization characteristics of Transformers to derive guidelines or rules that are more prescriptive for making selection decisions (rather than only reporting empirical behavior)?

2 comments

Thank you for pressing on what a practitioner should do with these results and on how far they generalize. The revision adds a Practical guidance paragraph with four conditional recommendations (Section 8), a cost table for every selection rule (Appendix E.1), new bit-width tests (Section 3) and a scale-rule measurement at 350M and 1B (Appendix D.3).

W1, Q3. Actionable guidance

The four recommendations of Section 8, in longer form:

  1. Do not carry a one-step error ranking across codebooks, bit widths or scale rules as a selection rule; after any such change, re-validate the choice by training. With 3-bit states at 4,000 steps the same ranking selected the lowest-loss representation under U (31 of 34 unit–learning-rate pairs), an intermediate one under F (12 of 12) and never the lowest-loss one under N (34 pairs). Under N the repeatable cost of its choice relative to the plain representation followed the first-moment width: it appeared with a 3-bit first moment (3+3 and 3+4 bits) and stayed below the preset threshold with a 4-bit one (4+3 bits, and 4+4 bits on two sets of units), where the selected representation was still not the lowest-loss one in every unit. This locates the change among the tested conditions. At ten times the budget (3+3 bits) the same test under N was inconclusive. Worked example. Within seven bits for the two moments, a 4-bit first moment with a 3-bit second moment trained better than the reverse for every representation in all six units, by 0.018–0.081 nats under N and 0.071–0.258 under U, and had the lower one-step error on every input (130M, learning rate 7e-4, 4,000 steps across 6 units and 3 representations; Section 3, Appendix B.5). The allocation of bits between the moments should be validated rather than assumed.
  2. For a workload trained repeatedly, fix the representation once on development units trained to the target budget, and require any per-run selector to beat that baseline at its own cost. The fixed choice stayed within 0.0045 nats of the best candidate at no per-unit cost (development 4.71 A100-hours, 0.79 per unit over the six confirmation units); the 1,000-step rule cost 0.69 A100-hours per unit (seven candidates and three controls to step 1,000) and showed no material gain over it; the error rule cost 0.03 A100-hours per unit and never selected the lowest-loss candidate (table in our reply to Reviewer EcxR, W4).
  3. For a one-off workload where no development baseline exists, use a short training pilot rather than the one-step error. The 1,000-step rule improved on the error rule beyond the threshold in four of six codec units, and in the external optimizer a short pilot selected the lowest-loss setting in all six units, against one for the error rule; when a development-fixed baseline is available, short training showed no material gain over it.
  4. Under the dynamic-exponent map with absolute-maximum scaling, check stability at the target working point before comparing quality. Every candidate became unstable at the two larger working points; the signed-maximum scale removed the reported instability (18 of 18 to 0 of 18 at 350M, 12 of 12 to 0 of 12 at 1B). Other configurations need their own check.

On Q3: on the optimization side, the new analyses point to a consistent pattern. The decomposition shows that the one-step update error is almost entirely directional (at least 98.5% of E2E^2 at 3+3 bits, 99.4% at 4+4), and the mixed-width tests show an asymmetry between Adam's moments: within the tested conditions, both the repeatable selection cost under N and the equal-budget quality gain (4+3 against 3+4 bits) track first-moment precision rather than second-moment precision (Section 3, Table 1, Appendix C.6). Because the training consequence of this asymmetry differed across codebooks and was inconclusive at the longer budget, Section 8 frames it as conditional empirical guidance rather than a closed-form architectural rule.

W2, Q1. An improved selection method, stable across settings

We tested two candidate selectors prospectively, the one-step error rule and a short-training rule, each against a choice fixed on development units and with its cost counted (cost table in our reply to Reviewer EcxR, W4). Neither established an advantage over the fixed choice in the codec tests. This comparison sets the empirical bar for any candidate selector: it must beat a representation fixed on development units once its own selection cost is counted (Section 6). The procedure our evidence supports is the validation-based one in recommendations 1–3.

Q2. A quantization method under which the one-step error is a stable criterion

The revision measures this at the working points where the instability occurred (Appendix D.3, Table 16). On the twelve 350M reference inputs (6 units × 2 probe steps), with the same state and the same next gradient, the signed scale lowered the error on the paper's primary coordinate set by 0.42–1.06% (0.34–1.01% at 130M); the order pre < post < plain and the error rule's choice stayed the same on all twelve, and the change was almost entirely in direction (norm ratio +0.003–0.005, cosine +0.0005–0.0007). The paired training runs went from 18 of 18 to 0 of 18 unstable. At 1B, where plain and pre were trained, the signed scale lowered the error by 0.32–0.33% for plain and 1.03–1.04% for pre, pre stayed below plain on all twelve inputs, and the paired runs went from 12 of 12 to 0 of 12 unstable. The absolute-scale values reproduce the previously recorded values at both working points. The error moved by about one percent while the stability outcome changed completely: at 350M and 1B the signed-maximum scale removed the measured instability directly, and training stability still has to be checked at the target working point rather than inferred from the one-step error on reference states.

W3. Other quantization classes

Our experiments test four scalar quantizer designs across two optimizer implementations: the uniform, floating-point and dynamic-exponent blockwise maps of our controlled codec family, and the quantile codebook with stochastic rounding of the external optimizer, run unmodified (Section 6, Appendix E.2). The findings are therefore not tied to a single scalar codec. Vector quantization is outside the empirical scope of this study (Section 8, Scope) and would need its own test under the same comparison against a development baseline.

Reviewer_EcxROpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper asks whether the one-step Adam update error introduced by a low-bit optimizer-state codec can serve as a prospective rule for choosing among representations of the same nominal bit width. It locks the lowest-error choice before any candidate is trained and then trains every candidate on matched, control-qualified units, covering three representations (plain, post, pre), three first-moment maps (uniform U, floating-point F, dynamic-exponent N), 3-bit and 4+4-bit states, 130M, 350M and 1B working points, and the external SOLO optimizer. The error ordering pre < post < plain is stable throughout, yet the selected pre codec was best under U (31 of 34 unit–learning-rate pairs), intermediate under F (12 of 12) and never best under N (34 of 34), and this mismatch did not recur at 4+4 bits. The paper also shows that signed-maximum scaling removes the instability of N runs at 350M and 1B while barely changing first-encoding error, and that in the codec tests neither the error rule nor a short-training rule beat a choice fixed on development units, although in SOLO the error rule beat the published default.

Critical strengths

The paper shows a rigorous prospective evaluation design. Every selection is locked before candidate training, units are paired and qualified by same-update controls against a preset threshold, locked, paired and post-hoc analyses are kept separate, and independent implementations recomputed every verdict and locked error. The paper has shown a great rounding space method (Zero-Inclusive Preconditioner-space Stochastic Rounding) for 4-bit optimizer innovations. This has provided an important step for further 4-bit low-precision training work.

Critical weaknesses

The decision evidence comes from one short-horizon regime. All codec selection verdicts come from 130M runs at only 0.244 tokens per parameter (4,000 steps × 32 × 256) with low-bit states simulated in fp32. The 1B selection test yields no verdict because every unit failed qualification, and at 350M every N candidate was unstable. The paper does not explain why the same ranking yields different training ranks across maps or why the scale rule changes stability: the registered extra-forgetting mechanism was not supported, and neither lower first-encoding error nor idempotence alone accounts for the stability change. The policy closest to the best candidate was a choice fixed on development units trained to the full target budget (regret ≤ 0.0045 nats in the codec tests), which is exactly the cost a cheap selector was meant to avoid, and no cheaper rule beat it (the 1,000-step rule showed no material gain).

Questions for authors

Can you repeat one core codec test (for example, set 1 at 7×10−4 under U, F and N) at a longer budget such as SOLO’s 40,000 steps (~2.44 tokens per parameter)? This would show whether the map-dependent training rank of the selected pre codec is a property of the codebooks or of the short-horizon regime. Can you decompose the one-step error into a magnitude (norm-ratio) and a direction (cosine) component and report how each relates to training rank? The uninvestigated finding that 4-bit post and pre beat exact AdamW by 0.024–0.038 nats in every unit suggests codecs may change the effective update size, which a relative ℓ2 error mixes with directional error; if one component tracked the decisions, the failure would lie in this aggregate rather than in one-step fidelity per se.

3 comments

Thank you for the close reading of the protocol and for proposing two concrete tests, a longer training budget and a magnitude–direction decomposition. We ran both; the revision also measures the scale rule at the larger working points and reports the cost of every selection rule.

W1, Q1. The same codec test at ten times the budget

We repeated the first-set test at 40,000 steps (2.44 tokens per parameter): the same six units, learning rate 7e-4, 3-bit states, the three representations under each of U, F and N, warmup scaled to 4,000 steps, and a 500M-token training cache of which the 4,000-step cache is a prefix. Each selection was locked before its candidates trained. All six units pass the same-math control check (worst deviation 0.0005–0.0023 nats against the 0.0082 threshold; control spread bjb_j 0.0001–0.0019), and the rule selected pre under every map in every unit, as at 4,000 steps.

MapRank of selected pre (6 units)Late loss of pre minus plain (nats)Verdictpre above AdamW (nats)
Ulowest in 6−0.103 to −0.069 (4,000 steps: −0.150 to −0.116)below preset cost0.049–0.073
Flowest in 5; 0.004 behind post in 1−0.082 to −0.061below preset cost0.016–0.030
Nlowest 1, middle 3, highest 2 (4,000 steps: highest in 6)+0.035, +0.090 (material loss); −0.015, −0.032, −1.441, −0.001inconclusive0.119–0.211

At 4,000 steps pre sat 0.05–0.07 nats above AdamW under both U and N. Under N, post had the lowest loss in five of six units (plain in all six at 4,000 steps); in the one unit where pre ranks first, plain's run did not settle (its seven late evaluation points span 1.31 nats, mean 5.09). All 18 runs under N rose more than 1 nat above their post-warmup minimum training loss, first between steps 4,645 and 12,497 and peaking 7–17 nats above it, and all returned to finite late losses; no run under U or F and no control run showed such a rise. The lowest loss among all nine candidates was an F representation in every unit. As in the 4,000-step tests, low-bit states are simulated in fp32, so the experiments test selection validity rather than realized memory savings (Section 8).

In the paper: Section 3 (paragraph "Ten times the budget" and the last row of Table 1) and Appendix B.6 (Tables 11 and 12); the abstract and the Scope paragraph state that the 130M codec test was read at both budgets.

W2. No verdict at 1B; every N candidate unstable at 350M

These two working points mark where the codec comparison stops being a quality comparison (Section 4, Appendix C.3–C.4). At 1B the selection test gives no quality verdict because in every unit at least one same-math control deviated from the AdamW reference by more than the 0.0082-nat bound inherited from 130M; at 350M every candidate under N became unstable, which turns the question into one of stability. Section 5 takes it up directly: replacing the absolute-maximum scale with the signed-maximum scale removed the reported instability in 18 of 18 paired runs at 350M and in 12 of 12 at 1B. The revised Scope paragraph states both limits in one place.

W3. Why the same ranking trains differently, and why the scale rule changes stability

The revision narrows both questions with three pieces of new evidence. Location. Under N the repeatable cost of the error's choice appears with a 3-bit first moment and stays below the preset threshold with a 4-bit one (Section 3). Components. Under N neither the magnitude nor the direction component selects the lowest-loss representation where the aggregate fails (Q2 below). Scale rule. The signed scale lowers the one-step error by 0.42–1.06% on the 350M reference states and 0.32–1.04% on the 1B ones without changing its order, while the paired runs go from 18 of 18 to 0 of 18 and from 12 of 12 to 0 of 12 unstable (Appendix D.3, Table 16). Section 5 rules out three explanations: extra forgetting of the first moment under repeated absolute-maximum encoding, a lower first-encoding error, and idempotence alone. The property of the scale rule that mediates the change remains open.

Q2. Decomposing the one-step error into magnitude and direction

Over one coordinate set, with r=∥u~∥/∥u∥r=\lVert\tilde u\rVert/\lVert u\rVert and θ\theta the angle between the codec's update u~\tilde u and the reference update uu,

E2=∥u~−u∥2∥u∥2=(r−1)2+2r (1−cos⁡θ),E^2=\frac{\lVert\tilde u-u\rVert^2}{\lVert u\rVert^2}=(r-1)^2+2r\,(1-\cos\theta),

which holds on every input (residual below 10−1110^{-11}). We computed it on the twelve reference inputs of the first set (6 units × 2 probe steps) at 3+3, 4+4 and both mixed widths; a second implementation, written independently from the raw captured states, reproduced every value (largest relative difference 3×10−113\times10^{-11}) and every selection.

The error is almost entirely directional. The magnitude term (r−1)2(r-1)^2 is at most 1.5% of E2E^2 at 3 bits and 0.6% at 4+4, and rr stays between 0.98 and 1.04 at 3 bits. The direction component makes the same selection as the aggregate on every input, at every width and under every map, so the aggregate does not hide a directional signal.

What each component selects (primary coordinate set; "lowest-loss" = inputs on which the selected representation had the lowest late loss under that map; mean gap to the lowest-loss representation in nats):

BitsMap∣r−1∣\lvert r-1\rvert selectsAggregate and direction: lowest-loss (gap)Magnitude: lowest-loss (gap)
3+3Upost 10, plain 212/12 (0)0/12 (0.033)
3+3Fpost 10, plain 1, pre 10/12 (0.014)10/12 (0.008)
3+3Npost 120/12 (0.036)0/12 (0.021)
3+4Upost 11, plain 112/12 (0)0/12 (0.024)
3+4Npre 10, plain 1, post 10/12 (0.027)1/12 (0.024)
4+3Upost 1212/12 (0)0/12 (0.007)
4+3Npre 9, post 310/12 (0.001)7/12 (0.002)
4+4Upost 128/12 (0.001)4/12 (0.004)
4+4Npost 6, pre 68/12 (0.001)6/12 (0.002)

The lowest-loss representation at 3+3 is pre under U, post under F and plain under N; direction selects pre and magnitude mostly post, so neither tracks the training outcome across maps. Apart from one input at 3+4 under N (1/12 against 0/12), the magnitude component selected the lowest-loss representation more often than the aggregate only under F at 3+3 (10/12, against its own 0/12 under U and N), which we note as a single-codebook post hoc observation. Under N with a 3-bit first moment neither component resolves the mismatch. The paper states its conclusions specifically for the relative ℓ2\ell_2 update error and its direction component.

The 4-bit runs below AdamW. The one-step error follows the first-moment width: pre under U has error 0.220–0.232 at 3+4 (0.221–0.233 at 3+3) and 0.097–0.102 at 4+3 (0.095–0.100 at 4+4). The runs below the AdamW reference follow the same width: with a 4-bit first moment every N representation finished below its reference (4+3: 0.006–0.031 nats; 4+4 on the second set: 0.014–0.045), and under U post and pre did (0.019–0.045) while plain stayed within 0.016; with a 3-bit first moment every representation finished above it (3+4 under N: 0.011–0.052). The norm ratio does not separate these cases: at 4+4 post and pre have rr 1.004–1.005 under U and 1.001–1.002 under N, closer to 1 than at 3 bits under U (1.023–1.028), and under N plain has rr 0.982–0.998 and post and pre 0.998–1.009 at every width. An inflated one-step norm therefore does not account for the runs below AdamW; since a norm ratio at one reference state does not describe the effective step over training, the cause remains open.

In the paper: Appendix C.6 (identity, Table 15, ranges of rr and cos⁡θ\cos\theta); the bit-width paragraph of Section 3 points to it, and the sentence that left the 4-bit effect "not investigated" is replaced there and in Appendix B.4.

W4. The fixed choice costs the target budget

We agree, and the revision reports that cost next to the regret. The table covers the six confirmation units (130M, 4,000 steps per run; A100-hours from the job accounting records). Full runs and the capture and scoring jobs are measured; a prefix cost is the prefix fraction of the full run's time, not a separately executed run.

RuleDevelopment cost, shared across unitsSelection cost per unitThen trains its choice (per unit)Regret (nats)Lowest-loss chosen
train every candidatenone2.34 A100-h (seven full runs)no, the winner is already trained06 of 6
one-step error rulenone0.03 A100-h, plus 0.05 CPU core-hone full run, 0.34 A100-h0.013–0.0380 of 6
1,000-step ruleshared with the fixed choice (its default)0.694 A100-h (0.586 for seven candidates, 0.108 for three controls)remaining 3,000 steps if continued, 0.24 A100-h0–0.0273 of 6
fixed development choice4.71 A100-h (14 full-budget runs on two units)noneone full run, 0.33 A100-h0–0.00453 of 6

The error rule needs the unit's AdamW reference up to the probe step, the state capture and the scoring (the CPU figure covers two implementations); the 1,000-step rule needs the first 1,000 steps of the seven candidates and of the unit's three controls, whose loss spread at the same steps decides whether the fixed choice is replaced; the fixed choice needs only the development runs.

Spread over the six confirmation units, the fixed choice's development cost is 0.79 A100-hours per unit. It is cheaper than training every candidate from the third unit on, and more expensive than the error rule unless reused on about 175 units (about 129 when each rule's final run is counted at its measured time). The 1,000-step rule adds 0.69 A100-hours per unit to the same development cost without a material gain in loss. The table therefore shows a cost–loss trade-off: the error rule is the cheapest and never selected the lowest-loss candidate; the fixed choice came closest to the best candidate among the rules cheaper than exhaustive training, at a development cost paid once per workload. For a one-off workload, short training is the cheapest tested rule that improved on the one-step error: the 1,000-step rule did so beyond the threshold in four of six codec units, and in the external optimizer a short pilot selected the lowest-loss setting in all six units, against one for the error rule. For a workload trained repeatedly, the development-fixed choice is the baseline to beat.

In the paper: Table 18 in Appendix E.1; Section 6 ("What each rule costs") and the Practical guidance paragraph of Section 8 state the development cost and the one-off case.

Comments on the paper

1 comment

We thank the reviewers for their careful reading. A revised manuscript is uploaded with this response. The revision preserves the submission's core research question and existing results while adding five empirical analyses and a rewritten presentation.

  1. Ten times the training budget (Reviewer EcxR). The first-set codec test was repeated at 40,000 steps, with each selection locked before its candidates trained. The one-step error selected the same representation under every codebook. Under the uniform and floating-point codebooks its choice had the lowest loss in six and five of six units; under the dynamic-exponent codebook the six-unit verdict is inconclusive, and every run under that codebook rose more than 1 nat above its post-warmup minimum before returning (Section 3, Table 1, Appendix B.6).
  2. Bit-width boundary (Reviewers Jn2B, EcxR, Vu4Y). Tests with a 3-bit first moment and a 4-bit second moment, the reverse, and 4+4 bits on a second set of units locate the repeatable cost of the error's choice: it appeared with a 3-bit first moment and stayed below the preset threshold with a 4-bit one. The abstract and Table 1 now present the study as a 3-bit result with these boundary tests (Section 3, Appendix B.5).
  3. Magnitude and direction components (Reviewer EcxR). The one-step error is decomposed exactly into a norm-ratio term and a cosine term. The direction component selects as the aggregate on every input; no component selects reliably across codebooks (Appendix C.6, Table 15).
  4. Cost of each selection rule and practical guidance (Reviewers EcxR, Vu4Y). A cost table reports each rule's development and per-unit cost next to its regret (Appendix E.1, Table 18), and Section 8 gains a Practical guidance paragraph with four conditional recommendations.
  5. Scale rule at the working points where stability changed (Reviewers EcxR, Vu4Y). On the 350M and 1B reference states the signed-maximum scale changes the one-step error by 0.32–1.06% and leaves its order unchanged, while the paired training runs go from 18 of 18 and 12 of 12 unstable to none (Appendix D.3, Table 16).
  6. Presentation (Reviewer Jn2B). The abstract is rewritten, terms are defined at first use, one name is used for the one-step update error and its ranking, Table 1 uses one range format, and the training recipe moved to Appendix A.2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.