- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper asks whether the one-step update error of a low-bit Adam state representation can be used to choose among representations before training. It finds that the error ranking stays stable across settings, but the training quality and stability of the chosen representation do not. The authors conclude that a stable fidelity ranking is not a reliable basis for selection.
Critical strengths
The paper addresses a practical question: how to choose among low-bit optimizer state representations without training each one to the full budget.
The experimental protocol is careful. The selected representation is fixed before training, all candidates are trained with the same seeds and data order, and the decision thresholds are set in advance.
The authors limit their claims to what was tested and openly report inconclusive results.
Critical weaknesses
The paper is hard to read. Many terms are used before they are defined, or are never clearly defined, for example "codec transitions", "matched units" and "fidelity ranking". "Matched units" appears in line 53, but "training unit" is only defined in line 111. Terms should be defined before they are used, so the reader can follow the paper on a first read.
The abstract is unclear, starting from the first sentence. In the first sentence, it is not clear what is being compared. In the second sentence, it is not clear what the "selector" selects. The abstract should use simple, direct language.
The paper uses different names for the same concept: "fidelity ranking" (line 26), "error ranking" (Figure 1a) and "error ordering" (lines 76 and 88). "Fidelity ranking" first appears in line 26 without a definition. The authors should choose one term, define it and use it throughout.
The presentation needs more care. For example, the values in the "range" column of Table 1 are not aligned. Overall, the paper feels rushed and does not meet the presentation standard expected at ICLR. It feels like the paper was written by AI, and the writing is not good.
Low-level details such as those in lines 106 to 109 (step counts, batch size, evaluation steps) belong in the appendix. The main body should be interesting to read and focus on the key ideas and insights.
Questions for authors
Almost all experiments use 3 bits, with only one small 4+4-bit test, and the main finding disappears at 4+4 bits. Why do the title and abstract not say this? The title, "Across Codebooks and Bit Widths", suggests a broader study of bit widths than the paper provides.
2 comments
Q1. The title and abstract do not say that the study is almost entirely at 3 bits
Agreed. The revised abstract states that the codec tests compare three 3-bit representations, that under the dynamic-exponent map the repeatable cost of the error's choice against the plain representation appeared with a 3-bit first moment and stayed below the preset threshold with a 4-bit one without the choice becoming the lowest-loss representation in every unit, and that the test at ten times the budget was inconclusive.
The title is unchanged. To test the bit-width axis in "Across Codebooks and Bit Widths" systematically alongside the 3+3-bit and 4+4-bit runs, the revision adds mixed-width tests (3+4 and 4+3 bits on the first set of units) and a second six-unit set at 4+4 bits, run under the protocol of the 3-bit test (six units per condition, selections locked before training, and the same-math controls and AdamW references recorded for these units in the 3-bit tests, which do not depend on the candidates' width). All eighteen units pass the control check, and the one-step error selected pre under every map at every width, as at 3+3 bits.
Dynamic-exponent map, learning rate 7e-4, 4,000 steps (Table 1; = late loss of the selected representation minus that of plain, range over six units; M/Unres./B = units with material loss / unresolved / below the preset cost threshold; m+v = bits of the first and second moment):
| Width (m+v) | Rank of selected pre | range (nats) | M/Unres./B | Verdict |
|---|---|---|---|---|
| 3+3, Set 1 (submission) | worst 6 | to | 6/0/0 | repeatable cost |
| 3+4, Set 1 (new) | middle 1, worst 5 | to | 6/0/0 | repeatable cost |
| 4+3, Set 1 (new) | best 5, middle 1 | to | 0/0/6 | below preset cost |
| 4+4, Set 1 (submission) | best 4, middle 2 | to | 0/0/6 | below preset cost |
| 4+4, Set 2 (new) | best 2, middle 4 | to | 0/0/6 | below preset cost |
With a 3-bit first moment the selected representation finished above plain in all six units (all above the preset margin) and had the highest loss in five of them. With a 4-bit first moment it finished below plain in all six, so the cost stayed below the threshold, but the rank mismatch did not vanish: at 4+3 post was lowest by 0.003 nats in one unit; at 4+4 post was lowest in four units of the second set (by 0.0003–0.004) and in two of the first (by 0.002). Under the uniform map the selected representation was the lowest-loss one in all six units at both mixed widths and in three of six at 4+4 on the second set (post in the other three, by 0.003–0.005). No run at these widths rose more than 1 nat above its post-warmup minimum.
In the tested conditions the repeatable cost of choosing by the one-step error therefore follows the first-moment width. The revision reports this first-moment bit-width boundary across the tested conditions and presents 4+4 bits and the mixed widths as boundary tests of the 3-bit result in the abstract, in Section 3 (the bit-width paragraph and the boundary rows of Table 1) and in Appendix B.5 (Table 10).
Thank you for the detailed, line-level list of presentation problems and for the question on scope. We made every change you list and added the bit-width tests that the title refers to. Page and line numbers below refer to the revised PDF.
W1–W5. Presentation