- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces CurveFP, a block-scaled logarithmic family of formats that computes products using integer-index operations and organizes accumulation over a finite set of phases determined by a rational radix. The paper proposes 8-bit formats for training (i.e., CurveFP8 E4C3/E5C2) and a 7-bit format for inference (i.e., CurveFP7 E3C3). The results support FP8 training parity at small model scale and comparable inference perplexity when CurveFP7 uses finer-grained scaling than FP8. A preliminary hardware comparison reports a modest tile-area reduction, but the practical advantage over existing formats remains unclear.
Critical strengths
- The core idea is interesting, and the encoding enables structured arithmetic.
- The explanation is dense and sometimes difficult to parse, but it captures the proposed family and explains several design trade-offs.
- The evaluation covers both training and inference with an FP8 comparison.
- The proposed format is accompanied by a hardware implementation and analysis; the authors acknowledge the limited scope of the hardware evaluation.
Critical weaknesses
- The added value over existing formats is not sufficiently established. The training results support parity with FP8, while the 7-bit inference results offer a modest storage reduction. These benefits need stronger context against microscaling formats such as MXFP8. Section 4.1 lists microscaling baselines, but no explicitly identified MX results are reported.
- The evaluation of the four 7B–9B models is limited to WikiText-2 perplexity (Figure 2). No zero-shot task accuracy or generative evaluation is reported for these models. The downstream experiments in Section 4.6 concern the much smaller pretrained models.
- Figure 2a changes scale granularity, element width, and data format simultaneously. It demonstrates a useful operating point but does not isolate the contribution of the proposed element format. Figure 2b varies scale granularity only for CurveFP7 and INT8 on Falcon-H1, leaving the corresponding FP8 comparison incomplete. The panels therefore indicate that tensorwise FP8 substantially outperforms tensorwise CurveFP7 on Falcon-H1. The repeated claim of “FP8-class quality at 7 bits” should consistently specify the block-scaled setting and scope the claims accordingly.
- Table 3 reports aggregate NMSE ratios without showing variation across the sampled windows or layers. The reported wins are encouraging, but per-window results would clarify how much the improvement varies. I assume stands for normalized mean squared error; acronyms should be defined at first use.
- The training runs in Section 4.5 are promising but use only 128.3M-parameter Llama-style models. The larger-model GEMM diagnostics do not establish training convergence or downstream parity at larger scale.
- The hardware comparison has limited practical reach. It compares two custom tiles with a common fixed-point accumulator and excludes scale handling, memory, etc. Peak throughput is unchanged, and the reported energy per MAC is slightly higher for CurveFP.
- The numerical evaluation covers CurveFP7 and the CurveFP8 E4C3/E5C2, whereas the hardware evaluation covers only CurveFP8 E4C3 against FP8 E4M3FN. It therefore does not establish the hardware trade-offs for 7-bit inference or backward-gradient arithmetic, or how costs scale with operand width, phase count, and dot-product length.
- The path from the proposed accelerator tile to a practical deployment remains unclear. The paper does not demonstrate acceleration on existing hardware or establish a system-level benefit sufficient to motivate dedicated hardware support.
- The contribution beyond existing logarithmic representations needs sharper explanation. Section 2 delineates CurveFP from existing logarithmic representations, but the paper does not clearly establish which additional design choices provide the decisive benefit for language models in particular.
- Some positioning statements need stronger support or qualification: (1) the introduction states that uniform integers increasingly require fine-grained scales as activation range widens, without a direct supporting citation or analysis, and (2) Section 2 attributes FP8-regime quality to the proposed structure, but the experiments do not isolate the effects of the element codebook, scale selection, and scale granularity.
Questions for authors
- What specific language model requirement motivates this representation, and which design choices address that requirement?
- Can the authors provide explicit MXFP8 comparisons and separate the effects of the element codebook, scaling granularity, and scale-selection method?
- Can one hardware implementation support different rational radices (p,q)? What additional resources (if any) are needed when the phase count changes?
- Do the authors have hardware results or analytical cost models for additional FP and CurveFP formats, including 7-bit inference and backward-gradient arithmetic? How do area and timing scale with operand width and dot-product length?
- Appendix C.1 applies rounded Q16 phase coefficients before accumulating into a common Q32 register, rather than retaining separate integer phase sums as in Section 3.2. What are the cost and numerical implications of this choice, and do the model evaluations emulate this arithmetic?
- Which deployment target is the proposed tile intended to represent, and what evidence indicates that its benefit survives scale handling, conversion, and integration into an accelerator?
- How does this work fit with the growing body of 4-bit training and inference research, and what trade-off motivates the 7- and 8-bit operating points?
5 comments
Zero-shot on the 7B–9B models (App. D.3). You are right that the submission reported only perplexity for these models. We added six tasks (ARC-E/C, HellaSwag, PIQA, WinoGrande, LAMBADA) on all four: CurveFP7 G64 changes mean accuracy by −0.0034, −0.0023, −0.0000 and −0.0025 (Llama-3, Qwen3, Qwen3.5, Falcon-H1). No task difference is significant (max |z| = 0.68), comparable to tensorwise FP8 (max |z| ≤ 0.91). We did not add free-form generation benchmarks in the rebuttal period; LAMBADA (last-word prediction) is the closest task in the suite.
Variation behind the training-GEMM table (Sec. 4.4, App. D.7). NMSE (normalized mean squared error) is now defined at first use. Across the 36 window comparisons the CurveFP8/FP8 ratio ranges from 0.866 to 0.940, so CurveFP8 wins every window (per-window table in App. D.7). Per layer role, CurveFP8 is lower in 96/96 Pythia-410M role × GEMM × window comparisons (ratios 0.80–0.98) and in 77/84 for Llama-3.2-3B; all seven exceptions are gradient GEMMs, six of them in one window.
Training scale (Sec. 4.5, App. D.5). We trained matched FP8 and CurveFP8 models at 0.32B and 1B parameters. Entries: BF16-inference held-out perplexity / format penalty (training-format minus BF16-inference perplexity of the same checkpoint).
| Model, tokens | FP8 | CurveFP8 |
|---|---|---|
| 128M, 3B (mean of 3 seeds) | 22.5407 / 0.0758 | 22.5366 / 0.0688 |
| 0.32B, 3B | 19.9927 / 0.0512 | 19.9914 / 0.0459 |
| 1.0B, 1B, seed 1 | 24.2539 / 0.0304 | 24.2822 / 0.0238 |
| 1.0B, 1B, seed 2 | 24.2201 / 0.0276 | 24.2023 / 0.0242 |
- No run is unstable, and quality stays at parity at every size (the downstream suite of Sec. 4.6 was not rerun at these sizes).
- Beyond parity, CurveFP8's format penalty is lower in all 10 matched training pairs (three sizes, three learning rates, up to three seeds), by 6–22%; it also stays lower against FP8 retrained with CurveFP8's exact scale recipe. Details are in our reply to UCLW.
Q1. Which language-model requirement, and which design choices address it.
- LLM weights and activations are heavy-tailed with wide dynamic range. Log-spaced levels give constant relative error, which is why CurveFP8 lowers weight and activation NMSE relative to E4M3 by 8–17% at equal width.
- Range must be local. Compact power-of-two block scales supply it at 0.06–0.08 bits per value, which is what makes the 7-bit point possible.
- Training needs a precision-oriented and a range-oriented role, hence E4C3/E5C2, mirroring E4M3/E5M2.
- The uniform curve index makes products closed, and the rational radix prices any other level density through the phase law.
Relative to earlier logarithmic formats (Sec. 2, App. D.8), the additions are the rational radix with its phase law, the compact power-of-two scale contract, and the FP8 role formats trained with standard AdamW.
Q7. Relation to 4-bit work. Our operating points target FP8-class training and calibration-free 7/8-bit inference with a single scalar algebra: 8 bits is the FP8 training width used in practice, and the 7-bit point removes one bit while staying within 1.32% of native without calibration. W4A4 methods rely on rotations, outlier migration or learned codebooks with calibration (QuaRot, SpinQuant, LO-BCQ, NestQuant; Sec. 2) and are complementary. We did not evaluate below 7 bits.
Q6. Deployment target and scope of the hardware study. The tile represents the multiply–accumulate array of a tensor-core-like matrix unit, the block an accelerator replicates many times. CurveFP is a numerical-format contribution whose primary evidence is model quality, so the tile is deliberately small: it demonstrates that the closed product maps to verified, timing-closed logic and that its arithmetic is the arithmetic evaluated in the models. It is far from an accelerator. We have no evidence yet on scale handling, conversion or integration and claim none; Sec. 5 leaves system-level efficiency to future work, and the paper claims area and product-stage depth only. You are right that peak throughput is equal and energy is on par (4.78 vs 4.75 pJ/MAC). On existing hardware the benefit is storage (7.06 bits per value at FP8-class quality, 12–14% below FP8 and MXFP8); the model-quality results do not depend on dedicated hardware.
Q5. Q16 coefficients vs. separate phase sums. Numerics: with accumulation that does not round, the two are identical by linearity (), and Q16 coefficients have relative error below . The model evaluations now emulate this arithmetic (Q16 coefficients, no intermediate rounding, one output rounding): perplexity is at the reported values in all four tested pairs, e.g. Qwen3 CurveFP7 7.0538 vs 7.0581 (App. D.4). Cost: the per-product form keeps one accumulator per output, whereas retaining phase sums needs one per phase () plus a final recombination. We measured only the former.
Q3. Other radices on one implementation. The datapath structure is the same for any , fixed at design time. Two parts change: the phase index becomes , a constant multiply and modulo on the rank sum instead of bit slicing, and the coefficient table grows from 8 to entries (9 for , 32 for ). We have not synthesized non-binary radices; the radix table motivates the 8-phase binary point as the default.
Q4. Other formats; scaling with width and dot length. We do not have routed tiles for 7-bit inference or backward GEMMs, and a credible physical-design study does not fit the rebuttal period, so we state what the format definitions imply. Product formation is one rank addition, a sign XOR and zero detection for every CurveFP format; it grows linearly with element width, and there is no significand multiplier to scale. E3C3 keeps the 8 phase coefficients and halves the exponent range of a product (16 vs 32 binades), which narrows the alignment shift. In backward GEMMs an E5C2 gradient multiplies an E4C3 operand; the product stays on the 8-phase grid and its exponent range widens to 48 binades, the same kind of widening that E5M2 brings to FP8. For every format the accumulator grows by one bit per doubling of the dot length. Routed area and timing for these points are future work.
Positioning statements and presentation. (1) The integer claim is now supported in place by Figure 2b: on Falcon-H1, INT8 degrades from 5.653 (G128) to 10.10 (tensorwise) while FP8 stays at 5.645–5.666. (2) Sec. 2 now says "with block scales" and points to the isolation in Sec. 4.2. (3) To ease the dense exposition you noted, Appendix A now has a worked product-and-accumulation example.
We hope the new isolation, task and training results address your main concerns, and we are glad to clarify further.
Thank you for the thorough review. Your central concern is the added value over existing formats, and several of your points ask for experiments that isolate it. We ran them; they are in the revised PDF (blue). This comment covers MXFP8 and the isolation (your question 2); the second covers task evaluation, variation and training scale; the third covers motivation, positioning and hardware (questions 1 and 3–7).
In short
Q2. Explicit MXFP8, and codebook, granularity and scale selection separated (Sec. 4.2, App. D.1). WikiText-2 perplexity; bits per value in parentheses. MXFP8 uses E4M3 elements with one E8M0 scale per 32 values, chosen by the OCP floor rule or by search. The three G64 columns share one power-of-two block-scale contract and selection rule, so only the element format differs.
Figure 2 and the setting of the 7-bit claim (App. D.2). We agree that Figure 2a alone does not isolate the element format; the table above does. For Figure 2b, a new table in App. D.2 completes the Falcon-H1 sweep with FP8 E4M3 and CurveFP8 at every granularity: both 8-bit formats are nearly granularity-invariant there, while CurveFP7's tensorwise collapse is a range effect (E3C3 spans 7.75 binades), which is why the format is specified with block scales. As you suggest, the revision says "block-scaled" consistently (Figure 2 caption, Secs. 2 and 4.2, conclusion) and states that the FP8 baselines of Figure 2a use standard FP32 absmax scales.