acceptodds
Under review as a conference paper at ICLR 2027

CurveFP: Co-Designing Numerical Representation and Product Arithmetic for Language Models

Abstract

Low-precision formats usually optimize scalar fidelity while inheriting conventional product arithmetic. We introduce CurveFP, a block-scaled family that distributes magnitudes across interleaved logarithmic curves. Uniform curve indices make every nonzero product an exact sign and integer-index update, while a rational radix exposes the finite phase schedule required for accumulation. We instantiate the algebra as CurveFP8 E4C3/E5C2 for training and CurveFP7 E3C3 for compact inference. On four 7B-9B models, CurveFP7 beats tensorwise FP8 perplexity with one fewer element bit and stays within 1.32% of native quality. CurveFP8 lowers error in all 36 paired training-GEMM comparisons. Across three matched 3B-token pretraining triplets, it reaches mean BF16-inference perplexity 22.5366 versus 22.5407 for FP8 and has a lower format penalty in every seed. Downstream evaluation shows transfer parity and a consistent WikiText-103 gain. In a preliminary 4x4 Nangate45 spatial accelerator tile, CurveFP8 uses one fewer product register and 4.6% less area than timing-closing FP8 at 500 MHz. These results support CurveFP as a numerical and arithmetic co-design, while leaving system-level efficiency to future study.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Quantization, Data Representation, Hardware Acceleration

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_z1suOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CurveFP, a block-scaled logarithmic family of formats that computes products using integer-index operations and organizes accumulation over a finite set of phases determined by a rational radix. The paper proposes 8-bit formats for training (i.e., CurveFP8 E4C3/E5C2) and a 7-bit format for inference (i.e., CurveFP7 E3C3). The results support FP8 training parity at small model scale and comparable inference perplexity when CurveFP7 uses finer-grained scaling than FP8. A preliminary hardware comparison reports a modest tile-area reduction, but the practical advantage over existing formats remains unclear.

Critical strengths
  • The core idea is interesting, and the encoding enables structured arithmetic.
  • The explanation is dense and sometimes difficult to parse, but it captures the proposed family and explains several design trade-offs.
  • The evaluation covers both training and inference with an FP8 comparison.
  • The proposed format is accompanied by a hardware implementation and analysis; the authors acknowledge the limited scope of the hardware evaluation.
Critical weaknesses
  • The added value over existing formats is not sufficiently established. The training results support parity with FP8, while the 7-bit inference results offer a modest storage reduction. These benefits need stronger context against microscaling formats such as MXFP8. Section 4.1 lists microscaling baselines, but no explicitly identified MX results are reported.
  • The evaluation of the four 7B–9B models is limited to WikiText-2 perplexity (Figure 2). No zero-shot task accuracy or generative evaluation is reported for these models. The downstream experiments in Section 4.6 concern the much smaller pretrained models.
  • Figure 2a changes scale granularity, element width, and data format simultaneously. It demonstrates a useful operating point but does not isolate the contribution of the proposed element format. Figure 2b varies scale granularity only for CurveFP7 and INT8 on Falcon-H1, leaving the corresponding FP8 comparison incomplete. The panels therefore indicate that tensorwise FP8 substantially outperforms tensorwise CurveFP7 on Falcon-H1. The repeated claim of “FP8-class quality at 7 bits” should consistently specify the block-scaled setting and scope the claims accordingly.
  • Table 3 reports aggregate NMSE ratios without showing variation across the sampled windows or layers. The reported wins are encouraging, but per-window results would clarify how much the improvement varies. I assume stands for normalized mean squared error; acronyms should be defined at first use.
  • The training runs in Section 4.5 are promising but use only 128.3M-parameter Llama-style models. The larger-model GEMM diagnostics do not establish training convergence or downstream parity at larger scale.
  • The hardware comparison has limited practical reach. It compares two custom tiles with a common fixed-point accumulator and excludes scale handling, memory, etc. Peak throughput is unchanged, and the reported energy per MAC is slightly higher for CurveFP.
  • The numerical evaluation covers CurveFP7 and the CurveFP8 E4C3/E5C2, whereas the hardware evaluation covers only CurveFP8 E4C3 against FP8 E4M3FN. It therefore does not establish the hardware trade-offs for 7-bit inference or backward-gradient arithmetic, or how costs scale with operand width, phase count, and dot-product length.
  • The path from the proposed accelerator tile to a practical deployment remains unclear. The paper does not demonstrate acceleration on existing hardware or establish a system-level benefit sufficient to motivate dedicated hardware support.
  • The contribution beyond existing logarithmic representations needs sharper explanation. Section 2 delineates CurveFP from existing logarithmic representations, but the paper does not clearly establish which additional design choices provide the decisive benefit for language models in particular.
  • Some positioning statements need stronger support or qualification: (1) the introduction states that uniform integers increasingly require fine-grained scales as activation range widens, without a direct supporting citation or analysis, and (2) Section 2 attributes FP8-regime quality to the proposed structure, but the experiments do not isolate the effects of the element codebook, scale selection, and scale granularity.
Questions for authors
  • What specific language model requirement motivates this representation, and which design choices address that requirement?
  • Can the authors provide explicit MXFP8 comparisons and separate the effects of the element codebook, scaling granularity, and scale-selection method?
  • Can one hardware implementation support different rational radices (p,q)? What additional resources (if any) are needed when the phase count changes?
  • Do the authors have hardware results or analytical cost models for additional FP and CurveFP formats, including 7-bit inference and backward-gradient arithmetic? How do area and timing scale with operand width and dot-product length?
  • Appendix C.1 applies rounded Q16 phase coefficients before accumulating into a common Q32 register, rather than retaining separate integer phase sums as in Section 3.2. What are the cost and numerical implications of this choice, and do the model evaluations emulate this arithmetic?
  • Which deployment target is the proposed tile intended to represent, and what evidence indicates that its benefit survives scale handling, conversion, and integration into an accelerator?
  • How does this work fit with the growing body of 4-bit training and inference research, and what trade-off motivates the 7- and 8-bit operating points?
5 comments

Thank you for the thorough review. Your central concern is the added value over existing formats, and several of your points ask for experiments that isolate it. We ran them; they are in the revised PDF (blue). This comment covers MXFP8 and the isolation (your question 2); the second covers task evaluation, variation and training scale; the third covers motivation, positioning and hardware (questions 1 and 3–7).

In short

  • Against explicit MXFP8, CurveFP7 is better on all four models with 14% fewer bits per value.
  • With scales, granularity and selection rule matched, the codebook lowers weight and activation error by 8–17% on all four models and lowers perplexity in all 15 settings on three.
  • Zero-shot accuracy shows no significant change on all four 7B–9B models, and matched training now reaches 1B parameters.

Q2. Explicit MXFP8, and codebook, granularity and scale selection separated (Sec. 4.2, App. D.1). WikiText-2 perplexity; bits per value in parentheses. MXFP8 uses E4M3 elements with one E8M0 scale per 32 values, chosen by the OCP floor rule or by search. The three G64 columns share one power-of-two block-scale contract and selection rule, so only the element format differs.

ModelNativeMXFP8 G32, OCP scales (8.25)MXFP8 G32, searched scales (8.25)FP8 E4M3 G64 (8.06)CurveFP8 G64 (8.06)CurveFP7 G64 (7.06)
Llama-3-8B6.13856.26846.22046.21986.21646.2194
Qwen3-8B7.00437.09137.05867.05797.05097.0582
Qwen3.5-9B6.98607.05017.02257.02237.01607.0186
Falcon-H1-7B5.64235.69665.64625.64515.66675.6633
  • Explicit MXFP8: CurveFP7 outperforms standard MXFP8 on all four models, by 0.03–0.05, with 14% fewer bits per value, and matches or beats searched-scale MXFP8 on Llama-3, Qwen3 and Qwen3.5.
  • Codebook at equal width: CurveFP8 lowers weight and activation NMSE relative to FP8 E4M3 by 8–17% on all four models. It lowers perplexity in all 15 matched settings on Llama-3, Qwen3 and Qwen3.5 (G64 and G512, MSE and ceil-absmax selection, plus a validation split), by 0.002–0.008; on Falcon-H1 all matched formats stay within 0.5% of native.
  • Selection rule: the CurveFP8-vs-FP8 ranking is the same under both rules on every model.
  • Granularity and the 7-bit point: the 7-bit quality rests on the block-scale contract, which is part of the CurveFP specification. A 7-bit FP control placed under our contract confirms this: it also reaches the FP8 regime, and CurveFP7 matches it within 0.3% at G64 on all four models, with 11% lower weight NMSE and a multiplier-free product (full matrix in App. D.1).

Figure 2 and the setting of the 7-bit claim (App. D.2). We agree that Figure 2a alone does not isolate the element format; the table above does. For Figure 2b, a new table in App. D.2 completes the Falcon-H1 sweep with FP8 E4M3 and CurveFP8 at every granularity: both 8-bit formats are nearly granularity-invariant there, while CurveFP7's tensorwise collapse is a range effect (E3C3 spans 7.75 binades), which is why the format is specified with block scales. As you suggest, the revision says "block-scaled" consistently (Figure 2 caption, Secs. 2 and 4.2, conclusion) and states that the FP8 baselines of Figure 2a use standard FP32 absmax scales.

Zero-shot on the 7B–9B models (App. D.3). You are right that the submission reported only perplexity for these models. We added six tasks (ARC-E/C, HellaSwag, PIQA, WinoGrande, LAMBADA) on all four: CurveFP7 G64 changes mean accuracy by −0.0034, −0.0023, −0.0000 and −0.0025 (Llama-3, Qwen3, Qwen3.5, Falcon-H1). No task difference is significant (max |z| = 0.68), comparable to tensorwise FP8 (max |z| ≤ 0.91). We did not add free-form generation benchmarks in the rebuttal period; LAMBADA (last-word prediction) is the closest task in the suite.

Variation behind the training-GEMM table (Sec. 4.4, App. D.7). NMSE (normalized mean squared error) is now defined at first use. Across the 36 window comparisons the CurveFP8/FP8 ratio ranges from 0.866 to 0.940, so CurveFP8 wins every window (per-window table in App. D.7). Per layer role, CurveFP8 is lower in 96/96 Pythia-410M role × GEMM × window comparisons (ratios 0.80–0.98) and in 77/84 for Llama-3.2-3B; all seven exceptions are gradient GEMMs, six of them in one window.

Training scale (Sec. 4.5, App. D.5). We trained matched FP8 and CurveFP8 models at 0.32B and 1B parameters. Entries: BF16-inference held-out perplexity / format penalty (training-format minus BF16-inference perplexity of the same checkpoint).

Model, tokensFP8CurveFP8
128M, 3B (mean of 3 seeds)22.5407 / 0.075822.5366 / 0.0688
0.32B, 3B19.9927 / 0.051219.9914 / 0.0459
1.0B, 1B, seed 124.2539 / 0.030424.2822 / 0.0238
1.0B, 1B, seed 224.2201 / 0.027624.2023 / 0.0242
  • No run is unstable, and quality stays at parity at every size (the downstream suite of Sec. 4.6 was not rerun at these sizes).
  • Beyond parity, CurveFP8's format penalty is lower in all 10 matched training pairs (three sizes, three learning rates, up to three seeds), by 6–22%; it also stays lower against FP8 retrained with CurveFP8's exact scale recipe. Details are in our reply to UCLW.

Q1. Which language-model requirement, and which design choices address it.

  1. LLM weights and activations are heavy-tailed with wide dynamic range. Log-spaced levels give constant relative error, which is why CurveFP8 lowers weight and activation NMSE relative to E4M3 by 8–17% at equal width.
  2. Range must be local. Compact power-of-two block scales supply it at 0.06–0.08 bits per value, which is what makes the 7-bit point possible.
  3. Training needs a precision-oriented and a range-oriented role, hence E4C3/E5C2, mirroring E4M3/E5M2.
  4. The uniform curve index makes products closed, and the rational radix prices any other level density through the phase law.

Relative to earlier logarithmic formats (Sec. 2, App. D.8), the additions are the rational radix with its phase law, the compact power-of-two scale contract, and the FP8 role formats trained with standard AdamW.

Q7. Relation to 4-bit work. Our operating points target FP8-class training and calibration-free 7/8-bit inference with a single scalar algebra: 8 bits is the FP8 training width used in practice, and the 7-bit point removes one bit while staying within 1.32% of native without calibration. W4A4 methods rely on rotations, outlier migration or learned codebooks with calibration (QuaRot, SpinQuant, LO-BCQ, NestQuant; Sec. 2) and are complementary. We did not evaluate below 7 bits.

Q6. Deployment target and scope of the hardware study. The tile represents the multiply–accumulate array of a tensor-core-like matrix unit, the block an accelerator replicates many times. CurveFP is a numerical-format contribution whose primary evidence is model quality, so the tile is deliberately small: it demonstrates that the closed product maps to verified, timing-closed logic and that its arithmetic is the arithmetic evaluated in the models. It is far from an accelerator. We have no evidence yet on scale handling, conversion or integration and claim none; Sec. 5 leaves system-level efficiency to future work, and the paper claims area and product-stage depth only. You are right that peak throughput is equal and energy is on par (4.78 vs 4.75 pJ/MAC). On existing hardware the benefit is storage (7.06 bits per value at FP8-class quality, 12–14% below FP8 and MXFP8); the model-quality results do not depend on dedicated hardware.

Q5. Q16 coefficients vs. separate phase sums. Numerics: with accumulation that does not round, the two are identical by linearity (∑hChSh=∑iCh(i)vi\sum_h C_h S_h = \sum_i C_{h(i)} v_i), and Q16 coefficients have relative error below 10−510^{-5}. The model evaluations now emulate this arithmetic (Q16 coefficients, no intermediate rounding, one output rounding): perplexity is at the reported values in all four tested pairs, e.g. Qwen3 CurveFP7 7.0538 vs 7.0581 (App. D.4). Cost: the per-product form keeps one accumulator per output, whereas retaining phase sums needs one per phase (H=8H = 8) plus a final recombination. We measured only the former.

Q3. Other radices on one implementation. The datapath structure is the same for any (p,q)(p, q), fixed at design time. Two parts change: the phase index becomes h=(p(nx+nw) mod qK)/gcd⁡(p,qK)h = (p(n_x + n_w) \bmod qK)/\gcd(p, qK), a constant multiply and modulo on the rank sum instead of bit slicing, and the coefficient table grows from 8 to HH entries (9 for r=28/9r = 2^{8/9}, 32 for r=23/4r = 2^{3/4}). We have not synthesized non-binary radices; the radix table motivates the 8-phase binary point as the default.

Q4. Other formats; scaling with width and dot length. We do not have routed tiles for 7-bit inference or backward GEMMs, and a credible physical-design study does not fit the rebuttal period, so we state what the format definitions imply. Product formation is one rank addition, a sign XOR and zero detection for every CurveFP format; it grows linearly with element width, and there is no significand multiplier to scale. E3C3 keeps the 8 phase coefficients and halves the exponent range of a product (16 vs 32 binades), which narrows the alignment shift. In backward GEMMs an E5C2 gradient multiplies an E4C3 operand; the product stays on the 8-phase grid and its exponent range widens to 48 binades, the same kind of widening that E5M2 brings to FP8. For every format the accumulator grows by one bit per doubling of the dot length. Routed area and timing for these points are future work.

Positioning statements and presentation. (1) The integer claim is now supported in place by Figure 2b: on Falcon-H1, INT8 degrades from 5.653 (G128) to 10.10 (tensorwise) while FP8 stays at 5.645–5.666. (2) Sec. 2 now says "with block scales" and points to the isolation in Sec. 4.2. (3) To ease the dense exposition you noted, Appendix A now has a worked product-and-accumulation example.

We hope the new isolation, task and training results address your main concerns, and we are glad to clarify further.

Reviewer_FS92OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CurveFP, a family of block-scaled logarithmic formats that jointly considers numerical representation and product arithmetic. Products are represented exactly in widened integer coordinates, while a rational exponent spacing determines the number of accumulation phases. Eight-bit formats are instantiated for forward and backward training and seven-bit for inference. Experiments on four 7B–9B models demonstrate that CurveFP7 achieves better perplexity than tensorwise FP8, and that CurveFP8 yields lower error rates across three matched 3B-token pretraining triplets. In addition, on a preliminary 4×4 Nangate45 spatial tile, CurveFP8 uses fewer product registers and a smaller area than FP8. These results indicate that CurveFP is an effective approach to numerical–arithmetic co-design, although system-level efficiency still requires further study.

Critical strengths
  1. A concrete connection between format design and arithmetic implementation: The paper explicitly connects logarithmic spacing to product coordinates and accumulation phases, and distinguishes exact product encoding from subsequent requantization. The matched, routed 4×4 tile comparison goes beyond operation-count arguments: Table 6 shows approximately 4.6% lower area with one fewer product pipeline stage at the same clock and peak throughput. Although limited in scope, this provides useful evidence that the representation can translate into a hardware benefit.

  2. Carefully controlled evidence for training viability: The matched pretraining triplets share initialization, data order, and optimization settings. Evaluating each checkpoint under both its training format and a common BF16 runtime usefully separates training effects from inference quantization loss. CurveFP8 also reduces aggregate NMSE in all 36 paired training-GEMM comparisons, covering forward, activation-gradient, and weight-gradient computations. These experiments provide credible evidence that the proposed formats can support sustained training.

Critical weaknesses
  1. The substantive novelty over prior logarithmic arithmetic is not sufficiently demonstrated: LNS-Madam already describes fractional-power-of-two representations, exponent-addition products, quotient/remainder decomposition, and accumulation grouped by remainder before fixed-coefficient recombination. This overlaps substantially with the binary-radix configuration’s central arithmetic. The rational-radix phase formula, compact binary scales, and role-specific formats offer possible extensions, but their incremental value is not isolated through a direct comparison or ablation against the closest LNS construction.

  2. The experiments do not yet establish a convincing accuracy–cost advantage under matched conditions: The headline inference comparison uses block-scaled CurveFP7 against tensorwise or rowwise FP8, confounding representation with scaling granularity. Although Section 4.1 mentions microscaling baselines, the reported results do not provide an explicit matched-block FP8/microscaling comparison across the four models. On the hardware side, the measured tile converts each product using a Q16 phase coefficient and accumulates in Q32; it does not measure the proposed deferred phase-bin reduction. Scale handling and boundary conversion are also excluded. Thus, the model-quality results and hardware savings are not connected through a demonstrated common numerical implementation.

Questions for authors
  1. Clarification of novelty: Relative to LNS-Madam’s fractional-base representation and remainder-grouped accumulation, which specific algorithmic or architectural capability is newly enabled by CurveFP? Can a matched comparison isolate the benefit of the proposed additions while keeping the optimizer and scaling policy fixed?

  2. Clarification of numerical fidelity: Do the model experiments emulate the Q16 coefficients, Q32 accumulation, and output conversion used by the hardware implementation? If not, what error bound or model-level validation connects the hardware arithmetic to the reported perplexities, and how much of the area advantage remains after scale handling and conversion are included?

2 comments

Thank you for the precise questions; both led to new experiments (blue in the revised PDF). This comment covers novelty; the next covers matched conditions and numerical fidelity.

In short

  • Novelty: binary-radix CurveFP shares LNS-Madam's lattice, and the paper now says so. New are the rational radix with its phase law, a compact power-of-two scale contract, and FP8-style role formats trained with standard AdamW; two matched comparisons isolate the first two.
  • Matched conditions: FP8 E4M3 under our exact block scales and explicit MXFP8 are now reported on all four models.
  • Fidelity: the models now run with the tile's arithmetic, and perplexity is match the reported values.

Novelty relative to LNS-Madam (Sec. 2, App. D.8). We agree that the binary-radix case shares LNS-Madam's lattice: flattening our fields gives n=Ke+kn = Ke + k, the base-21/82^{1/8} lattice (γ=8\gamma = 8) with exponent-addition products and remainder-grouped accumulation. Sec. 2 noted the shared LNS algebra and now names LNS-Madam. The capabilities CurveFP adds:

  1. Level densities between powers of two, at a known cost. LNS-Madam has γ\gamma levels per octave with γ\gamma a power of two. With r=2p/qr = 2^{p/q}, CurveFP has qK/pqK/p levels per octave and H=qK/gcd⁡(p,qK)H = qK/\gcd(p, qK) phases, e.g. 9 levels and 9 phases (r=28/9r = 2^{8/9}) or 10.7 levels and 32 phases (r=23/4r = 2^{3/4}). The phase law gives the accumulation cost of each choice in advance.
  2. Multiplier-free scaling with compact metadata. Scales are 4/5-bit power-of-two exponents (0.06–0.08 bits per value at G64) applied by a shift, instead of a real-valued scale per group.
  3. FP8 drop-in training. E4C3/E5C2 mirror the E4M3/E5M2 roles and train with unmodified AdamW and FP32 master weights instead of a multiplicative optimizer, now up to 1B parameters.

Matched comparisons with optimizer and scale policy fixed, as you asked.

  • Radix (the radix table, Table 2 in the submission): width, G512 scales and model are fixed, and the r=2r = 2 column is the γ=8\gamma = 8 LNS-Madam lattice. r=23/4r = 2^{3/4} lowers perplexity on all four models, by 0.0055–0.0259, at a cost the phase law states in advance (32 phases instead of 8).
  • Scale contract (new control, App. D.8): the same lattice at G64 under LNS-Madam's native contract (a real-valued FP32 scale per 64-value block, log-domain rounding) and under ours. The compact contract stays within 0.008–0.017 perplexity on all four models, while saving 0.44 bits per value and replacing a real multiply per block with a shift.
  • Rounding: log-domain and linear-domain nearest-rounding thresholds differ by under 0.1% in value at γ=8\gamma = 8, so the control above measures the scale contract.

Matched-block FP8 and microscaling on the four models (Sec. 4.2, App. D.1). WikiText-2 perplexity; bits per value in parentheses. The three G64 columns share one power-of-two block-scale contract and selection rule, so only the element format differs.

ModelNativeMXFP8 G32 (8.25)FP8 E4M3 G64 (8.06)CurveFP8 G64 (8.06)CurveFP7 G64 (7.06)
Llama-3-8B6.13856.26846.21986.21646.2194
Qwen3-8B7.00437.09137.05797.05097.0582
Qwen3.5-9B6.98607.05017.02237.01607.0186
Falcon-H1-7B5.64235.69665.64515.66675.6633
  • Microscaling: CurveFP7 is better than standard MXFP8 on all four models, by 0.03–0.05, with 1.19 fewer bits per value.
  • Element format only: CurveFP8 lowers weight and activation NMSE relative to FP8 E4M3 by 8–17% on all four models. It lowers perplexity in all 15 matched settings (G64 and G512, two selection rules, a validation split) on Llama-3, Qwen3 and Qwen3.5; on Falcon-H1 all matched formats stay within 0.5% of native. The full four-model matrix is in App. D.1.

Numerical fidelity (App. D.4). The model evaluations now emulate the tile: Q16 phase coefficients and binary block scales are represented exactly, products accumulate without intermediate rounding (as in the tile's Q32 accumulators), and the output is rounded once. WikiText-2 perplexity at G64, both columns with FP32 log-softmax for higher resolution:

Model, formatSimulator (arithmetic of the reported results)Tile's fixed-point arithmeticDifference
Qwen3-8B, CurveFP77.05817.0538−0.0043
Qwen3-8B, CurveFP87.05117.0507−0.0004
Llama-3-8B, CurveFP76.21976.2190−0.0007
Falcon-H1-7B, CurveFP85.66535.6644−0.0009
  • Model-level validation: perplexity is at the simulator in all four tested pairs.
  • Error bound: accumulation is exact and each Q16 coefficient has relative error below 10−510^{-5}, so a dot product deviates from the ideal by at most 10−510^{-5} times the sum of its absolute products.
  • Deferred phase-bin reduction: it computes the same sum by linearity (∑hChSh=∑iCh(i)vi\sum_h C_h S_h = \sum_i C_{h(i)} v_i), so the emulation covers both layouts. The tile uses the per-product form, which keeps one accumulator per output instead of one per phase.

Area after scale handling and conversion. These are excluded from both tiles and not measured, so we make no system-level claim. For CurveFP the scale is a binary exponent that adds into the shift amount, without a multiplier or an extra phase. As Sec. 5 states, the tile is a small feasibility demonstration, far from an accelerator. With the emulation above, the hardware and model results are now connected through one numerical implementation, as you asked.

Presentation. To make Sec. 3 easier to follow, Appendix A now has a worked product-and-accumulation example, and NMSE is defined at first use.

We hope this addresses both questions, and we are glad to clarify further.

Reviewer_UCLWOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CurveFP, a block-scaled "closed-product" codebook datatype family. Each element = a sign + a two's-complement exponent (E bits) + a curve index (C bits), under a shared power-of-two block scale 2^a; values are distributed across K=2^C interleaved logarithmic curves: |x| = 2^a · r^(e+k/K), with a rational radix r = 2^(p/q). Key properties: products are exactly closed in the code domain (sign XOR, curve-index addition mod K with carry into the exponent, exponent addition), requiring no general multiplier and no table lookup; the rational radix exposes a finite phase count H = qK/gcd(p,qK), binding each range–resolution choice to an explicit accumulation contract (the quality–phase law). It is instantiated as CurveFP8 E4C3/E5C2 (mirroring FP8 E4M3/E5M2 for training) and CurveFP7 E3C3 (7-bit inference, 8 phases). Evidence: (1) inference PTQ on four 7B–9B models — at G64, CurveFP7 beats tensorwise FP8 on all four, stays within 1.32% of native, and crosses INT8 at G512; (2) rational-radix ablation — 32 phases improve PPL over 8-phase binary by only 0.0055–0.0259; (3) operator-level NMSE on all three training GEMMs (Pythia-410M, Llama-3.2-3B captured operands) — CurveFP8 wins all 36 paired windows; (4) end-to-end matched pretraining — three BF16/FP8/CurveFP8 triplets, all at 128.3M params, 12 layers, 3B tokens, CurveFP8 mean PPL 22.5366 vs FP8 22.5407, which the paper explicitly characterizes as "parity, not statistical superiority"; (5) downstream transfer — WikiText-103 improved in all paired runs, PG-19/tasks mixed; (6) hardware — a preliminary 4×4 Nangate45 tile at 500 MHz, one fewer product register and 4.6% less area than timing-closing FP8 (4.1% smaller with the register restored).

Critical strengths
  1. A novel, elegant representation–arithmetic co-design. Turning multiplication into an exact closed update ("sign XOR + index addition + exponent addition," no multiplier, no lookup/projection) and using a rational radix to derive a finite phase count that binds every range–resolution choice to an explicit accumulation contract — this quality–phase law is a genuine conceptual contribution distinct from classical LNS/APoT/microscaling; one algebra spans both 8-bit training and 7-bit inference.
  2. Scrupulously honest framing and a concrete inference result. Throughout, results are characterized as parity/feasibility rather than overstated; CurveFP7 reaches FP8-class quality with one fewer bit, within 1.32% of native, crossing INT8 at G512 — a concrete, reproducible inference result.
Critical weaknesses
  1. The training evidence is both scale-limited and fails to establish equal-footing conditions in the main text, so it cannot support the paper's "training path" narrative. (Scale) The only end-to-end training is at 128.3M params / 3B tokens; note the rest is not training — the 410M/3.2B results are only operator-level NMSE, and the 7B–9B results are inference PTQ. The real risks of low-precision training (loss spikes, divergence, gradient underflow/overflow, activation/gradient dynamic-range growth) emerge precisely at scale; the range-oriented E5C2 gradient format designed for exactly this is never validated at any scale that would stress it, so 128M parity hardly extrapolates. (Fairness) The equal-footing conditions of the training comparison are not stated — the FP8 baseline's scale granularity (per-tensor/rowwise/block), delayed vs. JIT scaling, and gradient scaling/clipping are unspecified; whether CurveFP8 uses an STE, whether the block scale is learnable, and whether forward/backward dynamic scaling is identical to FP8 are likewise unstated (the main text gives only CurveFP8's own power-of-two + ceil-absmax). At a parity-level margin (−0.0110 ± 0.0217), any scaling/gradient asymmetry could dominate the conclusion — the comparison may be measuring the scaling strategy rather than the numerical algebra itself. The title/abstract foreground training, yet the evidence stops at small models, a short token budget, and unmatched recipes — a clear framing–evidence gap.
  2. As a "product-arithmetic co-design," the hardware-efficiency case is preliminary and omits the cost of the new mechanism. The evidence is a single preliminary 4×4 tile with a 4–4.6% area advantage and near-identical pJ/MAC (4.78 vs 4.75), and the comparison excludes the phase accumulation (H-phase fixed-coefficient recombination) and scale transport — which is exactly the new overhead the phase law introduces. The very machinery that would support "closed products are cheaper" is left out, so the core efficiency value is not yet demonstrated (the paper itself defers this to future work).
Questions for authors
  1. (Training credibility — key) Please disclose and match the full training recipe for both FP8 and CurveFP8 in the main text: scale granularity/type (per-tensor/rowwise/block; power-of-two vs. real-valued), delayed vs. JIT, gradient scaling/clipping, whether an STE is used, whether the block scale is learnable, and forward/backward dynamic-scaling consistency; and add a 128M→1B scaling law (format-penalty-vs-scale curve, presence/absence of loss spikes, LR/scale/block sensitivity). This is needed to show the parity is not an artifact of differing scaling strategies and that it extrapolates to real LLM training.
  2. (Hardware credibility — key) Could you include the phase-accumulation and scale-transport hardware in the area/energy comparison? Once this new overhead is counted, does 8-phase CurveFP8 still beat FP8 — this determines whether "closed products are cheaper" holds.
2 comments

Thank you for the careful review and for stating the two key questions so clearly. Both are answered with new experiments (blue in the revised PDF). This comment covers training; the next covers hardware.

In short

  • Recipe: the two recipes are identical except for the scale type, and giving FP8 CurveFP8's exact scales leaves parity and CurveFP8's lower penalty intact.
  • Scale: matched runs at 0.32B and 1B parameters train without loss spikes, and CurveFP8's format penalty is lower in all 10 matched pairs.

Recipe and equal footing (Sec. 4.5, App. D.5). You are right that the main text did not state the recipe; it does now. Item by item:

  • Granularity: one scale per tensor, for weights, activations and gradients, in both formats.
  • Delayed vs. JIT: just in time, recomputed at every step; no delayed scaling or amax history.
  • Scale type and selection (the only difference): FP8 used real-valued FP32 absmax scales, the standard FP8 recipe; CurveFP8 used power-of-two exponents, MSE-searched for weights and ceil-absmax for activations and gradients.
  • Learnable scales, loss scaling: none.
  • STE: yes. Both use the same quantized autograd (dX=QG(dY) QW(W)dX = Q_G(dY)\,Q_W(W), dW=QG(dY)⊤QA(X)dW = Q_G(dY)^\top Q_A(X)) with straight-through updates to FP32 master weights.
  • Gradient clipping: global norm 1.0.
  • Forward/backward consistency: within each format, the same JIT rule scales activations and gradients, and the backward pass reuses the forward-quantized operands.
  • Everything else: the same role split (E4M3/E4C3 forward, E5M2/E5C2 gradients), initialization, data order, optimizer and schedule.

To test whether the scale difference drives the result, we retrained FP8 with CurveFP8's exact scale recipe (middle column below).

Recipe control and scaling law (Sec. 4.5, App. D.5). Entries: BF16-inference perplexity / format penalty (training-format minus BF16-inference perplexity of the same checkpoint). Peak LR follows the GPT-3 size schedule and is shared by all formats.

Model, tokensFP8FP8 with CurveFP8's scale recipeCurveFP8
128M, 3B (mean of 3 seeds)22.5407 / 0.075822.5471 / 0.070822.5366 / 0.0688
0.32B, 3B19.9927 / 0.051220.0002 / 0.049219.9914 / 0.0459
1.0B, 1B, seed 124.2539 / 0.0304–24.2822 / 0.0238
1.0B, 1B, seed 224.2201 / 0.0276–24.2023 / 0.0242
  • Recipe: with matched scales CurveFP8 still has the lower penalty, in every 128M seed and at 0.32B, and quality stays at parity. At operator level its NMSE is 10.6–12.0% below recipe-matched FP8, in all 36 window × GEMM comparisons. The parity therefore does not come from the scaling strategy.
  • Penalty vs. scale: lower for CurveFP8 at every size.
  • Loss spikes, divergence: no run has a non-finite step. On identical data the moving-average CurveFP8 − FP8 training-loss difference stays within 0.015 nats after warmup and ends within 0.002: no format-specific spike.

Learning-rate sensitivity (128M, 1B tokens; App. D.5). Same entries; the last column is the difference in BF16-inference perplexity.

Peak LRFP8CurveFP8CurveFP8 − FP8
3e-430.4435 / 0.072230.4767 / 0.0651+0.0332
3e-4, seed 230.5620 / 0.073430.5615 / 0.0679−0.0005
6e-4 (paper's LR)27.4188 / 0.066727.3979 / 0.0599−0.0209
1.2e-326.4665 / 0.074426.4356 / 0.0650−0.0309

Over all 10 matched pairs in the two tables (128M seeds counted separately), CurveFP8's penalty is lower in every pair, by 6–22%, and the perplexity difference averages −0.002 (range −0.031 to +0.033; CurveFP8 lower in 7): parity.

Scale, block and range sensitivity (App. D.6).

  • Scale field: changing the gradient scale field at 1B (last bullet) moves perplexity by 0.017, within run-to-run variation.
  • Block size: training uses one scale per tensor for both formats, so there is no block size to tune; in inference CurveFP8 changes by at most 0.003 between 64- and 512-element blocks on all four models.
  • Overflow and range (the stress on E5C2): no gradient or activation value overflows at any size. 99.99% of gradient values lie within 26.5–27.25 binades of their tensor maximum, inside E5C2's 31.5-binade range, and the spread does not grow with scale.
  • Underflow: gradient magnitudes shrink with size, below the floor of our 5-bit scale-exponent field, which touches only values carrying under 5×10−75\times10^{-7} of the gradient energy. Re-biasing the same 5 bits reduces this to below 0.0006% of values, and a 1B run with the re-biased field reaches the same quality (24.265 vs 24.282).

An LR sweep at 1B did not fit the discussion period. With the recipe matched and parity reproduced at about eight times the original parameter count, the abstract and introduction now state the training evidence together with its scale.

Once the phase mechanism is counted, does 8-phase CurveFP8 still beat FP8? Yes: it is already counted. In the measured tile each processing element adds the two ranks, selects one of the eight fixed phase coefficients and shifts it into the accumulator (Sec. 5, App. C.1). The H-phase recombination is therefore folded into every product, and no separate recombination stage is left out. Our wording in Sec. 5 did not make this clear; it now does.

The reported comparison (4.6% less area than timing-closing FP8 with one fewer product register, 4.1% with the register restored, energy on par at 4.78 vs 4.75 pJ/MAC) thus includes the cost of the new mechanism.

Scale transport. It is excluded from both tiles. For CurveFP a scale is a binary exponent that adds into the shift amount (Sec. 3.2), so it needs no multiplier and adds no phase.

New: the tile's arithmetic reproduces the model results (App. D.4). We now run the models with the tile's fixed-point arithmetic (Q16 phase coefficients, accumulation without intermediate rounding, one output rounding). Perplexity is at the reported values in all four tested model–format pairs, e.g. Qwen3 CurveFP7 7.0538 vs 7.0581, so the measured datapath is the one that produces the reported quality.

Scope. The contribution is the numerical format and its arithmetic contract, evaluated mainly through model quality. The tile is a small demonstration that the closed product maps to verified, timing-closed logic: it shows implementability and local cost, and it is far from an accelerator. As Sec. 5 states, memory, interconnect and scale generation belong to a system-level study that we leave to future work.

We hope this resolves both key questions, and we are glad to clarify further.

Comments on the paper

1 comment

We thank the reviewers for careful and constructive reviews. We are glad that the reviewers found the co-design of level geometry and product algebra interesting and noted the careful framing and controls. The concerns were about evidence and scope. The revised PDF (one added page; every added or modified passage in blue) addresses each of them with new experiments or clarifications. All new experiments ran in the paper's pinned software environment, in which every paper result we re-ran (inference perplexities and all 72 training-GEMM NMSE values) reproduces exactly. Individual replies follow.

1. Is the element format isolated from scale granularity and selection? (FS92, z1su; Sec. 4.2, App. D.1–D.2) Yes, with two new comparisons. CurveFP7 with 64-element blocks (7.06 bits/value) outperforms standard MXFP8 (8.25 bits/value) on all four 7B–9B models, by 0.03–0.05 perplexity. With the element format as the only variable (identical power-of-two block scales and selection rule), CurveFP8 lowers weight and activation NMSE relative to FP8 E4M3 by 8–17% on all four models and lowers perplexity in all 15 matched settings on Llama-3, Qwen3 and Qwen3.5.

2. Task evaluation of the 7B–9B models (z1su; App. D.3). Over six zero-shot tasks on all four models, CurveFP7 changes mean accuracy by at most 0.0034, and no task difference is significant (max |z| = 0.68).

3. Training recipe, scale and stability (UCLW, z1su; Sec. 4.5, App. D.5–D.6). The full recipe is now in the main text, and FP8 retrained with CurveFP8's exact scale recipe keeps the higher format penalty in every seed. Matched runs at 0.32B and 1B parameters complete without instability and at parity in quality. Over all 10 matched FP8/CurveFP8 training pairs (three model sizes, three learning rates, up to three seeds), CurveFP8's format penalty is lower in every pair, by 6–22%.

4. Does the hardware arithmetic reproduce the model quality? (UCLW, FS92, z1su; Sec. 5, App. D.4) Yes. Running the models with the tile's fixed-point arithmetic gives perplexity exact at the reported values in all four tested model–format pairs, so the hardware and model results now share one numerical implementation. Sec. 5 also states explicitly that the phase recombination is inside the measured tile.

5. Relation to LNS-Madam (FS92, z1su; Sec. 2, App. D.8). Sec. 2 now names the shared binary-radix lattice and states what CurveFP adds; a new control isolates the scale contract.

6. Presentation (FS92, z1su). NMSE is defined at first use, the seven-bit claim names its block-scaled setting (Figure 2, Secs. 2 and 4.2, conclusion), and Appendix A gains a worked product-and-accumulation example.

A note on scope. CurveFP is a numerical-format contribution to a machine-learning venue: the level geometry, the closed product and the phase law, evaluated primarily through model quality in inference and training. The 4×4 tile is a deliberately small demonstration that this arithmetic maps to verified, timing-closed logic. For context, a tile is the small multiply–accumulate block that an accelerator replicates many times; comparing two tiles shows that an arithmetic is implementable and at what local cost, not how a complete chip would perform. As the paper states, it is a feasibility check and far from an accelerator: we keep memory, interconnect, scale handling and system-level efficiency to future work.

We hope these additions resolve the concerns raised, and we are glad to answer further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.