acceptodds
Under review as a conference paper at ICLR 2027

AngularQ: Geometry-Guided Gradients for Low-Bit Large Language Models

Abstract

Training large language models (LLMs) with one- to three-bit weights and activations offers major gains in efficiency, but preserving accuracy under extreme quantization remains a central challenge. The difficulty lies not only in representing discrete values but also in learning through them, since hard quantizers eliminate the derivatives required for gradient-based training. We introduce AngularQ, a mathematical framework for quantization-aware training that leverages the averaging effect of random embeddings to obtain a smooth population formulation of discrete low-bit computation. This formulation yields geometry-aware gradient surrogates that retain information from the quantized forward pass while providing an unbiased surrogate for the corresponding backward gradient. We characterize how different embedding constructions affect approximation accuracy and gradient quality, using these insights to guide a unified, computationally efficient approach to binary and multibit training. Beyond providing a comprehensive theoretical foundation for learning through quantization, this framework translates into substantial empirical gains in extreme-low-bit language modeling. Across language models from 45M to 1.7B parameters on FineWeb- Edu, AngularQ delivers its largest gains when both weights and activations are binary: 55.3% and 49.4% lower perplexity than matched-update QuEST at 0.6B and 1.7B, respectively. Two- and three-bit configurations retain substantial gains of 7.7–9.3%. Three paired training seeds at 140M consistently reproduce the binary advantage. At equal code-storage budgets, multibit quantization also outperforms binary projection expansion across the evaluated scales. These results establish angular geometry as a principled and effective foundation for learning extreme-low-bit language models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Low-Bit Large Language Models, Quantization-Aware Training, Surrogate Gradients, Angular Geometry, Binary Embeddings

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_qKuyOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AngularQ, a framework for low-bit LLM quantization-aware training based on random sign embeddings and a radial-angular decomposition of quantized inner products. The method uses hard low-bit forward operators while supplying geometry-aware surrogate gradients: a realized sampled radial component and a corrected tangent-space angular component. The paper proposes a unified sampled gradient intended to be efficient on GPUs and provides a teacher-student stationarity result.

Critical strengths

Novel perspective on surrogate gradients for extreme low-bit training. The analysis of multiple embedding families and the treatment of their finite-sample concentration properties provide more theoretical grounding than typical low-bit QAT methods.

Critical weaknesses
  1. Compared with QuEST, AngularQ changes several components at once: the embedding construction, hard forward representation, quantizer/codebook, normalization and radial parameterization, and backward rule. The authors explicitly frame the comparison as one between complete operators, which is reasonable for end-to-end benchmarking, but this prevents attributing the large gains specifically to the proposed angular surrogate gradient.

  2. In practice, the embedding is sampled once and fixed throughout training, while the weights and activations adapt to that same realization. Thus, the expectation-based unbiasedness statements do not directly characterize the gradient trajectory of the trained model. This issue is especially relevant for the discrete Rademacher/Walsh-Hadamard cases, where the guarantees are asymptotic and require delocalization assumptions, and for multibit Haar-frame training, where the paper explicitly uses a Gaussian joint-kernel approximation rather than claiming exact finite-dimensional population unbiasedness.

Questions for authors
  1. Can the authors provide a matched-forward ablation that holds the hard forward operator, embedding/frame, normalization, codebook, and training schedule fixed while changing only the backward rule?

  2. For the practical Haar-frame multibit implementation, can the authors quantify how closely the Gaussian joint-kernel approximation and the resulting supplied gradient match the corresponding finite-dimensional Haar population quantities during training?

  3. Can the authors report measured training cost and inference cost for the practical implementations, including wall-clock throughput, peak memory, and any overhead from materializing or applying the embedding frames?

3 comments
AuthorsOpenReview ↗

Response to Reviewer qKuy

We thank the reviewer for the comments and suggestions. Please find below our detailed response.

R4-01. Matched forward, frame, normalization, codebook and schedule

Can the authors provide a matched-forward ablation that holds the hard forward operator, embedding/frame, normalization, codebook, and training schedule fixed while changing only the backward rule?

Yes. In the added 140M experiment, both arms use identical hard forwards, frame tensors, normalization, binary codebook, FP initialization, training data and schedule. Only the full operand backward changes. Unified AngularQ achieves NLL 3.84313 versus 3.92692 for the adapted trust-masked STE, reducing PPL by 8.04% (Table 1). This answers the attribution question without changing the representation or quantizer.

R4-02. Finite-dimensional Haar multibit approximation

For the practical Haar-frame multibit implementation, can the authors quantify how closely the Gaussian joint-kernel approximation and the resulting supplied gradient match the corresponding finite-dimensional Haar population quantities during training?

We evaluated the Gaussian joint-kernel approximation against a finite-dimensional Haar reference at the actual model widths, codebooks and trained-state correlations. At the 0.6B multibit endpoint, the maximum sampled relative kernel-derivative error is 0.0463%. Median supplied-gradient cosines against the population reference are 0.977 for inputs and 0.982 for weights. These measurements support the accuracy of the approximation in the examined trained states. The 0.0463% quantity concerns the kernel derivative, rather than the entire network gradient.

R4-03. Fixed frames, Rademacher/Hadamard and delocalization

In practice, the embedding is sampled once and fixed throughout training, while the weights and activations adapt to that same realization.

We appreciate the reviewer’s thoughtful observation. The unbiasedness statements in our work are population-level identities: for fixed operands w,xw,x independent of the random embedding R\mathcal{R}, the expected sampled gradient agrees with the gradient of the smooth population operator,

=\nabla_w\mathbb{E}_{\mathcal{R}}\left[\mathcal{Q}_i(w,x)\right]$$ with exact or asymptotic interpretations depending on the embedding distribution. These identities characterize the underlying expected angular geometry of the quantized operator and provide a principled target for constructing its backward surrogate. They do not assert conditional unbiasedness along the training trajectory, where the embedding is fixed and the learned weights and intermediate activations may depend on its realization. Importantly, our sampled gradients explicitly retain information from the realized embedding and quantized forward codes, rather than relying solely on the population gradient. We also recognize that this adaptivity issue is a shared concern raised by the reviewer and Reviewer jtZN. To address it theoretically, we have strengthened our stationarity-to-recovery result to a uniform guarantee under additional geometric nondegeneracy conditions (please see response R3-01 to Reviewer jtZN). Specifically, the new result removes the requirement that the candidate $w$ be fixed independently of $\mathcal{G}$ and establishes a high probability recovery, that holds simultaneously for all admissible frame-dependent weights satisfying the stated geometric conditions. The key observation is that the restricted null-space property in the original proof already holds uniformly over all binary-code discrepancies. We will include this extension as the Uniform Recovery corollary, immediately following Theorem 2.2, with its proof in the appendix. This extension strengthens the theoretical foundation of AngularQ by establishing that exact recovery can be certified uniformly, even when the candidate weights adapt to the same fixed random embedding used during training. ### R4-04. Wall-clock and memory overhead > Can the authors report measured training cost and inference cost for the practical implementations, including wall-clock throughput, peak memory, and any overhead from materializing or applying the embedding frames? Table 5 includes measured training-update time and peak GPU allocation, including materializing and applying the dense frames; sharing limits 0.6B frame storage to 56 MiB. Table 6 adds packed structured-runtime measurements with all model buffers counted: **4.28× smaller resident storage**, with only **8.56 KiB** of transform state at 1.7B. At a common 4 GiB allocator cap, packing permits four times as many concurrent sequences and increases aggregate throughput by **2.99×** at a 512-token prompt and **2.83×** at a 2K prompt. Table 4 provides trained structured-frame accuracy evidence at 140M and 0.6B; the 1.7B hardware prototype establishes execution costs.
Reviewer_qKuyOpenReview ↗

Response to Authors

Thank you for the detailed clarification and for addressing both concerns.

Regarding R4-03, I appreciate the distinction between the population-level unbiasedness identities and conditional unbiasedness along a training trajectory with a fixed realization of the embedding. It is helpful to explicitly acknowledge that, once the frame is sampled and fixed, the learned weights and activations may adapt to that realization; hence the stated identities should not be interpreted as conditional unbiasedness of the actual training gradients.

The proposed uniform stationarity/recovery extension appears to be the right direction for addressing this issue. In the revision, please make the quantifiers and scope of the result especially explicit: namely, whether the high-probability event over the sampled frame holds uniformly over all admissible frame-dependent iterates/weights encountered during training, and which geometric assumptions must remain satisfied along that trajectory. It would also be useful to clearly separate (i) the population surrogate-gradient identity, (ii) the fixed-frame training behavior, and (iii) the uniform recovery guarantee in the main text, so readers do not conflate these claims.

Regarding R4-04, thank you for adding measured training-update time, peak GPU allocation, and packed structured-runtime/storage measurements. Reporting the cost of materializing and applying the frames is important, and the reported storage figures help clarify the practical overhead. Please ensure that the final paper presents these results with sufficient experimental context—for example, hardware, batch size, model/configuration, precision, and the corresponding baseline/QAT implementations—so that the wall-clock and memory comparisons are directly interpretable. If inference costs are not materially changed, it would also help to state this explicitly and report the relevant end-to-end inference measurements where applicable.

These additions substantially improve the paper, although I would like to verify the precise statement and proof of the uniform fixed-frame guarantee in the revision before considering this concern fully resolved.

AuthorsOpenReview ↗

Response to Reviewer qKuy: follow up

We thank the reviewer for the thoughtful follow-up and constructive suggestions. The manuscript has also been updated, with the changes highlighted in a contrasting color (purple). We have clarified the scope and quantifiers of the uniform fixed-frame recovery guarantee (please see the highlighted text following Theorem 2.2), included its proof in the appendix, and further distinguished population unbiasedness from fixed-frame training behavior (highlighted text following Eq. 7). We have also expanded the discussion of training and inference costs with the requested experimental details and qualifications (please see the updated experimental section and appendix). We appreciate the reviewer's careful consideration of these revisions.

Reviewer_jtZNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AngularQ, a geometry-aware gradient framework for 1–3 bit LLM quantization-aware training. It uses random embeddings to reveal a smooth angular structure beneath discrete quantized inner products, then derives sampled gradients that preserve both radial information and the realized projection frame. Experiments on 45M–1.7B models show especially large improvements over QuEST in W1A1 training.

Critical strengths
  • Novel and principled alternative to coordinate-wise STE.
  • Extensive theoretical analysis across multiple embeddings and precisions.
  • Strong W1A1 results, reducing PPL by 16.1–55.3% over QuEST.
  • Detailed experiments, implementation descriptions, and seed analysis.
Critical weaknesses
  • The theory mainly considers vectors independent of the random frame, whereas trained weights depend on the fixed frame.
  • Missing controlled ablations among STE, population, unified, and boundary-aware gradients.
  • The 1.7B W1A1 run uses an additional boundary-aware correction and a different schedule, complicating attribution.
  • No measured training cost, runtime memory, inference latency, or throughput.
  • Evaluation is limited to language-modeling loss, without downstream or reasoning benchmarks.
  • Some large-model comparisons do not match training history or computation.
Questions for authors
  • Can the theory be extended to trained weights that depend on the fixed frame?
  • How do the different gradient estimators compare under an identical forward operator and schedule?
  • What is the 1.7B W1A1 result before the final boundary-aware correction?
  • What are the actual training and inference costs?
  • Do the gains transfer to downstream and reasoning tasks?
2 comments
AuthorsOpenReview ↗

Response to Reviewer jtZN

We thank the reviewer for the comments and suggestions. Please find below our detailed response. The manuscript has also been updated, with the changes highlighted in a contrasting color.

R3-01. Frame-dependent trained parameters

Can the theory be extended to trained weights that depend on the fixed frame?

We thank the reviewer for raising this point. With only minimal modifications to Theorem 2.2, namely the addition of a geometric nondegeneracy condition, the recovery guarantee can be extended to a uniform almost-everywhere statement, allowing candidate weights to depend on the fixed random embedding. The key observation is that the restricted null-space property established in the proof of Theorem 2.2 already holds simultaneously over all possible binary-code discrepancies. By imposing a suitable condition on the realized backward subspace, this property yields a uniform recovery certificate outside a measure-zero exceptional set. We will include the following corollary immediately after Theorem 2.2 in the revised manuscript, with its proof provided in the appendix.

Corollary (Uniform Recovery of a Planted Model).

Under the assumptions and sample-size condition of Theorem 2.2 (except that ww need not be fixed independently of G\mathcal{G}), let S=sign⁡(GX)\mathcal{S}=\operatorname{sign}(\mathcal{G}X) and VG=range⁡(G⊤S)\mathcal{V}_{\mathcal{G}}=\operatorname{range}(\mathcal{G}^\top\mathcal{S}). Then, with probability at least 1−β1-\beta, whenever rank⁡(S)<p\operatorname{rank}(\mathcal{S})<p, the implication

⟹sign⁡(Gw)=sign⁡(Gw⋆)\quad\Longrightarrow\quad \operatorname{sign}(\mathcal{G}w)=\operatorname{sign}(\mathcal{G}w^\star)

holds simultaneously for all admissible w∈Sp−1∖VGw\in\mathbb{S}^{p-1}\setminus\mathcal{V}_{\mathcal{G}}.

Since \dim(\mathcal{V}&#95;{\mathcal{G}})\le\operatorname{rank}(\mathcal{S})<p, the exceptional set \mathcal{V}&#95;{\mathcal{G}}\cap\mathbb{S}^{p-1} has zero spherical measure. Thus, the corollary establishes uniform recovery almost everywhere, including for frame-dependent candidate weights outside this exceptional set.

R3-02. Controlled gradient comparison

How do the different gradient estimators compare under an identical forward operator and schedule?

The new 140M comparison changes only the backward rule while fixing the forward, initialization, frames, normalization, codebook, data and schedule. Unified AngularQ reaches PPL 46.6715, versus 50.7504 for the adapted trust-masked backward: an 8.04% improvement (Table 1). This provides direct evidence for the practical surrogate choice under an identical operator.

R3-03. Result before final boundary correction

What is the 1.7B W1A1 result before the final boundary-aware correction?

The reported result is reproduced without boundary correction under the same settings and update count. The remaining manuscript reference to that correction was an editing error and will be removed. The reported accuracy therefore does not require a boundary-correction stage.

R3-04. Actual training and inference costs

What are the actual training and inference costs?

Tables 5 and 6 report training and deployment measurements separately. Complete AngularQ QAT at 0.6B achieves approximately 95,504 tokens/s on the B200, with dense frames included in timing and peak allocation. For the packed structured inference prototype, model storage falls from 3.209 to 0.750 GiB. This permits a larger serving batch under the same allocator cap: aggregate throughput improves by 2.99×/2.83× at 4 GiB and 1.22×/1.48× at 6 GiB, for 512/2,048-token prompts respectively. The advantage is memory-limited serving capacity.

R3-05. Downstream/reasoning transfer

Do the gains transfer to downstream and reasoning tasks?

Table 3 evaluates the paper checkpoints on knowledge, science and commonsense questions at 0.6B and 1.7B. At 1.7B, AngularQ outperforms QuEST on all four tasks for both W1A1 and W3A3. For example, W3A3 improves MMLU-Redux by 16.22 percentage points and HellaSwag by 6.45 points. At 0.6B, it also improves all four W3A3 scores. The new evidence therefore extends the comparison beyond validation perplexity, using a shared zero-shot likelihood protocol and identical question subsets.

R3-06. Presentation and mismatched histories

Some large-model comparisons do not match training history or computation.

We will make the comparison design explicit: complete AngularQ versus complete QuEST measures the overall method, whereas the new 140M identical-forward experiment isolates the backward choice. The added selected-LR QuEST run matches the 0.6B W1A1 update budget and leaves a 51.24% PPL advantage for AngularQ. We will label phase-specific LRs and actual training budgets consistently. The manuscript's separate document-overlap sensitivity preserves the method ordering.

Dear Reviewer jtZN,

As the discussion period is nearing its end, we wanted to kindly follow up and ask whether our rebuttal and subsequent paper updates have addressed your concerns. If any questions or concerns remain, we would be grateful for the opportunity to clarify them. If our responses have affected your assessment of the paper, we would also appreciate it if this could be reflected in your final evaluation. Thank you again for your time and consideration.

-Authors

Reviewer_pujoOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AngularQ, a geometry-aware gradient surrogate for extreme-low-bit LLM training. Building on Grothendieck identities, it preserves the realized radial derivative while conditionally smoothing the discontinuous angular component. The authors derive a GPU-friendly unified surrogate, extend it to multilevel quantizers, and provide supporting theory. The largest empirical gains over QuEST occur in fully binary W1A1 training.

Critical strengths
  1. Principled gradient construction. Unlike heuristic STE-based methods, AngularQ derives its surrogate from the quantized operator's geometry, retaining sample-specific information while matching the population operator gradient exactly or asymptotically under stated assumptions.

  2. Strong theoretical and empirical results. The paper analyzes four embedding schemes, establishes variance comparisons under explicit assumptions, characterizes surrogate approximation quality, and relates stationarity to planted-code recovery under a teacher–student model. The unified rule reuses forward codes, and the W1A1 gains are consistent across three paired 140M seeds.

Critical weaknesses
  1. The source of the gains is insufficiently isolated.

AngularQ and QuEST differ in embedding transforms, reconstruction levels, codebooks, and norm-scaling treatment. Figure 2(a) and Appendix 9.4.1 provide a rotated RMS–STE control with a matching binary forward form, while Figure 9 examines embedding choices. These are useful partial ablations. However, the forward-matched control is limited to 45M and uses only an identity STE. No comparison at larger scales holds the forward operator and training protocol fixed while replacing only the backward rule, for example with QuEST-style trust-masked gradients or the population gradient. At the scales where the headline gains occur, the gains therefore cannot be attributed specifically to the proposed surrogate.

  1. Large-scale comparisons are not fully controlled.
  • At 0.6B, learning rates are transferred from smaller-scale screens without an independent sweep, and AngularQ and QuEST share the transferred learning rate. At 1.7B, however, QuEST’s own screen selects a learning rate four times higher than AngularQ’s, which suggests that QuEST may be outside its preferred range at 0.6B.
  • Table 16 notes that recorded values may reflect only the final launch phase, which may explain its difference from Table 5, but the complete trajectories are not shown.
  • At 1.7B W1A1, AngularQ uses validation-guided hold/decay and a boundary-aware tangent correction during the final 9,095 updates, while QuEST uses cosine decay. The 0.6B AngularQ runs also include an additional terminal decay, and the parent checkpoints of the 0.6B N2/N3 runs are reported to include boundary-corrected terminal training. The individual contributions of these modifications are not isolated.
  • AngularQ’s selected 1.7B learning rate also lies at the upper boundary of its screening grid. This would likely disadvantage AngularQ, but it indicates that tuning is not converged.
  1. Practical efficiency remains unverified.

Figure 3 and Appendix 9.7 report ideal parameter storage, excluding the random embedding matrices RR, and discuss sharing and regeneration as possible mitigations. However, no measured throughput, inference latency, or peak GPU memory is provided. Dense projections and additional backward computations may introduce substantial overhead, potentially offsetting the low-bit savings.

Questions for authors
  1. Can the authors provide complete 0.6B learning rate trajectories and a QuEST learning rate sweep? At 1.7B W1A1, what does AngularQ achieve with a cosine schedule and the unified gradient throughout? How much does each training modification contribute?

  2. Relationship to G-Net. Which theoretical and representation results are inherited from G-Net, and which are new? Does G-Net use a different training protocol, and if so, is a meaningful G-Net-inspired control possible?

  3. Training duration and efficiency. Figures 5 and 8 track AngularQ against continued FP, not QuEST. Does the AngularQ–QuEST gap persist under longer matched training or training from scratch? Can the authors also report actual training/inference costs and peak memory, including embedding overhead?

4 comments
AuthorsOpenReview ↗

Response To Reviewer pujo (part 1)

We thank the reviewer for the comments and suggestions. Please find below our detailed response (due to character limit it is split into 2 parts). The manuscript has also been updated, with the changes highlighted in a contrasting color.

R2-01. Attribution at the scales with headline gains

No comparison at larger scales holds the forward operator and training protocol fixed while replacing only the backward rule, for example with QuEST-style trust-masked gradients or the population gradient.

We added the requested control at 140M, extending the earlier 45M evidence. With the entire hard forward and training recipe fixed, replacing the adapted trust-masked backward by unified AngularQ improves PPL by 8.04% (Table 1). This isolates a backward-rule benefit. The larger complete-method improvements remain a separate result: even after the additional 0.6B QuEST LR run, AngularQ has 51.24% lower PPL (Table 2). The 140M pair is the scale of the new backward-only experiment.

R2-02. 0.6B tuning, complete schedules and search boundaries

Can the authors provide complete 0.6B learning rate trajectories and a QuEST learning rate sweep?

A 0.6B QuEST LR sweep was performed over {7.5e-5, 1e-4, 2e-4, 4e-4}. We now complete the full 74,095-update run at its preferred 4e-4 rate: NLL 3.94861, compared with 4.03537 for the originally reported 1e-4 run. AngularQ remains substantially better at NLL 3.23044. QuEST's trust-radius multiplier was also examined at 140M, where 1.0 outperformed 1.3; the 1.7B QuEST LR was independently selected as 4e-4. We will distinguish main and terminal LR settings in Tables 5/16 and include the recorded trajectories. We will also show the search ranges, including AngularQ's upper-grid 1.7B selection.

R2-03. Unified-only cosine and the individual training modifications

At 1.7B W1A1, what does AngularQ achieve with a cosine schedule and the unified gradient throughout? How much does each training modification contribute?

The reference to boundary correction was an editing error. The reported result is reproduced without boundary correction, using the same experimental settings and number of iterations; we will correct the text. Thus the result does not depend on adding that correction. An isolated comparison of the 1.7B cosine and hold/decay policies has not been performed, so we do not attribute a numerical gain to that schedule choice.

R2-04. Longer matched training and from scratch

Does the AngularQ–QuEST gap persist under longer matched training or training from scratch?

At 0.6B, the additional QuEST run completes the same 74,095-update budget at the stronger LR, and AngularQ retains 51.24% lower PPL. The 1.7B W3A3 comparison also uses matched 50K complete-cosine training budgets. These results address persistence over the reported QAT horizons. Longer-horizon convergence and training an LLM from random initialization are separate questions; our current claims concern FP-initialized QAT.

R2-05. Actual time and memory, including frames

Can the authors also report actual training/inference costs and peak memory, including embedding overhead?

Table 5 reports complete QAT-update time and peak GPU allocation, including dense frames. AngularQ takes 0.686 s/update, versus 0.532 s for QuEST and 0.382 s for FP; shared frame storage at 0.6B is 56 MiB. Table 6 adds a packed structured deployment prototype with 4.28× smaller resident model storage. At a common 4 GiB allocator cap it serves four times as many concurrent sequences, giving 2.99× aggregate throughput with a 512-token prompt and 2.83× with a 2K prompt. The corresponding gains at 6 GiB are 1.22× and 1.48×. These measurements quantify the serving capacity enabled by reduced storage.

AuthorsOpenReview ↗

Response To Reviewer pujo (part 2)

R2-06. G-Net relationship and a meaningful G-Net-inspired control

Which theoretical and representation results are inherited from G-Net, and which are new? Does G-Net use a different training protocol, and if so, is a meaningful G-Net-inspired control possible?

Response: Relationship to G-Net and the possibility of a G-Net-inspired control.

Although G-Net and AngularQ share the classical Grothendieck (Sheppard) identity as a mathematical foundation, they address fundamentally different problems, use distinct training protocols, and establish different theoretical guarantees.

Theoretical and representation contributions.

G-Net uses the Grothendieck identity to construct high-dimensional binary representations that approximate the computations of continuous neural networks. Its analysis primarily concerns representation accuracy and concentration as the embedding dimension increases. We build on the same classical identity and acknowledge G-Net's approximate two-sided Rademacher result. However, AngularQ develops a broader one- and two-sided formulation for rotationally invariant embeddings with potentially dependent rows, a separate one-sided Rademacher approximation, and new variance-reduction guarantees for orthogonal and structured embeddings.

More importantly, our central contributions concern optimization through discrete representations, rather than their approximation properties. These include the radial--angular formulation, boundary-aware sampled gradients obtained through conditional smoothing, GPU-friendly unified sampled gradients, their theoretical comparisons with population gradients, multibit generalizations, and stationarity-to-recovery guarantees. These developments are distinct from G-Net's theoretical framework.

Training protocols.

G-Net follows a two-stage procedure: a specially constructed continuous G-Net is trained first and subsequently converted into a high-dimensional binary network, without optimizing the resulting discrete model for the downstream task. Its approximation guarantees benefit from large embedding dimensions. AngularQ instead performs quantization-aware training directly on the discrete forward operator, starting from pretrained full-precision weights and updating the continuous latent parameters using geometry-aware surrogate gradients. Our framework is particularly effective with N≈pN\approx p, where the representation uses only one quantized coordinate per original dimension, rather than relying on large embedding expansions.

On a G-Net-inspired control.

We do not believe that a direct G-Net-inspired control would constitute a meaningful or scientifically well-controlled comparison under our experimental setting. G-Net requires a specialized continuous network architecture that is trained before being converted into a high-dimensional binary representation. In contrast, AngularQ operates directly on standard pretrained Transformer architectures through quantization-aware training. Reproducing G-Net would therefore require changing the underlying architecture, training procedure, and representation dimensions simultaneously, making it impossible to isolate the effect of the proposed optimization method. Conversely, simply applying random-sign embeddings to a pretrained Transformer without adopting G-Net's architectural construction would not constitute a faithful implementation of G-Net. Consequently, we believe that comparisons against established QAT methods operating under the same architecture, data, and training budgets provide a substantially more appropriate and informative evaluation of AngularQ.

In summary, G-Net studies how to represent a continuous network through high-dimensional binary embeddings, whereas AngularQ studies how to train low-bit networks directly, including in regimes where the embedding dimension is comparable to the original dimension.

Dear Reviewer pujo,

As the discussion period is nearing its end, we wanted to kindly follow up and ask whether our rebuttal and subsequent paper updates have addressed your concerns. If any questions or concerns remain, we would be grateful for the opportunity to clarify them. If our responses have affected your assessment of the paper, we would also appreciate it if this could be reflected in your final evaluation. Thank you again for your time and consideration.

-Authors

Thank you for the detailed response, which has addressed my major concerns. However, QAT is already a computationally expensive quantization approach, and the proposed method further reduces training throughput, introducing substantial additional computational costs. Nevertheless, I acknowledge that this work offers a valuable and promising perspective on improving the optimization process of QAT. Balancing its methodological contributions against the practical efficiency concerns, I have decided to maintain my score of 3.

Reviewer_xhyKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AngularQ is a QAT framework for 1-3 bit weights and activations in LLMs. It parameterizes binary codes as sign(Rw) with a random embedding R, and uses Grothendieck identities to get a smooth “norm times angle” population form, whose gradient serves as the backward surrogate. It then proposes a boundary-aware sampled gradient, averaging only along the sign-discontinuous normal direction, plus a GPU-friendly unified sampled gradient that extends to multibit quantizers. Theory covers embedding variance, surrogate comparisons, and a teacher-student recovery guarantee. It achieves higher acc than QUEST on 1,2,3-bit settings.

Critical strengths

A principled radial/angular view of quantized inner products, with boundary-conditional smoothing that extends naturally to multibit. Applied Variance reduction for Haar and Walsh-Hadamard, surrogate comparison thresholds, and exact recovery plus near-stationary stability in a teacher-student setting. Honest reporting: ParetoQ wins in some weight-only cases, there is an overlap-sensitivity analysis, a QuEST port fidelity check, and three paired seeds. It shows large W1A1 gains, and demonstrates a practical finding on how to spend bits.

Critical weaknesses

Efficiency is unverified. The main experiments use dense Haar frames, so Rx is a dense N by p float matmul. Storage excludes R, and there are no latency, FLOPs, or kernel numbers. Walsh-Hadamard is only tested at 140M with a single seed. The comparison is between whole operators (different quantizers, transforms, scales). There is no ablation holding the AngularQ forward fixed while swapping STE, trust-mask, population gradient, the unified sampled gradient, and the boundary-aware sampled gradient. The boundary-aware gradient is only used in the last 9,095 steps at 1.7B. Surrogate “tightness” is shown only on a toy plot (Fig. 1). Baseline tuning is unequal and the configs look inconsistent. AngularQ W1A1 gets validation-guided hold/decay and terminal decay, while QuEST gets standard cosine. LRs in Table 5 and Table 16 appear to disagree (e.g., 0.6B QuEST, 1.7B AngularQ W1A1). QuEST barely improves from 140M to 0.6B, which suggests under-tuning at scale and makes the 49-55% gap suspect. The “unbiased” claim concerns a smooth population function averaged over R, but in training R is fixed and correlated with w. Theorems 7.1-7.3 only favor the unified sampled gradient when |rho| is small (below 0.57 / 0.32), and rho is never measured. Theorem 2.2 is single-layer, assumes the data are independent of G, and guarantees nothing about reaching stationarity. Theorem 2.1 is the existing Grothendieck identity from G-Net, so the novelty boundary needs to be clearer

Questions for authors

What is the empirical distribution of rho during training, and what is the cosine similarity between the surrogate and the true smoothed gradient? Please explain the LR mismatches between Table 5 and Table 16. Does QuEST get the same hold/decay tuning, and how large is the gap then? What are the real inference FLOPs and latency for dense-Haar W1A1? Does Walsh-Hadamard keep the advantage at 0.6B/1.7B? Does the FP gap close with more training, and are there from-scratch results? Are there downstream-task results? What is new in Theorem 2.1 relative to G-Net? Can Theorem 2.2 extend to larger N/p or multiple layers?

4 comments
AuthorsOpenReview ↗

Response To Reviewer xhyK (Part 1)

We thank the reviewer for the comments and suggestions. Please find below our detailed response (due to character limit it is split into 3 parts). The manuscript has also been updated, with the changes highlighted in a contrasting color.

R1-01. Practical efficiency, dense frames and omitted storage

What are the real inference FLOPs and latency for dense-Haar W1A1?

A dense N-by-p frame adds approximately 2Np floating-point operations per transformed input. Table 5 measures complete dense-frame QAT; Table 6 evaluates a packed randomized-Hadamard runtime with fast full-width transforms. Sharing requires only three FP32 frames at 0.6B (56 MiB), while the 1.7B structured prototype needs 8.56 KiB of transform state. Resident model storage falls by 4.28×. Under a common 4 GiB allocator cap, packing supports batch 16 instead of batch 4 at a 512-token prompt, delivering 2.99× higher aggregate throughput; the corresponding gain with a 2K prompt is 2.83×. This demonstrates additional serving capacity under constrained memory. It does not establish a same-batch latency advantage.

R1-02. Does Hadamard preserve the advantage beyond 140M?

Does Walsh-Hadamard keep the advantage at 0.6B/1.7B?

Yes at 0.6B: the new W3A3 run achieves PPL 15.2838, compared with 15.2903 for Haar, using the same initialization, codebooks, data order, learning-rate trajectory and 30,095 updates (Table 4). This extends the earlier 140M result, where randomized Hadamard achieves PPL 46.5671 versus 46.6998 for Haar. The trained structured-frame accuracy evidence therefore covers both 140M and 0.6B; the 1.7B runtime measurements remain a hardware prototype.

R1-03. Forward-matched backward ablations

There is no ablation holding the AngularQ forward fixed while swapping STE, trust-mask, population gradient, the unified sampled gradient, and the boundary-aware sampled gradient.

We added a 140M experiment that fixes the hard forward, initialization, frame, normalization, codebook, data order and training schedule, and changes only the backward rule. Unified AngularQ reduces PPL from 50.7504 to 46.6715, an 8.04% improvement (Table 1). The comparator adapts a trust-masked STE to AngularQ's forward. This directly tests the practical backward choice; the complete AngularQ–QuEST comparisons test the full methods. The added training experiment compares these two rules, with population and conditional references examined through the gradient diagnostics.

R1-04. LR mismatch, QuEST scaling and equal tuning

Please explain the LR mismatches between Table 5 and Table 16. Does QuEST get the same hold/decay tuning, and how large is the gap then?

The LR discrepancy arose because Tables 5 and 16 describe different phases: the selected main-training rate and the terminal-phase launch setting. We will label these explicitly and provide the phase-resolved trajectories. More importantly, we completed a new 74,095-update QuEST run at the 0.6B sweep's preferred LR, 4e-4. Its PPL improves from 56.56 to 51.86; AngularQ still achieves 25.29, or 51.24% lower PPL (Table 2). QuEST uses its own schedule; an identical hold/decay-policy comparison has not been performed.

R1-05. rho distribution and cosine against a reference gradient

What is the empirical distribution of rho during training, and what is the cosine similarity between the surrogate and the true smoothed gradient?

We measured 73,728 activation–weight pairs across 72 layer/checkpoint samples. At the three binary 140M checkpoints, 99.82–99.85% of pairs satisfy |ρ| < 0.3157, the Haar/Walsh comparison threshold; all satisfy |ρ| < 0.5741, the Gaussian/Rademacher threshold. Thus the small-correlation regime is prevalent in the examined trained states. Median input/weight gradient cosines against the population reference are 0.702–0.753 / approximately 0.798 for binary and 0.977 / 0.982 for multibit. These are operator-level gradients; the theorem's conditional-estimator comparison and population-gradient alignment are distinct diagnostics.

R1-06. Fixed frames correlated with trained weights

The “unbiased” claim concerns a smooth population function averaged over R, but in training R is fixed and correlated with w.

The supplied AngularQ gradient retains the realized frame and forward codes; it is not obtained by replacing them with a population average. The population derivative supplies the angular sensitivity used in that construction. The unbiasedness statements average over frames for fixed operands, rather than asserting an unbiased optimization trajectory after weights adapt to a fixed frame. We will make this distinction explicit. The trained-checkpoint correlation and gradient measurements in R1-05 provide empirical support in the regime actually encountered during training.

AuthorsOpenReview ↗

Response To Reviewer xhyK (Part 2)

R1-07. More training, closing the FP gap and from-scratch results

Does the FP gap close with more training, and are there from-scratch results?

The paper's continued-training curves show recovery during QAT, but do not establish that the FP gap vanishes. Our additional full-budget 0.6B QuEST run confirms that the AngularQ advantage persists after the stronger LR choice: PPL 25.29 versus 51.86 after 74,095 updates. Our LLM results concern adaptation from FP checkpoints; we do not claim a new from-scratch LLM result.

R1-08. Downstream and reasoning tasks

Are there downstream-task results?

Yes. Table 3 adds zero-shot downstream evaluations of the paper checkpoints at both 0.6B and 1.7B. At 1.7B, AngularQ exceeds QuEST on all four tasks at both precisions. W3A3 improves MMLU-Redux from 21.48% to 37.70% and HellaSwag from 51.56% to 58.01%. At 0.6B, W3A3 wins all four tasks and W1A1 wins three. These results show that the language-modeling advantage transfers to knowledge and commonsense tasks; the table also shows the remaining distance to FP.

R1-09. Theorem 2.1 and G-Net novelty

What is new in Theorem 2.1 relative to G-Net?

The two-sided Grothendieck identity is a classical result, also known as Sheppard's formula, and we do not claim it as a new identity. Theorem 2.1 presents a more general and unified matrix formulation of both the one-sided and two-sided identities. In particular, the result is not restricted to Gaussian embeddings but holds for arbitrary rotationally invariant row distributions, with the appropriate distribution-dependent constant in the one-sided case. Importantly, the rows of the embedding matrix need not be independent, it suffices that they share the same rotationally invariant marginal distribution. This formulation accommodates structured embeddings with dependent rows, including Haar--Stiefel constructions, and provides a common foundation for both weight-only and joint weight--activation quantization.

More importantly, Theorem 2.1 serves as the mathematical starting point for a fundamentally different framework from G-Net. Whereas G-Net uses the Grothendieck identity to study hyper-dimensional binary representations and their approximation properties, AngularQ leverages the resulting smooth population geometry to develop gradient surrogates for training through discrete quantized operations. In particular, our contributions lie in the radial--angular formulation, the boundary-aware conditional smoothing that preserves the realized embedding geometry, the computationally efficient unified sampled gradients, and their extensions and theoretical guarantees. Thus, the central novelty of AngularQ is not the classical identity itself, but the new optimization framework developed from it.

AuthorsOpenReview ↗

Response To Reviewer xhyK (Part 3)

R1-10. Extension of the teacher–student result

Can Theorem 2.2 extend to larger N/p or multiple layers?

Larger embedding ratios.

Increasing N/pN/p is, in fact, favorable from the perspective of approximating the smooth population geometry. As our variance analysis illustrates, increasing the embedding dimension improves the concentration of random-embedding estimators around their population expectations. This connection can also be established directly for the proposed unified sampled gradient. In particular, consider the two-sided Gaussian construction with fixed w≠0w\neq 0, x∈Sp−1x\in\mathbb{S}^{p-1}, and ∣ρw,x∣<1|\rho_{w,x}|<1. For ψ(ρ)=2πarcsin⁡(ρ)\psi(\rho)=\frac{2}{\pi}\arcsin(\rho) and κ=2/π\kappa=\sqrt{2/\pi}, using the radial--angular decomposition and independence of the Gaussian embedding rows, we obtain the exact mean-squared deviation

\mathbb{E}&#95;G\left[\left\\|\widehat{\nabla_w}\mathcal{Q}&#95;{II}(w,x)-\nabla_w\mathbb{E}&#95;G[\mathcal{Q}&#95;{II}(w,x)]\right\\|_2^2\right]=\frac{1}{N}\left[1-\psi(\rho&#95;{w,x})^2+\frac{\psi'(\rho&#95;{w,x})^2}{\kappa^2}\left(p-1-\kappa^2(1-\rho&#95;{w,x}^2)\right)\right].

Consequently, for correlations bounded away from ±1\pm1, this deviation scales as O(p/N)\mathcal{O}(p/N). Thus, as N/pN/p increases, the sampled backward approaches the gradient of the smooth angular population operator. This provides a quantitative connection to continuous optimization in the large-embedding regime, for fixed operands independently of the random embedding.

The main motivation behind Theorem 2.2, however, is to establish a stronger, genuinely discrete recovery statement in the more challenging regime where NN is comparable to pp. Indeed, when N≈pN\approx p, the preceding mean-squared gradient deviation need not vanish, so concentration alone does not explain why stationary points should correspond to correct discrete representations. Theorem 2.2 instead establishes that, under its stated assumptions, a vanishing unified sampled gradient certifies exact recovery of the teacher's binary representation,

sign⁡(Gw)=sign⁡(Gw⋆),\operatorname{sign}(Gw)=\operatorname{sign}(Gw^\star),

with high probability. Importantly, this guarantee does not require N/p→∞N/p\to\infty and accommodates N≈pN\approx p, corresponding to only one binary coordinate per original weight coordinate. The theorem does not inherently impose N≤pN\le p either, although its sample-size requirement becomes more demanding as NN increases. Therefore, while larger embeddings improve concentration toward the continuous population model, our main theoretical interest is to show that meaningful exact-recovery guarantees are possible even with a limited binary representation budget, without requiring the sampled gradient to be an arbitrarily accurate approximation of its population counterpart.

Extension to multiple layers.

Extending such a stationarity-to-recovery guarantee to jointly trained multilayer networks is substantially more challenging. Even for conventional full-precision networks, the nonconvex composition of layers, parameter non-identifiability, hidden-layer symmetries, and the possibility of vanishing or cancelling backpropagated gradients make it difficult to infer recovery of individual layers from stationarity of the overall training objective. Quantization introduces additional complications, since intermediate activations depend on preceding quantized layers and their realized embeddings, leading to dependencies that are absent from the single-layer analysis.

Our objective in Theorem 2.2 is therefore to isolate and rigorously characterize the fundamental relationship between the proposed gradient surrogate and recovery of the underlying discrete representation. The teacher--student formulation provides a well-defined reference for comparing a quantized interaction against a planted target. This suggests a natural starting point for layerwise analysis: if reference intermediate activations are available, one can formulate a separate teacher--student recovery problem for each layer and investigate whether the corresponding assumptions hold. Such an approach, however, differs from establishing recovery under joint end-to-end optimization, where intermediate representations are coupled and the required independence and identifiability conditions may fail. We therefore view the single-layer result as a foundational recovery guarantee, rather than claiming that stationarity of an arbitrary deep quantized network implies recovery of all its constituent layers.

Dear Reviewer xhyK,

As the discussion period is nearing its end, we wanted to kindly follow up and ask whether our rebuttal and subsequent paper updates have addressed your concerns. If any questions or concerns remain, we would be grateful for the opportunity to clarify them. If our responses have affected your assessment of the paper, we would also appreciate it if this could be reflected in your final evaluation. Thank you again for your time and consideration.

-Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overall Response (Continued)

Table 5. Measured QAT implementation cost at 0.6B

MethodSeconds/updateTokens/sPeak allocated GiB
FP0.382171,54948.97
AngularQ W1A10.68695,50461.16
QuEST W1A10.532123,28949.10

All methods use the same NVIDIA B200, BF16 autocast, effective batch 32 and 2,048-token context. Medians cover 20 complete optimizer updates after warmup. Measurements include the dense frames and backward computation. In the 0.6B implementation, frame sharing requires only three distinct FP32 frames, totaling 56 MiB. Packed low-bit inference requires a dedicated runtime; these training measurements do not establish deployment speedups.

Table 6. Packed structured inference at 1.7B: hardware prototype

MeasurementBF16 FPPacked W1A1Benefit
Resident model storage (GiB)3.2090.7504.28× smaller
Tokens/s, 4 GiB cap, 512-token prompt1,334 (batch 4)3,987 (batch 16)2.99× higher
Tokens/s, 4 GiB cap, 2,048-token prompt410 (batch 1)1,161 (batch 4)2.83× higher
Tokens/s, 6 GiB cap, 512-token prompt4,968 (batch 16)6,050 (batch 32)1.22× higher
Tokens/s, 6 GiB cap, 2,048-token prompt1,258 (batch 4)1,860 (batch 8)1.48× higher

Measurements use the same B200 with equal PyTorch allocator caps, fresh processes and 128 generated tokens per sequence; they are not timings on physical 4/6 GiB GPUs. Each row uses the largest passing batch on the tested grid, with allocation failure verified at the next batch. Storage includes norms, embedding/head and transform state; randomized-Hadamard state requires only 8.56 KiB. Embedding/head and KV cache remain BF16. FP uses its faster validated native attention. Packing improves aggregate throughput by fitting more concurrent sequences; FP remains faster at the same batch size. The 1.7B prototype applies Hadamard frames to existing binary codes and measures execution costs, without claiming trained 1.7B Hadamard accuracy. Table 4 provides the trained structured-frame accuracy evidence.

AuthorsOpenReview ↗

Overall Response

General Response to All Reviewers

We thank the reviewers for their careful assessment and constructive suggestions. The new experiments strengthen the evidence in three ways: an identical-forward comparison isolates the benefit of AngularQ's backward rule; a full-budget QuEST run at the stronger learning rate tests whether the large binary gap survives additional baseline tuning; and downstream evaluations at 0.6B and 1.7B show transfer beyond language-modeling perplexity. We also quantify trained-state gradient agreement, test structured embeddings, and measure packed inference storage and memory-limited serving capacity. The revised manuscript highlights these additions in a different color.

Additional experimental evidence

Table 1. Changing only the backward rule at 140M

Backward rule with the same AngularQ forwardNLLPPL
Adapted trust-masked STE3.9269250.7504
Unified AngularQ3.8431346.6715

AngularQ reduces perplexity by 8.04%. The two runs share initialization, frames, codebook, normalization, training order, optimizer and the complete 9,604-update schedule. This is one paired-seed comparison of backward rules; the trust-masked control is adapted to AngularQ's forward rather than being the full QuEST method.

Table 2. Stronger QuEST learning rate at 0.6B

W1A1 methodMain peak LRUpdatesNLLPPL
QuEST, reported baseline1e-474,0954.0353756.5638
QuEST, additional LR run4e-474,0953.9486151.8631
AngularQ1e-474,0953.2304425.2908

The stronger LR improves QuEST perplexity by 8.31%. AngularQ still achieves 51.24% lower perplexity, so the main binary advantage persists against this improved baseline. Scores use the paper's shared fixed-window evaluation.

Table 3. Zero-shot downstream accuracy (%)

Scale / modelMMLU-ReduxARC-EasyARC-ChallengeHellaSwag
0.6B
FP initialization38.1852.3434.9647.17
Continued FP36.2359.3836.5252.25
QuEST W1A122.1732.4224.2225.20
AngularQ W1A121.8840.8226.5629.69
QuEST W3A323.1442.1928.7141.02
AngularQ W3A325.2052.7333.5946.19
1.7B
FP initialization58.9868.3640.4359.38
Continued FP61.0472.4648.0563.67
QuEST W1A122.3637.1124.2226.95
AngularQ W1A123.3447.0725.5937.99
QuEST W3A321.4860.3534.9651.56
AngularQ W3A337.7065.0439.4558.01

All models are scored on the same 3,072-question evaluation subsets: 1,024 MMLU-Redux questions, 512 each from ARC-Easy and ARC-Challenge, and 1,024 HellaSwag questions. We use zero-shot conditional likelihood, with the same scoring rule across models for each task. AngularQ and QuEST use the paper's matched budgets within each precision; FP initialization and continued FP are reference rows. Bold marks AngularQ improvements over QuEST at the same scale and precision.

At 1.7B, AngularQ improves every measured task at both precisions. W3A3 gains 16.22 points on MMLU-Redux and 6.45 points on HellaSwag over QuEST. At 0.6B, W3A3 also wins all four tasks; W1A1 wins three, with MMLU-Redux close (21.88% versus 22.17%). W3A3 approaches FP-initialization scores on ARC and HellaSwag, while MMLU remains the larger capability gap.

Table 4. Structured embeddings at 140M and 0.6B

Embedding familyNLLPPL
Haar3.84374046.6998
Gaussian3.87310048.0913
Rademacher3.87379548.1247
Randomized Hadamard3.84089546.5671

With the same FP initialization, 9,604-update recipe and final evaluation, randomized Hadamard matches Haar accuracy. This single-seed study supports the structured-frame choice at 140M; it is an accuracy comparison, rather than a deployment-speed measurement.

0.6B W3A3 embedding familyNLLPPL
Haar2.72721615.2903
Randomized Hadamard2.72679615.2838

The new 0.6B comparison uses the same FP initialization, codebooks, training examples, learning-rate trajectory and 30,095-update budget. Randomized Hadamard again matches Haar accuracy, extending the structured-frame evidence beyond 140M. Both comparisons use one seed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.