acceptodds
Under review as a conference paper at ICLR 2027

XOR-Trellis: Ultra-Low-Complexity Dequantization and Curvature-Aware Hadamard-Free LLM Quantization

Abstract

Trellis-coded quantization enables high-dimensional compression of large language model (LLM) weights at ultra-low bit widths without the exponentially large codebooks required by conventional vector quantization. Practical deployment, however, presents two challenges: reconstructing compressed weights at sufficient parallel throughput to avoid making dequantization an inference bottleneck, and maintaining quantization accuracy without costly incoherence transformations. We address these challenges with two complementary techniques. First, we introduce an ultra-low-complexity trellis dequantizer that uses a structured, hardware-efficient state-to-value mapping while preserving diverse reconstruction choices for trellis search. Second, we reformulate discrete trellis path optimization with a curvature-aware objective that reflects model sensitivity directly in the original coordinate space. Together, these techniques enable high-quality ultra-low-bit trellis quantization with inexpensive, highly parallel runtime reconstruction and without relying on Hadamard-based incoherence processing.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Post-Training Quantization, Ultra-Low-Bit Quantization, Vector Quantization, Trellis-Coded Quantization, Hardware-Efficient Inference, Curvature-Aware Quantization

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_KJAJOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper reduces the runtime cost of trellis in application to post-training quantization (PTQ) of LLMs. It does that by designing a simplified (relative to existing SOTA that is QTIP) decoding procedure that retains compression quality while allowing for much faster and hardware-friendly execution in hardware, referred to as XOR-Trellis. On top of it, the paper revisits the design choice of incoherence processing via structured randomized rotations (RHT). It proposes a number of optimization heuristics that boost non-RHT quantization quality, boosting its quality beyond RHT-enabled methods, while saving on inference complexity. The method is validated on PTQ of Llama 3, Mistral and Qwen 3 models up to 70B parameters. XOR-Trellis is shown to consistently outperform SOTA quantization methods, such as QTIP and LLVQ.

Critical strengths

The proposed encoding scheme is genuinely novel, smart, well-justified and hardware-friendly:

Trellis encoding for LLM weights have historically seen low adoption due to their complexity. It is nice to see this complexity addressed directly and thoroughly. Splitting the decoding problem into pallet selection and small lookup is well-justified, including theory (Appendix C). The hardware implications are thoroughly discussed, going as low-level as gate logic schemes. The additional optimization heuristics are well-justified and further simplify deployment by removing now-redundant runtime activation transforms. The experimental validation is adequate bar the weakness discussed below.

Critical weaknesses

[Addressed at rebuttal] One major concern:

It is unclear whether the comparison with baselines the paper presents is fair as XOR-Trellis uses a non-conventional scaling scheme (one E5M3 scale per 32 elements), when all the baselines use channel-wise scaling. This gives XOR-Trellis effectively 0.25-bit advantage over the baselines. A fair comparison would use the same scaling scheme for all methods, as the differences between the methods are virtually orthogonal to scaling. The authors shall address this discrepancy before I can fully recommend acceptance.

Questions for authors

[All addressed at rebuttal]

Minor concerns:

  • The paper does not fully explore or justify the role of permutation control q in the encoding scheme. This lever can seemingly accept an arbitrary number of bits from 0 to 5 (as there are 24 possible permutations of 4 elements). At 2-bit q, around half of taps are spend on q, the effect of which is unclear. What happens if 1,3 or 4 bits are used for describing mixing?
  • Curvature-aware Viterbi objective is not entirely novel, as GPTVQ (van Baalen et al.) optimizes the same local objective with a similar Hessian-diagonal-based cost adjustment. It is mentioned in Related Work in Appendix but nowhere in the main body. It is only a minor concern because the interaction with the Viterbi algorithm is genuinely novel, but it should be approached more honestly.
  • Energy ordering is very similar to "Static Activation Reordering" of Egiazarian et al.. Both traverse GPTQ in an order chosen by Hessian diagonal entries in a way that commutes with scaling to make it zero-cost runtime-wise. The difference is that energy ordering commutes due to block-constrained permutations whereas static activation reordering pre-computes the scales before the GPTQ pass. The group-wise reordering is still a novel way to approach this so this is more of a missing citation type of concern.

Questions:

  • The scaling scheme is rather unconventional. Why did you choose 32 if NVFP4 has group size of 16, which also matches T_x of your algorithm, reducing inter-block scaling dependencies?
  • It seems to me that this dichotomy of RHT/no-RHT is not as clear because of the existence of micro-rotated formats (Egiazarian et al.) that perform RHT only in small groups which has large implications for both accuracy impact and runtime implications. Is your quantization methods compatible with micro-rotations? How would you expect it to perform?

Editorial:

  • Section 4.6 title is initial caps, against the style recommendations and all the other sections.
4 comments

Q1. XOR-Trellis uses a non-conventional scaling scheme (one E5M3 scale per 32 elements)...

We thank the reviewer for raising this concern. We agree that the original comparison did not sufficiently isolate the contribution of E5M3/G32 scaling. We therefore conducted a matched-scaling ablation comparing QTIP-3INST and XOR-Trellis under both channel-wise and E5M3/G32 scaling, with and without RHT.

MethodRHTScalingWT2 ↓C4 ↓
QTIP-3INSTYesChannel-wise24.281130.4411
QTIP-3INSTYesE5M3/G3222.919728.9948
QTIP-3INSTNoE5M3/G3267.782665.3223
QTIP-3INSTNoChannel-wise12102.768766.14
XOR-TrellisYesChannel-wise25.071530.8098
XOR-TrellisYesE5M3/G3222.886528.9469
XOR-TrellisNoE5M3/G3221.378727.1569
XOR-TrellisNoChannel-wise27.188531.3113
XOR-TrellisNoE5M3/G51224.103530.1482

The results confirm that E5M3/G32 scaling benefits both methods, improving perplexity when RHT is enabled. However, scaling alone cannot explain the rotation-free performance of XOR-Trellis. Under identical E5M3/G32 scaling, removing RHT causes QTIP-3INST perplexity to deteriorate substantially (22.92/28.99 → 67.78/65.32), whereas XOR-Trellis achieves 21.38/27.16 without RHT.

Importantly, XOR-Trellis remains stable even with channel-wise scaling and no RHT (27.19/31.31), while QTIP-3INST collapses under the same conditions. XOR-Trellis for a group size of 512 (E5M3/G512) further improves the PPL to 24.1/30.14, almost at the same BPW as channel-wise scaling. Thus, fine-grained scaling helps accommodate local amplitude variation in the native basis, but is not sufficient by itself to eliminate RHT. These results demonstrate that the improved rotation-free behavior is not simply attributable to finer-grained scaling; rather, the XOR-Trellis reconstruction design and curvature-aware optimization are the primary drivers of its rotation-free advantage, beyond the benefits of additional scale metadata.

We also refer the reviewer to the matched bit-per-weight (BPW) comparisons in our response to Reviewer tDQk (Q2), which cover three different models. At the same effective BPW, rotation-free XOR-Trellis achieves better or comparable accuracy than QTIP-3INST with RHT across all three models. Combined with our other key contribution—an ultra-low-complexity XOR-Trellis decoder—this translates into up to 47% higher token generation throughput compared with QTIP HYB, as detailed in our response to Reviewer tDQk (Q3). These results further demonstrate that the benefits of XOR-Trellis extend beyond the choice of scaling scheme, enabling both accurate Hadamard-free quantization and more efficient inference.

We will revise the paper to report the 2.25-bpw effective rate explicitly and include both matched-scaling and matched-rate comparisons.

Q2. The paper does not fully explore or justify the role of permutation control q in the encoding scheme...

We agree that the role of permutation control (q) was insufficiently justified. In our decoder, q is not stored per weight; it is deterministically derived from the trellis history and controls how the four local reconstruction values are assigned to the four outgoing branches.

We therefore conducted a matched ablation on Llama-3.2-1B-Instruct, keeping the model, calibration data, Hessians, palettes, scaling, D-weighted Viterbi, energy ordering, and nominal 2.25-bpw rate fixed.

ControlPermutationsWT2 PPL ↓C4 PPL ↓
No q124.937630.3162
One bit: q0 only223.754130.1570
One bit: q1 only221.319627.1494
Original two-bit q421.380627.1560
Three-bit control821.337627.3519
Four-bit control1621.041327.2583
Five-bit control2420.533426.9140
No q, full state-bit coverage¹123.988530.0805

Relative to removing permutation control, the original two-bit q reduces WT2/C4 perplexity by 14.3%/10.4%, demonstrating the importance of history-dependent branch assignment. Interestingly, the two control bits contribute unequally: q_1 alone achieves comparable perplexity to the original two-bit design, whereas q_0 alone performs substantially worse. Increasing permutation diversity to 8 or 16 choices provides mixed improvements, while the full 24-permutation configuration achieves the best accuracy.

These results demonstrate that permutation control is beneficial, but its effectiveness depends on the specific history-dependent branch assignments rather than simply the number of available permutations. The original two-bit design represents a low-complexity operating point that captures most of the accuracy benefit using simple XOR logic without additional stored bits. We will clarify this accuracy–complexity tradeoff in the revised paper.

Q3. Curvature-aware Viterbi objective is ...

We thank the reviewer for pointing this out. We agree that Hessian-weighted reconstruction objectives are not novel by themselves. GPTVQ, for example, incorporates Hessian-aware distortion into vector quantization and codebook optimization.

Our contribution is specifically to incorporate LDL-diagonal sensitivity into the Viterbi branch metric, making trellis-path selection curvature-aware in the native, unrotated basis. This addresses the coordinate anisotropy that RHT would otherwise help mitigate, enabling accurate trellis quantization without Hadamard rotation.

We will revise the main text to cite GPTVQ alongside the curvature-aware objective and clarify that the novelty lies in integrating Hessian-aware weighting into trellis-based Viterbi optimization for rotation-free quantization, rather than in Hessian weighting itself.

Q4. Energy ordering is very similar to "Static Activation Reordering" of Egiazarian et al.....

We thank the reviewer for pointing out this relevant prior work. We agree that the general idea of ordering GPTQ updates based on Hessian-diagonal importance is related to existing activation-ordering methods, including MR-GPTQ’s Static Activation Reordering.

The key distinction lies in how compatibility with the runtime scaling structure is maintained. MR-GPTQ computes and fixes the quantization scales in the original column order, then applies Hessian-based column reordering during GPTQ before restoring the original order. Our energy ordering instead permutes entire scale-aligned groups according to their aggregate Hessian-diagonal energy, preserving each group’s membership and internal order. We recompute the LDL factorization in the reordered coordinates and restore the original layout before export, requiring no additional inference-time permutation.

We will cite MR-GPTQ in the main text and clarify that we do not claim Hessian-based ordering itself as novel. Our energy ordering is a scale-group-preserving adaptation tailored to trellis LDLQ: it reorders entire scale-aligned groups while preserving their membership and internal order, then restores the original layout before export, requiring no additional inference-time permutation.

Q5. The scaling scheme is rather unconventional. Why did you choose 32 if NVFP4 has group size of 16, which also matches T_x of your algorithm, reducing inter-block scaling dependencies?

We thank the reviewer for this observation. We agree that a group size of 16 aligns naturally with our 16-wide trellis/LDL blocks. However, group size 32 is not required by the XOR-Trellis decoder; rather, it represents an accuracy–storage tradeoff and follows the group size adopted by the original OCP Microscaling (MX) specification, in contrast to the 16-element scaling used by NVFP4. With a two-bit trellis stream and one eight-bit scale per group, the nominal storage rate is 2+8/G bits per weight. Thus, G32 halves the scale overhead relative to G16, giving 2.25 versus 2.50 bpw, excluding shared metadata and unquantized parameters.

Scale-group size and trellis-tile dimensions are separate parameters. In our implementation, each scale covers 32 consecutive input features within one output row, spanning two adjacent 16-column tile segments. Both segments read the same fixed scale independently; this sharing does not require communication between their trellis decoders. Nevertheless, the reviewer is correct that group size 16 would align scale boundaries with these segments and reduce their coupling during scale selection, potentially improving reconstruction accuracy at additional storage cost. We do not claim that group size 32 is optimal; a matched group-size ablation is needed to establish that tradeoff.

We conducted a matched group-size ablation on Llama-3.2-1B-Instruct, keeping the decoder, Hessians, calibration samples and seed, 16×16 trellis tiles, and LDL block size fixed. Both configurations use no RHT, D-weighted Viterbi, no energy ordering or fine-tuning, and identical WT2/C4 evaluation sequences at context length 8192.

Scale Group SizeTrellis + Scale BPWWT2 ↓C4 ↓
162.5018.642119.2161
322.2519.306920.5024

Reducing the scale-group size to 16 improves WT2 and C4 perplexity by 3.44% and 6.27%, respectively, at an additional 0.25 bpw. Therefore, we do not claim G32 is accuracy-optimal; rather, it provides a lower-storage operating point while retaining competitive accuracy.

For G32, adjacent 16-wide trellis blocks share a scale selected before LDL feedback, without requiring joint scale optimization during Viterbi search. G16 eliminates this scale sharing, although cross-block LDL error feedback remains. We will clarify this distinction and the difference between our G32 representation and standard NVFP4 in the revised paper.

Q6. It seems to me that this dichotomy of RHT/no-RHT is not as clear because of the existence of micro-rotated formats (Egiazarian et al.) that perform RHT only in small groups which has large implications for both accuracy impact and runtime implications. Is your quantization methods compatible with micro-rotations? How would you expect it to perform?

We agree that rotation should be viewed as a spectrum rather than a binary choice. XOR-Trellis is compatible with micro-rotations: for a block-diagonal orthogonal transform RR, we quantize WRWR using the transformed Hessian R⊤HRR^\top HR and apply the corresponding activation transform xRxR online. The XOR decoder and trellis structure remain unchanged. To examine this compatibility, we conducted a matched ten-configuration ablation on Llama-3.2-1B-Instruct, comparing five rotation sizes with Euclidean or D-weighted Viterbi. All configurations use the same calibration-derived Hessians (384 RedPajama sequences, context length 2048), G32 scaling, and no energy ordering or fine-tuning. Evaluation uses identical WT2/C4 token streams at a context length of 8192.

Rotation SizeRHT?Euclidean: WT2 / C4 ↓D-weighting: WT2 / C4 ↓
No rotationNo RHT25.0407 / 24.407419.3069 / 20.5024
16Micro-RHT24.1231 / 23.048322.1179 / 21.8814
32Micro-RHT19.4775 / 19.683819.2762 / 19.1291
128Micro-RHT18.7627 / 19.139718.3894 / 19.2306
Full input widthFull Hadamard18.3088 / 18.949818.3195 / 19.0948

The results confirm that XOR-Trellis supports micro-rotations, although accuracy does not improve monotonically with rotation size. A size-32 rotation improves C4 perplexity over rotation-free D-weighting while maintaining comparable WT2 accuracy, and size 128 improves both. Conversely, size 16 degrades both metrics. The benefit of D-weighting is largest without rotation and generally diminishes as rotation size increases, consistent with rotations reducing coordinate anisotropy. Micro-rotations partially reduce the coordinate anisotropy addressed by D-weighting, leaving room for additional improvements. As rotation becomes more extensive, the incremental benefit of D-weighting generally diminishes, consistent with our motivation for using it primarily in the rotation-free setting.

We will clarify that our contribution enables accurate fully rotation-free quantization, rather than claiming that all rotations are undesirable. Here, full-width rotation refers to an input-only normalized Hadamard transform, rather than QTIP’s two-sided randomized RHT. These experiments establish accuracy compatibility with micro-rotations but do not evaluate their runtime overhead or reproduce MR-GPTQ’s optimized implementation.

Q7. Section 4.6 title is initial caps, against the style recommendations and all the other sections.

Thank you for catching this inconsistency. We will change the Section 4.6 heading to sentence case to align with the venue’s style recommendations and the formatting used throughout the paper.

Reviewer_KJAJOpenReview ↗

Concerns resolved, score raised

Thank you for the follow-up.

My concerns have been resolved. I particularly appreciate the the Micro-RHT discussion because of how it encapsulates the interaction between D-weighting and whitening transforms.

I have raised my score accordingly.

Reviewer_Xab8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper targets two obstacles in ultra-low-bit (~2.25 bit/weight) trellis-coded quantization (TCQ) of LLM weights.

(1) XOR-Trellis decoder. A hardware/algorithm co-designed FP4 dequantizer whose state-to-value mapping is computed from a few masked XOR parity reductions plus a small fixed 4×4 FP4 palette lookup (Eq. 2), avoiding both a 2^16-entry state-to-value table and QTIP-3INST-style per-weight arithmetic. The authors report ~14 XOR2 gates per reconstruction and a 512-weight/cycle synthesis at 1,917 µm² (N3P).

(2) Curvature-aware, rotation-free optimization. The authors argue that the benefit of randomized Hadamard transforms (RHT) partly stems from masking a mismatch between the model-aware second-order objective (e^T H e) and the isotropic Euclidean metric used by Viterbi search. Using H = L D L^T, they weight each branch distortion by D_ii (Eq. 6), restoring residual curvature to path selection at negligible cost and with no runtime change. An auxiliary "energy ordering" of feature blocks by tr(H_Bb) is also proposed.

On Llama-3.1-8B, the rotation-free curvature-aware method reaches WT2 PPL 8.00, beating unrotated TCQ (8.69) and the RHT-based QTIP baseline (8.48). While the ideas are appealing, the experimental support is insufficient to establish the paper's central claims (see Weaknesses).

Critical strengths
  1. Interesting insight. The paper argues RHT works partly by hiding the mismatch between the Euclidean Viterbi metric and the curvature-aware objective.

  2. D-weighting is nearly free. It only rescales branch costs offline. Viterbi complexity and the decoder stay unchanged.

  3. Decoder is easy to reimplement. Palettes, XOR masks, and a gate-level design are all given.

Critical weaknesses
  1. No measured speedup. Sec. 5.4 reports only gate count (~14 XOR2) and area (1,917 µm²) — no end-to-end throughput or latency on real hardware.

  2. Core bandwidth claim unverified. The paper's main motivation is "dequantization must not bottleneck bandwidth," yet no experiment demonstrates this throughput advantage.

  3. Rotation-free claim relies on one model family. "Removing Hadamard" is validated essentially on Llama-3.1 only in the paper; cross-family results are deferred to the appendix.

  4. Contribution narrower than the title. Eq. 7 shows D-weighting must be disabled once RHT is applied — it is a fix for the rotation-free regime, not a general improvement.

  5. No mask sensitivity study. The decoder hinges on one XOR mask set, with no ablation over alternatives.

  6. Heavy calibration dependence, unanalyzed. H is estimated from 384 RedPajama sequences (seed 0), but there is no study of how calibration size, distribution, or seed affects results.

  7. D-weighting robustness unclear. Since D-weighting replys entirely on the H estimate, its benefit may degrade faster than RHT under calibration noise — untested.

Questions for authors
  1. Can you provide end-to-end throughput/latency measurements on real hardware (not only gate count and area) to substantiate the "not a bandwidth bottleneck" claim, together with the RTL/synthesis flow behind the 1,917 µm² result?
  2. How sensitive are results to the specific XOR mask set? Do other valid masks give comparable PPL, or is the reported mask tuned?
  3. How do calibration size, data distribution, and seed affect the results?
  4. In Table 2, YAQA-3INST without rotation collapses while XOR-Trellis stays stable — is this stability attributable to the palette geometry, the D-weighting, or both? Please ablate.

Thank you for your reply; I have decided to increase my score.

3 comments

Q1. Speedup measure.

We thank the reviewer for raising this concern. We have added measured end-to-end throughput and latency results on real hardware; please see our response to Reviewer tDQk, Q3. Inference speed., where we report and discuss these runtime measurements in detail.

Q2. Core bandwidth claim unverified.

XOR-Trellis is designed to minimize the computational overhead associated with decoding compressed weights. Furthermore, XOR-Trellis eliminates the need for the Random Hadamard Transform (RHT), reducing the additional computation and data movement associated with weight reconstruction.

The following table reports LLaMA-2 7B decode-only throughput (tokens/s) on an NVIDIA RTX A5000 GPU at batch size 1 for different context lengths:

Context LengthFP16QTIP HYBHYB (No RHT)XOR-Trellis (No RHT)
12846.65135.38161.25199.31
204840.6093.97106.11121.68

XOR-Trellis achieves 1.47× and 1.29× higher throughput than QTIP HYB for context lengths of 128 and 2048, respectively. Even when RHT is removed from HYB, XOR-Trellis maintains throughput improvements of 23.6% and 14.7%, demonstrating that its low-complexity decoding provides additional benefits beyond eliminating RHT.

For batch-size-1 decoding, reducing the computational overhead of weight reconstruction enables XOR-Trellis to process compressed weights more efficiently and sustain higher token throughput. By minimizing decoding overhead and eliminating RHT, XOR-Trellis enables more effective exploitation of the available memory bandwidth and further realizes the memory-bandwidth advantage of 2-bit compression.

Q3. Rotation-free claim relies on one model family. We agree that the main text places disproportionate emphasis on Llama-3.1, while much of the cross-family evidence is currently deferred to the appendix. However, the rotation-free result is not limited to Llama-3.1. We evaluate XOR-Trellis without RHT across Llama-2, Llama-3.2, Mistral, and Qwen models. Representative results include:

ModelQTIP-3INST + RHTXOR-Trellis, no RHTComparison
Llama-2-13B5.64 WT25.47rotation-free improves
Mistral-7B6.04 WT25.67rotation-free improves
Qwen3-8B11.35 WT210.25rotation-free improves

The corresponding rotation-free configurations also remain competitive on downstream accuracy: Mistral-7B reaches 67.95% average zero-shot accuracy, and Qwen3-8B reaches 67.75%, compared with 66.80% and 67.52% respectively for the rotated QTIP-3INST baselines.

Thus, the evidence for removing RHT spans not only different Llama generations, such as LLaMA-2, 3.1, 3.2, and sizes (1B, 7B, 8B, and 13B), but also the Mistral and Qwen model families. In the revision, we will move a representative cross-family comparison, including Mistral and Qwen, into the main text and make the scope of the rotation-free claim explicit.

Q4. Contribution narrower than the title.

We agree that D-weighting is specifically motivated by the rotation-free regime, rather than being intended as a universal improvement when RHT is already present. This is by design, as our primary objective is to reduce the reliance on Hadamard rotation while preserving quantization accuracy, rather than to further optimize quantization in the presence of RHT.

RHT approximately homogenizes the coordinate geometry, reducing the anisotropy that D-weighted Viterbi is meant to address. When RHT is removed, the native-basis LDL diagonal can vary substantially across coordinates, and using DiiD_{ii} into the branch metric explicitly accounts for this unequal sensitivity. Once RHT has homogenized the geometry, additional D-weighting is unnecessary; this is why Eq. 7 disables it in the rotated setting. This motivates the conditional formulation in Eq. 7.

Importantly, the contribution is not simply an improvement to the Viterbi objective, but enabling accurate ultra-low-bit quantization without the runtime overhead of Hadamard rotation. At the same bit width, our rotation-free D-weighted approach achieves better or comparable accuracy than existing rotation-based methods. Although demonstrated for trellis quantization, the underlying principle of aligning discrete reconstruction decisions with model sensitivity may also benefit other low-bit quantization and compression methods. Combined with our other key contribution—an ultra-low-complexity XOR-based dequantizer— this translates into up to 47% higher inference throughput than QTIP HYB in our additional GPU experiments, illustrating the practical benefit of eliminating RHT and reducing reconstruction complexity.

Nevertheless, we will clarify that D-weighted Viterbi is a targeted mechanism for enabling high-quality Hadamard-free quantization, rather than a claim of universal improvement over rotated quantization.

Q5. No mask sensitivity study.

We thank the reviewer for identifying this missing ablation. We have now evaluated the original mask set and nine alternatives: two replacing only the palette-selection masks, two replacing only the permutation-control masks, and five replacing all four masks.

All experiments use Llama-3.2-1B-Instruct without RHT and with D-weighting-weighted Viterbi. We keep the model, QTIP Hessians, calibration procedure (384 RedPajama sequences, context 2048, seed 0), trellis geometry, scaling, and evaluation protocol fixed. WT2 and C4 are evaluated using identical archived 8192-token sequences.

Alternative masks were sampled using predefined seeds and structural constraints (four taps per parity function, full history-bit coverage, and full-rank parity mappings), without perplexity-based selection. The four reconstruction palettes, permutation family, storage rate, and nominal parity-operation budget remain unchanged.

Mask VariantWT2 ↓C4 ↓
Original18.266418.8086
Palette-selection masks 118.287518.8289
Palette-selection masks 218.610419.3286
Permutation-control masks 118.171418.3074
Permutation-control masks 218.749719.0099
All masks 117.942518.5879
All masks 218.194318.5869
All masks 318.632418.7535
All masks 417.905318.1923
All masks 519.873119.7781

Four of the nine alternative mask sets improve both WT2 and C4 perplexity over the original, with the best achieving reductions of 1.98% and 3.28%, respectively. However, mask sensitivity is non-negligible, with the worst alternative increasing perplexity by 8.80% and 5.15%. Thus, the original mask set is neither uniquely necessary nor optimal, but arbitrary mask choices are not interchangeable.

Exhaustive enumeration confirms that all ten mappings share the same payload histogram, unordered local menus, duplicate-free numeric choices, and zero one-step signed and absolute transition correlations under uniform state enumeration.

These findings suggest that the arrangement of reconstruction choices along trellis paths, rather than these aggregate statistics alone, influences quantization accuracy.

Overall, these results demonstrate that the proposed XOR-Trellis framework is not dependent on a unique mask configuration, although mask selection affects reconstruction quality. We will include the exact masks, sampling seeds, and structural validation results in the appendix.

Q6. Heavy calibration dependence.

We thank the reviewer for highlighting this missing analysis. We conducted a 15-run calibration robustness study on Llama-3.2-1B-Instruct, varying calibration-set size, calibration corpus, and sampling seed. Specifically, we evaluate RedPajama using 128, 384, and 1024 sequences, and C4 and WikiText2 training splits using 384 sequences each, with three sampling seeds per configuration.

Importantly, we recompute the input Hessians for every calibration condition rather than changing the calibration data while reusing fixed Hessians. All other settings are held fixed: XOR-Trellis with D-weighted Viterbi and energy ordering, no RHT, identical trellis geometry and damping, a fixed quantization seed, and no layerwise or end-to-end fine-tuning. Each calibration sequence contains 2048 tokens, and WT2/C4 evaluation uses the same archived 8192-token sequences in every run.

Hessian Calibration CorpusSequencesWT2 ↓C4 ↓
RedPajama12820.0245 ± 0.053420.0068 ± 0.3520
RedPajama38419.4408 ± 1.010220.1351 ± 0.4977
RedPajama102419.6214 ± 0.419419.7708 ± 0.5357
C4 training split38419.8867 ± 0.423819.8368 ± 0.5223
WikiText2 training split38417.1211 ± 0.317620.1301 ± 0.2754

Values are mean ± sample standard deviation across three calibration-sampling seeds. Overall, the results show that XOR-Trellis is not strongly dependent on the original 384-sequence, seed-0 RedPajama calibration choice. Across RedPajama calibration sizes from 128 to 1024 sequences, perplexity remains within a relatively narrow range, with no monotonic dependence on calibration size. Variation across sampling seeds is also modest in most settings. Changing the calibration corpus from RedPajama to C4 produces similarly small changes in both WT2 and C4 perplexity, and the C4-calibrated results remain comparable to RedPajama.

WikiText2 calibration gives a notably lower WT2 perplexity without a corresponding C4 improvement, suggesting an in-domain calibration advantage rather than a general sensitivity of the method. Thus, within the calibration conditions tested, the method is broadly robust to calibration-set size and sampling seed and remains stable across RedPajama and C4 distributions, while also highlighting the potential benefit of in-domain calibration. We will include this analysis in the revised paper.

Q7. D-weighting robustness unclear.

We agree that robustness to Hessian estimation noise warrants a direct comparison with RHT. We therefore conducted 27 matched PTQ runs on Llama-3.2-1B-Instruct, varying calibration size (128/384/1024 sequences), sampling seed (three seeds), and quantization configuration: rotation-free Euclidean search, rotation-free D-weighted search, and RHT with Euclidean search. All configurations use the same XOR-Trellis mapping, corresponding calibration-derived Hessian estimates, and identical evaluation sequences, without energy ordering or fine-tuning. Evaluation uses identical WT2/C4 token streams at a context length of 8192 across all runs. Results are reported as mean ± sample standard deviation across three seeds.

Calibration SizeNo RHT + D: WT2 ↓No RHT + D: C4 ↓RHT: WT2 ↓RHT: C4 ↓
12820.62 ± 0.2820.72 ± 0.3821.98 ± 1.3921.31 ± 0.55
38420.26 ± 0.8220.89 ± 0.6521.45 ± 0.4221.12 ± 0.16
102420.44 ± 0.1620.85 ± 0.4720.65 ± 0.7220.18 ± 0.67

Reducing calibration size from 1024 to 128 sequences increases mean WT2 perplexity by only 0.91% for rotation-free DD-weighting, compared with 6.44% for RHT; the corresponding C4 changes are −0.59% and +5.56%, respectively. Furthermore, D-weighting outperforms matched rotation-free Euclidean search in all nine calibration-size/seed pairs on both datasets, reducing WT2 perplexity by 23.1–35.9% and C4 perplexity by 17.0–25.6%.

These results do not support the hypothesis that D-weighting degrades faster than RHT under reduced calibration in the evaluated setting. Its accuracy benefit persists even with 128 calibration sequences. We will include this analysis in the revision.

Reference configurations across Q5–Q7. Q5 holds the downloaded RelaxML input Hessians fixed, whereas Q6 recomputes input Hessians under explicitly controlled calibration conditions. These are different Hessian estimates, so their reference perplexities are not directly interchangeable; we do not attribute the difference solely to calibration size. Within Q6’s controlled study, increasing RedPajama calibration from 128 to 1024 sequences does not produce a monotonic improvement in perplexity. Q7 reuses Q6’s Hessian estimates but disables energy ordering to examine D-weighting separately. All evaluations use identical ctx8192 token streams and scoring.

Q8. In Table 2, YAQA-3INST without rotation collapses while XOR-Trellis stays stable — is this stability attributable to the palette geometry, the (D)-weighting, or both? Please ablate.

We thank the reviewer for suggesting this ablation. We conducted a controlled experiment on Llama-3.2-1B-Instruct (context length 2048), independently evaluating the contributions of XOR palette geometry and D-weighting while keeping RHT disabled.

DecoderRHTD-weightingXOR palettesWT2 PPLC4 PPL
3INSTNoNoNo10800.936918.11
3INSTNoYesNo3012.672552.78
XOR-TrellisNoNoYes38.329541.5805
XOR-TrellisNoYesYes27.188531.3113

The results demonstrate that both components contribute, but the XOR palette geometry is the primary factor preventing catastrophic degradation without RHT. Applying D-weighting to the original 3INST decoder substantially reduces perplexity, but does not prevent model collapse. In contrast, introducing the XOR palettes without D-weighting restores usable perplexity (38.33/41.58). Adding D-weighting further improves perplexity to 27.19/31.31. Both 3INST and XOR-Trellis include channel-wise scaling for the above runs.

Thus, the no-RHT stability is not attributable to (D)-weighting alone. These results establish that rotation-free stability is primarily enabled by the palette geometry, with D-weighted Viterbi providing complementary improvements through curvature-aware path selection. We will include this ablation in the revised paper.

Reviewer_tDQkOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes an XOR-based trellis code for decompressing weights to NVFP4 using a trellis. The paper then shows that their XOR-based LDL diagonal (D) adaptation frees the model from the need for Hadamard transforms. This is an impressive thing to remove, because Hadamard transforms are an extra overhead at inference time, but prior works made them seem necessary to make the weights look more Gaussian.

Critical strengths

The XOR trellis is very simple and seems like it would enable fast decoding.

The empirical results seem pretty good for a method that doesn't use a Hadamard transform! I really want to like this paper.

Critical weaknesses

I have two categories of concerns here.

The first one is about the comparison to the baseline methods of QTIP and YAQA. The selection of context lengths is very odd ("context lengths of 4096 for Llama-3.1-8B, 2048 for Llama-3.1-8B-Instruct, and 8192 for Llama-3.1-70B-Instruct"): why not use the same context length for all experiments and the same context length as the baseline work? And the result simply does not seem to match the published results in the QTIP paper. E.g. for Llama-3.1-70B-Instruct, which is the only model for which the context length for ppl matches that used in QTIP, this paper reports that QTIP at 2 bits has a perplexity of 6.31, but the number in the QTIP paper is 5.08 (their Table 8). That reported number would beat all the other numbers in the table, including the XOR-trellis numbers. No apples-to-apples comparison with published YAQA numbers is possible because the context length is different, but all this makes the baseline comparison suspect.

Also, QTIP (and, I think, YAQA) do not use a block scale like this paper does, right? So it seems like a comparison between XOR-trellis at 2.25 bits vs QTIP at 2 bits. Is that fair?

The other concern is about a lack of timing numbers. If part of the claim is that the XOR codebook is fast, shouldn't that be evaluated empirically? I worry about the lack of timing numbers, because it seems to me that the algorithm might not be able to be fast and accurate at the same time. This is because computing directly in NVFP4 requires using the NVFP4 tensor cores, which means the whole algorithm has to be in NVFP4, including the activations! But in this case precision is lost on the activations. Otherwise there needs to be a separate conversion step from NVFP4 to some other format, and that seems slow to do on the CUDA cores.

Questions for authors

What is the timing speed for inference using this XOR trellis? How does it compare to prior work?

What precision was used for the activations for the numbers reported in the figure? If a precision other than NVFP4 was used, how were the weights raised to that precision from FP4?

3 comments

Q1. Comparison to the baseline methods of QTIP and YAQA.

Thank you for the careful review and for recognizing the simplicity of the decoder and the significance of removing online Hadamard transforms. Your comments identify several distinctions that our presentation should make more explicit.

  • For the QTIP Table 8 result of 5.08 WT2 perplexity on Llama-3.1-70B-Instruct, we would like to clarify that this number corresponds to QTIP-HYB with layerwise fine-tuning and end-to-end fine-tuning. In contrast, all results reported in our paper are obtained without layerwise or end-to-end fine-tuning. Our baseline is QTIP-3INST, which we chose because it is the closest architectural comparison to XOR-Trellis: both are lookup-free trellis decoders. QTIP-HYB instead uses a learned runtime LUT/codebook together with additional fitting and fine-tuning, placing it in a different decoder-capacity and optimization regime. Therefore, the 5.08 result from QTIP Table 8 should not be compared directly with our 6.31 QTIP-3INST result. Importantly, disabling fine-tuning was an intentional experimental choice. Our goal is to isolate the contribution of the underlying quantization and decoder architecture itself, rather than improvements obtained from subsequent model adaptation. We therefore apply the same no-layerwise-FT and no-E2E-FT setting to both XOR-Trellis and the QTIP-3INST baseline. This choice was not intended to weaken the baseline or selectively favor our method; rather, it provides a controlled comparison of the fundamental trellis representations and decoding mechanisms before any recovery from fine-tuning.
  • We also agree that the different evaluation context lengths in the original submission made the comparisons unnecessarily difficult to interpret. We therefore re-evaluated the QTIP-3INST and XOR-Trellis configurations at a common 8192-token context length on llama3.1 models: | Variant | 8B WT2 ↓ | 8B C4 ↓ | 8B-Instruct WT2 ↓ | 8B-Instruct C4 ↓ | |---|---:|---:|---:|---:| | BF16 | 5.6089 | 7.0753 | 6.4997 | 8.0225 | | QTIP-3INST + RHT | 8.0317 | 9.2678 | 9.3762 | 10.6989 | | XOR + RHT | 7.7588 | 8.9925 | 8.8950 | 10.2499 | | XOR, no RHT | 8.3031 | 9.5381 | 9.3076 | 10.5281 | | XOR + D-weighting, no RHT | 7.7039 | 8.9327 | 8.5304 | 9.9091 | | XOR + D-weighting + energy, no RHT | 7.6281 | 8.8946 | 8.6457 | 9.9348 |

These common-context results preserve the same conclusion: under the same no-FT regime, rotation-free XOR-Trellis with curvature-aware search remains competitive with or better than QTIP-3INST + RHT.

  • In preparing the rebuttal, we also realized that YAQA already reports no-fine-tuning results directly comparable to our no-FT setting, and we should have used these results more clearly in the original submission. We therefore added the corresponding common 8192-token evaluation for XOR-Trellis: | Model | YAQA-A | YAQA-B | XOR-Trellis | |---|---|---|---| | Llama-3.1-8B-Instruct | 8.79 / 10.09 | 8.39 / 9.83 | 8.04 / 9.43 | | Llama-3.2-1B-Instruct | 17.22 / 17.64 | 16.66 / 17.36 | 15.17 / 16.59 | | Llama-3.2-3B-Instruct | 13.75 / 13.56 | 13.18 / 13.10 | 11.93 / 12.57 |

Values are WT2 / C4 perplexity at context length 8192. Under this matched no-FT evaluation setting, XOR-Trellis outperforms both YAQA-A and YAQA-B on all three models and both datasets. We thank the reviewer for prompting us to revisit this comparison; the omission of this directly comparable YAQA setting in the original submission was an oversight, not an intentional choice of a less favorable baseline. Due to the limited rebuttal timeline, we were not able to complete the corresponding 8192-token evaluation for Llama-3.1-70B-Instruct, but we will include it in the revised paper. We will also clearly distinguish calibration/Hessian context length from perplexity-evaluation context length and incorporate additional comparisons into the revised paper.

We hope these additional experiments and clarifications help resolve the concerns regarding baseline fairness, evaluation context, and the comparability of our results.

Q2. Also, QTIP (and, I think, YAQA) do not use a block scale like this paper does, right? So it seems like a comparison between XOR-trellis at 2.25 bits vs QTIP at 2 bits. Is that fair?

We agree that the original G32 configuration is not rate-matched to QTIP-3INST: it uses 2 trellis bits per weight plus one 8-bit scale per 32 weights, giving an effective rate of 2.25 bpw. We will clarify the distinction between the nominal trellis-path rate and the total effective storage rate.

To address this concern, we evaluated a separate, approximately rate-matched XOR-Trellis configuration. It retains the 16-bit trellis and learned 16×4 FP8 palettes, but replaces each 8-bit group scale with a 4-bit index into 16 shared FP16 gain levels. The additional scale-index bits are offset by omitting four predetermined trellis-path bits per 256-weight tile, resulting in a payload rate of $\frac{508\ \text{path bits}+4\ \text{gain-index bits}}{256\ \text{weights}} =\mathbf{2.0000\ bpw}. $

ModelQTIP-3INST + RHT WT2 / C4XOR-Trellis no RHT WT2 / C4bit-per-weight (BPW)
Llama-2-7B6.9489 / 9.07926.91533 / 9.041392.00246
Llama-3.2-1B-Instruct24.2811 / 30.441124.6118 / 28.97662.00024
Llama-3.2-3B-Instruct19.2991 / 20.906317.7582 / 20.42532.00014

Values are WT2 / C4 perplexity. At approximately matched effective bitrate, rotation-free XOR-Trellis without RHT remains competitive with RHT-based QTIP-3INST across all three models. It achieves lower perplexity on both datasets for Llama-2-7B and Llama-3.2-3B-Instruct, while showing a small WT2 degradation but improved C4 perplexity on Llama-3.2-1B-Instruct. These results demonstrate that XOR-Trellis’s rotation-free accuracy advantage is not solely attributable to the additional 0.25 bpw of the original G32 scaling scheme.

We will include both matched-rate and matched-scaling comparisons in the revised paper. For the specific effect of groupwise scaling, we refer the reviewer to our response to Reviewer KJAJ (Q1), which provides an explicit scaling ablation for both QTIP-3INST and XOR-Trellis.

Q3. Inference speed.

We thank the reviewer for raising this concern. We agree that evaluating decoding latency is important, particularly because the practical benefit of the proposed approach depends on achieving both high quantization accuracy and efficient execution.

We have evaluated an optimized GPU implementation that uses our simplified decoding path and Tensor Cores for matrix multiplication. The implementation leverages BF16 activations and BF16 Tensor Core multiplication by converting the decoded FP4 weights to BF16, with the conversion overhead included in the measurements. We reduce decoding and data-preparation overhead through aligned compressed-data loads that reuse overlapping trellis bits, fused input preparation, and fused output processing (including reduction, scaling, and conversion). These optimizations eliminate separate GPU kernel launches and intermediate buffers, substantially reducing execution overhead across the model’s linear projections.

We compare the resulting decode-only end-to-end inference throughput (tokens/s) against an optimized QTIP HYB implementation using the same GPU execution framework, based on the optimized HYB kernels released by the QTIP authors.

NVIDIA RTX A5000 — Llama-2-7B, batch size 1

Context LengthFP16QTIP HYBHYB (No RHT)XOR-Trellis (No RHT)
12846.65135.38161.25199.31
204840.6093.97106.11121.68

NVIDIA B200 — Llama-2-7B, batch size 1

Context LengthQTIP HYBHYB (No RHT)XOR-Trellis (No RHT)
128260.75346.18460.94
2048234.61313.50403.69

On the A5000, XOR-Trellis achieves 47.2% and 29.5% higher throughput than QTIP HYB for context lengths of 128 and 2048, respectively. Importantly, even after removing RHT from HYB, XOR-Trellis maintains throughput improvements of 23.6–33.1% across the evaluated configurations.

Q4. What precision was used for the activations for the numbers reported in the figure? If a precision other than NVFP4 was used, how were the weights raised to that precision from FP4?

The reported accuracy results use weight-only quantization, with activations retained at the model’s native 16-bit precision (BF16 for our Llama-3.x checkpoints), rather than NVFP4. The trellis decoder produces an FP4 reconstruction value qiq_i, and the corresponding weight is reconstructed as w^i=sgqi\hat w_i=s_gq_i, using an E5M3 scale shared by 32 weights. For evaluation, these reconstructed weights are represented in BF16 and multiplied by BF16 activations. This dequantization does not recover discarded weight information: it represents the already-quantized reconstruction in the arithmetic format used for computation.

I do really wish there was some fine-tuning experiment here, because a lot of the gaps between PTQ methods evaporate (or even reverse) after fine-tuning is done, and we generally always want to do fine-tuning when doing PTQ in practice. But I do think there's a solid technical contribution here regardless so I am adjusting my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.