- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper reduces the runtime cost of trellis in application to post-training quantization (PTQ) of LLMs. It does that by designing a simplified (relative to existing SOTA that is QTIP) decoding procedure that retains compression quality while allowing for much faster and hardware-friendly execution in hardware, referred to as XOR-Trellis. On top of it, the paper revisits the design choice of incoherence processing via structured randomized rotations (RHT). It proposes a number of optimization heuristics that boost non-RHT quantization quality, boosting its quality beyond RHT-enabled methods, while saving on inference complexity. The method is validated on PTQ of Llama 3, Mistral and Qwen 3 models up to 70B parameters. XOR-Trellis is shown to consistently outperform SOTA quantization methods, such as QTIP and LLVQ.
Critical strengths
The proposed encoding scheme is genuinely novel, smart, well-justified and hardware-friendly:
Trellis encoding for LLM weights have historically seen low adoption due to their complexity. It is nice to see this complexity addressed directly and thoroughly. Splitting the decoding problem into pallet selection and small lookup is well-justified, including theory (Appendix C). The hardware implications are thoroughly discussed, going as low-level as gate logic schemes. The additional optimization heuristics are well-justified and further simplify deployment by removing now-redundant runtime activation transforms. The experimental validation is adequate bar the weakness discussed below.
Critical weaknesses
[Addressed at rebuttal] One major concern:
It is unclear whether the comparison with baselines the paper presents is fair as XOR-Trellis uses a non-conventional scaling scheme (one E5M3 scale per 32 elements), when all the baselines use channel-wise scaling. This gives XOR-Trellis effectively 0.25-bit advantage over the baselines. A fair comparison would use the same scaling scheme for all methods, as the differences between the methods are virtually orthogonal to scaling. The authors shall address this discrepancy before I can fully recommend acceptance.
Questions for authors
[All addressed at rebuttal]
Minor concerns:
- The paper does not fully explore or justify the role of permutation control
qin the encoding scheme. This lever can seemingly accept an arbitrary number of bits from 0 to 5 (as there are 24 possible permutations of 4 elements). At 2-bitq, around half of taps are spend onq, the effect of which is unclear. What happens if 1,3 or 4 bits are used for describing mixing? - Curvature-aware Viterbi objective is not entirely novel, as GPTVQ (van Baalen et al.) optimizes the same local objective with a similar Hessian-diagonal-based cost adjustment. It is mentioned in Related Work in Appendix but nowhere in the main body. It is only a minor concern because the interaction with the Viterbi algorithm is genuinely novel, but it should be approached more honestly.
- Energy ordering is very similar to "Static Activation Reordering" of Egiazarian et al.. Both traverse GPTQ in an order chosen by Hessian diagonal entries in a way that commutes with scaling to make it zero-cost runtime-wise. The difference is that energy ordering commutes due to block-constrained permutations whereas static activation reordering pre-computes the scales before the GPTQ pass. The group-wise reordering is still a novel way to approach this so this is more of a missing citation type of concern.
Questions:
- The scaling scheme is rather unconventional. Why did you choose 32 if NVFP4 has group size of 16, which also matches T_x of your algorithm, reducing inter-block scaling dependencies?
- It seems to me that this dichotomy of RHT/no-RHT is not as clear because of the existence of micro-rotated formats (Egiazarian et al.) that perform RHT only in small groups which has large implications for both accuracy impact and runtime implications. Is your quantization methods compatible with micro-rotations? How would you expect it to perform?
Editorial:
- Section 4.6 title is initial caps, against the style recommendations and all the other sections.
4 comments
Q3. Curvature-aware Viterbi objective is ...
We thank the reviewer for pointing this out. We agree that Hessian-weighted reconstruction objectives are not novel by themselves. GPTVQ, for example, incorporates Hessian-aware distortion into vector quantization and codebook optimization.
Our contribution is specifically to incorporate LDL-diagonal sensitivity into the Viterbi branch metric, making trellis-path selection curvature-aware in the native, unrotated basis. This addresses the coordinate anisotropy that RHT would otherwise help mitigate, enabling accurate trellis quantization without Hadamard rotation.
We will revise the main text to cite GPTVQ alongside the curvature-aware objective and clarify that the novelty lies in integrating Hessian-aware weighting into trellis-based Viterbi optimization for rotation-free quantization, rather than in Hessian weighting itself.
Q4. Energy ordering is very similar to "Static Activation Reordering" of Egiazarian et al.....
We thank the reviewer for pointing out this relevant prior work. We agree that the general idea of ordering GPTQ updates based on Hessian-diagonal importance is related to existing activation-ordering methods, including MR-GPTQ’s Static Activation Reordering.
The key distinction lies in how compatibility with the runtime scaling structure is maintained. MR-GPTQ computes and fixes the quantization scales in the original column order, then applies Hessian-based column reordering during GPTQ before restoring the original order. Our energy ordering instead permutes entire scale-aligned groups according to their aggregate Hessian-diagonal energy, preserving each group’s membership and internal order. We recompute the LDL factorization in the reordered coordinates and restore the original layout before export, requiring no additional inference-time permutation.
We will cite MR-GPTQ in the main text and clarify that we do not claim Hessian-based ordering itself as novel. Our energy ordering is a scale-group-preserving adaptation tailored to trellis LDLQ: it reorders entire scale-aligned groups while preserving their membership and internal order, then restores the original layout before export, requiring no additional inference-time permutation.
Q5. The scaling scheme is rather unconventional. Why did you choose 32 if NVFP4 has group size of 16, which also matches T_x of your algorithm, reducing inter-block scaling dependencies?
We thank the reviewer for this observation. We agree that a group size of 16 aligns naturally with our 16-wide trellis/LDL blocks. However, group size 32 is not required by the XOR-Trellis decoder; rather, it represents an accuracy–storage tradeoff and follows the group size adopted by the original OCP Microscaling (MX) specification, in contrast to the 16-element scaling used by NVFP4. With a two-bit trellis stream and one eight-bit scale per group, the nominal storage rate is 2+8/G bits per weight. Thus, G32 halves the scale overhead relative to G16, giving 2.25 versus 2.50 bpw, excluding shared metadata and unquantized parameters.
Scale-group size and trellis-tile dimensions are separate parameters. In our implementation, each scale covers 32 consecutive input features within one output row, spanning two adjacent 16-column tile segments. Both segments read the same fixed scale independently; this sharing does not require communication between their trellis decoders. Nevertheless, the reviewer is correct that group size 16 would align scale boundaries with these segments and reduce their coupling during scale selection, potentially improving reconstruction accuracy at additional storage cost. We do not claim that group size 32 is optimal; a matched group-size ablation is needed to establish that tradeoff.
We conducted a matched group-size ablation on Llama-3.2-1B-Instruct, keeping the decoder, Hessians, calibration samples and seed, 16×16 trellis tiles, and LDL block size fixed. Both configurations use no RHT, D-weighted Viterbi, no energy ordering or fine-tuning, and identical WT2/C4 evaluation sequences at context length 8192.
| Scale Group Size | Trellis + Scale BPW | WT2 ↓ | C4 ↓ |
|---|---|---|---|
| 16 | 2.50 | 18.6421 | 19.2161 |
| 32 | 2.25 | 19.3069 | 20.5024 |
Reducing the scale-group size to 16 improves WT2 and C4 perplexity by 3.44% and 6.27%, respectively, at an additional 0.25 bpw. Therefore, we do not claim G32 is accuracy-optimal; rather, it provides a lower-storage operating point while retaining competitive accuracy.
For G32, adjacent 16-wide trellis blocks share a scale selected before LDL feedback, without requiring joint scale optimization during Viterbi search. G16 eliminates this scale sharing, although cross-block LDL error feedback remains. We will clarify this distinction and the difference between our G32 representation and standard NVFP4 in the revised paper.
Q6. It seems to me that this dichotomy of RHT/no-RHT is not as clear because of the existence of micro-rotated formats (Egiazarian et al.) that perform RHT only in small groups which has large implications for both accuracy impact and runtime implications. Is your quantization methods compatible with micro-rotations? How would you expect it to perform?
We agree that rotation should be viewed as a spectrum rather than a binary choice. XOR-Trellis is compatible with micro-rotations: for a block-diagonal orthogonal transform , we quantize using the transformed Hessian and apply the corresponding activation transform online. The XOR decoder and trellis structure remain unchanged. To examine this compatibility, we conducted a matched ten-configuration ablation on Llama-3.2-1B-Instruct, comparing five rotation sizes with Euclidean or D-weighted Viterbi. All configurations use the same calibration-derived Hessians (384 RedPajama sequences, context length 2048), G32 scaling, and no energy ordering or fine-tuning. Evaluation uses identical WT2/C4 token streams at a context length of 8192.
| Rotation Size | RHT? | Euclidean: WT2 / C4 ↓ | D-weighting: WT2 / C4 ↓ |
|---|---|---|---|
| No rotation | No RHT | 25.0407 / 24.4074 | 19.3069 / 20.5024 |
| 16 | Micro-RHT | 24.1231 / 23.0483 | 22.1179 / 21.8814 |
| 32 | Micro-RHT | 19.4775 / 19.6838 | 19.2762 / 19.1291 |
| 128 | Micro-RHT | 18.7627 / 19.1397 | 18.3894 / 19.2306 |
| Full input width | Full Hadamard | 18.3088 / 18.9498 | 18.3195 / 19.0948 |
The results confirm that XOR-Trellis supports micro-rotations, although accuracy does not improve monotonically with rotation size. A size-32 rotation improves C4 perplexity over rotation-free D-weighting while maintaining comparable WT2 accuracy, and size 128 improves both. Conversely, size 16 degrades both metrics. The benefit of D-weighting is largest without rotation and generally diminishes as rotation size increases, consistent with rotations reducing coordinate anisotropy. Micro-rotations partially reduce the coordinate anisotropy addressed by D-weighting, leaving room for additional improvements. As rotation becomes more extensive, the incremental benefit of D-weighting generally diminishes, consistent with our motivation for using it primarily in the rotation-free setting.
We will clarify that our contribution enables accurate fully rotation-free quantization, rather than claiming that all rotations are undesirable. Here, full-width rotation refers to an input-only normalized Hadamard transform, rather than QTIP’s two-sided randomized RHT. These experiments establish accuracy compatibility with micro-rotations but do not evaluate their runtime overhead or reproduce MR-GPTQ’s optimized implementation.
Q7. Section 4.6 title is initial caps, against the style recommendations and all the other sections.
Thank you for catching this inconsistency. We will change the Section 4.6 heading to sentence case to align with the venue’s style recommendations and the formatting used throughout the paper.
Concerns resolved, score raised
Thank you for the follow-up.
My concerns have been resolved. I particularly appreciate the the Micro-RHT discussion because of how it encapsulates the interaction between D-weighting and whitening transforms.
I have raised my score accordingly.
Q1. XOR-Trellis uses a non-conventional scaling scheme (one E5M3 scale per 32 elements)...
We thank the reviewer for raising this concern. We agree that the original comparison did not sufficiently isolate the contribution of E5M3/G32 scaling. We therefore conducted a matched-scaling ablation comparing QTIP-3INST and XOR-Trellis under both channel-wise and E5M3/G32 scaling, with and without RHT.
The results confirm that E5M3/G32 scaling benefits both methods, improving perplexity when RHT is enabled. However, scaling alone cannot explain the rotation-free performance of XOR-Trellis. Under identical E5M3/G32 scaling, removing RHT causes QTIP-3INST perplexity to deteriorate substantially (22.92/28.99 → 67.78/65.32), whereas XOR-Trellis achieves 21.38/27.16 without RHT.
Importantly, XOR-Trellis remains stable even with channel-wise scaling and no RHT (27.19/31.31), while QTIP-3INST collapses under the same conditions. XOR-Trellis for a group size of 512 (E5M3/G512) further improves the PPL to 24.1/30.14, almost at the same BPW as channel-wise scaling. Thus, fine-grained scaling helps accommodate local amplitude variation in the native basis, but is not sufficient by itself to eliminate RHT. These results demonstrate that the improved rotation-free behavior is not simply attributable to finer-grained scaling; rather, the XOR-Trellis reconstruction design and curvature-aware optimization are the primary drivers of its rotation-free advantage, beyond the benefits of additional scale metadata.
We also refer the reviewer to the matched bit-per-weight (BPW) comparisons in our response to Reviewer tDQk (Q2), which cover three different models. At the same effective BPW, rotation-free XOR-Trellis achieves better or comparable accuracy than QTIP-3INST with RHT across all three models. Combined with our other key contribution—an ultra-low-complexity XOR-Trellis decoder—this translates into up to 47% higher token generation throughput compared with QTIP HYB, as detailed in our response to Reviewer tDQk (Q3). These results further demonstrate that the benefits of XOR-Trellis extend beyond the choice of scaling scheme, enabling both accurate Hadamard-free quantization and more efficient inference.
We will revise the paper to report the 2.25-bpw effective rate explicitly and include both matched-scaling and matched-rate comparisons.
Q2. The paper does not fully explore or justify the role of permutation control q in the encoding scheme...
We agree that the role of permutation control (q) was insufficiently justified. In our decoder, q is not stored per weight; it is deterministically derived from the trellis history and controls how the four local reconstruction values are assigned to the four outgoing branches.
We therefore conducted a matched ablation on Llama-3.2-1B-Instruct, keeping the model, calibration data, Hessians, palettes, scaling, D-weighted Viterbi, energy ordering, and nominal 2.25-bpw rate fixed.
qq0onlyq1onlyqq, full state-bit coverage¹Relative to removing permutation control, the original two-bit q reduces WT2/C4 perplexity by 14.3%/10.4%, demonstrating the importance of history-dependent branch assignment. Interestingly, the two control bits contribute unequally: q_1 alone achieves comparable perplexity to the original two-bit design, whereas q_0 alone performs substantially worse. Increasing permutation diversity to 8 or 16 choices provides mixed improvements, while the full 24-permutation configuration achieves the best accuracy.
These results demonstrate that permutation control is beneficial, but its effectiveness depends on the specific history-dependent branch assignments rather than simply the number of available permutations. The original two-bit design represents a low-complexity operating point that captures most of the accuracy benefit using simple XOR logic without additional stored bits. We will clarify this accuracy–complexity tradeoff in the revised paper.