-
The core ideas and several technical formulations substantially overlap with prior work, leaving the independent novelty of the proposed method insufficiently established.
-
The paper does not adequately establish the underlying relationship between the observed representation geometry distortion and downstream performance degradation.
-
There are discrepancies between some optimization objectives and their theoretical motivations, while the advantages of the proposed components over existing approaches remain unclear.
-
Although the method achieves performance improvements, the current experiments do not sufficiently identify the primary sources of these gains or systematically validate several key design choices.
-
The paper provides insufficient analysis of applicability across quantization configurations, algorithmic stability, and the trade-offs between performance improvements and computational overhead.
-
Other Comments
-
The novelty of the proposed approach requires further clarification.
A major concern with this paper is the substantial overlap between its core ideas and existing research.
First, preserving pairwise similarities, distances, or angular relationships between representations has been extensively explored in model compression. For instance, [1] introduces similarity-preserving knowledge distillation by maintaining pairwise similarity structures between teacher and student representations, while [2] further investigates the preservation of relational information through distances and angles. Therefore, using Gram matrices to characterize and preserve geometric relationships between token representations is not, by itself, a sufficiently novel contribution.
[3] addresses essentially the same problem as this paper. It also investigates error accumulation and anisotropic representation distortion in output-driven 1-bit quantization and proposes true-target output reconstruction and token-relation-preserving updates. In particular, the true-target reconstruction objective in [3] is identical to Equation (6) of this paper. Moreover, the geometry-related objective in Equation (9) is mathematically equivalent to the corresponding formulation in [3], up to a constant factor and notational differences, with both relying on the same Gram-matrix-based affinity. In addition, [3] already employs gradient-based masking to filter reconstruction-optimal updates and applies output-driven refinement to the final fully connected layer of each Transformer block. The relationship between the proposed method and [3] therefore requires explicit clarification.
Furthermore, [4] also investigates the limitations of output alignment and the importance of preserving internal representation geometry in low-bit LLM compression. Although [4] focuses on quantization-aware distillation rather than 1-bit PTQ, its central motivation is closely related to that of this paper.
In my view, the potentially distinctive contributions of this work lie primarily in the local second-order constrained projection and geometry-aware salient-mask selection, rather than the general principle of geometry preservation. I encourage the authors to provide direct comparisons with these closely related studies, clearly identify the technical problems that remain unresolved by prior approaches, and quantitatively demonstrate the incremental benefits of the newly introduced components.
-
Section 3 does not sufficiently establish a causal relationship between geometry distortion and performance degradation.
Section 3 motivates the proposed approach through Figures 1(a)–(c). Although these experiments provide useful observations, the evidence is insufficient to fully support the claimed relationship between geometry distortion and downstream performance degradation.
First, Figure 1(a) shows that MSE-based optimization favors high-magnitude channels, while Figure 1(b) demonstrates that output-driven quantization can introduce greater token geometry distortion. However, the causal connection between these observations remains unclear. In particular, it is not established whether the magnitude-dependent optimization bias directly leads to geometry distortion or whether other factors, such as binary quantization constraints and parameter interactions, are responsible.
Second, Figure 1(c) examines geometry distortion and perplexity under perturbations with identical Frobenius norms. While the results suggest a correlation, controlling the overall reconstruction error does not isolate the effect of geometry distortion. Different perturbation directions may also alter errors in important feature channels or directions to which subsequent layers are particularly sensitive. Therefore, the observed performance degradation cannot be attributed solely to geometry distortion. Furthermore, the relatively large perturbation magnitude of 0.5 raises questions about whether these artificial perturbations are representative of actual 1-bit quantization errors.
I suggest that the authors conduct additional controlled experiments using realistic quantization errors and investigate whether geometry distortion remains strongly associated with downstream degradation after accounting for other factors, such as feature importance and downstream sensitivity. This would provide stronger evidence for the proposed motivation.
-
The geometry constraint in Section 4 is not fully consistent with the theoretical motivation in Section 3.
Section 3 characterizes token geometry using a normalized Gram matrix, whose entries correspond to pairwise cosine similarities between token representations. Accordingly, the proposed motivation primarily concerns preserving representation directions and relational structures rather than absolute feature magnitudes.
However, Equation (9) in Section 4 introduces an unnormalized Gram-matrix affinity to maintain a tractable quadratic optimization structure. This surrogate is not equivalent to the normalized token geometry measure introduced earlier.
Consider a simple case:
[
\hat{Y}=cY,\qquad c>0.
]
In this setting, all pairwise cosine similarities remain unchanged. Nevertheless, the proposed geometry affinity satisfies
\frac{c^2}{2}|YY^\top|F^2.
]
Therefore, the affinity changes with the global scaling factor even though the relative geometry of token representations remains identical.
Consequently, the non-decreasing constraint on (S{\mathrm{Reg}}) in Equation (10) does not strictly guarantee preservation of the normalized token geometry emphasized in Section 3. Although the appendix provides visual evidence supporting the empirical effectiveness of this surrogate, its theoretical justification remains insufficient.
The authors should more clearly distinguish unnormalized Gram affinity from cosine-similarity-based geometry and provide a quantitative analysis of their relationship. Furthermore, comparisons with alternative geometry-preserving objectives, such as normalized Gram-matrix distance, cosine similarity loss, or Centered Kernel Alignment would help establish the advantages of the proposed formulation.
-
The theoretical contribution and practical necessity of the local second-order projection are insufficiently demonstrated.
Sections 4.3–4.5 introduce a geometry-preserving projection based on a local second-order approximation. However, Proposition 1 only guarantees non-decreasing geometry affinity under the diagonal Hessian approximation, a property that largely follows from the mask definition in Equation (15), rather than providing a stronger guarantee for the original constrained optimization problem.
Moreover, the coordinate-wise constraints may be overly conservative, as individually infeasible updates could potentially become feasible when combined. More importantly, prior work [3] has already employed first-order gradient information to select parameter updates, making the additional benefits of second-order information unclear.
The results in Appendix Table 8 further raise concerns. On OPT-6.7B, changing (\beta) from 0 to 1 improves C4 perplexity only from 16.24 to 16.18, while slightly worsening WikiText2 perplexity from 14.56 to 14.59. These marginal differences call into question the practical contribution of the second-order term.
I suggest that the authors provide more direct comparisons between first-order and second-order masking, including their effects on geometry preservation, convergence behavior, and downstream performance, to better justify the necessity of the proposed second-order approximation.
-
The fairness and statistical reliability of the experimental evaluation require further investigation.
First, the proposed method allocates different representational capacities to different weight columns. Although the paper reports average bit widths of approximately 1.08–1.11 bits, it remains unclear whether scaling parameters, mask metadata, and other storage overheads are consistently accounted for across competing methods.
Second, while some OPT and LLaMA results are averaged over three random seeds, the Qwen experiments and several ablation studies lack comparable statistical analysis. Given the use of random projections and mask selection, additional evaluations are needed to establish the stability of the reported improvements, particularly considering the large variance observed in some LLaMA-2-7B results on PTB.
Furthermore, Appendix Table 7 shows non-monotonic performance changes with increasing calibration set sizes, but provides limited explanation.
I suggest that the authors clarify the effective storage budgets, report statistical variations for key experiments, and further evaluate robustness to calibration data, quantization configurations, and random initialization.
-
The applicability and generalizability of the proposed method require further discussion.
Section 4.8 and Appendix E.2 indicate that the proposed output-driven refinement is applied only to the final fully connected layer of each Transformer block, while the remaining linear layers continue to use ARB-RC.
This design may help avoid excessive modifications to intermediate representations and reduce computational overhead. However, the paper does not provide sufficient experimental evidence to justify this particular layer-selection strategy.
In particular, the central motivation of this work emphasizes the importance of preserving token geometry for subsequent Transformer computations. Nevertheless, attention projection layers and other intermediate linear layers may also substantially influence representation geometry. If geometry preservation is indeed a critical mechanism, its effectiveness could vary considerably depending on which layers are refined.
It would therefore be valuable to investigate alternative layer-selection strategies, including applying the proposed refinement only to attention projection layers, only to MLP layers, or to all linear layers, in addition to the current configuration.
Such experiments would help justify the implementation choices and clarify the applicability and limitations of the proposed geometry-preserving mechanism.
References
[1] F. Tung and G. Mori, "Similarity-Preserving Knowledge Distillation," Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1365–1374, 2019.
[2] W. Park, D. Kim, Y. Lu, and M. Cho, "Relational Knowledge Distillation," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3967–3976, 2019.
[3] D. A. Hoang et al., "Rethinking Output Alignment for 1-bit Post-Training Quantization of Large Language Models," arXiv preprint arXiv:2512.21651, 2025.
[4] F. Tu et al., "Beyond Output Matching: Preserving Internal Geometry in NVFP4 LLM Distillation," arXiv preprint arXiv:2606.05682, 2026.