acceptodds
Withdrawn from ICLR 2027

SubRot: Signed Gradient Subspace Calibration for VLM Rotation Quantization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Post-training quantization has emerged as an important technique for reducing the deployment cost of vision-language models (VLMs). Unlike conventional large language models, VLMs jointly process visual and textual tokens, which differ substantially in their distributions, quantities, and semantics. Existing VLM quantization methods primarily characterize such heterogeneity at either the modality or token level. However, the number, semantic roles, and visual content of tokens vary considerably across samples, making it difficult to establish stable token-level correspondences across samples. In contrast, the channel space provides a shared coordinate system across samples, making it a more natural domain in which to accumulate stable task-sensitive structures. In this work, we employ the empirical Fisher matrix as a approximation to the local curvature of the autoregressive loss with respect to intermediate activations, and perform eigendecomposition to identify a gradient-based subspace along the channel dimension. This subspace exhibits three desirable properties: cross-sample stability, clear sensitivity separation, and consistent effects along certain directions. Based on these observations, we employ a Fisher-based curvature term to suppress excessive quantization errors along sensitive directions, while using a signed first-order term to align unavoidable quantization perturbations with loss-decreasing directions. Consequently, our method goes beyond merely reducing the overall magnitude of quantization errors, further ensuring that errors within the task-sensitive subspace are both magnitude-controlled and directionally favorable. Extensive experiments demonstrate the effectiveness of our method for low-bit rotation quantization of vision-language models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language Models, Post-Training Quantization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AEd9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SubRot proposes a calibration objective for post-training quantization of vision-language models. It uses activation-gradient statistics to identify a sensitive channel subspace, then separates directions according to whether their gradient signs are consistent across samples. Calibration combines reconstruction MSE, signed guidance in the stable directions, and error suppression in the remaining sensitive directions. Experiments cover five models and five benchmarks under W4A6 and W4A4. The reported average improvements over FlatQuant are 0.60 and 0.52 percentage points, respectively, with additional component ablations, perturbation analyses, and deployment measurements.

Critical strengths

The improvements extend across several models and quantization settings. Tables 1 and 2 report better average scores than FlatQuant in all ten model and precision combinations, with gains of up to 1.4 percentage points. The method retains FlatQuant's inference graph, so these accuracy gains do not require additional inference computation.

The component ablation supports using the two added objectives together. In Table 3, the average increases from 45.0 for reconstruction alone to 45.3 with signed guidance, 45.5 with error suppression, and 46.1 with both. This is useful evidence that the signed term contributes beyond suppressing errors in sensitive directions.

Critical weaknesses

The argument connecting stable gradient signs to beneficial quantization errors is incomplete. Equation (8) establishes sign consistency after averaging directional derivatives over tokens. Equation (9), however, applies that sign to residuals that can vary independently across tokens. These are different quantities. For example, projected token gradients (g=(3,-1)) have a positive mean, but residuals (e=(0,-\epsilon)), with (\epsilon>0), make the signed mean residual negative while giving a positive first-order loss change (g^\top e=\epsilon). Figure 3 tests imposed perturbations rather than the residuals produced by quantization. The ablation supports the usefulness of the term, but the claim that it directs actual quantization errors toward lower task loss needs more direct evidence.

Several choices that define the method are missing. The manuscript does not specify the chosen rank (r), the criterion for declaring a direction sign-stable, or the values of (\alpha) and (\beta). Equation (9) also combines a mean with a squared Frobenius norm, making the relative weighting depend on the number of tokens and selected directions unless further normalization is used. These are central parts of the calibration objective, not incidental implementation details, and their omission prevents a faithful reproduction of the method.

Questions for authors

Using the existing calibrated models, can you check whether the signed objective agrees with the true first-order contribution (\langle G,\Delta Y\rangle_F) on held-out examples? The relevant errors here are the actual quantization residuals, rather than fixed perturbations. This would address my main concern about the proposed mechanism without requiring another benchmark suite.

What rank, sign-stability criterion, loss coefficients, and normalization were used for the reported results? Please also state whether the quantizer and transformation optimization settings were otherwise unchanged from FlatQuant. These details would resolve the uncertainty about what was implemented and what accounts for the comparison.

Reviewer_xqMAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SubRot, a signed gradient subspace calibration method for post-training quantization of vision-language models (VLMs). The method aims to address the limitations of existing modality- or token-level gradient statistics, particularly their sensitivity to variations in token composition and their loss of channel-wise and signed gradient information. SubRot constructs a channel-wise empirical Fisher matrix from activation gradients and extracts a sensitive subspace through eigendecomposition. Within this subspace, directions exhibiting consistent gradient signs across calibration samples receive signed first-order guidance, while the remaining sensitive directions are regularized through a quadratic penalty. These terms are combined with the original MSE reconstruction objective to guide quantization calibration. Experiments on five VLMs and five benchmarks under W4A6 and W4A4 settings report average improvements of 0.60 and 0.52 percentage points over FlatQuant, respectively. The authors also provide ablation studies and analyses of subspace stability, sensitivity separation, and signed first-order effects.

Critical strengths
  1. A meaningful channel-space perspective on gradient-sensitive quantization. The paper identifies limitations of modality- and token-level gradient aggregation and proposes a structured alternative based on the eigenspaces of the empirical Fisher matrix. The distinction between sign-stable and sign-unstable directions is an interesting extension of conventional reconstruction-based quantization objectives. This provides a reasonable motivation for exploring more fine-grained, task-aware calibration.

  2. Consistent empirical improvements without additional inference overhead. SubRot is evaluated across multiple VLM architectures, model sizes, and quantization settings. The reported average improvements over FlatQuant are consistently positive across the evaluated model configurations, with a particularly noticeable improvement of 1.4 percentage points on LLaVA-NeXT-7B under W4A4. The deployment results also indicate that the method retains FlatQuant's inference-time efficiency. These findings support the practical potential of the proposed approach.

Critical weaknesses
  1. Insufficient theoretical justification for the connection between Fisher-based sensitivity, signed gradient guidance, and the final optimization objective. The main concern is that the theoretical motivation does not fully justify the actual calibration objective. First, Eq. (5) introduces a second-order Taylor expansion involving the Hessian of the loss with respect to intermediate activations, while Eq. (6) replaces this curvature information with a channel-wise empirical Fisher matrix. The approximation and the assumptions required to reduce the full activation-space curvature to this channel-wise representation are not sufficiently established. More importantly, although the sensitive directions are obtained through Fisher eigendecomposition, the second-order regularization term in Eq. (9) uses an unweighted squared Frobenius norm. It does not explicitly incorporate the corresponding Fisher eigenvalues, making it unclear to what extent the objective actually preserves the estimated directional curvature information. Similarly, the first-order guidance term relies on the signs of aggregated directional gradients rather than their full magnitudes. The sign-stability property in Eq. (8) characterizes average directional effects across samples, but it does not guarantee that the actual token-dependent quantization residuals will align with loss-decreasing directions. Consequently, the proposed objective appears to be a reasonable heuristic inspired by first- and second-order sensitivity analysis, rather than a sufficiently justified approximation to the stated loss expansion. This distinction matters because the paper's central contribution relies on the claimed mechanism of Fisher-informed, loss-aware quantization error shaping.

  2. Limited empirical gains and insufficient evidence isolating the contribution of the proposed mechanisms. Although SubRot consistently improves average accuracy over FlatQuant, the overall gains remain relatively modest: 0.60 percentage points under W4A6 and 0.52 percentage points under W4A4. Several model configurations show improvements of only 0.1–0.4 percentage points. The paper does not report results across multiple random seeds or independently sampled calibration sets, nor does it provide confidence intervals or other measures of experimental variability. Given the magnitude of the reported improvements, such evidence is important for establishing their robustness. More importantly, the existing ablations do not sufficiently distinguish the benefits of Fisher-based subspace selection and signed gradient guidance from those of simply introducing additional regularization terms into the FlatQuant objective. Table 3 only examines the inclusion or exclusion of the two proposed losses on one model under W4A6, without comparing against alternative subspace selections, randomized sign guidance, or curvature-weighted variants. Furthermore, the signed-effect validation in Figure 3 uses manually directed perturbations on only 16 held-out samples, rather than analyzing the actual quantization residuals produced by the optimized model. As a result, the current experiments demonstrate that the modified calibration objective can improve performance, but do not convincingly establish that the improvements arise specifically from the proposed Fisher-sensitive subspace and signed first-order mechanisms. This is particularly consequential because these mechanisms constitute the paper's primary claimed contribution.

Questions for authors

Q1. Theoretical relationship between Fisher sensitivity and the calibration objective. Could the authors clarify the assumptions under which the channel-wise empirical Fisher matrix in Eq. (6) approximates the curvature relevant to Eq. (5)? Why does the second-order term in Eq. (9) use an unweighted squared norm instead of explicitly incorporating the Fisher eigenvalues? Would an eigenvalue-weighted objective yield different results?

Q2. Effectiveness of signed guidance under actual quantization errors. The analysis in Figure 3 uses perturbations deliberately oriented along loss-decreasing directions. Could the authors provide evidence that the actual quantization residuals learned by SubRot exhibit the same behavior? In particular, an analysis of the first-order loss contribution from real quantization residuals would help establish whether signed guidance operates as theoretically intended.

Q3. Robustness of the reported performance improvements. Could the authors report the mean and standard deviation of performance across multiple random calibration sets or training seeds? This is particularly important for configurations where the improvements over FlatQuant are below 0.5 percentage points.

Reviewer_33SyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Existing VLM quantization methods rely on unstable token- or modality-level gradient statistics and discard gradient signs, limiting calibration accuracy across diverse multimodal inputs. Eigendecomposes the channel-wise empirical Fisher matrix of activation gradients to identify a cross-sample stable, task-sensitive subspace. Partitions this subspace into directions with cross-sample consistent gradient signs versus remaining sensitive directions with unstable signs. Calibrates rotation parameters using a joint objective that steers quantization errors along loss-decreasing sign-stable directions, suppresses error magnitudes along sign-unstable directions, and retains MSE for reconstruction.

Critical strengths

The proposed method replaces token/modality heuristics with a channel-eigenspace Fisher decomposition, establishing a sequence-invariant coordinate frame that models cross-channel correlations and prevents catastrophic low-bit activation collapse across variable visual token layouts.

Critical weaknesses

The paper is not easy to follow for non-experts in quantization research, and the performance gains are marginal.

Questions for authors
  1. In section 4.1., it is mentioned "Quantization is applied to the language model component". Why is the vision encoder left in full precision if the method claims to solve end-to-end VLM quantization?
  2. Given identical inference latency and memory footprints to FlatQuant, isn't the technical contribution confined entirely to an offline calibration loss?
  3. Are the marginal 0.1–0.4 percentage point gains over FlatQuant on most evaluated models statistically significant beyond calibration noise?
  4. How sensitive is the method to tuning the rank rr and loss weights (α,β\alpha, \beta ), and what prevents the first-order guidance term from violating the local Taylor trust region?
  5. How can cross-sample sign consistency be convincingly established across diverse multimodal distributions when empirical validation relies on only 16 samples?
  6. What exact statistical threshold and resulting dimensionality determine whether an eigenvector is classified into the sign-stable guidance set versus the second-order suppression set?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.