- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SubRot proposes a calibration objective for post-training quantization of vision-language models. It uses activation-gradient statistics to identify a sensitive channel subspace, then separates directions according to whether their gradient signs are consistent across samples. Calibration combines reconstruction MSE, signed guidance in the stable directions, and error suppression in the remaining sensitive directions. Experiments cover five models and five benchmarks under W4A6 and W4A4. The reported average improvements over FlatQuant are 0.60 and 0.52 percentage points, respectively, with additional component ablations, perturbation analyses, and deployment measurements.
Critical strengths
The improvements extend across several models and quantization settings. Tables 1 and 2 report better average scores than FlatQuant in all ten model and precision combinations, with gains of up to 1.4 percentage points. The method retains FlatQuant's inference graph, so these accuracy gains do not require additional inference computation.
The component ablation supports using the two added objectives together. In Table 3, the average increases from 45.0 for reconstruction alone to 45.3 with signed guidance, 45.5 with error suppression, and 46.1 with both. This is useful evidence that the signed term contributes beyond suppressing errors in sensitive directions.
Critical weaknesses
The argument connecting stable gradient signs to beneficial quantization errors is incomplete. Equation (8) establishes sign consistency after averaging directional derivatives over tokens. Equation (9), however, applies that sign to residuals that can vary independently across tokens. These are different quantities. For example, projected token gradients (g=(3,-1)) have a positive mean, but residuals (e=(0,-\epsilon)), with (\epsilon>0), make the signed mean residual negative while giving a positive first-order loss change (g^\top e=\epsilon). Figure 3 tests imposed perturbations rather than the residuals produced by quantization. The ablation supports the usefulness of the term, but the claim that it directs actual quantization errors toward lower task loss needs more direct evidence.
Several choices that define the method are missing. The manuscript does not specify the chosen rank (r), the criterion for declaring a direction sign-stable, or the values of (\alpha) and (\beta). Equation (9) also combines a mean with a squared Frobenius norm, making the relative weighting depend on the number of tokens and selected directions unless further normalization is used. These are central parts of the calibration objective, not incidental implementation details, and their omission prevents a faithful reproduction of the method.
Questions for authors
Using the existing calibrated models, can you check whether the signed objective agrees with the true first-order contribution (\langle G,\Delta Y\rangle_F) on held-out examples? The relevant errors here are the actual quantization residuals, rather than fixed perturbations. This would address my main concern about the proposed mechanism without requiring another benchmark suite.
What rank, sign-stability criterion, loss coefficients, and normalization were used for the reported results? Please also state whether the quantizer and transformation optimization settings were otherwise unchanged from FlatQuant. These details would resolve the uncertainty about what was implemented and what accounts for the comparison.