- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes UniDec, a method for compressing attention layers in vision transformers without fine-tuning. For each query–key and value–output pair, it factorizes one weight matrix and absorbs a factor into the other matrix. The side to decompose is selected separately for each head using the reconstruction error of truncated SVD. The method also allows different ranks across pairs and layers. Experiments cover image classification, the vision tower of LLaVA, object detection, and semantic segmentation. The paper reports better accuracy and latency than independent per-weight decomposition, and studies compatibility with other decomposition methods and token merging.
Critical strengths
- The empirical evaluation covers a useful range of models and tasks. The improvements over independent per-weight decomposition are consistent across several vision backbones and extend to multimodal evaluation, detection, and segmentation. Reporting inference latency alongside parameter counts is also helpful. These results suggest that the method could be a useful compression component beyond a single model family.
- The method is simple and works with existing compression pipelines. The main operations are offline SVD and matrix multiplication, and the head-wise selection does not introduce an inference-time decision process. Table 4 provides useful evidence that UniDec complements existing methods: for example, adding it to SVD-LLM v2 improves accuracy from 70.5% to 72.8% at the same reported parameter and FLOP budget.
Critical weaknesses
- The experiments do not establish the contribution relative to the closest prior method. COMCAT (Xiao et al., ICML 2023, Section 3.2 and Table 1) already exploits the products (W^Q(W^K)^\top) and (W^V W^O), explains the advantage of head-level factorization over independent matrix factorization, and reports results without fine-tuning. Although the paper cites this work, it does not clearly explain the methodological distinction or compare against its head-level compression. This is particularly important because the additional benefit of adaptive side selection is modest in the paper’s own ablation. In Table 9, dynamic selection improves over always decomposing K and O by only 0.0–0.4 percentage points across the six reported settings. A matched comparison with a strong paired-factorization baseline is needed to determine how much UniDec adds beyond the existing attention-pair formulation.
- The compressed attention formula appears to change the attention temperature. Equation (3) scales the attention logits by (1/\sqrt{d_h}), while Equation (8) uses (1/\sqrt{r_{QK}}). Equations (6)–(7) do not specify a compensating rescaling of the factors. As written, this changes the attention distribution even if the low-rank factors reproduce the original QK product exactly. It is unclear whether this is a notation error or an intentional part of the implementation. The distinction matters because the reported results could depend on both low-rank approximation and a change in attention scaling.
Questions for authors
- Closest baseline: How does UniDec compare with COMCAT’s head-level/product-SVD compression when using the same checkpoint, compression budget, attention scaling, and fine-tuning allowance? This comparison would clarify the contribution of unilateral decomposition.
- Attention scaling: Is the (1/\sqrt{r_{QK}}) factor in Equation (8) intentional? If so, are the factors rescaled to preserve the original normalization? If the temperature is intentionally changed, how much does that choice contribute to the results?
- Adaptive selection: How well does the individual-weight reconstruction error predict the error of the resulting paired product or attention output? Given the small gains over fixed K/O selection in Table 9, is there evidence that adaptive selection provides a consistent benefit on the other evaluated architectures?
3 comments
Official Comment for Reviewer cjfW (2/2)
(Q3) Consistency of Adaptive Selection
Table I. Selection agreement (%) with paired-weight and attention-output relative reconstruction error criteria.
| Model | Q–K paired weight | Q–K attention output | V–O paired weight | V–O attention output |
|---|---|---|---|---|
| DeiT-Small | 100.0 | 86.5 | 100.0 | 96.9 |
| DeiT-Small distilled | 93.3 | 82.1 | 100.0 | 98.5 |
| DeiT-Base | 93.1 | 61.2 | 89.7 | 85.1 |
| Average | 95.5 | 76.6 | 96.6 | 93.5 |
Table J. Ablation on dynamic selection across different model architectures.
| Selection | DeiT3-Large | ViT-Large | DeiT-Small | DeiT-Base |
|---|---|---|---|---|
| MHA Parameter Reduction | 50% | 50% | 60% | 60% |
| Fixed (Q, V) | 77.9 | 72.8 | 39.7 | 61.8 |
| Fixed (Q, O) | 81.6 | 76.8 | 59.0 | 70.3 |
| Fixed (K, V) | 81.3 | 70.5 | 46.5 | 65.6 |
| Fixed (K, O) | 82.4 | 73.7 | 65.3 | 71.7 |
| Dynamic | 83.0 | 77.7 | 65.4 | 72.1 |
We thank the reviewer for this question. We directly evaluated whether the individual-weight reconstruction criterion selects the decomposition side that yields lower paired-product or attention-output relative reconstruction error. As shown in Table I, across DeiT-Small, DeiT-Small distilled, and DeiT-Base, mean selection agreement for the reported cases is 95.5% (Q–K) and 96.6% (V–O) for paired-product error. For attention-output error, the corresponding rates are 76.6% and 93.5%, respectively. These results provide direct evidence that the individual-weight criterion is informative for selecting the decomposition side based on both paired-product and attention-output reconstruction. We included this analysis in Table 13 of the revised manuscript and discussed it in Section 4.12, “Validation of the Selection Criterion.”
We further extended the fixed-selection ablation to DeiT3-Large and ViT-Large. As shown in Table J, dynamic selection achieves absolute accuracy gains of 0.6% on DeiT3-Large (83.0% vs. 82.4%) and 4.0% on ViT-Large (77.7% vs. 73.7%) over fixed K/O. Notably, the most effective fixed combination for ViT-Large is Q/O rather than K/O. Dynamic selection still exceeds fixed Q/O by 0.9% (77.7% vs. 76.8%) on ViT-Large. Overall, dynamic selection achieves the highest accuracy among all compared strategies in each of the four reported settings. These findings support its ability to accommodate differences across architectures without manually choosing a fixed decomposition pair. We added the DeiT3-Large and ViT-Large results to Table 9 of the revised manuscript and discussed these findings in Section 4.10, “Effectiveness of Dynamic Selection.”
We hope that these clarifications and additional experiments resolve the reviewer's concerns. We would be glad to address any remaining questions, and we would sincerely appreciate it if the reviewer could consider these revisions in the final assessment.
Follow-up on Our Response and Additional Experiments
Dear Reviewer cjfW,
Thank you again for your time and for reviewing our paper. As the discussion period draws to a close, we would like to kindly follow up on our response.
In brief, we added direct comparisons with COMCAT under matched compression budgets and attention scaling, demonstrating accuracy gains of 1.3–11.2 percentage points across all seven models without fine-tuning (Table G). We also clarified the attention-scaling implementation and added an ablation supporting the adopted scaling (Table H). To validate the individual-weight selection criterion, we analyzed its agreement with paired-product and attention-output reconstruction-error criteria (Table I). Finally, our expanded selection ablation shows that the best fixed configuration varies across architectures, supporting dynamic selection without manually choosing a fixed configuration for each architecture (Table J).
We would be grateful to know whether our response and the additional results address your concerns, and we would be happy to provide any further clarification.
Best regards,
Authors
Official Comment for Reviewer cjfW (1/2)
We sincerely appreciate the reviewer's valuable feedback. The comments on the comparison with COMCAT, the attention scaling, and the adaptive selection have helped us clarify our contributions and strengthen the experimental evidence. We address each point below.
(W1 & Q1) Methodological Distinction and Direct Comparison with COMCAT
Table G. Top-1 accuracy (%) comparison of UniDec and COMCAT across vision models.
To address the reviewer’s concern regarding COMCAT, we expanded its methodological description in Appendix A.1.2 of the revised manuscript to clarify its distinction from UniDec. Following the reviewer’s comment, we also conducted direct comparisons and summarized the results in Table G. For a fair comparison, we used identical checkpoints for each model (DeiT (Small, Small-distilled, Base), DeiT3-L, ViT-L, Swin-L, and DINOv2), with matched compression budgets and attention scaling, and no fine-tuning. As shown in Table G, UniDec outperforms COMCAT across all evaluated models. Although the magnitude of the gains varies across models, UniDec achieves gains of up to 11.2% on the DeiT family, 9.0% on DINOv2-B, and improvements of 1.3–2.8% on DeiT3-L, ViT-L, and Swin-L. These results support the effectiveness of UniDec’s unilateral decomposition in preserving information compared with COMCAT. We included these results in Tables 1 and 5 of the revised manuscript. We hope these comparisons help clarify our contribution and address the reviewer’s concern.
(W2 & Q2) Attention Scaling and Ablation Results
Table H. Ablation on the attention scaling factor in UniDec.
We thank the reviewer for highlighting this distinction. The 1/rQK factor in Eq.(8) is intentional and used in our implementation. The factors in Eqs.(6)–(7) are used without compensating rescaling. This design choice is motivated by the dimension-based scaling of standard attention, with rQK reflecting the reduced Q–K dimension after low-rank decomposition. We clarified these details in Section 3.2, “Unilateral Weight Decomposition,” of the revised manuscript.
We conducted an ablation study comparing the adopted scaling 1/rQK with the original scaling 1/dh, as shown in Table H. The adopted scaling consistently improves Top-1 accuracy across all five evaluated models, with gains ranging from 0.3 to 2.3%. These results support this design choice across diverse vision architectures. Since COMCAT uses the same scaling in Table G, the advantage of UniDec over COMCAT is not attributable to the scaling choice. We included these results in Table 12 of the revised manuscript and added the corresponding discussion in Section 4.13, “Ablation on Attention Scaling.”