acceptodds
Under review as a conference paper at ICLR 2027

Exploiting Weight Space Symmetries for Vision Attention Compression

11 views

Abstract

Transformers have achieved remarkable success across diverse domains, yet their large model sizes incur considerable computation and latency. Accordingly, weight decomposition has emerged as an effective compression technique. However, previous methods do not consider the unique mechanism of Multi-Head Attention (MHA), where query-key (-) and value–output (-) computations are linear operations. To effectively leverage this mechanism, we propose Unilateral Decomposition for vision model compression (UniDec), a novel framework that applies decomposition to only one side of the - or - weight pairs. This approach effectively addresses approximation error at the unilateral weight level while exploiting the linear characteristics of the attention. Additionally, since --- weights exhibit varying sensitivities to low-rank approximation across heads, UniDec adaptively selects which side to be decomposed according to the rank sensitivity, thereby preserving the important information of weights. Extensive experiments demonstrate that UniDec generalizes to various transformer-based structures. Demonstrating exceptional performance even under aggressive compression (i.e., 60% MHA reduction), UniDec improves Top-1 accuracy by +34.0% / +46.3% on DeiT-Small / DeiT-S Distill while maintaining a +0.7% gain on DeiT-Base.

Primary area:
optimization
Keywords:
weight decomposition, weight compression

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cjfWOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes UniDec, a method for compressing attention layers in vision transformers without fine-tuning. For each query–key and value–output pair, it factorizes one weight matrix and absorbs a factor into the other matrix. The side to decompose is selected separately for each head using the reconstruction error of truncated SVD. The method also allows different ranks across pairs and layers. Experiments cover image classification, the vision tower of LLaVA, object detection, and semantic segmentation. The paper reports better accuracy and latency than independent per-weight decomposition, and studies compatibility with other decomposition methods and token merging.

Critical strengths
  1. The empirical evaluation covers a useful range of models and tasks. The improvements over independent per-weight decomposition are consistent across several vision backbones and extend to multimodal evaluation, detection, and segmentation. Reporting inference latency alongside parameter counts is also helpful. These results suggest that the method could be a useful compression component beyond a single model family.
  2. The method is simple and works with existing compression pipelines. The main operations are offline SVD and matrix multiplication, and the head-wise selection does not introduce an inference-time decision process. Table 4 provides useful evidence that UniDec complements existing methods: for example, adding it to SVD-LLM v2 improves accuracy from 70.5% to 72.8% at the same reported parameter and FLOP budget.
Critical weaknesses
  1. The experiments do not establish the contribution relative to the closest prior method. COMCAT (Xiao et al., ICML 2023, Section 3.2 and Table 1) already exploits the products (W^Q(W^K)^\top) and (W^V W^O), explains the advantage of head-level factorization over independent matrix factorization, and reports results without fine-tuning. Although the paper cites this work, it does not clearly explain the methodological distinction or compare against its head-level compression. This is particularly important because the additional benefit of adaptive side selection is modest in the paper’s own ablation. In Table 9, dynamic selection improves over always decomposing K and O by only 0.0–0.4 percentage points across the six reported settings. A matched comparison with a strong paired-factorization baseline is needed to determine how much UniDec adds beyond the existing attention-pair formulation.
  2. The compressed attention formula appears to change the attention temperature. Equation (3) scales the attention logits by (1/\sqrt{d_h}), while Equation (8) uses (1/\sqrt{r_{QK}}). Equations (6)–(7) do not specify a compensating rescaling of the factors. As written, this changes the attention distribution even if the low-rank factors reproduce the original QK product exactly. It is unclear whether this is a notation error or an intentional part of the implementation. The distinction matters because the reported results could depend on both low-rank approximation and a change in attention scaling.
Questions for authors
  1. Closest baseline: How does UniDec compare with COMCAT’s head-level/product-SVD compression when using the same checkpoint, compression budget, attention scaling, and fine-tuning allowance? This comparison would clarify the contribution of unilateral decomposition.
  2. Attention scaling: Is the (1/\sqrt{r_{QK}}) factor in Equation (8) intentional? If so, are the factors rescaled to preserve the original normalization? If the temperature is intentionally changed, how much does that choice contribute to the results?
  3. Adaptive selection: How well does the individual-weight reconstruction error predict the error of the resulting paired product or attention output? Given the small gains over fixed K/O selection in Table 9, is there evidence that adaptive selection provides a consistent benefit on the other evaluated architectures?
3 comments
AuthorsOpenReview ↗

Official Comment for Reviewer cjfW (1/2)

We sincerely appreciate the reviewer's valuable feedback. The comments on the comparison with COMCAT, the attention scaling, and the adaptive selection have helped us clarify our contributions and strengthen the experimental evidence. We address each point below.


(W1 & Q1) Methodological Distinction and Direct Comparison with COMCAT

Table G. Top-1 accuracy (%) comparison of UniDec and COMCAT across vision models.

MethodDeiT-SmallDeiT-Small distilledDeiT-BaseDINOv2-BaseDeiT3-LargeViT-LargeSwin-Large
Baseline79.980.981.881.286.884.386.3
MHA Parameter Reduction60%60%60%20%50%50%50%
COMCAT54.263.665.764.281.774.979.2
UniDec (Ours)65.469.372.173.283.077.781.2
Gain over COMCAT (%)+11.2+5.7+6.4+9.0+1.3+2.8+2.0

To address the reviewer’s concern regarding COMCAT, we expanded its methodological description in Appendix A.1.2 of the revised manuscript to clarify its distinction from UniDec. Following the reviewer’s comment, we also conducted direct comparisons and summarized the results in Table G. For a fair comparison, we used identical checkpoints for each model (DeiT (Small, Small-distilled, Base), DeiT3-L, ViT-L, Swin-L, and DINOv2), with matched compression budgets and attention scaling, and no fine-tuning. As shown in Table G, UniDec outperforms COMCAT across all evaluated models. Although the magnitude of the gains varies across models, UniDec achieves gains of up to 11.2% on the DeiT family, 9.0% on DINOv2-B, and improvements of 1.3–2.8% on DeiT3-L, ViT-L, and Swin-L. These results support the effectiveness of UniDec’s unilateral decomposition in preserving information compared with COMCAT. We included these results in Tables 1 and 5 of the revised manuscript. We hope these comparisons help clarify our contribution and address the reviewer’s concern.


(W2 & Q2) Attention Scaling and Ablation Results

Table H. Ablation on the attention scaling factor in UniDec.

ModelMHA parameter reductionUniDec (1/dh1/\sqrt{d_h})UniDec (1/rQK1/\sqrt{r_{QK}})Scaling contribution (%)
DeiT-B60%70.572.1+1.6
DeiT3-L50%81.983.0+1.1
ViT-L50%75.477.7+2.3
Swin-L50%79.781.2+1.5
DINOv2-B20%72.973.2+0.3

We thank the reviewer for highlighting this distinction. The 1/rQK1/\sqrt{r_{QK}} factor in Eq.(8) is intentional and used in our implementation. The factors in Eqs.(6)–(7) are used without compensating rescaling. This design choice is motivated by the dimension-based scaling of standard attention, with rQKr_{QK} reflecting the reduced Q–K dimension after low-rank decomposition. We clarified these details in Section 3.2, “Unilateral Weight Decomposition,” of the revised manuscript.

We conducted an ablation study comparing the adopted scaling 1/rQK1/\sqrt{r_{QK}} with the original scaling 1/dh1/\sqrt{d_h}, as shown in Table H. The adopted scaling consistently improves Top-1 accuracy across all five evaluated models, with gains ranging from 0.3 to 2.3%. These results support this design choice across diverse vision architectures. Since COMCAT uses the same scaling in Table G, the advantage of UniDec over COMCAT is not attributable to the scaling choice. We included these results in Table 12 of the revised manuscript and added the corresponding discussion in Section 4.13, “Ablation on Attention Scaling.”

AuthorsOpenReview ↗

Official Comment for Reviewer cjfW (2/2)


(Q3) Consistency of Adaptive Selection

Table I. Selection agreement (%) with paired-weight and attention-output relative reconstruction error criteria.

ModelQ–K paired weightQ–K attention outputV–O paired weightV–O attention output
DeiT-Small100.086.5100.096.9
DeiT-Small distilled93.382.1100.098.5
DeiT-Base93.161.289.785.1
Average95.576.696.693.5

Table J. Ablation on dynamic selection across different model architectures.

SelectionDeiT3-LargeViT-LargeDeiT-SmallDeiT-Base
MHA Parameter Reduction50%50%60%60%
Fixed (Q, V)77.972.839.761.8
Fixed (Q, O)81.676.859.070.3
Fixed (K, V)81.370.546.565.6
Fixed (K, O)82.473.765.371.7
Dynamic83.077.765.472.1

We thank the reviewer for this question. We directly evaluated whether the individual-weight reconstruction criterion selects the decomposition side that yields lower paired-product or attention-output relative reconstruction error. As shown in Table I, across DeiT-Small, DeiT-Small distilled, and DeiT-Base, mean selection agreement for the reported cases is 95.5% (Q–K) and 96.6% (V–O) for paired-product error. For attention-output error, the corresponding rates are 76.6% and 93.5%, respectively. These results provide direct evidence that the individual-weight criterion is informative for selecting the decomposition side based on both paired-product and attention-output reconstruction. We included this analysis in Table 13 of the revised manuscript and discussed it in Section 4.12, “Validation of the Selection Criterion.”

We further extended the fixed-selection ablation to DeiT3-Large and ViT-Large. As shown in Table J, dynamic selection achieves absolute accuracy gains of 0.6% on DeiT3-Large (83.0% vs. 82.4%) and 4.0% on ViT-Large (77.7% vs. 73.7%) over fixed K/O. Notably, the most effective fixed combination for ViT-Large is Q/O rather than K/O. Dynamic selection still exceeds fixed Q/O by 0.9% (77.7% vs. 76.8%) on ViT-Large. Overall, dynamic selection achieves the highest accuracy among all compared strategies in each of the four reported settings. These findings support its ability to accommodate differences across architectures without manually choosing a fixed decomposition pair. We added the DeiT3-Large and ViT-Large results to Table 9 of the revised manuscript and discussed these findings in Section 4.10, “Effectiveness of Dynamic Selection.”


We hope that these clarifications and additional experiments resolve the reviewer's concerns. We would be glad to address any remaining questions, and we would sincerely appreciate it if the reviewer could consider these revisions in the final assessment.

AuthorsOpenReview ↗

Follow-up on Our Response and Additional Experiments

Dear Reviewer cjfW,

Thank you again for your time and for reviewing our paper. As the discussion period draws to a close, we would like to kindly follow up on our response.

In brief, we added direct comparisons with COMCAT under matched compression budgets and attention scaling, demonstrating accuracy gains of 1.3–11.2 percentage points across all seven models without fine-tuning (Table G). We also clarified the attention-scaling implementation and added an ablation supporting the adopted scaling (Table H). To validate the individual-weight selection criterion, we analyzed its agreement with paired-product and attention-output reconstruction-error criteria (Table I). Finally, our expanded selection ablation shows that the best fixed configuration varies across architectures, supporting dynamic selection without manually choosing a fixed configuration for each architecture (Table J).

We would be grateful to know whether our response and the additional results address your concerns, and we would be happy to provide any further clarification.

Best regards,

Authors

Reviewer_DNf1OpenReview ↗
Rating
2
Confidence
3(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

UniDec compresses vision-transformer attention by factorizing one matrix from each Q–K and V–O pair and absorbing a factor into its counterpart. It selects the decomposed side per head using SVD reconstruction error. Experiments report improved accuracy and latency over per-weight decomposition across classification, detection, segmentation, and vision-language tasks.

Critical strengths
  • UniDec consistently improves the accuracy–efficiency trade-off over per-weight decomposition across multiple architectures and tasks (Tables 1–3). For example, detection improves from 50.1 to 55.4 mAP at the same parameter count and fewer FLOPs, supporting applicability beyond classification.
  • The method operates without fine-tuning and complements existing compression methods. Table 4 reports gains of 1.9 and 2.3 percentage points when combined with SVD-LLM and SVD-LLM v2 at matched parameter and FLOP budgets.
Critical weaknesses
  • The advance over the closest prior work looks insufficiently established. COMCAT (ICML 2023), §3.2, already exploits Q–K and V–O products for head-level compression and reports results without fine-tuning. Although cited, it is not experimentally compared. Gains over independent per-weight SVD therefore do not establish the benefit of unilateral approximation over existing paired-matrix decomposition.
  • The evidence does not seem to support the claimed necessity of dynamic selection. In Table 9, always decomposing K and O is within 0.0–0.4 percentage points of dynamic selection across all six settings. The highlighted 10.3-point improvement instead compares against the weakest fixed choice. This substantially weakens the demonstrated significance of a central contribution.
Questions for authors
  • How does UniDec compare with COMCAT's direct decomposition of paired products using identical checkpoints, per-layer ranks, parameter budgets, and no fine-tuning?
  • Equation (8) replaces the original denominator dh\sqrt{d_h} with rQK\sqrt{r_{QK}}, changing attention temperature beyond the low-rank approximation. Is this intentional and implemented? How do results change when retaining the original scaling?
  • Does dynamic selection provide a material advantage over fixed K/O selection on the other evaluated architectures when rank schedules are held identical?
3 comments
AuthorsOpenReview ↗

Official Comment for Reviewer DNf1

We sincerely appreciate the reviewer's thorough review and the insightful comments. In particular, the suggestions regarding the comparison with COMCAT, the attention scaling, and the effectiveness of dynamic selection have helped us substantially strengthen the paper. We address each point below.


(W1 & Q1) Direct Comparison with COMCAT

Table D. Top-1 accuracy (%) comparison of UniDec and COMCAT across vision models.

MethodDeiT-SmallDeiT-Small distilledDeiT-BaseDINOv2-BaseDeiT3-LargeViT-LargeSwin-Large
Baseline79.980.981.881.286.884.386.3
MHA Parameter Reduction60%60%60%20%50%50%50%
COMCAT54.263.665.764.281.774.979.2
UniDec (Ours)65.469.372.173.283.077.781.2
Gain over COMCAT (%)+11.2+5.7+6.4+9.0+1.3+2.8+2.0

To address the reviewer’s concern regarding the lack of an experimental comparison with COMCAT, we compared UniDec and COMCAT on the model checkpoints used in Tables 1 and 5 of the manuscript under fair comparison settings with identical rank configurations, parameter budgets, and training-free evaluation. As shown in Table D, UniDec consistently preserves performance better than COMCAT across all evaluated models. Although the magnitude of the gains varies across models, UniDec achieves improvements of up to 11.2 % on the DeiT family and meaningful gains of 1.3–2.8 % on large models. We included these results in Tables 1 and 5 of the revised manuscript and hope that this direct comparison addresses the reviewer’s concern.


(Q2) Attention Scaling and Ablation Results

Table E. Ablation on the attention scaling factor in UniDec.

ModelMHA Parameter ReductionUniDec (1/dh1/\sqrt{d_h})UniDec (1/rQK1/\sqrt{r_{QK}})Scaling contribution (%)
DeiT-Base60%70.572.1+1.6
DeiT3-Large50%81.983.0+1.1
ViT-Large50%75.477.7+2.3
Swin-Large50%79.781.2+1.5
DINOv2-Base20%72.973.2+0.3

We thank the reviewer for the thoughtful comment. Following the dimension-based scaling of standard attention, we intentionally use 1/rQK1/\sqrt{r_{QK}} in Eq. (8) and our implementation to reflect the reduced Q–K dimension after low-rank decomposition.

We ablate the attention scaling by replacing 1/rQK1/\sqrt{r_{QK}} with the original 1/dh1/\sqrt{d_h}. Table E shows that 1/rQK1/\sqrt{r_{QK}} yields higher Top-1 accuracy on each of the five evaluated models, with improvements of 0.3–2.3%. This consistent trend supports the chosen scaling across different vision architectures.

We included these results in Table 12 of the revised manuscript and added the corresponding discussion in Section 4.13, “Ablation on Attention Scaling.”


(W2 & Q3) Effectiveness of Dynamic Selection across Different Models

Table F. Ablation on dynamic selection across different model architectures.

SelectionDeiT3-LargeViT-LargeDeiT-SmallDeiT-Base
MHA Parameter Reduction50%50%60%60%
Fixed (Q, V)77.972.839.761.8
Fixed (Q, O)81.676.859.070.3
Fixed (K, V)81.370.546.565.6
Fixed (K, O)82.473.765.371.7
Dynamic83.077.765.472.1

We appreciate the reviewer’s suggestion to evaluate dynamic selection on additional architectures. We extended the fixed-selection ablation to DeiT3-L and ViT-L, using identical per-layer rank schedules across all selection strategies within each model. As shown in Table F, dynamic selection outperforms fixed K/O by 0.6% on DeiT3-L (83.0 vs. 82.4) and 4.0% on ViT-L (77.7 vs. 73.7). Notably, the most effective fixed combination for ViT-L is Q/O rather than K/O.

These results show that the advantage over fixed K/O depends on the architecture and can be substantially larger than the gains observed on DeiT-Small and DeiT-Base. The best fixed configuration also changes across models, supporting head-wise selection that adapts the decomposed side without manually choosing a global decomposition pair for each architecture. We added the DeiT3-Large and ViT-Large results to Table 9 of the revised manuscript and expanded the discussion in Section 4.10, “Effectiveness of Dynamic Selection,” to emphasize comparisons with the strongest fixed configuration for each model.


We hope that these responses and the additional experiments address the reviewer's concerns. We would be happy to provide further clarification if any questions remain, and we would greatly appreciate it if the reviewer could take these updates into account in the final evaluation.

Thank you for the additional experiments. They substantially address my original concerns. However, Tables D–E suggest that much of the gain on several large models comes from changing attention scaling. A comparison allowing COMCAT the same scaling adjustment would help establish the contribution of the proposed decomposition. I therefore maintain my current rating.

AuthorsOpenReview ↗

Official Comment by Authors

Table K. Accuracy (%) comparison between COMCAT and UniDec under matched attention scaling.

ModelMHA ReductionCOMCAT(1/dh1/\sqrt{d_h})UniDec (1/dh1/\sqrt{d_h})COMCAT (1/rQK1/\sqrt{r_{QK}})UniDec (1/rQK1/\sqrt{r_{QK}})
DeiT-Small60%53.363.9 (+10.6)54.265.4 (+11.2)
DeiT-Small Distilled60%64.467.8 (+3.4)63.669.3 (+5.7)
DeiT-Base60%65.170.5 (+5.4)65.772.1 (+6.4)
DeiT3-Large50%81.481.9 (+0.5)81.783.0 (+1.3)
ViT-Large50%74.775.4 (+0.7)74.977.7 (+2.8)
Swin-Large50%79.279.7 (+0.5)79.281.2 (+2.0)
DINOv2-Base20%64.772.9 (+8.2)64.273.2 (+9.0)

We thank the reviewer for the helpful suggestion. We would like to clarify that our previously reported comparisons with COMCAT already used the same adjusted attention scaling, 1/rQK1/\sqrt{r_{QK}}, for both methods to ensure a fair comparison. We apologize for not making this setting sufficiently clear.

To further address the reviewer’s concern, we additionally compare COMCAT and UniDec using the original scaling, 1/dh1/\sqrt{d_h}, for both methods. Table K presents the results grouped by scaling choice. Values in parentheses indicate UniDec’s gains over COMCAT under the same scaling, in percentage points.

UniDec consistently outperforms COMCAT across all seven models under both scaling choices, with gains of 0.5–10.6 percentage points under the original scaling and 1.3–11.2 percentage points under the adjusted scaling. These comparisons demonstrate the contribution of the proposed decomposition beyond the scaling adjustment, including when COMCAT receives the same adjustment.

Reviewer_fdtSOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Transformers are a high performing architecture for a multitude of tasks across NLP, vision and multi-modal tasks. However, due to their size, they can incur large computation and latency requirements. Prior works have addressed this through quantization, weight decomposition, pruning and distillation. This work focuses on weight decomposition for vision transformer models to reduce computation and improve latency without fine-tuning. They exploit the structure of Multi-head attention (MHA) that allows their method, UniDec, to decompose one projection from each of the query/key and value/output pairs. That being either queries or keys, or the values or outputs. UniDec dynamically selects the member of the pair to decompose at each layer using Frobenius-norm as the selection criteria. UniDec achieves lower parameter counts and lower latency compared to the base models. As well as maintaining more performance compared to per-weight decomposition. When combined with other weight decomposition methods or token merging methods for transformers or multi-modal VLMs, UniDec is able to improve performance relative to without UniDec. When fine-tuning a model with UniDec, the lost performance relative to the base model can be recovered.

Critical strengths
  • clear explanation of the method and Multi-head attention Section 3 provides a clear explanation of multi-head attention, establishing useful context for the introduction of UniDec. This makes the proposed method and its application easy to understand.

  • extensive experimental results The authors conduct experiments comparing UniDec across multiple vision and multi modal tasks/benchmarks with different model architectures as well. This helps contextualize UniDec in the literature so it can be easily referenced how this method affects other architectures and tasks.

Critical weaknesses
  • presentation of efficiency improvements could use more context in the presentation. The computational efficiency gains are primarily presented in terms of GFLOPs. However, reporting throughput in images or samples per second would make the practical benefits easier for readers to interpret and compare.

The results in Table 8 do not convincingly establish that the Frobenius norm is superior to alternative selection criteria. Further analysis or discussion would help justify its selection

Questions for authors
  • The fine-tuning results are quite convincing, can the authors provide more fine-tuning results on the other tasks to show if the trend continues on other tasks?
3 comments
AuthorsOpenReview ↗

Official Comment for Reviewer fdtS

We sincerely thank the reviewer for the positive assessment and the constructive suggestions. We address each point below.


(W1) Additional Throughput Evaluation

Table A. Summary of throughput measurements added to Tables 1, 5, 6, and 7 of the revised manuscript.

Model / settingBatch sizeBaseline (samples/sec)UniDec (samples/sec)
DINOv264496.5545.6
DeiT3-Large64213.3260.1
ViT-Large64214.3261.4
Swin-Large64311.1363.2
DeiT-Small, 20–60% reduction1281942.01975.8–2334.5
DeiT-Small distilled, 20–60% reduction1281929.31967.3–2323.5
DeiT-Base, 20–60% reduction128675.8748.0–836.5
ToMe + UniDec on DeiT-Base128675.81361.1

We thank the reviewer for this constructive suggestion. Following this feedback, we have added throughput measurements in samples/sec to Tables 1, 5, 6, and 7 of the revised manuscript. All measurements were performed on a single NVIDIA RTX 3090 GPU with an input resolution of 224 × 224, using batch sizes selected to utilize as much of its 24 GB GPU memory as possible. Table A summarizes these results to make the practical efficiency gains easier to interpret and compare. We hope these additional measurements address the reviewer’s concern.


(W2) Justification for the Frobenius Norm as the Default Selection Criterion

Table B. Selection agreement with the paired-product relative reconstruction error criterion at 50% rank retention.

Selection criterionQ–K (%)V–O (%)Average (%)
Frobenius norm80.584.282.4
Spectral norm76.670.773.7
Activation-aware scaling75.986.581.2

We appreciate the reviewer’s thoughtful comment. The purpose of the experiments in Table 8 was to analyze different error metrics within our proposed unilateral decomposition and dynamic selection framework. Specifically, our framework accommodates different error metrics, allowing the most suitable criterion to be used for each model architecture and rank-retention setting. We apologize for the confusion caused by our original explanation and revised the relevant text in the manuscript to clarify this point.

Nevertheless, we adopted the Frobenius norm as our default selection metric because it achieved the best or tied-best accuracy in six of the twelve settings in Table 8 of our draft. To address the reviewer’s concern regarding this choice, we evaluated selection accuracy on DeiT-B, DeiT3-L, ViT-L, and Swin-L at 50% rank retention and summarized the results in Table B. For each original Q–K or V–O weight pair, selection accuracy measures agreement with the reference choice determined by relative reconstruction error. The Frobenius norm achieved the highest selection accuracy for Q–K. Although its selection accuracy for V–O was slightly lower than that of activation-aware scaling, it achieved the highest average selection accuracy across the two pairs at 82.4%, providing additional support for its use as our default selection criterion. We included this additional analysis in Table 10 of the revised manuscript and clarified the rationale for this choice.


(Q1) Additional Fine-Tuning Results across Tasks

Table C. Fine-tuning results on vision-language and downstream vision tasks.

ModelMetricBaselineAfter decompositionAfter fine-tuning
LLaVA-1.5-7BVQAv2 test-dev78.574.778.5
LLaVA-1.5-7BGQA61.959.861.8
LLaVA-1.5-7BMME Perception1504.61408.01482.6
DINO Swin-LCOCO mAP57.255.457.0
SETR-NaiveCityscapes mIoU77.476.977.6

We appreciate the reviewer’s interest in whether the fine-tuning trend generalizes to other tasks. To examine this, we conducted additional fine-tuning experiments on the vision-language tasks in Table 2 and the downstream vision tasks in Table 3 of the manuscript. We used COCO train2017 for the vision-language tasks and object detection, and the Cityscapes training set for semantic segmentation, with 10 additional fine-tuning epochs for all tasks. As shown in Table C, the results show at least 98.5% performance retention across all benchmarks, supporting the trend observed in our original experiments.


We hope these responses address the reviewer's concerns, and we would be happy to provide any further clarification.

I thank the authors for their thorough response and additional experiments. I believe all of my original concerns and questions have been adequately addressed so I maintain my score.

The additional throughput measurements provide useful evidence of the practical computational benefits of UniDec. The new analysis of selection criteria provides a reasonable justification for using the Frobenius norm as the default, although its advantage over is relatively small. Finally, the additional fine-tuning experiments across vision and vision-language tasks provide convincing evidence that the observed performance recovery generalizes across tasks.

Dear Reviewer fdtS,

We are pleased to hear that our responses have fully addressed your concerns and questions. We sincerely thank you for your time and effort in reviewing our manuscript, and for your positive recognition of our work.

Best regards, Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response (Revision summary)

We sincerely thank all reviewers for their constructive feedback. We have revised the manuscript and added experiments and analyses to address the reviewers’ concerns. Unless otherwise specified, table and section numbers below refer to the revised manuscript.


Summary of revisions

  • Practical efficiency through throughput measurements (fdtS). We added throughput measurements demonstrating UniDec’s practical efficiency gains across architectures and compression settings, including integration with ToMe.

    • Revised manuscript. We added throughput results to Tables 1, 5, 6, and 7.
  • Justification for the default Frobenius-norm criterion (fdtS). We conducted an additional analysis showing that the Frobenius norm achieves the highest average agreement with the paired-product reconstruction-error criterion at 82.4%, supporting its use as the default.

    • Revised manuscript. We clarified the purpose of the experiments in Table 8 and added the selection-agreement analysis to Table 10.
  • Additional fine-tuning results across tasks (fdtS). We conducted additional fine-tuning experiments on vision-language tasks, object detection, and semantic segmentation, recovering at least 98.5% of the original model’s performance across all evaluated benchmarks.

    • Revised manuscript. We incorporated these results into Tables 2 and 3.
  • Methodological distinction and direct comparison with COMCAT (DNf1, cjfW). We conducted direct comparisons with COMCAT under matched evaluation settings without fine-tuning, demonstrating accuracy gains of 1.3–11.2 percentage points across all seven evaluated models.

    • Revised manuscript. We clarified the methodological distinction in Appendix A.1.2 and added the comparison results to Tables 1 and 5.
  • Clarification and ablation of attention scaling (DNf1, cjfW). We clarified that the scaling factor 1/rQK1/\sqrt{r_{QK}} is intentional and implemented without compensating rescaling. We conducted an ablation showing accuracy improvements of 0.3–2.3 percentage points over the original scaling across five models. We used matched attention scaling in the COMCAT comparisons.

    • Revised manuscript. We clarified the implementation in Sec. 3.2 and added the scaling ablation to Sec. 4.13 and Table 12.
  • Effectiveness of dynamic selection across architectures (DNf1, cjfW). We extended the selection ablation to DeiT3-Large and ViT-Large and found that the best fixed selection differs across architectures. This highlights the importance of our dynamic selection method, which adapts the decomposed side without requiring a manually chosen fixed configuration for each architecture.

    • Revised manuscript. We extended Table 9 and expanded Sec. 4.10 to emphasize the variation in the best fixed selection across architectures and the importance of dynamic selection.
  • Validation of the individual-weight selection criterion (cjfW). We analyzed selection agreement across the evaluated DeiT models and found mean agreement rates of 95.5–96.6% for paired-product reconstruction error and 76.6–93.5% for attention-output reconstruction error, supporting the individual-weight criterion.

    • Revised manuscript. We added this analysis to Sec. 4.12 and Table 13.

We hope these revisions address the reviewers’ concerns, and we would be glad to clarify any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.