- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates catastrophic forgetting in continual learning of contrastive vision-language models, focusing on the relationship between representation drift and changes in cross-modal geometry. The authors argue that existing methods may unnecessarily penalize representation changes that do not affect image-text similarities. Through empirical analysis on a fixed continual learning sequence consisting of seven image-text retrieval datasets, they observe that approximately 39%–48% of representation drift can be explained by a shared orthogonal transformation of visual and textual embeddings, which preserves cross-modal similarities and retrieval rankings. Motivated by this observation, the authors propose Decomposed Relative Geometry Preservation. The method first uses orthogonal Procrustes alignment to factor out representation drift attributable to a shared orthogonal transformation between the previous and current models. It then exactly decomposes the remaining alignment residual into three components: visual deformation, textual deformation, and cross-modal orthogonal alignment mismatch, allowing each component to be regularized separately. In addition, the authors introduce a budget-based adaptive weighting mechanism that dynamically adjusts the regularization weights using exponential moving averages of the losses to control the relative contributions of different loss terms to the overall training objective.
Critical strengths
- The paper is generally well organized, with a coherent progression from research motivation to methodological design and experimental analysis. The authors introduce the proposed approach through intuitive examples and mathematical derivations, making the paper relatively easy to follow.
- The experimental results provide some empirical support for the effectiveness of the proposed method, although the necessity of its core design and its distinct advantages over existing approaches remain to be further established.
Critical weaknesses
-
Insufficient distinction between the proposed method and existing work. The main motivation of this paper is based on the observation that shared orthogonal transformations can induce substantial representation drift without changing image-text similarities. However, orthogonal invariance is a well-established mathematical property, and related ideas have already been explored in Procrustes-based geometric distillation, relational distillation, and topology preservation for continual vision-language learning, such as Relational Knowledge Distillation[1], Mod-X[2], CTP[[[3], and Knowledge Distillation through Geometry-Aware Representational Alignment[4]. Although these studies do not necessarily propose the same three-component decomposition as Decomp-RGP, the paper does not sufficiently clarify its substantive differences from existing geometry-preserving approaches. In particular, the lack of direct comparisons with simpler geometry-invariant objectives makes it difficult to assess the distinct contribution and novelty of the proposed method. [1]Relational Knowledge Distillation(CVPR2019) [2]Continual Vision-Language Representation Learning with Off-Diagonal Information(ICML2023) [3]CTP: Towards Vision-Language Continual Pretraining via Compatible Momentum Contrast and Topology Preservation(ICCV2023) [4] Knowledge Distillation through Geometry-Aware Representational Alignment(arXiv2025)
-
Insufficient validation of the justification and necessity of the three-component geometric decomposition. Although the proposed decomposition of RGP into three residual components is mathematically valid, this algebraic identity does not necessarily establish that the components accurately characterize harmful geometric changes during continual learning. A shared orthogonal transformation is only a sufficient condition for preserving cross-modal similarities, and residual changes that cannot be explained by such a transformation do not necessarily lead to degraded retrieval performance. Furthermore, the cross-modal asynchronous term measures the difference between the optimal objectives under independent and shared alignments, rather than directly measuring changes in cross-modal similarities or forgetting. More importantly, Table 2 shows that the full Decomp-RGP improves performance by approximately 6.79 percentage points over the undecomposed RGP. However, the current experiments do not sufficiently distinguish whether this improvement arises from the proposed geometric decomposition itself or from independent regularization and more flexible loss weighting. Consequently, the necessity of the core design and its underlying mechanism remain insufficiently supported.
-
Limited experimental validation of the generalizability of the proposed method. The evaluation is primarily conducted on a fixed seven-task image-text retrieval sequence constructed by the authors. However, continual learning performance can be sensitive to task ordering and domain distribution shifts, and results from a single task sequence may not be sufficient to establish the robustness of the method. In addition, Decomp-RGP applies geometric constraints using only current-task data and a frozen model from the previous training stage. It therefore remains unclear whether preserving geometry on the current data distribution can reliably protect knowledge acquired from previous tasks with substantially different distributions. Although the paper compares against several representative continual vision-language learning methods, its coverage of recent geometry-preserving approaches remains limited[5-7], and some baselines were originally developed for continual learning settings different from continual image-text retrieval. Finally, while Decomp-RGP achieves the highest overall retrieval accuracy, it does not obtain the best backward transfer performance, suggesting that the claims regarding its stability-plasticity advantages should be more carefully qualified. [5]Pi-CCA: Prompt-Invariant CCA Certificates for Replay-Free Continual Multimodal Learning(ICLR 2026) [6] Hyperbolic Multimodal Continual Learning(ICML 2026) [7] HypCL: Adapting CLIP in Hyperbolic Space for Continual Learning(ICML 2026)
Questions for authors
- Novelty and distinction from existing methods. Prior studies have already explored Procrustes-based geometric distillation, relational distillation, and topology or cross-modal geometry preservation in continual vision-language learning. Could the authors further clarify the substantive technical differences between Decomp-RGP and these existing approaches? In particular, why would directly preserving Gram matrices or cross-modal similarity matrices be insufficient to address the shared orthogonal drift considered in this paper? Comparisons with these simpler geometry-invariant objectives under a unified experimental protocol would help establish the distinct value of the proposed method.
- Justification of the three-component decomposition and sources of performance improvement. Could the authors provide further theoretical or empirical evidence explaining how the three components—visual deformation, textual deformation, and cross-modal orthogonal alignment mismatch—relate to harmful geometric changes and catastrophic forgetting? In particular, given the substantial improvement of approximately 6.79 percentage points achieved by the full Decomp-RGP over the undecomposed RGP, could the authors conduct component-wise ablations or fair comparisons with alternative geometry-preserving losses using the same adaptive weighting mechanism? Such evidence would help clarify whether the improvement primarily stems from the specific geometric decomposition or from the flexibility of independently weighting multiple loss terms.
- Experimental robustness and preservation of historical knowledge. Could the authors further evaluate the robustness of Decomp-RGP under different task orders and varying degrees of domain distribution shift? Moreover, since the method imposes geometric constraints using only current-task data and a frozen previous-stage model, could the authors clarify to what extent this mechanism can preserve historical knowledge from tasks whose distributions differ substantially from the current task? Such analyses would help establish the scope of the paper's conclusions and the generalizability of the reported stability-plasticity trade-off. Overall, I find the motivation and formulation of this work interesting, and the reported empirical improvements are encouraging. However, the concerns above currently limit my confidence in the novelty and necessity of the proposed approach. I would be willing to reconsider my assessment and potentially raise my rating if the authors can adequately address these concerns with convincing explanations and supporting evidence.
5 comments
Response to Reviewer PZ8r (W2, Q2)
We thank the reviewer for this insightful comment and for highlighting the importance of clarifying the contributions of the three-component decomposition and Budget-based Adaptive Weighting to the observed performance improvements.
Geometric Changes and Catastrophic Forgetting
The three components measure different types of geometric changes, and we do not assume that every residual change necessarily causes forgetting. RGP uses shared orthogonal alignment to preserve joint image–text geometry, while allowing geometry-preserving representation changes. Intra-modal deformation does not necessarily degrade retrieval performance as long as the correct image–text rankings are preserved. However, without appropriate constraints, such changes may accumulate across tasks and disrupt previously learned relationships, increasing the risk of catastrophic forgetting.
The cross-modal orthogonal alignment mismatch measures the additional residual introduced by shared alignment compared with independent modality-specific alignments. Although it does not directly measure changes in cross-modal similarities or forgetting, it captures the disagreement between the optimal visual and textual alignments. Fig. 1(b) shows that the synchronization between visual and textual transformations affects retrieval performance even at the same drift magnitude, supporting the practical relevance of this component. Furthermore, Fig. 5(b)(c) shows that different budget allocations lead to different ACC and BWT, supporting the value of independently controlling these components during continual learning.
Contributions of Decomposition and Budget-based Adaptive Weighting
To clarify the source of the 6.79 percentage-point improvement, we provide a more detailed comparison of Feature Distillation, RGP, and Decomp-RGP under different weighting strategies.
| Method | ACC ↑ | BWT ↑ |
|---|---|---|
| Feature Distillation | 51.03 | +0.35 |
| RGP | 60.11 | −1.41 |
| RGP | 53.82 | +0.19 |
| Decomp-RGP | 60.17 | −1.38 |
| Decomp-RGP | 61.89 | -2.64 |
| Decomp-RGP 5%, 5%, 30% | 60.21 | −0.58 |
| Decomp-RGP 8%, $\rho_t = $ 2%, $ \rho_a = $ 30% | 60.61 | −0.75 |
With the same fixed coefficient , RGP improves ACC over Feature Distillation by 9.08 percentage points (60.11% vs. 51.03%), demonstrating the benefit of removing shared orthogonal drift. As established in Eq. (6), Decomp-RGP is mathematically equivalent to RGP when the three coefficients are equal. Their nearly identical ACC (60.17% vs. 60.11%) confirms that the decomposition alone does not explain the performance improvement.
The decomposition enables independent control of the three components. With fixed coefficients (5,5,30), Decomp-RGP achieves 61.89% ACC but suffers greater forgetting (BWT −2.64%). Budget-based Adaptive Weighting with (5%,5%,30%) improves BWT to −0.58%, with ACC at 60.21%. Adjusting the budget allocation to (8%,2%,30%) further improves ACC to 60.61%, with BWT of −0.75%.
The 6.79-point gain highlighted by the reviewer comes from comparing RGP with a total adaptive budget of 40% (ACC 53.82%) against Decomp-RGP with separately allocated budgets (8%, 2%, and 30% with ACC 60.61%). This improvement therefore reflects the combined effect of decomposition and adaptive budget allocation, rather than decomposition alone. RGP provides the main benefit by removing geometry-preserving drift, while Decomp-RGP further allows us to regulate the three components independently, achieving a better balance between stability and plasticity.
Response to Reviewer PZ8r (W3, Q3)
We thank the reviewer for the thoughtful and constructive feedback, as well as for recognizing the motivation and empirical results of our work. We appreciate the opportunity to further clarify the robustness of Decomp-RGP and its ability to preserve historical knowledge under distribution shifts.
Robustness Across Task Orders, Domain Shifts, and Learning Protocols
We thank the reviewer for raising concerns about experimental generalizability. To evaluate the sensitivity of Decomp-RGP to task ordering, we conducted additional experiments using a different seven-task retrieval sequence. As reported in the revised Table 4, Decomp-RGP achieves 60.12% ACC, outperforming the strongest baseline (56.56%) by 3.56%. This result indicates that the effectiveness of Decomp-RGP is not restricted to the original task order.
Furthermore, Fig. 4 presents the zero-shot performance after continual learning across the seven datasets in our retrieval protocol. Decomp-RGP outperforms state-of-the-art methods on five datasets, with only a slightly lower result on one dataset.
To further evaluate our method under a different learning protocol, we conducted additional experiments on the 11-dataset Multimodal Task-Incremental Learning (MTIL) classification benchmark used in ZSCL. Following the reviewer's suggestions, we also included five additional methods:
- Knowledge Distillation through Geometry-Aware Representational Alignment(arXiv2025)
- CTP: Towards Vision-Language Continual Pretraining via Compatible Momentum Contrast and Topology Preservation(ICCV2023)
- Pi-CCA: Prompt-Invariant CCA Certificates for Replay-Free Continual Multimodal Learning(ICLR 2026)
- Hyperbolic Multimodal Continual Learning(ICML 2026)
- HypCL: Adapting CLIP in Hyperbolic Space for Continual Learning(ICML 2026)
Knowledge Distillation through Geometry-Aware Representational Alignment was originally developed for language model distillation. To adapt its Procrustes-based objective to CLIP, we concatenate the image and class-text embeddings and apply Procrustes alignment between the joint representations of the current model and the frozen previous-stage model. CTP was originally designed for vision-language continual pretraining. We adapted it to our CLIP-based setting using the image and text embeddings produced by CLIP.
Under this MTIL protocol, Decomp-RGP outperforms the strongest baseline by 4.01% in average accuracy. The additional comparisons are reported in Table 3 of the revised manuscript. These results further support the effectiveness of Decomp-RGP across different task orders, datasets, and continual learning settings.
Preservation of Historical Knowledge
We thank the reviewer for raising this question. Decomp-RGP operates without storing or replaying historical training data. At each stage, the frozen previous-stage model provides a geometric reference for current-task samples. Our method allows visual and textual representations to undergo shared orthogonal transformations during adaptation, while constraining the remaining geometric changes after alignment. Historical knowledge is therefore preserved indirectly, and the effectiveness of this mechanism depends on how well current-task samples reflect the representation relationships relevant to previous tasks.
After learning all seven tasks, Decomp-RGP achieves a Mean R@1 of 83.89% on Flickr30K. In comparison, fine-tuning without regularization achieves only 73.08% (Fig. 3 and Table 1). Across the full sequence, Decomp-RGP achieves a BWT of −0.75%, compared with −20.40% for fine-tuning CLIP without regularization (Table 2). These results show that geometric constraints applied using only current-task data can substantially reduce forgetting. Although the tasks have different data distributions, Decomp-RGP can still reduce forgetting by preserving the relative geometry of the previous-stage model. Our method allows shared orthogonal transformations while constraining the remaining geometric changes. This helps preserve previously learned knowledge while allowing the model to adapt to new task.
Response to Reviewer PZ8r (W1, Q1)
We thank the reviewer for highlighting these related works.
Distinction from Existing Geometry-Preserving Methods
Relational Knowledge Distillation (RKD) preserves relational information by distilling pairwise distances and triplet angles, while Mod-X selectively distills the off-diagonal information of contrastive matrices. Geometry-Aware Representational Alignment uses Gram-matrix matching and Procrustes distance to distill representations from a teacher language model to a student.
RGP takes a different approach to modeling representation drift in continual vision-language learning. It finds a single shared orthogonal transformation that jointly aligns the visual and textual embeddings of the frozen previous-task model with those of the current model, and penalizes only the residual drift after this alignment. Changes in image–image or text–text similarities cannot be explained by a shared orthogonal transformation and therefore contribute to the RGP loss, even when the cross-modal image–text similarity matrix used by Mod-X remains unchanged.
CTP preserves cross-modal and intra-modal sample topology by matching similarity relationships between the current and previous-task models. Building on RGP, Decomp-RGP takes a different approach by exactly decomposing the residual drift into visual deformation, textual deformation, and cross-modal orthogonal alignment mismatch. The first two terms measure the residual deformation after removing the best modality-specific orthogonal alignment, while the third captures the additional residual incurred by enforcing a shared alignment for both modalities. Budget-based Adaptive Weighting further allows us to assign separate target budgets to these components, enabling stronger regularization of cross-modal orthogonal alignment mismatch while independently controlling visual and textual deformation.
Comparison with Cross-modal Similarity KD and Gram Preservation
Cross-modal Similarity KD, Joint Gram KD, and RGP are all invariant to a shared orthogonal transformation applied to both modalities. However, similarities between images or between texts can still change even when image–text similarities remain unchanged. By penalizing the residual feature displacement after the optimal shared orthogonal alignment, RGP also constrains these intra-modal geometric changes, helping preserve the relationships within each modality during continual adaptation.
As established in Appendix A.2, matching the joint Gram matrices is equivalent to the existence of a shared orthogonal transformation. Accordingly, both Joint Gram KD and RGP have zero loss when the previous and current joint embeddings can be perfectly aligned by a shared orthogonal transformation. However, they measure deviations differently. Joint Gram KD penalizes changes in pairwise similarities across the joint Gram matrix, whereas RGP directly measures the minimum residual feature displacement after optimal shared alignment. Thus, changes in feature representations can affect multiple pairwise similarities, while RGP measures the remaining displacement directly.
The cross-modal block in Block Gram KD penalizes changes in image–text similarities, including those caused by visual or textual deformation. In contrast, Decomp-RGP measures the additional residual incurred by enforcing a shared orthogonal alignment rather than allowing separate modality-specific alignments. This decomposition allows us to control visual deformation, textual deformation, and cross-modal orthogonal alignment mismatch separately. It also allows us to assign a larger target budget to the latter while independently controlling the regularization of the two modalities.
Experimental Comparison
With λ=40, RGP improves ACC over Similarity KD and Joint Gram KD by 4.88 and 8.44 points, respectively. Under ρ=40%, the gains are 2.18 and 1.10 points. With the same (5%,5%,30%) allocation, Decomp-RGP outperforms Block Gram KD by 9.52 points in ACC, although Block Gram KD achieves better BWT.
RGP primarily improves plasticity, while decomposition and Budget-based Adaptive Weighting improve stability by independently controlling the three residual components, leading to a better stability–plasticity trade-off. The corresponding results have been updated in Table 2, and additional analysis has been added to Section 4.4 of the revised manuscript.