acceptodds
Under review as a conference paper at ICLR 2027

Beyond Representation Drift: Decomposed Relative Geometry Preservation for Continual Vision-Language Learning

Abstract

Continual adaptation of contrastive vision-language models leads to catastrophic forgetting, accompanied by substantial drift in their visual and textual representations. Existing methods mitigate forgetting by constraining changes in model parameters, feature representations, or cross-modal affinities, yet they do not explicitly separate changes that preserve the joint image–text geometry from those that alter it. We find that this distinction is substantial in practice: across a seven-task continual sequence, nearly half of the accumulated representation drift is attributable to a shared orthogonal transformation that leaves image–text retrieval unchanged. Motivated by this observation, we propose Decomposed Relative Geometry Preservation (Decomp-RGP), which measures drift after factoring out shared orthogonal transformation through closed-form Procrustes alignment and exactly decomposes the remaining residual into visual deformation, textual deformation, and cross-modal orthogonal alignment mismatch. We further introduce a budget-based adaptive weighting scheme to control the contributions of these components in fine-tuning process. Across seven image–text retrieval tasks, Decomp-RGP improves Mean R@1 by 5.32% over the strongest baseline, while maintaining stronger zero-shot performance across multiple CLIP backbones.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Vision Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PZ8rOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates catastrophic forgetting in continual learning of contrastive vision-language models, focusing on the relationship between representation drift and changes in cross-modal geometry. The authors argue that existing methods may unnecessarily penalize representation changes that do not affect image-text similarities. Through empirical analysis on a fixed continual learning sequence consisting of seven image-text retrieval datasets, they observe that approximately 39%–48% of representation drift can be explained by a shared orthogonal transformation of visual and textual embeddings, which preserves cross-modal similarities and retrieval rankings. Motivated by this observation, the authors propose Decomposed Relative Geometry Preservation. The method first uses orthogonal Procrustes alignment to factor out representation drift attributable to a shared orthogonal transformation between the previous and current models. It then exactly decomposes the remaining alignment residual into three components: visual deformation, textual deformation, and cross-modal orthogonal alignment mismatch, allowing each component to be regularized separately. In addition, the authors introduce a budget-based adaptive weighting mechanism that dynamically adjusts the regularization weights using exponential moving averages of the losses to control the relative contributions of different loss terms to the overall training objective.

Critical strengths
  1. The paper is generally well organized, with a coherent progression from research motivation to methodological design and experimental analysis. The authors introduce the proposed approach through intuitive examples and mathematical derivations, making the paper relatively easy to follow.
  2. The experimental results provide some empirical support for the effectiveness of the proposed method, although the necessity of its core design and its distinct advantages over existing approaches remain to be further established.
Critical weaknesses
  1. Insufficient distinction between the proposed method and existing work. The main motivation of this paper is based on the observation that shared orthogonal transformations can induce substantial representation drift without changing image-text similarities. However, orthogonal invariance is a well-established mathematical property, and related ideas have already been explored in Procrustes-based geometric distillation, relational distillation, and topology preservation for continual vision-language learning, such as Relational Knowledge Distillation[1], Mod-X[2], CTP[[[3], and Knowledge Distillation through Geometry-Aware Representational Alignment[4]. Although these studies do not necessarily propose the same three-component decomposition as Decomp-RGP, the paper does not sufficiently clarify its substantive differences from existing geometry-preserving approaches. In particular, the lack of direct comparisons with simpler geometry-invariant objectives makes it difficult to assess the distinct contribution and novelty of the proposed method. [1]Relational Knowledge Distillation(CVPR2019) [2]Continual Vision-Language Representation Learning with Off-Diagonal Information(ICML2023) [3]CTP: Towards Vision-Language Continual Pretraining via Compatible Momentum Contrast and Topology Preservation(ICCV2023) [4] Knowledge Distillation through Geometry-Aware Representational Alignment(arXiv2025)

  2. Insufficient validation of the justification and necessity of the three-component geometric decomposition. Although the proposed decomposition of RGP into three residual components is mathematically valid, this algebraic identity does not necessarily establish that the components accurately characterize harmful geometric changes during continual learning. A shared orthogonal transformation is only a sufficient condition for preserving cross-modal similarities, and residual changes that cannot be explained by such a transformation do not necessarily lead to degraded retrieval performance. Furthermore, the cross-modal asynchronous term measures the difference between the optimal objectives under independent and shared alignments, rather than directly measuring changes in cross-modal similarities or forgetting. More importantly, Table 2 shows that the full Decomp-RGP improves performance by approximately 6.79 percentage points over the undecomposed RGP. However, the current experiments do not sufficiently distinguish whether this improvement arises from the proposed geometric decomposition itself or from independent regularization and more flexible loss weighting. Consequently, the necessity of the core design and its underlying mechanism remain insufficiently supported.

  3. Limited experimental validation of the generalizability of the proposed method. The evaluation is primarily conducted on a fixed seven-task image-text retrieval sequence constructed by the authors. However, continual learning performance can be sensitive to task ordering and domain distribution shifts, and results from a single task sequence may not be sufficient to establish the robustness of the method. In addition, Decomp-RGP applies geometric constraints using only current-task data and a frozen model from the previous training stage. It therefore remains unclear whether preserving geometry on the current data distribution can reliably protect knowledge acquired from previous tasks with substantially different distributions. Although the paper compares against several representative continual vision-language learning methods, its coverage of recent geometry-preserving approaches remains limited[5-7], and some baselines were originally developed for continual learning settings different from continual image-text retrieval. Finally, while Decomp-RGP achieves the highest overall retrieval accuracy, it does not obtain the best backward transfer performance, suggesting that the claims regarding its stability-plasticity advantages should be more carefully qualified. [5]Pi-CCA: Prompt-Invariant CCA Certificates for Replay-Free Continual Multimodal Learning(ICLR 2026) [6] Hyperbolic Multimodal Continual Learning(ICML 2026) [7] HypCL: Adapting CLIP in Hyperbolic Space for Continual Learning(ICML 2026)

Questions for authors
  1. Novelty and distinction from existing methods. Prior studies have already explored Procrustes-based geometric distillation, relational distillation, and topology or cross-modal geometry preservation in continual vision-language learning. Could the authors further clarify the substantive technical differences between Decomp-RGP and these existing approaches? In particular, why would directly preserving Gram matrices or cross-modal similarity matrices be insufficient to address the shared orthogonal drift considered in this paper? Comparisons with these simpler geometry-invariant objectives under a unified experimental protocol would help establish the distinct value of the proposed method.
  2. Justification of the three-component decomposition and sources of performance improvement. Could the authors provide further theoretical or empirical evidence explaining how the three components—visual deformation, textual deformation, and cross-modal orthogonal alignment mismatch—relate to harmful geometric changes and catastrophic forgetting? In particular, given the substantial improvement of approximately 6.79 percentage points achieved by the full Decomp-RGP over the undecomposed RGP, could the authors conduct component-wise ablations or fair comparisons with alternative geometry-preserving losses using the same adaptive weighting mechanism? Such evidence would help clarify whether the improvement primarily stems from the specific geometric decomposition or from the flexibility of independently weighting multiple loss terms.
  3. Experimental robustness and preservation of historical knowledge. Could the authors further evaluate the robustness of Decomp-RGP under different task orders and varying degrees of domain distribution shift? Moreover, since the method imposes geometric constraints using only current-task data and a frozen previous-stage model, could the authors clarify to what extent this mechanism can preserve historical knowledge from tasks whose distributions differ substantially from the current task? Such analyses would help establish the scope of the paper's conclusions and the generalizability of the reported stability-plasticity trade-off. Overall, I find the motivation and formulation of this work interesting, and the reported empirical improvements are encouraging. However, the concerns above currently limit my confidence in the novelty and necessity of the proposed approach. I would be willing to reconsider my assessment and potentially raise my rating if the authors can adequately address these concerns with convincing explanations and supporting evidence.
5 comments
AuthorsOpenReview ↗

Response to Reviewer PZ8r (W1, Q1)

We thank the reviewer for highlighting these related works.

Distinction from Existing Geometry-Preserving Methods

Relational Knowledge Distillation (RKD) preserves relational information by distilling pairwise distances and triplet angles, while Mod-X selectively distills the off-diagonal information of contrastive matrices. Geometry-Aware Representational Alignment uses Gram-matrix matching and Procrustes distance to distill representations from a teacher language model to a student.

RGP takes a different approach to modeling representation drift in continual vision-language learning. It finds a single shared orthogonal transformation that jointly aligns the visual and textual embeddings of the frozen previous-task model with those of the current model, and penalizes only the residual drift after this alignment. Changes in image–image or text–text similarities cannot be explained by a shared orthogonal transformation and therefore contribute to the RGP loss, even when the cross-modal image–text similarity matrix used by Mod-X remains unchanged.

CTP preserves cross-modal and intra-modal sample topology by matching similarity relationships between the current and previous-task models. Building on RGP, Decomp-RGP takes a different approach by exactly decomposing the residual drift into visual deformation, textual deformation, and cross-modal orthogonal alignment mismatch. The first two terms measure the residual deformation after removing the best modality-specific orthogonal alignment, while the third captures the additional residual incurred by enforcing a shared alignment for both modalities. Budget-based Adaptive Weighting further allows us to assign separate target budgets to these components, enabling stronger regularization of cross-modal orthogonal alignment mismatch while independently controlling visual and textual deformation.

Comparison with Cross-modal Similarity KD and Gram Preservation

Cross-modal Similarity KD, Joint Gram KD, and RGP are all invariant to a shared orthogonal transformation applied to both modalities. However, similarities between images or between texts can still change even when image–text similarities remain unchanged. By penalizing the residual feature displacement after the optimal shared orthogonal alignment, RGP also constrains these intra-modal geometric changes, helping preserve the relationships within each modality during continual adaptation.

As established in Appendix A.2, matching the joint Gram matrices is equivalent to the existence of a shared orthogonal transformation. Accordingly, both Joint Gram KD and RGP have zero loss when the previous and current joint embeddings can be perfectly aligned by a shared orthogonal transformation. However, they measure deviations differently. Joint Gram KD penalizes changes in pairwise similarities across the joint Gram matrix, whereas RGP directly measures the minimum residual feature displacement after optimal shared alignment. Thus, changes in feature representations can affect multiple pairwise similarities, while RGP measures the remaining displacement directly.

The cross-modal block in Block Gram KD penalizes changes in image–text similarities, including those caused by visual or textual deformation. In contrast, Decomp-RGP measures the additional residual incurred by enforcing a shared orthogonal alignment rather than allowing separate modality-specific alignments. This decomposition allows us to control visual deformation, textual deformation, and cross-modal orthogonal alignment mismatch separately. It also allows us to assign a larger target budget to the latter while independently controlling the regularization of the two modalities.

Experimental Comparison

MethodACC (%) ↑BWT (%) ↑
Similarity KD λ=40\lambda=4055.23-0.93
Similarity KD ρ=40%\rho=40\%51.64+0.53
Joint Gram KD: λ=40\lambda=4051.67+1.58
Joint Gram KD: ρ=40%\rho=40\%52.72+0.28
Block Gram KD: ρv\rho_v=5%, ρt\rho_t=5%, ρa\rho_a=30%50.69+1.42
RGP: λ=40\lambda=4060.11−1.41
RGP: ρ=40%\rho=40\%53.82+0.19
Decomp-RGP: ρv\rho_v=5%, ρt\rho_t=5%, ρa\rho_a=30%60.21−0.58

With λ=40\lambda=40, RGP improves ACC over Similarity KD and Joint Gram KD by 4.88 and 8.44 points, respectively. Under ρ=40%\rho=40\%, the gains are 2.18 and 1.10 points. With the same (5%,5%,30%) allocation, Decomp-RGP outperforms Block Gram KD by 9.52 points in ACC, although Block Gram KD achieves better BWT.

RGP primarily improves plasticity, while decomposition and Budget-based Adaptive Weighting improve stability by independently controlling the three residual components, leading to a better stability–plasticity trade-off. The corresponding results have been updated in Table 2, and additional analysis has been added to Section 4.4 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer PZ8r (W2, Q2)

We thank the reviewer for this insightful comment and for highlighting the importance of clarifying the contributions of the three-component decomposition and Budget-based Adaptive Weighting to the observed performance improvements.

Geometric Changes and Catastrophic Forgetting

The three components measure different types of geometric changes, and we do not assume that every residual change necessarily causes forgetting. RGP uses shared orthogonal alignment to preserve joint image–text geometry, while allowing geometry-preserving representation changes. Intra-modal deformation does not necessarily degrade retrieval performance as long as the correct image–text rankings are preserved. However, without appropriate constraints, such changes may accumulate across tasks and disrupt previously learned relationships, increasing the risk of catastrophic forgetting.

The cross-modal orthogonal alignment mismatch measures the additional residual introduced by shared alignment compared with independent modality-specific alignments. Although it does not directly measure changes in cross-modal similarities or forgetting, it captures the disagreement between the optimal visual and textual alignments. Fig. 1(b) shows that the synchronization between visual and textual transformations affects retrieval performance even at the same drift magnitude, supporting the practical relevance of this component. Furthermore, Fig. 5(b)(c) shows that different budget allocations lead to different ACC and BWT, supporting the value of independently controlling these components during continual learning.

Contributions of Decomposition and Budget-based Adaptive Weighting

To clarify the source of the 6.79 percentage-point improvement, we provide a more detailed comparison of Feature Distillation, RGP, and Decomp-RGP under different weighting strategies.

MethodACC ↑BWT ↑
Feature Distillation λ=40\lambda=4051.03+0.35
RGP λ=40\lambda=4060.11−1.41
RGP ρ=40%\rho=40\%53.82+0.19
Decomp-RGP λv=40, λt=40, λa=40\lambda_v = 40,\ \lambda_t = 40,\ \lambda_a = 4060.17−1.38
Decomp-RGP λv=5, λt=5, λa=30\lambda_v = 5,\ \lambda_t = 5,\ \lambda_a = 3061.89-2.64
Decomp-RGP ρv=\rho_v = 5%, ρt=\rho_t = 5%, ρa=\rho_a = 30%60.21−0.58
Decomp-RGP ρv=\rho_v = 8%, $\rho_t = $ 2%, $ \rho_a = $ 30%60.61−0.75

With the same fixed coefficient λ=40\lambda=40, RGP improves ACC over Feature Distillation by 9.08 percentage points (60.11% vs. 51.03%), demonstrating the benefit of removing shared orthogonal drift. As established in Eq. (6), Decomp-RGP is mathematically equivalent to RGP when the three coefficients are equal. Their nearly identical ACC (60.17% vs. 60.11%) confirms that the decomposition alone does not explain the performance improvement.

The decomposition enables independent control of the three components. With fixed coefficients (5,5,30), Decomp-RGP achieves 61.89% ACC but suffers greater forgetting (BWT −2.64%). Budget-based Adaptive Weighting with (5%,5%,30%) improves BWT to −0.58%, with ACC at 60.21%. Adjusting the budget allocation to (8%,2%,30%) further improves ACC to 60.61%, with BWT of −0.75%.

The 6.79-point gain highlighted by the reviewer comes from comparing RGP with a total adaptive budget of 40% (ACC 53.82%) against Decomp-RGP with separately allocated budgets (8%, 2%, and 30% with ACC 60.61%). This improvement therefore reflects the combined effect of decomposition and adaptive budget allocation, rather than decomposition alone. RGP provides the main benefit by removing geometry-preserving drift, while Decomp-RGP further allows us to regulate the three components independently, achieving a better balance between stability and plasticity.

AuthorsOpenReview ↗

Response to Reviewer PZ8r (W3, Q3)

We thank the reviewer for the thoughtful and constructive feedback, as well as for recognizing the motivation and empirical results of our work. We appreciate the opportunity to further clarify the robustness of Decomp-RGP and its ability to preserve historical knowledge under distribution shifts.

Robustness Across Task Orders, Domain Shifts, and Learning Protocols

We thank the reviewer for raising concerns about experimental generalizability. To evaluate the sensitivity of Decomp-RGP to task ordering, we conducted additional experiments using a different seven-task retrieval sequence. As reported in the revised Table 4, Decomp-RGP achieves 60.12% ACC, outperforming the strongest baseline (56.56%) by 3.56%. This result indicates that the effectiveness of Decomp-RGP is not restricted to the original task order.

Furthermore, Fig. 4 presents the zero-shot performance after continual learning across the seven datasets in our retrieval protocol. Decomp-RGP outperforms state-of-the-art methods on five datasets, with only a slightly lower result on one dataset.

To further evaluate our method under a different learning protocol, we conducted additional experiments on the 11-dataset Multimodal Task-Incremental Learning (MTIL) classification benchmark used in ZSCL. Following the reviewer's suggestions, we also included five additional methods:

  • Knowledge Distillation through Geometry-Aware Representational Alignment(arXiv2025)
  • CTP: Towards Vision-Language Continual Pretraining via Compatible Momentum Contrast and Topology Preservation(ICCV2023)
  • Pi-CCA: Prompt-Invariant CCA Certificates for Replay-Free Continual Multimodal Learning(ICLR 2026)
  • Hyperbolic Multimodal Continual Learning(ICML 2026)
  • HypCL: Adapting CLIP in Hyperbolic Space for Continual Learning(ICML 2026)

Knowledge Distillation through Geometry-Aware Representational Alignment was originally developed for language model distillation. To adapt its Procrustes-based objective to CLIP, we concatenate the image and class-text embeddings and apply Procrustes alignment between the joint representations of the current model and the frozen previous-stage model. CTP was originally designed for vision-language continual pretraining. We adapted it to our CLIP-based setting using the image and text embeddings produced by CLIP.

Under this MTIL protocol, Decomp-RGP outperforms the strongest baseline by 4.01% in average accuracy. The additional comparisons are reported in Table 3 of the revised manuscript. These results further support the effectiveness of Decomp-RGP across different task orders, datasets, and continual learning settings.

Preservation of Historical Knowledge

We thank the reviewer for raising this question. Decomp-RGP operates without storing or replaying historical training data. At each stage, the frozen previous-stage model provides a geometric reference for current-task samples. Our method allows visual and textual representations to undergo shared orthogonal transformations during adaptation, while constraining the remaining geometric changes after alignment. Historical knowledge is therefore preserved indirectly, and the effectiveness of this mechanism depends on how well current-task samples reflect the representation relationships relevant to previous tasks.

After learning all seven tasks, Decomp-RGP achieves a Mean R@1 of 83.89% on Flickr30K. In comparison, fine-tuning without regularization achieves only 73.08% (Fig. 3 and Table 1). Across the full sequence, Decomp-RGP achieves a BWT of −0.75%, compared with −20.40% for fine-tuning CLIP without regularization (Table 2). These results show that geometric constraints applied using only current-task data can substantially reduce forgetting. Although the tasks have different data distributions, Decomp-RGP can still reduce forgetting by preserving the relative geometry of the previous-stage model. Our method allows shared orthogonal transformations while constraining the remaining geometric changes. This helps preserve previously learned knowledge while allowing the model to adapt to new task.

Reviewer_fKAJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that continual learning methods for vision–language models should not constrain all representation change, since representations naturally drift during continual adaptation. The authors assume that a shared orthogonal transformation, under which visual and textual embeddings move together, is a benign change that preserves image–text geometry. They propose Relative Geometry Preservation (RGP), which removes this shared component via closed-form Procrustes alignment and measures only the residual drift. Decomp-RGP further decomposes the residual into visual deformation, textual deformation, and cross-modal alignment mismatch, and controls each term with a budget-based adaptive weighting scheme. On a seven-task continual image–text retrieval sequence with CLIP, the method improves Mean R@1 by 5.32pp over the strongest baseline and better preserves zero-shot classification performance across multiple CLIP backbones.

Critical strengths
  • Simple and principled formulation of drift: The paper decomposes representation drift with a closed-form Procrustes alignment, separating a shared orthogonal component from visual deformation, textual deformation, and cross-modal alignment mismatch. The decomposition is exact (L_RGP = L_v + L_t + L_async) and adds no learnable parameters. This matters to my assessment because it gives a clean and interpretable lens on what kind of change occurs during continual adaptation, rather than treating all representation change as uniformly harmful, and the regularizer is easy to adopt in existing CLIP fine-tuning pipelines.
Critical weaknesses

W1. The core premise and the source of the improvement are insufficiently supported.: The paper’s central claim is that shared orthogonal drift is a natural, benign change that existing methods unnecessarily constrain. However, Fig. 1(a)(b) only illustrate the well-known invariance of inner products under synthetic rotations, and the Gaussian-noise control in Fig. 1(c) (lines 81–82) matches only the displacement magnitude, not the covariance or low-dimensional structure of learned drift. More importantly, the paper does not show that constraining this component harms learning: in Table 2, factoring out the shared rotation (RGP) improves ACC by only ~1.7pp over budget-matched feature distillation, while most of the gain comes from the decomposed, separately weighted objective. The premise regarding prior work is also unclear, since baselines that distill similarity or affinity matrices (e.g., Mod-X, DKR) appear to be invariant to shared rotation by construction, and no analysis is provided on whether this is what limits their performance. This is consequential because the paper’s motivation and novelty rest on this premise, yet the evidence attributes most of the improvement to a different component.

W2. The evaluation is narrow and the hyperparameter selection is not well justified.: Both the motivation and the main evaluation focus on a single seven-task retrieval sequence with one task order, without justification for the dataset or sequence choice, and with CLIP as the only model family. It remains unclear whether the conclusions hold for continual classification (e.g., ZSCL-style MTIL), other task orders, or models such as SigLIP. In addition, the decomposition introduces new hyperparameters that appear to be selected on the same evaluation sequence (Fig. 5) without a separate validation split, while Table 2 suggests limited benefit from the specific ratio and adaptive weighting (fixed-λ 60.17, 5:5:30 60.21, 8:2:30 60.61). This is consequential because it makes it difficult to judge whether the reported gains generalize beyond the specific setting and tuning used in the paper.

Questions for authors

Q1. Can the authors show that constraining shared orthogonal drift actually harms learning? In Table 2, RGP improves ACC by only ~1.7pp over budget-matched feature distillation, while most of the gain comes from the decomposed objective. Would the noise contrast in lines 81–82 hold under a control matching the covariance of learned drift?

Q2. Do the compared baselines actually constrain shared orthogonal transformation? Similarity-based methods (e.g., Mod-X, DKR) appear invariant to it by construction. How does the method compare with direct Gram-matrix distillation, which Appendix A.2 shows is equivalent to preserving geometry up to a shared rotation?

Q3. Why were these datasets and this single task order chosen? Does the method hold under other orders and under continual classification settings such as MTIL?

Q4. Were the hyperparameters in Fig. 5 selected on the evaluation sequence itself? Given that fixed-λ Decomp-RGP already reaches 60.17, what does adaptive weighting add?

5 comments
AuthorsOpenReview ↗

Response to Reviewer fKAJ (W1, Q1)

We thank the reviewer for the valuable suggestions regarding our motivation and experimental analysis.

Orthogonal drift in real training and the Covariance-matched Gaussian control.

We agree with the reviewer that Figs.1(a) and 1(b) use controlled perturbations rather than representations obtained from actual continual training. These experiments are designed to illustrate how shared orthogonal transformations and general feature perturbations affect image-text relationships differently. Fig. 1(c) further examines representation drift during actual continual learning, showing that a shared orthogonal transformation accounts for approximately 39% to 48% of the total drift across the seven training stages.

Following the reviewer's suggestion, we further constructed a covariance-matched Gaussian control using the joint mean and covariance of the observed image-text displacements, while matching the overall displacement magnitude. Shared orthogonal alignment explains 38.22% of the Gaussian drift, compared with 3.06% for isotropic Gaussian noise and 47.62% for real training. These results show that matching the covariance structure explains part of the shared orthogonal component, although the proportion remains higher in real training. We sincerely thank the reviewer for this helpful suggestion and have accordingly refined the wording in the Introduction to better reflect the scope of our observation.

We clarify that the covariance-matched Gaussian control examines the statistical properties of observed drift, while RGP addresses how such drift should be constrained. By aligning representations through a shared orthogonal transformation, RGP avoids penalizing geometry-preserving rotations. Its decomposition and Budget-based Adaptive Weighting further enable independent control of residual geometric changes. Together with our observations from real training, these properties motivate our approach.

Effect of Constraining Shared Orthogonal Drift.

Under the same adaptive budget of ρ=40\rho=40%, RGP improves ACC over Feature Distillation by 1.70 percentage points (53.82% vs. 52.12%). However, adaptive weighting adjusts the regularization coefficients according to the loss magnitudes, so the same budget does not necessarily produce identical coefficients. We therefore also compare the two methods with a fixed λ=40\lambda=40. In this setting, Feature Distillation penalizes both the RGP residual and the additional differences attributable to shared orthogonal alignment, whereas RGP penalizes only the residual after alignment.

With λ=40\lambda=40, RGP achieves 60.11% ACC, compared with 51.03% for Feature Distillation, a difference of 9.08 percentage points. This result provides further evidence that penalizing shared orthogonal drift can restrict the model's ability to adapt to new tasks.

Source of Performance Improvement

MethodACC ↑BWT ↑
Feature Distillation Fixed λ=40\lambda=4051.03+0.35
RGP λ=40\lambda=4060.11−1.41
RGP ρ=40%\rho=40\%53.82+0.19
Decomp-RGP λv=40, λt=40, λa=40\lambda_v = 40,\ \lambda_t = 40,\ \lambda_a = 4060.17−1.38
Decomp-RGP λv=5, λt=5, λa=30\lambda_v = 5,\ \lambda_t = 5,\ \lambda_a = 3061.89-2.64
Decomp-RGP ρv\rho_v = 5%, ρt\rho_t = 5%, ρa\rho_a = 30%60.21−0.58
Decomp-RGP ρv\rho_v = 8%, ρt\rho_t = 2%, ρa\rho_a = 30%60.61−0.75

As shown above, under the same fixed λ=40\lambda=40, RGP achieves 60.11% ACC compared with 51.03% for Feature Distillation, an improvement of +9.08 percentage points. This supports the benefit of penalizing only the remaining feature differences after shared orthogonal alignment, allowing greater flexibility when learning new tasks. This is the main source of the plasticity improvement in our method. Applying Budget-based Adaptive Weighting to RGP improves BWT from −1.41% to +0.19%, showing its role in improving stability by adjusting the regularization strength during training.

With equal fixed coefficients (λv=λt=λa=40\lambda_v=\lambda_t=\lambda_a=40), RGP and Decomp-RGP achieve nearly identical ACC (60.11% vs. 60.17%) and BWT, consistent with the mathematical equivalence in Eq. (6). Based on the ablations in Fig. 5, we select ρv=8\rho_v=8%, ρt=2\rho_t=2%, and ρa=30\rho_a=30%, equivalent to the previously reported budget allocation of 20%, 5%, and 75% (previous version). We have updated the labels in Fig. 5 to match Table 2, expressing each component budget as a percentage of the total loss (revised version).

In summary, RGP primarily improves plasticity but provides relatively limited stability, while Decomposition and Budget-based Adaptive Weighting help achieve a better balance between plasticity and stability. Additional ablation experiments have been added to Table 2, and further analysis has been included in Section 4.4.

AuthorsOpenReview ↗

Response to Reviewer fKAJ (W1, Q2)

Distinction from Existing Similarity-based Methods

We thank the reviewer for raising this point.

Similarity-based distillation methods preserve image–text relationships by matching the cross-modal similarity or affinity distributions between the previous and current models. Specifically, Mod-X selectively distills the off-diagonal information in the contrastive matrices, while DKR filters and rectifies incorrect image–text affinities during knowledge distillation. For any R∈O(d)R\in O(d), applying the same orthogonal transformation to both the visual and textual representations of the current model gives

=V_nRR^\top T_n^\top =V_nT_n^\top.$$ Therefore, **a shared orthogonal transformation leaves the corresponding distillation losses of Mod-X and DKR unchanged**. RGP finds the optimal shared orthogonal transformation that jointly aligns the visual and textual representations of the previous and current models, and penalizes the residual feature differences after alignment. Even when image–text similarities within the current batch remain unchanged, pairwise similarities among images or among texts may still change. **RGP therefore provides a regularization signal for intra-modal geometric changes that are not reflected in cross-modal similarities**, helping the model preserve the geometric relationships within both modalities while adapting to new tasks. Decomp-RGP further decomposes the residual after joint alignment into three components: visual deformation, textual deformation, and cross-modal orthogonal alignment mismatch. The first two terms measure the residual feature differences after independently finding the optimal modality-specific orthogonal alignments for the visual and textual representations, respectively. The third term captures the additional residual incurred by enforcing a shared orthogonal alignment rather than separate modality-specific alignments. By assigning separate target budgets to these three components, we can place greater emphasis on regularizing cross-modal orthogonal alignment mismatch while independently controlling the regularization strengths for visual and textual deformation. **Comparison with Gram-matrix Distillation** As shown in Appendix A.2, the old and new joint Gram matrices are identical if and only if the corresponding embeddings can be aligned exactly by a single shared orthogonal transformation. Joint Gram KD and RGP therefore share the same zero-loss condition. When the geometry changes, Joint Gram KD penalizes discrepancies in pairwise similarities, whereas RGP uses Procrustes alignment to find the best shared orthogonal transformation and penalizes the discrepancy that remains after alignment. The two losses can thus assign different penalties to the same geometric change. To examine this difference, we conducted additional comparisons with Joint-Gram KD and Block-Gram KD under corresponding fixed and adaptive weighting settings. | Method | ACC (%) ↑ | BWT (%) ↑ | |---|---:|---:| | Joint Gram KD: $\lambda=40$ | 51.67 (+8.25) | +1.58 | | Joint Gram KD: $\rho=40\%$ | 52.72 (+9.30) | +0.28 | | Block Gram KD: $\rho_v$=5\%, $\rho_t$=5\%, $\rho_a$=30\%\$ | 50.69 (+7.27) | +1.42 | | RGP: $\lambda=40$ | 60.11 (+16.69) | -1.41 | | RGP: $\rho=40\%$ | 53.82 (+10.40) | +0.19 | | Decomp-RGP: $\rho_v$=5\%, $\rho_t$=5\%, $\rho_a$=30\%\$ | **60.21 (+16.79)** | -0.58 | Under the same fixed coefficient $\lambda=40$, RGP achieves 60.11% ACC, outperforming Joint-Gram KD by 8.44 percentage points. With the same adaptive budget of $\rho=40%$, RGP also improves ACC by 1.10 percentage points (53.82% vs. 52.72%), while both methods maintain BWT close to zero. Furthermore, under the same component-wise budget allocation 5%,5%,30%, Decomp-RGP achieves 60.21% ACC, compared with 50.69% for Block-Gram KD, an improvement of 9.52 percentage points. These results demonstrate that, despite their shared orthogonal invariance, Procrustes-based residual preservation and direct Gram-matrix distillation lead to different stability–plasticity trade-offs. In the evaluated settings, RGP provides greater plasticity, while the adaptive weighting helps control the associated forgetting. The Gram KD ablation results have been added to Table 2, with the corresponding analysis presented in Section 4.4 of the revised manuscript.
AuthorsOpenReview ↗

Response to Reviewer fKAJ (W2, Q3, Q4)

We thank the reviewer for the constructive feedback and valuable suggestions, which have helped us improve the clarity and strengthen the experimental evaluation of our work.

Dataset Selection and Generalization Across Task Orders

We selected these datasets because they contain rich natural language descriptions, such as detailed image captions, instead of simple category-based descriptions such as "a photo of [class]". This allows us to evaluate continual learning in settings where the model needs to learn and preserve detailed image-text semantic relationships, including the joint image-text geometry.

For SUN397, which originally provides category labels, we use BLIP-2 to generate text descriptions conditioned on the images and their corresponding scene categories. We further use Qwen2.5-VL to assess the quality of the generated image-text pairs and filter out inaccurate descriptions. Detailed dataset splits, preprocessing procedures, and evaluation protocols have been added to Appendix A.9 of the revised manuscript.

The original seven-task sequence was randomly selected without using test performance to determine the task order. To further evaluate the robustness of Decomp-RGP across different task orders, we conducted an additional experiment with another randomly selected sequence, reported in Table 4 of the revised manuscript. Under this new task order, Decomp-RGP achieves 60.12% average ACC, outperforming the strongest baseline (56.56%) by 3.56 percentage points. This result demonstrates that the performance improvements of Decomp-RGP extend to another task order.

Generalization to Continual Classification and CLIP Backbones

To further evaluate the applicability of Decomp-RGP to continual classification, we conducted additional experiments on the ZSCL-style MTIL benchmark, with results reported in Table 3 of the revised manuscript.

We follow the MTIL task sequence while maintaining the same training configuration as our continual image-text retrieval experiments, including a fixed learning rate of 1×10−51\times10^{-5} across all tasks. Unlike the original ZSCL training recipe, which uses dataset-specific learning rates and 100K ImageNet reference data, our experiments use a unified learning rate and do not incorporate ImageNet reference data. Under this setting, Decomp-RGP achieves 83.82% average classification accuracy, outperforming the compared baselines. These results demonstrate that Decomp-RGP is also effective in continual image classification.

Furthermore, we extended our evaluation to different CLIP backbones, including ViT-B/32, ViT-B/16, ViT-L/14, and ViT-L/14@336px, as reported in Table 5. Decomp-RGP consistently improves continual image-text retrieval performance across these backbones, demonstrating its effectiveness across different model scales and input resolutions.

Hyperparameter Selection and Stability–Plasticity Trade-off

We would like to clarify that all hyperparameters, including the total regularization budget and its allocation, were selected using the validation sets independently of the test sets. Fig. 5 reports the test results under different budget configurations to illustrate the stability–plasticity trade-off. Detailed dataset splits are provided in Appendix A.9.

The main improvement in plasticity comes from RGP, while Decomposition and Budget-based Adaptive Weighting provide independent control over the three residual components and help balance stability and plasticity during continual adaptation.

At the beginning of each task, the current and frozen previous models are identical, making the initial RGP residual nearly zero. Meanwhile, CLIP and regularization loss magnitudes vary across tasks, making fixed coefficients unreliable. Budget-based Adaptive Weighting dynamically adjusts the coefficients according to loss magnitudes, keeping each component's contribution close to its target budget. This prevents over- or under-regularization relative to the CLIP loss and improves the stability–plasticity trade-off.

The results in Table 2 further illustrate this effect.

MethodSettingACC ↑BWT ↑
Decomp-RGPλv=40, λt=40, λa=40\lambda_v = 40,\ \lambda_t = 40,\ \lambda_a = 4060.17−1.38
Decomp-RGPλv=5, λt=5, λa=30\lambda_v = 5,\ \lambda_t = 5,\ \lambda_a = 3061.89-2.64
Decomp-RGPρv\rho_v = 5%, ρt\rho_t = 5%, ρa\rho_a = 30%60.21−0.58
Decomp-RGPρv\rho_v = 8%, ρt\rho_t = 2%, ρa\rho_a = 30%60.61−0.75

The table above shows that Budget-based Adaptive Weighting improves BWT when ACC is high but BWT is relatively poor. Its purpose is to balance plasticity and stability.

Additional ablation experiments and analyses have been added to Table 2 and Section 4.4, respectively. We have also revised the labels in Fig. 5 to report the contribution of each component as a proportion of the total loss, consistent with Table 2, rather than as a percentage of the total regularization budget as previously shown.

Reviewer_fCRROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies representation drift during continual adaptation of contrastive vision-language models. Relative Geometry Preservation (RGP) removes a shared orthogonal transformation through Procrustes alignment before penalizing residual drift. The residual is decomposed into visual deformation, textual deformation, and the additional cost of requiring a shared rather than modality-specific alignment. An adaptive weighting scheme assigns loss budgets to these components. Experiments on a seven-domain retrieval sequence report a 5.32 percentage-point improvement over the strongest evaluated baseline, alongside zero-shot classification results.

Critical strengths
  1. The Procrustes formulation is concise, and the closed-form objectives and nonnegative decomposition are correctly derived. Explicitly measuring drift and controlling its components provides a useful framework for investigating the stability-plasticity trade-off.

  2. The reported retrieval improvement is substantial and the zero-shot classification evaluation provides additional evidence that the improvement is not confined to the trained retrieval tasks.

Critical weaknesses
  1. I’m not yet convinced the paper clearly separates its contribution from existing relational objectives, or point out where the gains come from. Cross-modal similarity objectives are already invariant to a shared orthogonal transformation because (VR)(TR)^T = VT^T, and Mod-X already discusses synchronized rotations across modalities. So for my understanding, showing that point-wise feature matching has this limitation doesn’t show that relational distillation has it too.

  2. The decomposition satisfies L_RGP = L_v + L_t + L_async, which means that with equal coefficients the objective doesn’t change at all. Yet Table 2 compares adaptive-budget RGP against a fixed-weight Decomp-RGP, and the only detail given for the latter’s coefficients is “lambda=40.” As far as I can tell, this mixes the effect of the decomposition with the effect of reweighting. I’m not claiming the numbers contradict each other; my concern is that I can’t tell which change is responsible for the improvement, and the setup isn’t fully specified. I think controls with matched weighting are needed before the central methodological claim holds up. As a research paper this is not clear enough.

  3. Several evaluation details are unclear to me, including the data splits, the candidate pools for retrieval, how the SUN397 text is constructed, and how the budgets are selected on validation data. Only one task order is reported, and Table 3 doesn’t include paired baseline results for the additional backbones. Taken together, these make it hard for me to be confident about how general the gains are, or about the explanation offered for them.

Questions for authors
  1. What exactly are the component coefficients in the fixed-weight Decomp-RGP row of Table 2? I couldn’t find them beyond “lambda=40.” Would it be possible to add a fixed-weight RGP baseline with matched weighting, plus an equal-weight decomposition? That would let me see how much of the gain comes from the weighting versus the decomposition itself.

  2. When the weighting strategy and tuning budget are matched, what does explicit Procrustes preservation actually buy you over similarity-based or joint-Gram preservation, which are already rotation-invariant?

6 comments
AuthorsOpenReview ↗

Response to Reviewer fCRR W1

We thank the reviewer for this important comment and the opportunity to clarify the distinction between RGP and existing relational distillation methods.

W1. Clarification of Contributions Relative to Existing Methods.

We thank the reviewer for this important comment. We agree that Similarity KD and Mod-X are already invariant to shared orthogonal transformations. Point-wise feature matching is sensitive to such rotations, whereas relational distillation preserves this invariance. Our contribution is to decompose the residual after shared orthogonal alignment and independently control visual deformation, textual deformation, and cross-modal orthogonal alignment mismatch during continual learning.

Similarity KD directly preserves cross-modal similarities, while Mod-X selectively distills the off-diagonal entries of the cross-modal contrastive matrix. Mod-X also analyzes how asynchronous rotations between visual and textual representations affect retrieval. However, both objectives primarily constrain cross-modal relationships and are invariant to a shared orthogonal transformation applied to both modalities, since (VR)(TR)⊤=VT⊤(VR)(TR)^\top=VT^\top:

VnTn⊤≈VoTo⊤⏟Cross-modal similarity\underbrace{V_nT_n^{\top}\approx V_oT_o^{\top}}_{\text{Cross-modal similarity}}

However, preserving cross-modal similarities does not guarantee intra-modal geometric consistency. Even when image-text similarities within the current batch remain unchanged, similarities between images or between texts may still change. Such changes can disrupt previously learned semantic structures and affect the model's ability to retain useful distinctions for subsequent tasks.

RGP addresses this limitation by explicitly optimizing a single shared orthogonal matrix R∈O(d)R\in O(d) to jointly align the visual and textual representations of the current and previous models:

Vn≈VoR,Tn≈ToR.V_n\approx V_oR,\qquad T_n\approx T_oR.

After this alignment, RGP measures the remaining representation differences. Using the same RR for both modalities jointly preserves three geometric relationships:

VnTn⊤≈VoTo⊤⏟Cross-modal alignment\underbrace{V_nT_n^{\top}\approx V_oT_o^{\top}}_{\text{Cross-modal alignment}}

VnVn⊤≈VoVo⊤⏟Visual intra-modal geometry\underbrace{V_nV_n^{\top}\approx V_oV_o^{\top}}_{\text{Visual intra-modal geometry}}

TnTn⊤≈ToTo⊤⏟Textual intra-modal geometry\underbrace{T_nT_n^{\top}\approx T_oT_o^{\top}}_{\text{Textual intra-modal geometry}}

Unlike cross-modal similarity distillation alone, RGP also captures changes in visual-visual and text-text relationships that may not be reflected in the current image-text similarities. By constraining these changes after shared orthogonal alignment, RGP helps retain previously learned semantic distinctions that may remain important for subsequent tasks, without penalizing shared orthogonal transformations.

We report the comparison with Mod-X in Table 1 and the additional comparison with Cross-modal Similarity KD in Table 2. The relevant results are summarized below:

MethodSettingACC (%) ↑BWT (%) ↑
Cross-modal Similarity KDλ=40\lambda = 4055.23-0.93
Cross-modal Similarity KDΣρ=40%\Sigma\rho=40\%51.64+0.53
Mod-XOfficial setting51.01+0.71
RGPλ=40\lambda = 4060.11-1.41
RGPΣρ=40%\Sigma\rho=40\%53.82+0.19
Decomp-RGPρv\rho_v = 8%, ρt\rho_t = 2%, ρa\rho_a = 30%60.61−0.75

With fixed weights (λ=40\lambda=40), RGP improves ACC over Cross-modal Similarity KD from 55.23% to 60.11%, while BWT decreases from −0.93% to −1.41%. Under the same 40% adaptive budget, RGP achieves 53.82% ACC, outperforming Cross-modal Similarity KD (51.64%) and Mod-X (51.01%) by 2.18 and 2.81 percentage points, respectively, while maintaining positive BWT (+0.19%). These results support the benefit of jointly preserving intra-modal geometry and cross-modal alignment. Adaptive weighting generally achieves lower ACC than fixed weights because it adjusts the regularization strength according to the contrastive loss magnitude. This can improve BWT at the cost of some ACC, with the primary goal of balancing stability and plasticity during continual learning. Decomp-RGP further improves ACC to 60.61% through independent control of the residual geometric components, with a BWT of −0.75%, reflecting a trade-off between plasticity and stability. The additional Similarity KD ablation results have been added to Table 2, with the corresponding analysis included in Section 4.4 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer fCRR (W2, Q1)

W2, Q1. Weighting and controls in Table 2

We thank the reviewer fCRR for pointing out the ambiguity in Table 2. Following the reviewer's suggestion, we have added the fixed-weight RGP baseline (λ=40\lambda=40) and clarified the settings in Table 2. Specifically, we renamed “Decomp-RGP: decompose into Lv/Lt/LasyncL_v/L_t/L_{\mathrm{async}} (λ=40\lambda=40)” to “Decomp-RGP: λv=λt=λasync=40\lambda_v=\lambda_t=\lambda_{\mathrm{async}}=40” to make the weighting scheme explicit. We also added a new control experiment with λv=5\lambda_v=5, λt=5\lambda_t=5, and λasync=30\lambda_{\mathrm{async}}=30. For ease of comparison, we summarize the results in the table below.

MethodSettingACC ↑BWT ↑
Feature DistillationFixed λ=40\lambda=4051.03+0.35
RGPλ=40\lambda=4060.11−1.41
RGPρ=40%\rho=40\%53.82+0.19
Decomp-RGPλv=40, λt=40, λa=40\lambda_v = 40,\ \lambda_t = 40,\ \lambda_a = 4060.17−1.38
Decomp-RGPλv=5, λt=5, λa=30\lambda_v = 5,\ \lambda_t = 5,\ \lambda_a = 3061.89-2.64
Decomp-RGPρv\rho_v = 5%, ρt\rho_t = 5%, ρa\rho_a = 30%60.21−0.58
Decomp-RGPρv\rho_v = 8%, ρt\rho_t = 2%, ρa\rho_a = 30%60.61−0.75

As shown in the table above, Decomp-RGP with λv=40\lambda_v=40, λt=40\lambda_t=40, and λa=40\lambda_a=40 achieves nearly identical results to RGP with λ=40\lambda=40, consistent with the mathematical equivalence established in Eq. (6). The slight differences may be attributed to stochastic variations during training. The key difference between RGP and feature distillation is that RGP penalizes only non-orthogonal changes, while using a shared orthogonal transformation RR to preserve intra-modal visual and textual relationships, as well as cross-modal relationships. Under the same fixed weight of λ=40\lambda=40, RGP achieves 60.11% ACC, compared with 51.03% for feature distillation, an improvement of 9.08 percentage points. This demonstrates that RGP provides greater plasticity for learning new tasks.

Our Budget-based Adaptive Weighting method is designed to balance plasticity and stability. When plasticity is high but stability is insufficient, it automatically adjusts the regularization weights to strengthen stability and reduce catastrophic forgetting. This can be observed from two comparisons. First, compared with RGP using a fixed λ=40\lambda=40, RGP with an adaptive budget of ρ=40\rho=40% improves BWT from −1.41% to +0.19%, a gain of 1.60 percentage points. Similarly, compared with Decomp-RGP using fixed weights λv=5\lambda_v=5, λt=5\lambda_t=5, and λa=30\lambda_a=30, our adaptive setting with ρv=5\rho_v=5%, ρt=5\rho_t=5%, and ρa=30\rho_a=30% improves BWT from −2.64% to −0.58%, while ACC decreases by only 1.68 percentage points.

However, decomposition alone does not improve performance. Its main purpose is to work with Budget-based Adaptive Weighting, enabling finer control over the three components, which the original RGP loss cannot achieve. Compared with RGP using ρ=40\rho=40%, Decomp-RGP with ρv=5\rho_v=5%, ρt=5\rho_t=5%, and ρa=30\rho_a=30% provides greater plasticity, improving ACC from 53.82% to 60.21%, while maintaining BWT at −0.58%, indicating limited forgetting.

Based on the stability–plasticity trade-off shown in Fig. 5(c), we select ρv=8\rho_v=8%, ρt=2\rho_t=2%, and ρa=30\rho_a=30%, corresponding to a total regularization budget of Σρ=40\Sigma\rho=40%. These values specify the target loss contributions of visual deformation, textual deformation, and cross-modal orthogonal alignment mismatch, respectively.

We have added the additional ablation experiments to Table 2 and revised the row labels to make the component settings clearer. The corresponding analysis has also been updated in Section 4.4 of the revised manuscript. We have also made the budget labels in Fig. 5 consistent with Table 2. Previously expressed as percentages of the total regularization budget, they now represent percentages of the total loss.

AuthorsOpenReview ↗

Response to Reviewer fCRR W3

Evaluation Protocols and Additional Experiments

We thank the reviewer for the constructive feedback and valuable suggestions, which have helped us improve the clarity and completeness of our experimental evaluation. Following the reviewer's suggestion, we have added details on the data splits, splitting criteria, and the model used for SUN397 text generation in Appendix A.9. We have also included the complete dataset statistics for our final evaluation protocol in Table 8.

For SUN397, we use BLIP-2 to generate image-specific text descriptions conditioned on both the images and their category labels. Rather than using simple class-name templates such as "a photo of [class]", we use complete descriptions that capture richer scene information. This allows the retrieval task to rely on more detailed image-text correspondences rather than primarily matching category labels.

Furthermore, we use Qwen2.5-VL to assess the quality of both the official Lexica text descriptions and the BLIP-2-generated SUN397 descriptions. We remove descriptions that fail to meet our quality criteria, along with their corresponding images. We will publicly release the text generation and quality filtering code, as well as the final text description files for all datasets, together with our Decomp-RGP implementation to ensure reproducibility.

Following the reviewer's suggestion, we have conducted additional experiments using a new task order: SUN397 → Visual Genome → COCO → Flickr30K → Kream → Lexica → Wikido. The results are reported in Table 4.

We have also extended our evaluation to the MTIL (Multi-Task Incremental Learning) protocol adopted from ZSCL, which focuses on continual learning for multimodal classification tasks. The results are presented in Table 3, where we additionally compare our method with CTP, HMCL, HypCL, Procrustes KD, and Pi-CCA to further evaluate the generalizability of RGP across different continual learning settings.

Furthermore, we have added the zero-shot performance of the baseline models to Table 5, providing a more comprehensive comparison with Decomp-RGP.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.