acceptodds
Under review as a conference paper at ICLR 2027

Don't Care About Encoders: Encoder-Agnostic Class-Incremental Cross-Modal Retrieval

Abstract

Real-world retrieval systems must continuously evolve to incorporate new semantic classes and upgraded backbone architectures while maintaining retrieval across modalities. However, existing class-incremental learning (CIL) methods often assume fixed encoder architectures, while backward-compatible training typically focuses on unimodal scenarios. To address this challenge, we propose Don't Care About Encoders (DCAE), an encoder-agnostic framework designed for class-incremental cross-modal retrieval. DCAE uses lightweight residual adapters on top of frozen backbone encoders to map feature vectors into a shared normalized representation space. To promote geometric stability across modalities and encoder versions, we anchor representations to fixed class prototypes with simplex equiangular tight frame (ETF) geometry. Our method further encourages preservation of backbone relationships through a relational matching loss and mitigates catastrophic forgetting via supervised contrastive distillation and replay. Experimental results on audio-visual retrieval tasks demonstrate that DCAE outperforms the evaluated baselines in final retrieval accuracy, while compatibility matrices assess retrieval against old gallery embeddings without re-encoding the corresponding items. The code will be made available after acceptance.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
cross-modal retrieval, class-incremental learning, backward-compatible representation learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_cyf5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose DCAE. It keeps the pretrained encoders frozen and trains small residual adapters to map features from different modalities and encoder versions into a shared space, anchored by fixed simplex-ETF class prototypes. The training objective combines classification, prototype alignment, relationship preservation, and supervised contrastive distillation using stored representations from replay examples. The paper analyzes how prototype alignment and shifts in the representations affect compatibility in the expected cosine distance for each class pair. It evaluates final accuracy and retrieval across encoder versions on AVE-CI, VGGSound-CI, and MSCOCO-CI.

Critical strengths
  1. The paper tackles a practical combination of challenges: a system must keep learning new classes and replace different modality encoders, while still retrieving items in its existing gallery. Fixed class prototypes give different modalities and encoder versions a shared reference, which fits the goal of the paper.
  2. The reported results show that DCAE has higher final class retrieval accuracy than the tested baselines on all three data settings. On AVE-CI, it improves on the strongest baseline, NC Terminus, by 9.09 percentage points for audio-to-image retrieval and 11.21 points for image-to-audio retrieval. The compatibility matrices also directly show how updated queries perform against older galleries.
Critical weaknesses
  1. The paper says on page 8 that all methods use the same replay memory size, but Appendix E.3 says NC Terminus uses 20 examples per class, while DCAE uses 10% of the data in its main setup. The paper does not explain whether these settings amount to the same number of examples at each stage, and it does not report results with a strictly matched budget. Since replay has a clear effect on DCAE's performance, this makes it harder to tell how much of the improvement comes from the method itself.
  2. Table 7 mainly reports final accuracy. It does not show AC, ACA, or compatibility matrices when each loss is removed. As a result, it is hard to tell how much ETF alignment, relationship preservation, and supervised contrastive distillation each help retrieval against older galleries.
Questions for authors
  1. With the replay budget and training setup kept the same, how do AC, ACA, and the full cross-version compatibility matrices change when ETF alignment, relationship preservation, and supervised contrastive distillation are removed one at a time?
  2. When an encoder is replaced, how is its adapter initialized or carried over? How are replay examples processed by the new encoder, and when are the stored target representations updated?
4 comments

We thank the Reviewer for their valuable feedback. We are pleased that the proposed setting has been recognized as both interesting and practically relevant.


W1: NC-Terminus protocol clarifications

We thank the Reviewer for highlighting the imprecise wording, which we clarified in the revised version. In our experiments, we aimed to preserve the original methodological choices of each baseline as closely as possible to those intended by their respective authors. As such, we followed NC Terminus' original memory allocation strategy of 20 samples per class, resulted in 560 stored samples for the 28 classes of AVE-CIL. We provide an additional experiment in which we set the NC Terminus replay buffer to be equal to ours, in which we observe that the both NC Terminus runs have comparable results:

DirectionMethodMemory BudgetFA ↑AA ↑AC ↑ACA ↑
Audio-to-ImageNC Terminus560 samples69.17 ± 3.4630.11 ± 2.450.39 ± 0.2512.47 ± 10.78
NC Terminus10% memory68.29 ± 3.5131.65 ± 2.380.39 ± 0.1711.98 ± 8.62
DCAE (ours)10% memory77.47 ± 0.6241.57 ± 0.710.83 ± 0.1630.95 ± 4.61
Image-to-AudioNC Terminus560 samples73.08 ± 5.0440.22 ± 7.450.61 ± 0.3423.55 ± 18.43
NC Terminus10% memory72.87 ± 4.5040.78 ± 6.890.50 ± 0.1720.79 ± 13.76
DCAE (ours)10% memory84.29 ± 1.3448.52 ± 0.470.94 ± 0.0941.48 ± 4.90

Our new experiment indicates that this implementation choice has little impact on NC Terminus performance. Moreover, the results from Table 11 of the Appendix confirm that DCAE consistently outperforms NC Terminus, even with a smaller replay buffer (100 samples). This demonstrates that DCAE’s performance gains are not attributed by the difference in memory budget or the shared ETF structure, strengthening out proposed method as a whole. We improved the clarity and reproducibility in the revised manuscript by expanding the discussion of baseline adaptations in Appendix E.3.

W2, Q1: Contribution of the training objectives to compatibility

We have restructured the ablation studies in the revised manuscript, replacing the original Table 7 with two more comprehensive tables that incorporate the requested compatibility metrics. Specifically, Table 3 presents a cumulative ablation study, starting from a cross-entropy-only baseline and progressively incorporating ETF alignment, supervised contrastive distillation, relational matching, and the residual block, reporting all four metrics (FA, AA, AC, and ACA) at each stage. Table 8 complements this analysis by systematically evaluating all eight possible combinations of the three loss components, allowing us to assess their individual contributions and interactions.

Ablations on main components, AVE-CI dataset, Audio-to-Image

ConfigurationFA ↑AA ↑AC ↑ACA ↑
CE only74.9427.790.173.09
+ ETF alignment75.5234.680.5012.91
+ Sup. Con. Distil.75.2135.360.5016.79
+ Relational matching76.3138.510.6720.60
+ Residual block (full DCAE)78.0141.780.8334.32

Ablations on main components, AVE-CI dataset, Image-to-Audio

ConfigurationFA ↑AA ↑AC ↑ACA ↑
CE only82.6032.830.000.00
+ ETF alignment83.6744.140.8334.12
+ Sup. Con. Distil.83.7346.270.8337.18
+ Relational matching77.0146.021.0041.02
+ Residual block (full DCAE)84.8549.281.0045.37

Individual Loss ablation, AVE-CI dataset, Audio-to-Image

ETFRelSupConFA ↑AA ↑AC ↑ACA ↑
✗✗✗74.9427.790.173.09
✗✗✓78.6430.270.177.83
✗✓✗72.8430.930.3314.05
✓✗✗75.5234.680.5012.91
✗✓✓68.4939.450.5023.11
✓✗✓77.1938.370.6723.70
✓✓✗77.4339.610.6735.51
✓✓✓78.0141.780.8334.32

Individual Loss ablation, AVE-CI dataset, Image-to-Audio

ETFRelSupConFA ↑AA ↑AC ↑ACA ↑
✗✗✗82.6032.830.000.00
✗✗✓74.8438.840.000.00
✗✓✗79.9942.900.000.00
✓✗✗83.6744.140.8334.12
✗✓✓79.4149.850.3327.28
✓✗✓83.1549.500.8339.82
✓✓✗82.1548.950.8340.58
✓✓✓84.8549.281.0045.37

From the new ablations, we can extract several key aspects:

  1. ETF alignment is the strongest individual objective for compatibility. It is the only standalone auxiliary loss that achieves nonzero Image-to-Audio AC (0.50), and it delivers substantially higher ACA than either Rel or SupCon alone.
  2. Relational matching complements ETF. Combining ETF with Rel increases ACA from 25.00 to 35.51 for Audio-to-Image and from 24.13 to 40.58 for Image-to-Audio. This allows the adapter to keep the backbone local neighbourhood structure around each anchor, so retrieval is more accurate whenever compatibility holds.
  3. SupCon contributes to the complete model but is less influential individually. Removing SupCon from full DCAE reduces AC from 0.83 to 0.67 in Audio-to-Image and from 1.00 to 0.83 in Image-to-Audio. This decrease is expected since SupCon keeps the representation of the memory samples to previous-assigned vertices, solving the representation drift.

Q2: Adapter initialization

We would like to clarify how the adapters are initialized and trained following an encoder change. To illustrate this process, we consider the adapter ϕm(t)\phi_m^{(t)} at incremental step tt, where the underlying encoder for modality mm is replaced by a different architecture (e.g., from CLIP to SigLIP).

  1. Initialization: Following an encoder change, a new adapter (ϕm(t)\phi_m^{(t)}) is initialized with zero weights. Given its residual architecture, this initialization initially preserves the embeddings produced by the new encoder, facilitating faster convergence. Importantly, no parameters are transferred from the previous adapter (ϕm(t−1)\phi_m^{(t-1)}), as the encoder architectures may differ in both output dimensionality and feature distributions (e.g., CLIP produces 512-dimensional embeddings, whereas SigLIP produces 768-dimensional embeddings).
  2. Replay and compatibility preservation: The replay memory stores raw clips alongside their corresponding labels. Following an encoder change, these stored clips are re-encoded using the new backbone and processed by the newly initialized adapter, producing the embeddings (zi,m(t)\mathbf{z}_{i,m}^{(t)}). Their corresponding anchors, denoted by (z~i,m\tilde{\mathbf{z}}_{i,m}), are obtained from the encoder–adapter pair associated with the incremental step at which each class was introduced. Together with the fixed class prototypes, these anchors guide the new adapter to align its outputs with the previously established embedding space, thereby preserving compatibility with the legacy gallery.

We have incorporated these clarifications into Section 4.2 and Section 5 of the revised manuscript. Additionally, we included the following pseudocode to Appendix E.2, referenced in Section 4.2.

def swap_encoder(E_new, d):
    E = freeze(E_new)       # Keep encoder weights fixed.
    phi = fresh_adapter(d)  # New weights; pad inputs to d.
    return E, phi

def continual_train(tasks, schedule, d, K, budget):
    E, phi, M = {}, {}, empty_memory()
    mu = fixed_simplex_ETF(K, d)  # Keep prototypes fixed.

    for t, D_t in enumerate(tasks, start=1):
        # Before replacing an encoder, compute replay targets
        # with the previous step's encoder and adapter.
        targets = {}
        if not M.empty():
            with no_grad():  # Targets do not change during training.
                for m in modalities:
                    H_prev = E[m](M.inputs(m))
                    targets[m] = phi[m](H_prev)

        for m in modalities:
            if t == 1 or schedule.replaces(t, m):
                E[m], phi[m] = swap_encoder(
                    schedule[t, m], d)
            # Otherwise keep the existing encoder and adapter.

        # Preserve all earlier class-to-prototype assignments.
        assign_unused_prototypes(mu, D_t.new_classes)

        for B_new in training_batches(D_t):
            # Combine new examples with replay examples.
            R = sample_replay(M)
            B = concatenate(B_new, R)
            # Both use the current encoders and adapters.
            H = {m: E[m](B.inputs(m)) for m in modalities}
            Z = {m: phi[m](H[m]) for m in modalities}
            # Full loss: only the SupCon term uses stored targets.
            # R.ids selects the targets of this replay batch.
            L = DCAE_loss(H, Z, B.labels,
                          mu, targets, R.ids)
            update_adapters_and_classification_heads(L)

        # Select examples to replay in the next task.
        M = update_memory(M, D_t, budget)

    return E, phi  # Classification heads are not used at inference.

Dear Reviewer cyf5,

We are grateful for your time, your effort and for the provided review.

Since the discussion period is nearly over, we would welcome any indication of whether these address your concerns, and are happy to clarify or run further analyses in the remaining time.

Best regards,

DCAE authors

Reviewer_3AQmOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies class-incremental cross-modal retrieval, where modality-specific encoders may be replaced asynchronously during continual learning. The goal is to learn new classes and upgrade encoders while maintaining compatibility between updated query embeddings and historical gallery embeddings, without re-encoding the existing database.The authors propose DCAE (Don't Care About Encoders), which uses lightweight residual MLP adapters to map features from frozen pretrained encoders into a shared representation space. Fixed Simplex-ETF class prototypes serve as geometric anchors, while prototype alignment, relational matching, and supervised contrastive distillation are employed to align representations, preserve backbone feature structures, and mitigate forgetting. The paper also provides sufficient conditions for backward compatibility in terms of class-conditional expected cosine distances.Experiments on AVE-CI, VGGSound-CI, and MSCOCO-CI show that DCAE outperforms the evaluated baselines in final retrieval accuracy and several backward compatibility metrics.

Critical strengths
  1. Practically relevant problem. In real-world retrieval systems, new classes and stronger pretrained encoders are continuously introduced, while re-encoding an entire historical database can be computationally expensive. Maintaining backward compatibility during encoder upgrades is therefore an important problem.

  2. Relatively simple framework. DCAE does not require modifying the pretrained backbones. Instead, lightweight residual adapters map heterogeneous features into a shared geometric space. This modular design offers practical advantages.

  3. Evaluation across multiple modalities and encoder replacement settings. The authors conduct experiments on three datasets and demonstrate promising results in terms of final retrieval accuracy and backward compatibility.

Critical weaknesses
  1. Missing experimental comparison with the closely related IAM-CL²R method. IAM-CL²R (Biondi et al., 2024; 2026) also addresses encoder replacement and backward compatibility in class-incremental learning using a fixed simplex geometry. Although the authors discuss this method in the related work, no experimental comparison is provided. Given the similarity in their underlying ideas, I believe that at least a unimodal comparison under the same encoder replacement setting would be valuable, even though IAM-CL²R was originally designed for unimodal retrieval. Such a comparison would help establish the advantages of DCAE over existing methods.

  2. Insufficient comparison with simpler baselines using the same adapter architecture. DCAE combines a residual MLP adapter with multiple training objectives, but it remains unclear whether the improvements mainly come from the adapter's representation capacity or the proposed geometric alignment and knowledge preservation objectives. Although Table 7 provides some ablations, controlled comparisons using the same adapter architecture, replay budget, and training settings with simpler objectives (e.g., CE-only or CE + ETF) are still missing, particularly in terms of backward compatibility.

  3. Limited evaluation of fine-grained cross-modal retrieval. The paper primarily evaluates retrieval using class-level CMC@1, where retrieving any gallery sample from the same class is considered correct. Since DCAE explicitly aligns representations toward fixed class prototypes, this metric may favor class-level representation learning and does not fully demonstrate whether instance-level cross-modal correspondences are preserved. The authors should discuss this limitation or consider additional instance-level retrieval experiments.

Questions for authors

Q1. IAM-CL²R has already investigated class-incremental learning and encoder replacement using fixed simplex geometry. Why was it not included in the experimental comparisons? Could the authors at least provide a unimodal comparison under the same encoder replacement setting?

Q2. Under identical residual MLP adapter architectures, parameter counts, training data, and replay budgets, how do simpler objectives such as Cross-Entropy alone or Cross-Entropy + ETF Alignment compare with the full DCAE in terms of FA, AC, and ACA?

Q3. The current class-level CMC@1 primarily measures retrieval of samples belonging to the same class. Could the authors further discuss how this differs from instance-level cross-modal retrieval? Would the proposed method maintain similar advantages under instance-level retrieval metrics based on ground-truth matched pairs?

5 comments

We thank the Reviewer for the constructive feedback and for recognizing the practical relevance and modular design of DCAE. In the revised version, we have expanded our experiments to address the three raised concerns.


W1 & Q1: IAM-CL2R baseline

Thank you for suggesting this experimental comparison. We initially omitted this because of its unimodal scenario. We have now included IAM-CL²R, adapted to our cross-modal setting, on all three datasets using the same task splits, encoder replacement schedules, and replay budgets. As reported in Tables 1 and 2, DCAE outperforms IAM-CL²R on every reported metric in both retrieval directions. With the expanded set of baselines, DCAE retains the highest final accuracy (FA), average accuracy (AA), average compatibility (AC), and average compatible accuracy (ACA) across all three datasets, when considering both the CMC@1 metric (class-level) and for the newly introduced R@10 (instance-level).

AVE-CI — CMC@1-based evaluation:

DirectionMethodFA ↑AA ↑AC ↑ACA ↑
Audio-to-ImageIAM-CL²R27.71 ± 1.4222.95 ± 0.180.72 ± 0.3516.26 ± 8.09
DCAE77.47 ± 0.6241.57 ± 0.710.83 ± 0.1630.95 ± 4.61
Image-to-AudioIAM-CL²R33.93 ± 1.0925.35 ± 0.620.50 ± 0.0014.18 ± 0.30
DCAE84.29 ± 1.3448.52 ± 0.470.94 ± 0.0941.48 ± 4.90

VGG-CI — CMC@1-based evaluation:

DirectionMethodFA ↑AA ↑AC ↑ACA ↑
Audio-to-ImageIAM-CL²R23.52 ± 0.5013.24 ± 0.130.79 ± 0.0111.19 ± 0.16
DCAE67.35 ± 0.6928.90 ± 0.921.00 ± 0.0028.45 ± 0.91
Image-to-AudioIAM-CL²R25.24 ± 0.6712.28 ± 0.120.64 ± 0.049.20 ± 0.42
DCAE64.25 ± 0.9830.23 ± 0.730.93 ± 0.0826.68 ± 3.69

MSCOCO-CI — CMC@1-based evaluation:

DirectionMethodFA ↑AA ↑AC ↑ACA ↑
Image-to-TextIAM-CL²R9.05 ± 0.238.83 ± 0.080.50 ± 0.334.44 ± 2.96
DCAE50.88 ± 0.2227.59 ± 1.151.00 ± 0.0027.24 ± 1.64
Text-to-ImageIAM-CL²R10.18 ± 0.269.70 ± 0.100.44 ± 0.194.37 ± 1.77
DCAE52.24 ± 1.0527.87 ± 0.990.94 ± 0.1025.21 ± 1.98

AVE-CI — R@10-based incremental evaluation: These are the additional experiments in which FA, AA, AC, and ACA are computed using instance-level R@10 rather than class-level CMC@1.

DirectionMethodFA ↑AA ↑AC ↑ACA ↑
Audio-to-ImageIAM-CL²R7.52 ± 0.306.50 ± 0.070.50 ± 0.003.23 ± 0.07
DCAE28.80 ± 0.9613.41 ± 0.360.83 ± 0.0013.01 ± 0.81
Image-to-AudioIAM-CL²R10.43 ± 0.118.94 ± 0.250.28 ± 0.102.70 ± 0.90
DCAE36.46 ± 2.1319.90 ± 0.271.00 ± 0.0017.81 ± 0.40

We attribute the performance improvements of DCAE to its Relational loss and the SupCon loss as opposed to IAM-CL2R's HOC loss. Therefore, IAM-CL2R gets compatibility from fixed class directions plus a per-sample tie to the previous model, while DCAE pins features to fixed class position, keeps the encoder's local structure (Rel loss) and adjusts to previously extracted embeddings (SupCon), so it keeps learning while staying compatible. The revised manuscript includes this new baseline.

W2 & Q2: New ablation study

To make clear the role of each component, we redesigned the ablation studied. Firstly, we design our main ablations (previously Table 7) around the incremental addition of the components, and moved it to the main paper (now Table 3). Additionally we added a controlled loss ablation comparing CE-only, CE + ETF, and the remaining loss combinations under the same residual adapter architecture and training conditions (new Table 8, in Appendix). We also extended the two tables to include compatibility metrics (AA, AC and ACA).

Ablations on main components, AVE-CI dataset, Audio-to-Image

ConfigurationFA ↑AA ↑AC ↑ACA ↑
CE only74.9427.790.173.09
+ ETF alignment75.5234.680.5012.91
+ Sup. Con. Distil.75.2135.360.5016.79
+ Relational matching76.3138.510.6720.60
+ Residual block (full DCAE)78.0141.780.8334.32

Ablations on main components, AVE-CI dataset, Image-to-Audio

ConfigurationFA ↑AA ↑AC ↑ACA ↑
CE only82.6032.830.000.00
+ ETF alignment83.6744.140.8334.12
+ Sup. Con. Distil.83.7346.270.8337.18
+ Relational matching77.0146.021.0041.02
+ Residual block (full DCAE)84.8549.281.0045.37

Individual Loss ablation, AVE-CI dataset, Audio-to-Image

ETFRelSupConFA ↑AA ↑AC ↑ACA ↑
✗✗✗74.9427.790.173.09
✗✗✓78.6430.270.177.83
✗✓✗72.8430.930.3314.05
✓✗✗75.5234.680.5012.91
✗✓✓68.4939.450.5023.11
✓✗✓77.1938.370.6723.70
✓✓✗77.4339.610.6735.51
✓✓✓78.0141.780.8334.32

Individual Loss ablation, AVE-CI dataset, Image-to-Audio

ETFRelSupConFA ↑AA ↑AC ↑ACA ↑
✗✗✗82.6032.830.000.00
✗✗✓74.8438.840.000.00
✗✓✗79.9942.900.000.00
✓✗✗83.6744.140.8334.12
✗✓✓79.4149.850.3327.28
✓✗✓83.1549.500.8339.82
✓✓✗82.1548.950.8340.58
✓✓✓84.8549.281.0045.37

Compared with CE-only, full DCAE increases AA from 27.65/32.93 to 41.78/49.28, AC from 0.17/0.00 to 0.83/1.00 and ACA from 3.70/0.00 to 34.32/45.37 for audio-to-image and image-to-audio retrieval, respectively. For the CE+ETF scenario, DCAE showcases similar improvements, excepting the AA for image-to-audio retrieval. Nonetheless, these results support the contribution of the combined objectives beyond the adapter’s representation capacity, particularly in preserving compatibility across updates, which represents our main goal.

Additionally, we present a new comparison of the replay budget in the same 3 scenarios: full DCAE, CE-only and CE + ETF. The comparison is available in Figure 4, in the Appendix. The CE-only baseline does not preserve compatibility, while the CE + ETF gives noisy results, especially for larger sizes of the budget. However, DCAE’s results showcase that the compatibility is maintained under our proposed setting, outperforming the two baselines.

W3 & Q3: Instance-level metrics

We agree that all the presented results, based on CMC@1 alone, which is a class-level metric,do not establish whether instance-level correspondences are preserved. We therefore added the instance-level recall at rank 10, denoted by R@10, which measures whether the ground-truth matched item appears among the ten highest-ranked results. We selected the rank 10, as it is a common reference point for instance-level retrieval, which is a harder task by design. We evaluated all the baselines (previously and newly added) on AVE. The results are available in the revised Appendix, in Table 14.

Instance-level metrics, based on R@10, AVE-CI dataset, Audio-to-Image

MethodFA ↑AA ↑AC ↑ACA ↑
OrthoAlign11.25 ± 0.009.18 ± 0.000.00 ± 0.000.00 ± 0.00
XBT4.21 ± 0.242.25 ± 0.180.22 ± 0.101.65 ± 0.03
AV-CIL23.06 ± 0.6812.62 ± 0.530.33 ± 0.295.09 ± 3.77
NC Terminus16.87 ± 1.468.40 ± 0.710.33 ± 0.003.10 ± 0.74
MKP-Adapt11.21 ± 2.956.15 ± 0.560.33 ± 0.002.01 ± 0.32
IAM-CL²R7.52 ± 0.306.50 ± 0.070.50 ± 0.003.23 ± 0.07
NPT17.06 ± 0.399.03 ± 0.330.65 ± 0.108.20 ± 0.62
Omniv. VE21.18 ± 0.5313.14 ± 0.500.61 ± 0.197.51 ± 2.34
DCAE (ours)28.80 ± 0.9613.41 ± 0.360.83 ± 0.0013.01 ± 0.81

Instance-level metrics, based on R@10, AVE-CI dataset, Image-to-Audio

MethodFA ↑AA ↑AC ↑ACA ↑
OrthoAlign17.17 ± 0.0014.32 ± 0.000.06 ± 0.000.96 ± 0.00
XBT3.99 ± 0.102.03 ± 0.820.11 ± 0.100.89 ± 0.57
AV-CIL18.18 ± 0.177.45 ± 0.010.28 ± 0.101.68 ± 1.04
NC Terminus29.82 ± 0.9217.03 ± 0.320.56 ± 0.198.96 ± 2.08
MKP-Adapt17.64 ± 7.0112.28 ± 3.600.44 ± 0.256.21 ± 3.59
IAM-CL²R10.43 ± 0.118.94 ± 0.250.28 ± 0.102.70 ± 0.90
NPT27.63 ± 1.7515.38 ± 0.510.72 ± 0.1010.27 ± 2.03
Omniv. VE29.82 ± 2.0517.40 ± 0.320.78 ± 0.1014.67 ± 1.19
DCAE (ours)36.46 ± 2.1319.90 ± 0.271.00 ± 0.0017.81 ± 0.40

These results demonstrate that DCAE outperforms all baselines across both retrieval directions even in instance-level retrieval. In particular, our method achieves a substantial improvement in the final R@10 (FA) and compatibility aware R@10 (ACA). Moreover, our results on average compatibility (AC) denote the ability to retrieve the same instance even after encoder updates.

Comments on the paper

9 comments

We thank the Reviewer for recognizing the relevance of the setting and the improvements in retrieval accuracy and compatibility. We address the related concerns together below.


W1-W2, Q3 Methodological contribution and ablation evidence

DCAE’s methodological contribution is how it learns a shared representation across changing modalities, classes, and encoders. The same adapter outputs are trained jointly toward fixed class prototypes while retaining relationships from the backbone and remaining consistent with previous representations. These requirements interact: moving features toward prototypes changes their pairwise relationships, while preserving the new backbone’s relationships alone does not establish compatibility with previous embeddings.

This interaction is central to Eq. 10. The prototype-alignment loss LETF\mathcal{L}_{\mathrm{ETF}} (Eq. 7) draws representations toward their class prototypes; LRel\mathcal{L}_{\mathrm{Rel}} (Eq. 8) constrains the resulting distortion of backbone relationships; and supervised contrastive distillation LSupCon\mathcal{L}_{\mathrm{SupCon}} (Eq. 9) encourages consistency with stored previous representations. All three terms act on the same learned representation space.

We explicitly build on the fixed geometry of Biondi et al. (2024; 2026) and Yang et al. (2023) [1–3]. We do not claim a new simplex construction or independently new loss primitives. Our analytical contribution concerns imperfect alignment: Proposition 4.1 accounts for class-mean components orthogonal to the prototypes and bounds how their changes can counteract alignment gains. This characterizes compatibility under departures from the prototype axes, rather than assuming that fixed prototypes alone guarantee compatibility.

In the new ablation studies, we refer the Reviewer to the new Table 3, we measure the benefit beyond ETF alignment in both final accuracy (FA) and compatibility metrics with previous galleries (AA, AC and ACA). We report the cumulative ablation from fewer to more components on both retrieval directions, i.e. the audio-to-image and image-to-audio, on AVE-CIL, on a single run:

Ablations on main components, AVE-CI dataset, Audio-to-Image

ConfigurationFA ↑AA ↑AC ↑ACA ↑
CE only74.9427.790.173.09
+ ETF alignment75.5234.680.5012.91
+ Sup. Con. Distil.75.2135.360.5016.79
+ Relational matching76.3138.510.6720.60
+ Residual block (full DCAE)78.0141.780.8334.32

Ablations on main components, AVE-CI dataset, Image-to-Audio

ConfigurationFA ↑AA ↑AC ↑ACA ↑
CE only82.6032.830.000.00
+ ETF alignment83.6744.140.8334.12
+ Sup. Con. Distil.83.7346.270.8337.18
+ Relational matching77.0146.021.0041.02
+ Residual block (full DCAE)84.8549.281.0045.37

These results support the contribution of each component of DCAE. While the ETF substantially improves bot AA and AC, highlighting its importance in establishing a shared, and therefore compatible, embedding space, the subsequent addition of supervised contrastive distillation and relational matching further improve compatibility-aware metrics in both retrieval direction, i.e. audio-to-image and image-to-audio.

The 9.09/11.21-point improvements over NC Terminus in Table 1 describe the overall A2I/I2A method comparison, averaged over three runs. We agree that NC-Terminus already relies on the ETF prototypes, and, as shown in Table 3, our method improves on this premise. Therefore, our improvements over NC Terminus come from our design choices of DCAE, which ensure better compatibility with the previous galleries.

Q1 and Q2: NPT and Omnivorous Vision Encoder

We agree that NPT and Omnivorous Vision Encoder (Omniv. VE) are relevant references and that their omission made the original comparison incomplete. We have added comparisons with adapted versions of both methods on AVE-CI, VGGSound-CI, and MSCOCO-CI, in Tables 1 and 2, respecting the existing convention of mean$\pm$std of three different seed runs.

AVE-CI — Audio-to-Image

MethodFA ↑AA ↑AC ↑ACA ↑
NPT66.23 ± 1.2833.48 ± 1.230.67 ± 0.1723.14 ± 7.25
Omniv. VE43.61 ± 3.1537.86 ± 1.660.56 ± 0.2523.70 ± 10.15
DCAE (ours)77.47 ± 0.6241.57 ± 0.710.83 ± 0.1630.95 ± 4.61

AVE-CI — Image-to-Audio

MethodFA ↑AA ↑AC ↑ACA ↑
NPT68.40 ± 1.8440.16 ± 0.500.89 ± 0.1033.20 ± 3.71
Omniv. VE46.28 ± 3.5837.93 ± 1.410.17 ± 0.176.98 ± 6.86
DCAE (ours)84.29 ± 1.3448.52 ± 0.470.94 ± 0.0941.48 ± 4.90

VGG-CI — Audio-to-Image

MethodFA ↑AA ↑AC ↑ACA ↑
NPT37.81 ± 4.4716.04 ± 1.090.78 ± 0.0711.78 ± 1.25
Omniv. VE24.60 ± 0.3325.82 ± 0.820.44 ± 0.0612.96 ± 2.18
DCAE (ours)67.35 ± 0.6928.90 ± 0.921.00 ± 0.0028.45 ± 0.91

VGG-CI — Image-to-Audio

MethodFA ↑AA ↑AC ↑ACA ↑
NPT44.18 ± 1.7719.40 ± 0.120.82 ± 0.1015.44 ± 2.25
Omniv. VE21.51 ± 2.0220.20 ± 1.450.21 ± 0.035.25 ± 1.27
DCAE (ours)64.25 ± 0.9830.23 ± 0.730.93 ± 0.0826.68 ± 3.69

MSCOCO-CI — Image-to-Text

MethodFA ↑AA ↑AC ↑ACA ↑
NPT44.60 ± 3.4616.87 ± 3.740.39 ± 0.387.59 ± 7.96
Omniv. VE43.98 ± 1.5924.96 ± 0.670.89 ± 0.1026.08 ± 1.22
DCAE (ours)50.88 ± 0.2227.59 ± 1.151.00 ± 0.0027.24 ± 1.64

MSCOCO-CI — Text-to-Image

MethodFA ↑AA ↑AC ↑ACA ↑
NPT42.55 ± 2.3314.57 ± 0.340.61 ± 0.107.13 ± 2.22
Omniv. VE43.25 ± 1.4021.45 ± 1.440.89 ± 0.1024.83 ± 1.86
DCAE (ours)52.24 ± 1.0527.87 ± 0.990.94 ± 0.1025.21 ± 1.98

DCAE obtains higher mean FA, AA, AC, and ACA than these two adapted methods in both directions (image-to-audio/audio-to-image, and text-to-image/image-to-text) on AVE-CI, VGGSound-CI (Table 1) and MS-COCO (Table 2). These results establish their relevance as comparisons and support DCAE under the evaluated replacement protocols. They do not imply that either original method is inherently inapplicable to continual retrieval.

NPT [4] encourages learned text representations to have ETF pairwise similarities and aligns image representations with them (Eqs. 5–6 of NPT). These learned targets do not explicitly fix class coordinates across encoder replacements. DCAE uses the same fixed class prototypes across modality branches and encoder versions, jointly with backbone relationship preservation and supervision from previous outputs.

Omniv VE [5] combines cross-modal alignment with anchoring to a frozen teacher, not to the simplex geometrical structure. We acknowledge its adapter architecture as related. Omniv VE is using it for closely-related vision modalities (RGB, Depth and Segmentation), while ours aligns semantically different modalities: RGB images and audio waveforms. Its reported setting processes visual modalities through a shared encoder and it does not evaluate class increments with independently replaced modality encoders, and, as such, their design choice is suboptimal for our cross-modal retrieval scenario. That setting is addressed by DCAE through joint training against fixed class prototypes, current backbone relationships, and stored previous representations.

W3: Baseline adaptations

All the evaluated baselines required adaptations because our setting is novel, combining cross-modality retrieval, backbone encoder replacements and class incremental learning. As such, their original protocols do not completely match the setting studied here. NC Terminus studies class-incremental classification; IAM-CL²R studies compatible lifelong retrieval with encoder replacements in a unimodal setting; and AV-CIL addresses audio-visual incremental classification. OrthAlign, XBT, and MKP-Adapter address compatibility across multimodal models, without the class-incremental protocol. XBT includes sequential model upgrades, but does not introduce disjoint class increments. These methods, also described in Table 5, therefore provide relevant baselines, but cannot be evaluated unchanged under our protocol.

However, our evaluation builds on public datasets and established class-incremental splits. We extend AVE-CI and VGGSound-CI, applying the same task splits as AV-CIL, with encoder replacements and retrieval against historical galleries, replacement schedule, and retrieval evaluation to the compared methods. Adaptations are therefore necessary for this common comparison and their necessity does not establish that the comparison is unfair. To make the adaptation more transparent, we extended Appendix E.3 to specify the changes required by our protocol for each baseline.

W4: Dependence on replay

We added the replay-budget ablation in Figure 4 to assess performance under storage constraints. Across budgets from 1% to 100%, full DCAE maintains substantially higher compatibility than the CE-only and CE+ETF baselines; increasing their replay budgets does not close this gap. Full DCAE maintains high compatibility at 1% and reaches AC = 1.00 in both directions from 10% onward. Table 12 further reports FA of 76.03/84.52 with 500 training examples and 73.36/76.82 with 100 examples. Regarding privacy, replay retains training examples, while query and gallery sets remain separate. Backward compatibility specifically avoids accessing and re-encoding users’ original gallery data after an encoder upgrade. DCAE therefore supports deployments where those raw gallery items cannot be retained or reused, provided training examples remain available for replay. This discussion prompted us to correct an overly broad statement in Section G. We now distinguish restrictions on retaining users’ gallery data, which is addressed by the compatibility, from the storage required for training replay. Privacy-sensitive gallery deployment is not inherently a limitation of DCAE. Storage constraints remain relevant, but the new ablation and fixed-budget results demonstrate strong performance with limited replay.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.