- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors propose DCAE. It keeps the pretrained encoders frozen and trains small residual adapters to map features from different modalities and encoder versions into a shared space, anchored by fixed simplex-ETF class prototypes. The training objective combines classification, prototype alignment, relationship preservation, and supervised contrastive distillation using stored representations from replay examples. The paper analyzes how prototype alignment and shifts in the representations affect compatibility in the expected cosine distance for each class pair. It evaluates final accuracy and retrieval across encoder versions on AVE-CI, VGGSound-CI, and MSCOCO-CI.
Critical strengths
- The paper tackles a practical combination of challenges: a system must keep learning new classes and replace different modality encoders, while still retrieving items in its existing gallery. Fixed class prototypes give different modalities and encoder versions a shared reference, which fits the goal of the paper.
- The reported results show that DCAE has higher final class retrieval accuracy than the tested baselines on all three data settings. On AVE-CI, it improves on the strongest baseline, NC Terminus, by 9.09 percentage points for audio-to-image retrieval and 11.21 points for image-to-audio retrieval. The compatibility matrices also directly show how updated queries perform against older galleries.
Critical weaknesses
- The paper says on page 8 that all methods use the same replay memory size, but Appendix E.3 says NC Terminus uses 20 examples per class, while DCAE uses 10% of the data in its main setup. The paper does not explain whether these settings amount to the same number of examples at each stage, and it does not report results with a strictly matched budget. Since replay has a clear effect on DCAE's performance, this makes it harder to tell how much of the improvement comes from the method itself.
- Table 7 mainly reports final accuracy. It does not show AC, ACA, or compatibility matrices when each loss is removed. As a result, it is hard to tell how much ETF alignment, relationship preservation, and supervised contrastive distillation each help retrieval against older galleries.
Questions for authors
- With the replay budget and training setup kept the same, how do AC, ACA, and the full cross-version compatibility matrices change when ETF alignment, relationship preservation, and supervised contrastive distillation are removed one at a time?
- When an encoder is replaced, how is its adapter initialized or carried over? How are replay examples processed by the new encoder, and when are the stored target representations updated?
4 comments
W2, Q1: Contribution of the training objectives to compatibility
We have restructured the ablation studies in the revised manuscript, replacing the original Table 7 with two more comprehensive tables that incorporate the requested compatibility metrics. Specifically, Table 3 presents a cumulative ablation study, starting from a cross-entropy-only baseline and progressively incorporating ETF alignment, supervised contrastive distillation, relational matching, and the residual block, reporting all four metrics (FA, AA, AC, and ACA) at each stage. Table 8 complements this analysis by systematically evaluating all eight possible combinations of the three loss components, allowing us to assess their individual contributions and interactions.
Ablations on main components, AVE-CI dataset, Audio-to-Image
| Configuration | FA ↑ | AA ↑ | AC ↑ | ACA ↑ |
|---|---|---|---|---|
| CE only | 74.94 | 27.79 | 0.17 | 3.09 |
| + ETF alignment | 75.52 | 34.68 | 0.50 | 12.91 |
| + Sup. Con. Distil. | 75.21 | 35.36 | 0.50 | 16.79 |
| + Relational matching | 76.31 | 38.51 | 0.67 | 20.60 |
| + Residual block (full DCAE) | 78.01 | 41.78 | 0.83 | 34.32 |
Ablations on main components, AVE-CI dataset, Image-to-Audio
| Configuration | FA ↑ | AA ↑ | AC ↑ | ACA ↑ |
|---|---|---|---|---|
| CE only | 82.60 | 32.83 | 0.00 | 0.00 |
| + ETF alignment | 83.67 | 44.14 | 0.83 | 34.12 |
| + Sup. Con. Distil. | 83.73 | 46.27 | 0.83 | 37.18 |
| + Relational matching | 77.01 | 46.02 | 1.00 | 41.02 |
| + Residual block (full DCAE) | 84.85 | 49.28 | 1.00 | 45.37 |
Individual Loss ablation, AVE-CI dataset, Audio-to-Image
| ETF | Rel | SupCon | FA ↑ | AA ↑ | AC ↑ | ACA ↑ |
|---|---|---|---|---|---|---|
| ✗ | ✗ | ✗ | 74.94 | 27.79 | 0.17 | 3.09 |
| ✗ | ✗ | ✓ | 78.64 | 30.27 | 0.17 | 7.83 |
| ✗ | ✓ | ✗ | 72.84 | 30.93 | 0.33 | 14.05 |
| ✓ | ✗ | ✗ | 75.52 | 34.68 | 0.50 | 12.91 |
| ✗ | ✓ | ✓ | 68.49 | 39.45 | 0.50 | 23.11 |
| ✓ | ✗ | ✓ | 77.19 | 38.37 | 0.67 | 23.70 |
| ✓ | ✓ | ✗ | 77.43 | 39.61 | 0.67 | 35.51 |
| ✓ | ✓ | ✓ | 78.01 | 41.78 | 0.83 | 34.32 |
Individual Loss ablation, AVE-CI dataset, Image-to-Audio
| ETF | Rel | SupCon | FA ↑ | AA ↑ | AC ↑ | ACA ↑ |
|---|---|---|---|---|---|---|
| ✗ | ✗ | ✗ | 82.60 | 32.83 | 0.00 | 0.00 |
| ✗ | ✗ | ✓ | 74.84 | 38.84 | 0.00 | 0.00 |
| ✗ | ✓ | ✗ | 79.99 | 42.90 | 0.00 | 0.00 |
| ✓ | ✗ | ✗ | 83.67 | 44.14 | 0.83 | 34.12 |
| ✗ | ✓ | ✓ | 79.41 | 49.85 | 0.33 | 27.28 |
| ✓ | ✗ | ✓ | 83.15 | 49.50 | 0.83 | 39.82 |
| ✓ | ✓ | ✗ | 82.15 | 48.95 | 0.83 | 40.58 |
| ✓ | ✓ | ✓ | 84.85 | 49.28 | 1.00 | 45.37 |
From the new ablations, we can extract several key aspects:
- ETF alignment is the strongest individual objective for compatibility. It is the only standalone auxiliary loss that achieves nonzero Image-to-Audio AC (0.50), and it delivers substantially higher ACA than either Rel or SupCon alone.
- Relational matching complements ETF. Combining ETF with Rel increases ACA from 25.00 to 35.51 for Audio-to-Image and from 24.13 to 40.58 for Image-to-Audio. This allows the adapter to keep the backbone local neighbourhood structure around each anchor, so retrieval is more accurate whenever compatibility holds.
- SupCon contributes to the complete model but is less influential individually. Removing SupCon from full DCAE reduces AC from 0.83 to 0.67 in Audio-to-Image and from 1.00 to 0.83 in Image-to-Audio. This decrease is expected since SupCon keeps the representation of the memory samples to previous-assigned vertices, solving the representation drift.
Q2: Adapter initialization
We would like to clarify how the adapters are initialized and trained following an encoder change. To illustrate this process, we consider the adapter at incremental step , where the underlying encoder for modality is replaced by a different architecture (e.g., from CLIP to SigLIP).
- Initialization: Following an encoder change, a new adapter () is initialized with zero weights. Given its residual architecture, this initialization initially preserves the embeddings produced by the new encoder, facilitating faster convergence. Importantly, no parameters are transferred from the previous adapter (), as the encoder architectures may differ in both output dimensionality and feature distributions (e.g., CLIP produces 512-dimensional embeddings, whereas SigLIP produces 768-dimensional embeddings).
- Replay and compatibility preservation: The replay memory stores raw clips alongside their corresponding labels. Following an encoder change, these stored clips are re-encoded using the new backbone and processed by the newly initialized adapter, producing the embeddings (). Their corresponding anchors, denoted by (), are obtained from the encoder–adapter pair associated with the incremental step at which each class was introduced. Together with the fixed class prototypes, these anchors guide the new adapter to align its outputs with the previously established embedding space, thereby preserving compatibility with the legacy gallery.
We have incorporated these clarifications into Section 4.2 and Section 5 of the revised manuscript. Additionally, we included the following pseudocode to Appendix E.2, referenced in Section 4.2.
def swap_encoder(E_new, d):
E = freeze(E_new) # Keep encoder weights fixed.
phi = fresh_adapter(d) # New weights; pad inputs to d.
return E, phi
def continual_train(tasks, schedule, d, K, budget):
E, phi, M = {}, {}, empty_memory()
mu = fixed_simplex_ETF(K, d) # Keep prototypes fixed.
for t, D_t in enumerate(tasks, start=1):
# Before replacing an encoder, compute replay targets
# with the previous step's encoder and adapter.
targets = {}
if not M.empty():
with no_grad(): # Targets do not change during training.
for m in modalities:
H_prev = E[m](M.inputs(m))
targets[m] = phi[m](H_prev)
for m in modalities:
if t == 1 or schedule.replaces(t, m):
E[m], phi[m] = swap_encoder(
schedule[t, m], d)
# Otherwise keep the existing encoder and adapter.
# Preserve all earlier class-to-prototype assignments.
assign_unused_prototypes(mu, D_t.new_classes)
for B_new in training_batches(D_t):
# Combine new examples with replay examples.
R = sample_replay(M)
B = concatenate(B_new, R)
# Both use the current encoders and adapters.
H = {m: E[m](B.inputs(m)) for m in modalities}
Z = {m: phi[m](H[m]) for m in modalities}
# Full loss: only the SupCon term uses stored targets.
# R.ids selects the targets of this replay batch.
L = DCAE_loss(H, Z, B.labels,
mu, targets, R.ids)
update_adapters_and_classification_heads(L)
# Select examples to replay in the next task.
M = update_memory(M, D_t, budget)
return E, phi # Classification heads are not used at inference.
Dear Reviewer cyf5,
We are grateful for your time, your effort and for the provided review.
Since the discussion period is nearly over, we would welcome any indication of whether these address your concerns, and are happy to clarify or run further analyses in the remaining time.
Best regards,
DCAE authors
We thank the Reviewer for their valuable feedback. We are pleased that the proposed setting has been recognized as both interesting and practically relevant.
W1: NC-Terminus protocol clarifications
We thank the Reviewer for highlighting the imprecise wording, which we clarified in the revised version. In our experiments, we aimed to preserve the original methodological choices of each baseline as closely as possible to those intended by their respective authors. As such, we followed NC Terminus' original memory allocation strategy of 20 samples per class, resulted in 560 stored samples for the 28 classes of AVE-CIL. We provide an additional experiment in which we set the NC Terminus replay buffer to be equal to ours, in which we observe that the both NC Terminus runs have comparable results:
Our new experiment indicates that this implementation choice has little impact on NC Terminus performance. Moreover, the results from Table 11 of the Appendix confirm that DCAE consistently outperforms NC Terminus, even with a smaller replay buffer (100 samples). This demonstrates that DCAE’s performance gains are not attributed by the difference in memory budget or the shared ETF structure, strengthening out proposed method as a whole. We improved the clarity and reproducibility in the revised manuscript by expanding the discussion of baseline adaptations in Appendix E.3.