acceptodds
Under review as a conference paper at ICLR 2027

CIRCE: Counterfactually Informed Reliable Component Evidence for Continual Phrase-to-Patch Alignment

Abstract

Continual adaptation enables vision-language models to acquire new concepts from sequential data streams. Existing methods mainly assess whether global predictions and image-text representations remain stable. They do not determine whether a model retains the local visual evidence required to understand a phrase in a complex image. We formulate continual phrase-to-patch alignment, which studies whether a vision language model retains phrase conditioned local evidence as it adapts to successive stages. The setting reveals a gap between global retention and local retention. A model preserves its global phrase decision while the patch evidence supporting that decision drifts or becomes diffuse. To close this gap, we propose CIRCE, Counterfactually Informed Reliable Component Evidence, which preserves and selects reliable local evidence. CIRCE first learns stable phrase conditioned mixture anchors through counterfactual supervision, spatial compactness, and perturbation stability, then freezes the acquired local state. Given an image and target phrase, CIRCE selects the retained state with the strongest mixture density, spatial concentration, and prediction stability. Experiments on COCO80-CI8 and VG16-CI4 show that CIRCE achieves superior Grounded Counterfactual Accuracy and preserves local phrase region evidence more effectively than representative continual learning baselines.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Z4t8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces continual phrase-to-patch alignment, studying whether vision-language models retain local grounding capabilities during sequential learning without replay or region-level supervision. The proposed CIRCE framework learns phrase-conditioned mixture anchors using target and counterfactual phrases, freezes stage-specific grounding experts, and selects an expert through Gaussian modeling of target-phrase embeddings. Experiments report improvements over continual learning baselines.

Critical strengths
  • It highlights a limitation of prior continual learning: retaining global image–text predictions does not necessarily preserve correct local grounding. Studying phrase-to-patch retention extends evaluation toward fine-grained visual understanding.
  • CIRCE reports higher average GCA than the strongest evaluated baselines on both COCO80-CI8 and PACO-CI8, without replay or region-level training supervision.
Critical weaknesses
  1. In PACO-CI8, alternative phrases describe sibling parts of the same object, which may also be present in the image. Since each phrase independently pools visual evidence, a model could correctly ground both the target and an alternative. It is therefore unclear why the designated target must receive a higher score. Please clarify whether negatives are false for the entire image or only for the target region, and whether the dataset contains mutually conflicting ranking constraints.
  2. The definition in A.3 includes the final checkpoint in the maximization, so the resulting forgetting value must be nonnegative. However, Tables 1 and 2 report negative values for CIRCE-CF. The authors should reconcile the formula and the reported results.
  3. Correct several minor presentation issues: unresolved equation references “Equations ?? through ??” and the unresolved PACO-LVIS citation “(?)” (line 679).
Questions for authors

See the weakness

6 comments

Comment 1. Counterfactual Construction and Validation.


We appreciate the reviewer’s concern about alternatives that may remain valid in the image. We clarify the construction protocol below and examine the sensitivity of our results to such alternatives.

(1) PACO construction rules and residual ambiguity.

PACO alternatives are generated through controlled substitutions, as implemented in \color{blue}{\text{scripts/build_paco_class_incremental.py}}. For part queries, the replacement is drawn from sibling parts associated with the same parent object. For color, material, and pattern queries, we substitute a value from the same attribute group while keeping the part and parent object unchanged.

These constraints limit the semantic edit, but do not establish that the resulting description is unsupported throughout the image. An object may contain both the queried part and its sibling, and another visible instance may satisfy a substituted attribute. We therefore agree that validity must be checked against the full image, given that each phrase independently pools its supporting evidence.

(2) Whole-image validation and filtered evaluation on PACO.

We conducted an additional audit of the original PACO-CI8 evaluation set using Qwen3.5-27B. Alternatives were examined individually and labeled supported, unsupported, or uncertain, according to whether any visible instance supports the complete description.

Among the 8,000 evaluation records, 103 (1.29%) contained at least one supported alternative. An additional 129 (1.61%) were classified as uncertain. Removing records with supported alternatives retained 7,897 records (98.71%); excluding the additional uncertain records retained 7,768 records (97.10%).

GPM and CIRCE were evaluated on identical retained records. Filtering removed entire records, leaving the counterfactual sets of retained records and the original evaluation aggregation unchanged.

PACO-CI8 evaluation setSamplesRetention rateGPM Avg. GCACIRCE Avg. GCAGain (pp)
Original set8,000100.00%22.5124.85+2.34
Excluding records with supported alternatives7,89798.71%22.8825.31+2.43
Further excluding uncertain records7,76897.10%23.1625.68+2.52

CIRCE’s average GCA advantage is 2.52 percentage points on the conservative subset, compared with 2.34 points on the original set. Thus, its advantage persists after removing the records identified as potentially valid or uncertain by this audit.


Additional validation on COCO80-CI8.

We also applied this analysis to COCO80-CI8. Its construction procedure is provided in \color{blue}{\text{scripts/build_coco80_candidate_pools.py}}. For object queries derived from native COCO annotations, candidate replacements exclude categories appearing in the image’s non-crowd annotations, with preference given to alternatives in the same supercategory. Attribute replacements stay within the same group. Action replacements use supplied negative verbs or predefined substitutions, while relation replacements preserve the participating entities.

Under the same whole-image audit protocol, Qwen3.5-27B identified 1.30% of evaluation records as containing potentially valid alternatives and an additional 1.00% as uncertain. We again used common filtered subsets for both methods, preserving each retained record’s counterfactual set and the original aggregation.

COCO80-CI8 evaluation setSamplesRetention rateGPM Avg. GCACIRCE Avg. GCAGain (pp)
Original set8,000100.00%61.3967.08+5.69
Excluding records with potentially valid alternatives7,89698.70%61.6667.74+6.08
Further excluding uncertain records7,81697.70%62.0167.88+5.87

On the conservative COCO subset, CIRCE maintains a 5.87-percentage-point advantage over GPM, compared with 5.69 points originally. Both benchmarks therefore show that the observed advantage remains after filtering the ambiguity identified by the model-based audit.

Comment 2. Reconcile the forgetting definition with the negative values in Tables 1–2.


We appreciate the reviewer’s careful examination of the metric definition.

The reported values follow the implemented calculation; Appendix A.3 contains an incorrect maximization bound. In the submitted implementation, \color{blue}{\text{scripts/analyze_circe_anchor_selection.py}}, the historical maximum is computed over checkpoints strictly preceding the final checkpoint:

FGCA=1T−1∑k=1T−1(max⁡k≤t<TRt,k−RT,k).F_{\mathrm{GCA}} = \frac{1}{T-1} \sum_{k=1}^{T-1} \left( \max_{k \leq t < T} R_{t,k} - R_{T,k} \right).

We will amend Appendix A.3\color{blue}{\text{Appendix A.3}} to use this bound and align the written definition with the implementation underlying Tables 1–2.

Under this definition, forgetting can be negative when final performance on earlier groups exceeds their best pre-final performance on average. This corresponds to positive backward transfer, an interpretation discussed by Chaudhry et al. [1].

Such improvement remains possible with category-disjoint stages because different categories can share transferable visual knowledge. Related discussions of knowledge sharing and transfer appear in Ke et al. [2] and van de Ven et al. [3]. For CIRCE-CF, the expert bank also expands as learning proceeds: a newly acquired expert may provide better evidence for an earlier query, improving old-task performance even though previously completed experts remain frozen.

Negative forgetting values also appear in prior empirical results. Powers et al. [4, Table 1] report −0.2 ± 0.2 for SANE on Miner and interpret this as backward transfer. Ke et al. [5, Table 2] report forgetting rates of −1.09 for macro-F1 and −0.60 for accuracy with DAS. These studies employ different forgetting formulations, but illustrate the possibility of improved earlier-task performance following subsequent learning.


[1] Arslan Chaudhry, Puneet K. Dokania, Thalaiyasingam Ajanthan, and Philip H. S. Torr. Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence. European Conference on Computer Vision (ECCV), pp. 532–547, 2018.

[2] Zixuan Ke, Bing Liu, and Xingchang Huang. Continual Learning of a Mixed Sequence of Similar and Dissimilar Tasks. Advances in Neural Information Processing Systems, 33, 2020.

[3] Gido M. van de Ven, Tinne Tuytelaars, and Andreas S. Tolias. Three Types of Incremental Learning. Nature Machine Intelligence, 4:1185–1197, 2022.

[4] Sam Powers, Eliot Xing, and Abhinav Gupta. Self-Activating Neural Ensembles for Continual Reinforcement Learning. Conference on Lifelong Learning Agents (CoLLAs), PMLR 199:683–704, 2022.

[5] Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. Continual Pre-training of Language Models. International Conference on Learning Representations (ICLR), 2023.

Comment 3. Some equation references are missing.


We appreciate the reviewer’s careful reading. We will replace the unresolved equation references with the correct numbers, including the instance on Page 5, Line 253. We will also check all cross-references to equations, figures, and tables to ensure they resolve correctly in the revised manuscript.

Reviewer_zpoBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces continual phrase-to-patch alignment, which evaluates whether vision-language models retain local visual evidence for individual phrases during sequential adaptation. The proposed method CIRCE learns phrase-conditioned Gaussian mixture anchors using target phrases and minimally edited counterfactuals. After each stage, the local pooling, text adaptation and mixture modules are frozen as a stage expert. At inference, Gaussian summaries of stage-specific phrase embeddings select an expert using only the target phrase. An optional CIRCE-CF variant additionally uses counterfactual phrases for expert verification. Experiments on COCO80-CI8 and PACO-CI8 report improved Grounded Counterfactual Accuracy (GCA) and near-zero forgetting.

Critical strengths
  1. The paper highlights a meaningful limitation of conventional continual learning in vision-language models: stable global predictions do not necessarily imply preserved local grounding. Studying phrase-level discrimination together with spatial localization is a useful direction.
  2. The manuscript is well-written and easy to follow. The motivation is clearly stated.
  3. The experiments are sufficient and the proposed CIRCE achieves substantial GCA improvements over the compared baselines on both benchmarks.
Critical weaknesses
  1. The technical novelty appears limited. The method mainly combines existing local alignment, mixture modeling, and expert-routing techniques, and its contribution beyond this combination needs clearer justification.
  2. It is unclear how much of the improvement comes from the proposed evidence modeling rather than simply storing and freezing a separate expert for each stage. More controlled comparisons are needed.
  3. The efficiency evaluation is insufficient. Although storage costs are reported, actual training time, inference speed, and computational costs relative to the baselines are missing.
  4. The evaluation is limited to two constructed benchmarks and one backbone. Broader experiments are needed to establish generality and scalability in realistic continual learning settings.
  5. Some of the Equation references are missing in the manuscript, e.g., Page 5 Line 253. Please revise the manuscript carefully.
Questions for authors

Please refer to the weaknesses.

6 comments

Comment 1. Clarify the technical novelty beyond combining existing techniques.


We thank the reviewer and clarify our contributions relative to representative prior work below.

① Locally discriminative alignment without region supervision.

CLIP learns global image–text alignment [1] (Radford et al., 2021), while β-CLIP extends alignment through text-conditioned spatial pooling and multi-granular contrastive learning [2] (Zohra et al., 2026). These objectives supervise similarities between text and global or pooled visual representations, but do not explicitly enforce a patch-level target–counterfactual margin\color{red}{\text{do not explicitly enforce a patch-level target–counterfactual margin}}. Building on β-CLIP’s local pooling, CIRCE adds evidence-weighted patch counterfactual supervision\color{blue}{\text{evidence-weighted patch counterfactual supervision}}. Patches receiving high evidence weights are trained to favor the target over its controlled alternatives. This supplies a local discrimination signal without bounding-box, mask, or region-proposal supervision during training.

In Table 3\color{brown}{\textbf{Table 3}}, removing only the patch counterfactual term preserves the complete architecture but reduces COCO80-CI8 average/final GCA from 67.08/64.02 to 64.12/60.88, demonstrating its contribution beyond image-level ranking. Tables 1–2\color{brown}{\textbf{Tables 1–2}} report the overall performance under this region-supervision-free training protocol.

② Phrase-conditioned mixtures that directly construct local evidence.

Prototype Mixture Models already represent diverse visual regions using multiple prototypes [3] (Yang et al., 2020). Their prototypes are estimated through expectation–maximization from mask-annotated support images; they are not generated from the queried phrase\color{red}{\text{not generated from the queried phrase}}. CIRCE instead learns phrase-conditioned mixtures coupled to the evidence field\color{blue}{\text{phrase-conditioned mixtures coupled to the evidence field}}. It generates mixture parameters from the phrase and softly assigns phrase-conditioned patch features to components. Component responsibilities and learned contribution weights directly determine the evidence field used for prediction. The complete construction is trained through image-level ranking and patch-level counterfactual discrimination, without annotated support regions.

Table 6\color{brown}{\textbf{Table 6}} shows that increasing the component count from K = 1 to K = 4 improves average GCA from 64.91 to 67.08 on COCO80-CI8 and from 23.41 to 24.85 on PACO-CI8. Table 3\color{brown}{\textbf{Table 3}} further shows that removing the learned component contribution weights reduces COCO80-CI8 average GCA to 65.58, supporting their role in constructing useful evidence.

③ Task-ID-free access to retained local grounding states.

MoE-Adapters already combines expert freezing with automatic selection for continual vision–language recognition [4] (Yu et al., 2024). Its distribution selector uses image-feature reconstruction errors and does not condition selection on the requested phrase\color{red}{\text{does not condition selection on the requested phrase}}. CIRCE preserves each locally trained pooler, text adapter, and mixture head, then uses target-phrase-only Gaussian routing\color{blue}{\text{target-phrase-only Gaussian routing}} to select a phrase-compatible grounding expert in the shared frozen text space. The Gaussian summaries capture both the center and variation of each stage’s phrase distribution. Standard inference requires only the image and target phrase.

In Table 3\color{brown}{\textbf{Table 3}}, Gaussian routing achieves 67.08 average GCA on COCO80-CI8, compared with 65.96 for mean-only routing and 62.73 for selecting the most-recent expert. These controlled comparisons support the benefit of phrase-distribution-based selection within the retained grounding expert bank.


[1] Radford, A., et al. Learning Transferable Visual Models From Natural Language Supervision. ICML, 2021.

[2] Zohra, F., Zhao, C., Itani, H., and Ghanem, B. β-CLIP: Text-Conditioned Contrastive Learning for Multi-Granular Vision-Language Alignment. CVPR, 2026.

[3] Yang, B., Liu, C., Li, B., Jiao, J., and Ye, Q. Prototype Mixture Models for Few-shot Semantic Segmentation. ECCV, 2020, pp. 763–778.

[4] Yu, J., Zhuge, Y., Zhang, L., Hu, P., Wang, D., Lu, H., and He, Y. Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters. CVPR, 2024, pp. 23219–23230.

Comment 2. Disentangle the benefit of evidence modeling from retaining frozen stage experts.


We thank the reviewer and provide two controlled analyses below.

① Controlling expert retention and storage capacity.

We compare three methods: an ordinary frozen expert bank trained with image-level phrase ranking, without mixture anchors or patch counterfactual supervision, and with closely matched storage; CIRCE without the patch counterfactual term, retaining the complete mixture-anchor architecture (Table 3\color{brown}{\textbf{Table 3}}); and full CIRCE, using both mixture anchors and patch counterfactual supervision. All three share one frozen backbone, eight frozen experts, the same Gaussian router, data stream, training schedule, and random seeds.

MethodFrozen expertsAdditional retained state (MiB)Avg. GCA ↑Final GCA ↑
Ordinary frozen expert bank8120.5462.12 ± 1.1258.26 ± 0.96
CIRCE without patch counterfactual term8118.1464.12 ± 0.9560.88 ± 0.72
CIRCE8118.1467.08 ± 0.9964.02 ± 0.58

CIRCE exceeds the ordinary frozen expert bank by 4.96 points in average GCA and 5.76 points in final GCA, while using 2.40 MiB less retained storage. Under the identical CIRCE architecture, adding the patch counterfactual term improves average/final GCA by 2.96/3.14 points. These comparisons demonstrate gains beyond retaining multiple frozen experts\color{blue}{\text{These comparisons demonstrate gains beyond retaining multiple frozen experts}}.

Comment 3. Report training time, inference speed, and computational costs relative to the baselines.


We thank the reviewer for suggesting a more complete efficiency evaluation.

① Efficiency under a matched measurement protocol.

We compare representative baselines and CIRCE using the same GPU (H200), input resolution, and numerical precision, with training batch size 64 and inference batch size 1. Total training time covers all eight stages, including backbone feature extraction and method-specific updates such as Gaussian fitting or gradient-projection updates. Inference latency includes image/text encoding, routing, and local prediction, measured after warm-up with GPU synchronization.

MethodTotal training time (min) ↓Inference latency (ms/query) ↓Inference GFLOPs/query ↓Final GCA
GPM78.404.8041.4057.94
Ordinary frozen expert bank44.804.9041.4058.26
CIRCE48.805.3042.1064.02
CIRCE-CF48.808.6062.8068.65

Standard CIRCE evaluates only one local expert per query\color{blue}{\text{only one local expert per query}}. Although the retained bank grows across stages, each standard prediction executes only the selected expert after phrase-based routing.

② Additional inference cost of CIRCE-CF.

CIRCE-CF checks the selected expert using the supplied counterfactual phrases and evaluates additional experts when verification fails. As reported in Table 8\color{brown}{\textbf{Table 8}}, it evaluates an average of 3.21 experts per query on COCO80-CI8. Shared frozen backbone features can be reused across these expert evaluations.

This variant improves final GCA from 64.02 to 68.65, at additional inference cost. We report CIRCE-CF separately because it uses counterfactual text inputs and additional expert evaluations. The timing protocol includes processing those supplied texts; it excludes counterfactual generation.

Reviewer_BticOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies continual phrase-to-patch alignment, focusing on the observation that global predictions can remain correct while local visual evidence deteriorates. CIRCE learns phrase-conditioned mixture anchors using a local counterfactual objective, freezes each stage’s expert, and selects an expert through Gaussian modeling of target-phrase embeddings. Training requires neither region supervision nor replay, and standard inference uses only the image and target phrase.

Critical strengths
  1. The distinction between global prediction retention and local evidence retention motivates a meaningful research problem. GCA jointly evaluates localization and counterfactual discrimination.
  2. Standard CIRCE achieves average GCA of 67.08% and 24.85%, compared with 61.39% and 22.51% for the strongest baseline. Ablations support the contributions of mixture anchors and the local counterfactual term.
Critical weaknesses
  1. Counterfactual validity is unclear. PACO part queries use sibling parts as alternatives, although these parts may also appear in the image. Requiring the designated target to outrank other valid descriptions needs justification.
  2. Capacity and routing confounds remain. Frozen experts inherently avoid parameter overwriting, but storage-matched comparisons are missing. Moreover, category-disjoint stages may allow phrase-only routing to recover acquisition provenance without demonstrating competence-based selection under overlapping distributions.
  3. The forgetting definition contradicts reported values. Appendix A.3 includes the final checkpoint in the maximum, making forgetting necessarily nonnegative, whereas Tables 1–2 report negative values for CIRCE-CF.
Questions for authors
  1. How are valid alternative descriptions excluded from counterfactual sets? Can you report their frequency and results after filtering?
  2. Can you provide storage-matched comparisons and evaluate routing when categories or phrases recur across stages?
  3. Which forgetting formula is implemented? Please reconcile the negative values and report Point and Disc separately to identify the source of improvement.
10 comments

Comment 1. Counterfactual Construction and Validation.


We thank the reviewer for highlighting the potential ambiguity of counterfactual alternatives in PACO. Below, we clarify the construction rules and provide an additional whole-image audit to assess the impact of this ambiguity.

(1) Controlled counterfactual construction in PACO.

The PACO counterfactual construction rules are implemented in \color{blue}{\text{scripts/build_paco_class_incremental.py}}. Part queries replace the target part with a sibling part belonging to the same parent object. Color, material, and pattern queries replace an attribute value within the corresponding group while preserving the part and parent object.

These rules constrain the semantic changes, but we agree that they do not guarantee that every alternative is false for the entire image. In particular, sibling parts may coexist, and a substituted attribute may describe another visible instance. Because each phrase independently pools visual evidence, an alternative may remain supported elsewhere in the image, motivating additional whole-image validation.

(2) Additional whole-image audit and sensitivity analysis on PACO.

To assess this residual ambiguity, we additionally audited the original PACO-CI8 evaluation set using Qwen3.5-27B. Each counterfactual alternative was assessed separately against the full image as supported, unsupported, or uncertain, considering whether any visible instance satisfies the description.

The audit flagged 103 records (1.29%) as containing at least one potentially valid alternative, with a further 129 records (1.61%) flagged as uncertain. Both percentages are calculated relative to the original 8,000 evaluation records. We first excluded entire records containing supported alternatives, retaining 7,897 records (98.71%), then additionally excluded uncertain records to obtain a more conservative subset of 7,768 records (97.10%). Both methods were evaluated on exactly the same retained records, preserving each record’s original counterfactual set and the original evaluation aggregation.

PACO-CI8 evaluation setSamplesRetention rateGPM Avg. GCACIRCE Avg. GCAGain (pp)
Original set8000100.00%22.5124.85+2.34
Excluding records with supported alternatives789798.71%22.8825.31+2.43
Further excluding uncertain records776897.10%23.1625.68+2.52

After conservative filtering, CIRCE retains a 2.52-percentage-point advantage in average GCA over GPM, compared with 2.34 percentage points on the original set. This sensitivity analysis shows that CIRCE’s advantage persists after excluding the potentially valid and uncertain records identified by the model-based audit.


Additional check on COCO.

For completeness, we also examined counterfactual construction and potential ambiguity in COCO80-CI8. The construction rules are implemented in \color{blue}{\text{scripts/build_coco80_candidate_pools.py}}. For object queries derived from native COCO annotations, we exclude categories present in the image’s non-crowd annotations and prioritize alternatives within the same supercategory. Attribute queries replace a value within the same group; action queries use supplied negative verbs or predefined substitutions; and relation queries replace the relation while preserving the participating entities.

Using the same whole-image audit and record-level filtering protocol, Qwen3.5-27B flagged 1.30% of COCO80-CI8 records as containing potentially valid alternatives and a further 1.00% as uncertain. Both methods were evaluated on the same filtered subsets, again preserving the retained records’ original counterfactual sets and evaluation aggregation.

COCO80-CI8 evaluation setSamplesRetention rateGPM Avg. GCACIRCE Avg. GCAGain (pp)
Original set8000100.00%61.3967.08+5.69
Excluding records with potentially valid alternatives789698.70%61.6667.74+6.08
Further excluding uncertain records781697.70%62.0167.88+5.87

CIRCE retains a 5.87-percentage-point advantage after conservative filtering, compared with 5.69 percentage points on the original set, providing additional evidence that the observed advantage persists under this ambiguity filtering.

Comment 2. Capacity and routing confounds.


We thank the reviewer for suggesting this comparison.

(1) Storage-matched comparison.

CIRCE shares a single frozen vision–language backbone across all stages. Each stage retains only its local pooler, text adapter, mixture-anchor module, and aggregate statistics. As reported in Appendix A.4\color{blue}{\text{Appendix A.4}}, the eight-stage expert bank occupies approximately 110.12 MiB, and the routing statistics occupy 8.02 MiB. The total retained state beyond the shared 584.73 MiB backbone is therefore 118.14 MiB. To isolate the contribution of CIRCE’s local grounding design while controlling for parameter isolation and storage capacity, we additionally evaluate an ordinary frozen expert bank\color{brown}{\textbf{we additionally evaluate an ordinary frozen expert bank}}. This baseline also trains and freezes one expert per stage, retains eight experts, and uses the same Gaussian router\color{brown}{\textbf{uses the same Gaussian router}}. Each expert uses a standard local pooler and text adapter trained with image-level phrase ranking. We adjust the adapter width to closely match CIRCE’s retained storage budget, with all added parameters participating in training and inference. The data stream, training schedule, and random seeds are kept identical.

We also include the loss ablation from Table 3\color{blue}{\text{Table 3}}, which preserves CIRCE’s complete expert architecture and removes only the patch counterfactual loss, As shown in the table below.

MethodFrozen expertsAdditional retained state (MiB)Avg. GCA ↑Final GCA ↑
Ordinary frozen expert bank8120.5462.12 ± 1.1258.26 ± 0.96
CIRCE without patch counterfactual term8118.1464.12 ± 0.9560.88 ± 0.72
CIRCE8118.1467.08 ± 0.9964.02 ± 0.58

CIRCE outperforms the ordinary frozen expert bank by 4.96 points in average GCA and 5.76 points in final GCA while using 2.40 MiB less retained storage. Comparing the two CIRCE variants under identical architecture, freezing mechanism, and storage, the patch counterfactual objective contributes a further 2.96 and 3.14 points, respectively. These results show that CIRCE’s local grounding design provides gains beyond those obtained by retaining multiple frozen experts.

(2) Routing under overlapping distributions.

① Clarification of the continual learning setting and expected effects of category recurrence. We appreciate this question and would first clarify how category recurrence changes the learning setting. Our main experiments deliberately adopt a strict class-incremental protocol, where target categories do not recur across stages. This evaluates whether previously acquired local grounding knowledge can be retained without renewed training on those categories. As discussed by [1] Cossu et al. (2022), excluding category repetition can exacerbate forgetting, whereas naturally recurring concepts can help reinforce previous knowledge. Allowing categories to recur introduces additional learning opportunities, as studied in class-incremental learning with repetition by [2] Hemati et al. (2023). Related tasks can also benefit from selectively transferring shared knowledge, as demonstrated by [3] Ke et al. (2020).

These findings motivate our expectation that category recurrence can ease retention and improve grounding performance even at a fixed expert count, through renewed exposure to earlier concepts and opportunities to learn shared visual features. We therefore view overlapping distributions as a complementary setting with different opportunities for knowledge acquisition and retention. Motivated by this expectation and the reviewer’s question, we provide the additional experiments below, comparing routing strategies within the same trained expert bank to assess whether CIRCE’s routing remains effective under category recurrence.

② Additional routing evaluation under category recurrence. To examine CIRCE under these conditions and address the reviewer’s suggestion, we additionally evaluate streams with \color{brown}{\textbf{25\\% and 50\\% category recurrence}}. Within each setting, all routing strategies use the same bank of eight frozen experts\color{brown}{\textbf{same bank of eight frozen experts}}, holding the acquired knowledge and storage capacity fixed while varying only expert selection. We compare CIRCE’s Gaussian router with selecting the most-recent expert, a provenance reference that selects the expert associated with each record’s source stage, and an evaluator oracle that selects the best-performing expert for each evaluation record.

Routing strategy25% category recurrence: Final GCA (%)50% category recurrence: Final GCA (%)
Most-recent expert60.4561.98
Provenance reference62.2563.87
CIRCE Gaussian router65.2066.80
Evaluator oracle — upper bound71.3072.32

CIRCE achieves final GCA of 65.20% and 66.80%, exceeding the provenance reference by 2.95 and 2.93 percentage points, respectively. It also improves over selecting the most-recent expert by 4.75 and 4.82 points. These results demonstrate that CIRCE remains effective under category recurrence and that its phrase-based routing provides gains over acquisition-provenance selection, even when both strategies access exactly the same expert bank.


[1] Andrea Cossu, Gabriele Graffieti, Lorenzo Pellegrini, Davide Maltoni, Davide Bacciu, Antonio Carta, and Vincenzo Lomonaco. Is Class-Incremental Enough for Continual Learning? Frontiers in Artificial Intelligence, 5:829842, 2022.

[2] Hamed Hemati, Andrea Cossu, Antonio Carta, Julio Hurtado, Lorenzo Pellegrini, Davide Bacciu, Vincenzo Lomonaco, and Damian Borth. Class-Incremental Learning with Repetition. Proceedings of the 2nd Conference on Lifelong Learning Agents (CoLLAs), PMLR 232:437–455, 2023.

[3] Zixuan Ke, Bing Liu, and Xingchang Huang. Continual Learning of a Mixed Sequence of Similar and Dissimilar Tasks. Advances in Neural Information Processing Systems, 33 (NeurIPS), 2020.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.