- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces continual phrase-to-patch alignment, studying whether vision-language models retain local grounding capabilities during sequential learning without replay or region-level supervision. The proposed CIRCE framework learns phrase-conditioned mixture anchors using target and counterfactual phrases, freezes stage-specific grounding experts, and selects an expert through Gaussian modeling of target-phrase embeddings. Experiments report improvements over continual learning baselines.
Critical strengths
- It highlights a limitation of prior continual learning: retaining global image–text predictions does not necessarily preserve correct local grounding. Studying phrase-to-patch retention extends evaluation toward fine-grained visual understanding.
- CIRCE reports higher average GCA than the strongest evaluated baselines on both COCO80-CI8 and PACO-CI8, without replay or region-level training supervision.
Critical weaknesses
- In PACO-CI8, alternative phrases describe sibling parts of the same object, which may also be present in the image. Since each phrase independently pools visual evidence, a model could correctly ground both the target and an alternative. It is therefore unclear why the designated target must receive a higher score. Please clarify whether negatives are false for the entire image or only for the target region, and whether the dataset contains mutually conflicting ranking constraints.
- The definition in A.3 includes the final checkpoint in the maximization, so the resulting forgetting value must be nonnegative. However, Tables 1 and 2 report negative values for CIRCE-CF. The authors should reconcile the formula and the reported results.
- Correct several minor presentation issues: unresolved equation references “Equations ?? through ??” and the unresolved PACO-LVIS citation “(?)” (line 679).
Questions for authors
See the weakness
6 comments
Comment 2. Reconcile the forgetting definition with the negative values in Tables 1–2.
We appreciate the reviewer’s careful examination of the metric definition.
The reported values follow the implemented calculation; Appendix A.3 contains an incorrect maximization bound. In the submitted implementation, \color{blue}{\text{scripts/analyze_circe_anchor_selection.py}}, the historical maximum is computed over checkpoints strictly preceding the final checkpoint:
We will amend to use this bound and align the written definition with the implementation underlying Tables 1–2.
Under this definition, forgetting can be negative when final performance on earlier groups exceeds their best pre-final performance on average. This corresponds to positive backward transfer, an interpretation discussed by Chaudhry et al. [1].
Such improvement remains possible with category-disjoint stages because different categories can share transferable visual knowledge. Related discussions of knowledge sharing and transfer appear in Ke et al. [2] and van de Ven et al. [3]. For CIRCE-CF, the expert bank also expands as learning proceeds: a newly acquired expert may provide better evidence for an earlier query, improving old-task performance even though previously completed experts remain frozen.
Negative forgetting values also appear in prior empirical results. Powers et al. [4, Table 1] report −0.2 ± 0.2 for SANE on Miner and interpret this as backward transfer. Ke et al. [5, Table 2] report forgetting rates of −1.09 for macro-F1 and −0.60 for accuracy with DAS. These studies employ different forgetting formulations, but illustrate the possibility of improved earlier-task performance following subsequent learning.
[1] Arslan Chaudhry, Puneet K. Dokania, Thalaiyasingam Ajanthan, and Philip H. S. Torr. Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence. European Conference on Computer Vision (ECCV), pp. 532–547, 2018.
[2] Zixuan Ke, Bing Liu, and Xingchang Huang. Continual Learning of a Mixed Sequence of Similar and Dissimilar Tasks. Advances in Neural Information Processing Systems, 33, 2020.
[3] Gido M. van de Ven, Tinne Tuytelaars, and Andreas S. Tolias. Three Types of Incremental Learning. Nature Machine Intelligence, 4:1185–1197, 2022.
[4] Sam Powers, Eliot Xing, and Abhinav Gupta. Self-Activating Neural Ensembles for Continual Reinforcement Learning. Conference on Lifelong Learning Agents (CoLLAs), PMLR 199:683–704, 2022.
[5] Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. Continual Pre-training of Language Models. International Conference on Learning Representations (ICLR), 2023.
Comment 3. Some equation references are missing.
We appreciate the reviewer’s careful reading. We will replace the unresolved equation references with the correct numbers, including the instance on Page 5, Line 253. We will also check all cross-references to equations, figures, and tables to ensure they resolve correctly in the revised manuscript.
Comment 1. Counterfactual Construction and Validation.
We appreciate the reviewer’s concern about alternatives that may remain valid in the image. We clarify the construction protocol below and examine the sensitivity of our results to such alternatives.
(1) PACO construction rules and residual ambiguity.
PACO alternatives are generated through controlled substitutions, as implemented in \color{blue}{\text{scripts/build_paco_class_incremental.py}}. For part queries, the replacement is drawn from sibling parts associated with the same parent object. For color, material, and pattern queries, we substitute a value from the same attribute group while keeping the part and parent object unchanged.
These constraints limit the semantic edit, but do not establish that the resulting description is unsupported throughout the image. An object may contain both the queried part and its sibling, and another visible instance may satisfy a substituted attribute. We therefore agree that validity must be checked against the full image, given that each phrase independently pools its supporting evidence.
(2) Whole-image validation and filtered evaluation on PACO.
We conducted an additional audit of the original PACO-CI8 evaluation set using Qwen3.5-27B. Alternatives were examined individually and labeled supported, unsupported, or uncertain, according to whether any visible instance supports the complete description.
Among the 8,000 evaluation records, 103 (1.29%) contained at least one supported alternative. An additional 129 (1.61%) were classified as uncertain. Removing records with supported alternatives retained 7,897 records (98.71%); excluding the additional uncertain records retained 7,768 records (97.10%).
GPM and CIRCE were evaluated on identical retained records. Filtering removed entire records, leaving the counterfactual sets of retained records and the original evaluation aggregation unchanged.
CIRCE’s average GCA advantage is 2.52 percentage points on the conservative subset, compared with 2.34 points on the original set. Thus, its advantage persists after removing the records identified as potentially valid or uncertain by this audit.
Additional validation on COCO80-CI8.
We also applied this analysis to COCO80-CI8. Its construction procedure is provided in \color{blue}{\text{scripts/build_coco80_candidate_pools.py}}. For object queries derived from native COCO annotations, candidate replacements exclude categories appearing in the image’s non-crowd annotations, with preference given to alternatives in the same supercategory. Attribute replacements stay within the same group. Action replacements use supplied negative verbs or predefined substitutions, while relation replacements preserve the participating entities.
Under the same whole-image audit protocol, Qwen3.5-27B identified 1.30% of evaluation records as containing potentially valid alternatives and an additional 1.00% as uncertain. We again used common filtered subsets for both methods, preserving each retained record’s counterfactual set and the original aggregation.
On the conservative COCO subset, CIRCE maintains a 5.87-percentage-point advantage over GPM, compared with 5.69 points originally. Both benchmarks therefore show that the observed advantage remains after filtering the ambiguity identified by the model-based audit.