CIRCE: Counterfactually Informed Reliable Component Evidence for Continual Phrase-to-Patch Alignment

Continual adaptation enables vision-language models to acquire new concepts from sequential data streams. Existing methods mainly assess whether global predictions and image-text representations remain stable. They do not determine whether a model retains the local visual evidence required to understand a phrase in a complex image. We formulate continual phrase-to-patch alignment, which studies whether a vision language model retains phrase conditioned local evidence as it adapts to successive stages. The setting reveals a gap between global retention and local retention. A model preserves its global phrase decision while the patch evidence supporting that decision drifts or becomes diffuse. To close this gap, we propose CIRCE, Counterfactually Informed Reliable Component Evidence, which preserves and selects reliable local evidence. CIRCE first learns stable phrase conditioned mixture anchors through counterfactual supervision, spatial compactness, and perturbation stability, then freezes the acquired local state. Given an image and target phrase, CIRCE selects the retained state with the strongest mixture density, spatial concentration, and prediction stability. Experiments on COCO80-CI8 and VG16-CI4 show that CIRCE achieves superior Grounded Counterfactual Accuracy and preserves local phrase region evidence more effectively than representative continual learning baselines.