acceptodds
Under review as a conference paper at ICLR 2027

CIVG: Causal Inference for Accurate and Robust Generalized Visual Grounding

Abstract

Generalized visual grounding, encompassing generalized referring expression comprehension and generalized referring expression segmentation, extends classical visual grounding to multi-target and no-target scenarios. However, existing methods suffer from a critical limitation: they tend to learn superficial co-occurrence patterns induced by dataset biases, and they often fail to reliably reject no-target queries due to spurious correlations. Causal inference offers a principled solution by disentangling dataset-induced biases and enabling counterfactual reasoning about target existence. To this end, we propose CIVG, a novel framework that explicitly integrates causal inference into generalized visual grounding. CIVG introduces a Causality-Aware Deconfounding Module (CADM) that performs front-door adjustment to mitigate confounding biases from both visual and textual modalities simultaneously. We further design a Counterfactual Relevance Regularization (CFR) branch that leverages counterfactual contrastive learning to enhance the model's ability to reject no-target queries. In addition, we present CIVG-Union, a dual-branch prediction strategy that harnesses the complementary strengths of models at different scales. Extensive experiments on two tasks and four benchmarks demonstrate that CIVG achieves state-of-the-art performance, with substantial gains in both accuracy and robustness, particularly in no-target scenarios. Our results validate the effectiveness of integrating causal inference into generalized visual grounding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual Grounding, Casual Inference, Referring Expression Segmentation, Referring Expression Comprehension

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NC7rOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CIVG for generalized visual grounding, covering both comprehension and segmentation with single, multiple, or absent targets. Built on InstanceVG, it introduces prototype-memory-based multimodal deconfounding and counterfactual relevance regularization. A dual-model variant combines target-existence prediction from a smaller model with localization from a larger model. Experiments report improvements across several grounding benchmarks.

Critical strengths

The framework addresses the practically relevant challenge of rejecting no-target expressions while maintaining localization accuracy. Experiments cover both grounding tasks and include a reproduced InstanceVG baseline under matched settings. Single-model improvements and component ablations provide evidence that the proposed additions offer benefits beyond model scaling or dual-model combination.

Critical weaknesses

The central causal interpretation is not sufficiently supported by the identification assumptions or the connection between the adjustment formula and its attention-based implementation. The experiments demonstrate predictive improvements but do not clearly attribute them to deconfounding or counterfactual reasoning. This gap limits the methodological contribution, while the dual-model results require stronger controls for decision calibration and computational cost.

Questions for authors
  1. The manuscript does not establish that the proposed mediators satisfy the front-door identification conditions. In particular, the residual connection retains original input features, and its relationship to the assumed causal graph is unclear. The identification limitations acknowledged in the appendix should also be reflected in the main causal claims.

  2. Equation 3 defines an expectation over the marginal training distribution, whereas Equation 4 uses input-dependent attention weights. These are not generally equivalent. The approximation requires further justification and a controlled comparison with ordinary prototype-memory attention to establish benefits beyond feature aggregation.

  3. Expressions from different images are not necessarily invalid for the current image, so source-ID filtering cannot guarantee correct negative labels. Please clarify how false negatives are handled and whether original no-target pairs are treated as positives. A comparison with standard image–text matching supervision would also help assess CFR’s contribution.

  4. Standard benchmark improvements do not directly demonstrate reduced reliance on background or linguistic shortcuts. Controlled background changes, meaning-preserving paraphrases, and shifted object co-occurrences would better test the claimed robustness and deconfounding effects.

  5. CIVG-Union should be compared with the same cascade applied to baseline models and with threshold-calibrated alternatives. Please also report target-present recall alongside no-target accuracy, and provide the combined parameter count and measured inference cost of the two-model system.

3 comments

We thank the reviewer for the positive assessment and constructive feedback. The responses to your concerns, including the compute-matched comparisons and controlled Bias Test, have been fully incorporated into the revised manuscript. Below we directly address these points.

1. Mediators and Residual Connections

The front-door criterion requires the mediator MM to intercept paths from XX to YY, be d-separated from confounders, and directly affect YY. Our mediators MvM_v and MtM_t are built solely from encoded features via prototype-memory cross-attention. They intercept the path and feed directly into prediction heads, ensuring the X→M→YX \to M \to Y path. Regarding residual connections, following GOAT[2], we retain them as an architectural choice to stabilize training and preserve non-confounded cues, not as a formal identification guarantee.

2. Approximation in Eq. (4) and Matched Non-causal Control

The strict theoretical formula of front-door adjustment is difficult to apply in practice, making approximation a common and accepted approach. Eq. (4) is an empirical query-based estimator rather than an exact equivalence.

Mathematically, the prototype memory banks (size K=256K=256) act as a discrete empirical proxy for the marginal distribution P(X)P(X). The softmax-attention weights dynamically computed from the current input function as a non-parametric kernel regression, approximating the conditional distribution P(M∣X=x)P(M \mid X=x). This attention-based pooling is a standard, differentiable mechanism for estimating complex high-dimensional expectations in causal vision-language models (PCMCI[1], GOAT[2], CausalVTG[3]).

To verify that the gains are not merely from increased capacity, a parameter-matched non-causal attention control (using identical memory and attention mechanisms but without the front-door decomposition) yields significantly inferior results (Table 1), confirming the causal mechanism.

Table 1: Comparison with the non-causal prototype-attention control.

VariantF1scoregIoUcIoUN-acc
Non-causal attention72.7773.1868.9372.09
CADM (ours)74.0674.0169.6674.75

3. CFR Negative Sampling and False Negatives

To mitigate false negatives, we filter out same-source augmentations and identical expressions during pairing. Crucially, original no-target expressions are factual positives (label 1), not negatives. Cross-image mismatched pairs serve as negatives, enforced by a ranking margin to focus on causal alignments.

4. Controlled Bias Test

We constructed an out-of-distribution (OOD) test set (2,000 pairs, 600 no-target) by pasting sheep instances onto cat backgrounds. This intentionally breaks visual co-occurrence (e.g., grass backgrounds).

In this test, the baseline frequently hallucinates targets due to spurious background cues. In contrast, CIVG-B accurately rejects these no-target expressions, outperforming the baseline by +8.66% N-acc (Table 2). This directly proves the model has shifted from relying on background shortcuts to causal visual evidence.

Table 2: Performance on the controlled OOD bias test set.

MethodF1scoregIoUcIoUN-acc
InstanceVG (Baseline)47.4048.7831.6085.67
Non-causal attention48.4049.4932.2087.17
CIVG-B (Ours)51.8552.5432.9194.33

5. CIVG-Union: Recall and Cost

Recall: Target accuracy (T-acc) represents target-present recall. CIVG-Union maintains high recall while boosting no-target accuracy (Table 3), alongside strict precision (PR@7-9).

Table 3: CIVG-Union performance on gRefCOCO.

SplitN-accT-acc (Recall)gIoUcIoU[email protected][email protected][email protected]
val80.5396.2577.3972.1177.5766.1828.00
testA75.8796.1977.0876.1680.5770.1834.04
testB68.9891.3869.1168.3266.7657.7734.38

Cascade Comparison: As reported in Section 5.5, the pure baseline cascade (without CADM/CFR) yields 75.64 F1score and 75.21 N-acc, far inferior to our CIVG-Union (77.74 F1, 80.53 N-acc). This proves the gain comes from deconfounding, not the cascade.

Cost: Using an early-exit cascade, CIVG-Union adds marginal parameters (+45.65M) but achieves slightly faster inference (18.3 FPS vs. 17.18 FPS, Table 4).

Table 4: Parameter count and inference speed comparison.

ModelParams (M)Inference Speed (FPS)
InstanceVG-B170.8973.99
InstanceVG-L619.3927.34
InstanceVG-Union (Baseline)790.2817.18
CIVG-B195.5271.48
CIVG-L640.4126.65
CIVG-Union (Ours)835.9318.30

[1]Shaowu Xu, et al. "Progressive Cross-Modal Causal Intervention for Long-Term Action Recognition." CVPR, 2026.

[2]Liuyi Wang, et al. "Vision-and-language navigation via causal learning." CVPR, 2024.

[3] Qiyi Wang, et al. "CausalVTG: Towards Robust Video Temporal Grounding via Causal Inference." NeurIPS, 2025.

Thank you for the additional explanations and experiments. The non-causal attention comparison, controlled bias test, and cascade evaluation address several of my previous concerns and improve the experimental evidence. However, some important issues remain:

  1. The front-door identification conditions are still not rigorously established. In particular, the role of residual connections and their potential impact on the assumed causal paths remain unclear.
  2. The kernel-regression explanation of Eq. (4) provides some intuition, but the theoretical connection to Eq. (3) is still insufficiently justified. The empirical comparison alone does not establish the validity of the causal approximation.
  3. The potential false negatives in CFR and the comparison with threshold-calibrated cascade alternatives have not been fully addressed.

The revisions are helpful, and I acknowledge the authors' efforts by increasing my score from 1 to 2. However, the fundamental methodological concerns, particularly regarding causal identification and theoretical justification, remain unresolved.

Reviewer_T5CJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses a meaningful problem in generalized visual grounding and offers a plausible causal modeling direction, but several technical and empirical issues should be addressed in the camera-ready version.

Critical strengths
  1. The literature review is thorough and reflects a solid understanding of the field.
    The paper covers classical and recent visual grounding work, generalized referring expression comprehension and segmentation, and causal inference for vision language tasks. This gives the proposed method a clear context and makes the motivation for jointly modeling visual and textual confounders easy to follow.

  2. The problem is important and the overall framework is reasonable.
    Introducing a structural causal model with visual and textual confounders and combining front door adjustment with counterfactual relevance regularization is a sensible way to target spurious correlations in generalized grounding. The experiments on gRefCOCO, Ref ZOM, and R RefCOCO variants show competitive results and suggest that the causal modules bring measurable gains, especially on no target accuracy.

Critical weaknesses
  1. The causal identification and interpretability need stronger justification.
    The structural causal model is described at a high level, and the front door adjustment is implemented through prototype memory and attention with a separable additive approximation. The paper should clarify the identification assumptions and provide targeted sanity checks showing that the learned mediators actually remove visual or textual confounders rather than simply adding capacity.

  2. The empirical study is not fully rigorous and some comparisons are not compute matched.
    The paper reports single runs on two RTX 3090 GPUs without multiple seed variance or significance tests, and CIVG Union is essentially a two model cascade compared with single model baselines. The authors should add repeated runs, parameter and latency matched comparisons, and a clearer analysis of how much of the gain comes from the causal modules versus the dual branch design.

  3. The presentation and reproducibility need improvement.
    Several tables are incomplete or hard to parse, and important details such as prototype memory size, gating behavior, and counterfactual hard negative construction are only briefly described. The paper should clean up the tables, provide a clearer method diagram and implementation details, and release code and configurations to support reproducibility.

Questions for authors

See weakness.

1 comment

We thank the reviewer for the constructive comments. All responses to your concerns have been incorporated into the revised manuscript. Below we provide our detailed responses.

1. Causal Identification and Sanity Checks

To demonstrate that the learned mediators remove confounders rather than simply adding capacity, we provide the following direct evidence:

  • Matched Non-causal Control: We compared CADM with a parameter- and training-matched non-causal attention control (identical prototype memory and cross-attention, but without front-door decomposition or residual fusion). As shown in Table 1, the control falls significantly behind (e.g., 72.77 vs. 74.06 F1score), confirming that the gain stems from the causal mechanism rather than mere capacity expansion.
  • Controlled OOD Bias Test: We synthesized a test set by pasting sheep instances onto cat backgrounds to break visual co-occurrence. As shown in Table 2, CIVG-B improves no-target accuracy (N-acc) by 8.66% over the baseline, proving the mediators effectively resist spurious background shortcuts.

Table 1: Comparison with the non-causal prototype-attention control.

VariantF1scoregIoUcIoUN-acc
Non-causal prototype attention72.7773.1868.9372.09
CADM (ours)74.0674.0169.6674.75

Table 2: Performance on the controlled OOD bias test set.

MethodF1scoregIoUcIoUN-acc
InstanceVG (Baseline)47.4048.7831.6085.67
Non-causal prototype attention48.4049.4932.2087.17
CIVG-B (Ours)51.8552.5432.9194.33

2. Compute-Matched Comparison and Dual-Branch Analysis

We have conducted strict compute-matched comparisons and latency analyses.

  • Dual-branch vs. Causal modules: As shown in Table 5 of Section 5.5, applying the same cascade to the pure baseline (InstanceVG-Union) without CADM and CFR only achieves an F1score of 75.64, which is significantly lower than our full CIVG-Union (77.74). This proves the gain is not from the dual-branch design.
  • Latency and Parameters: CIVG-Union has 835.93M parameters and achieves 18.3 FPS. In comparison, the baseline cascade (InstanceVG-Union) has 790.28M parameters and achieves 17.18 FPS. CIVG-Union is actually slightly faster despite more parameters.

3. Implementation Details and Reproducibility

We provide the exact implementation details for the core components:

  • Prototype Memory & Gating: The prototype memory size is K=256K=256, initialized by K-means clustering and updated via EMA (momentum 0.999, temperature 0.1). The gating behavior uses an adaptive weight λ\lambda (Eq. 7) to fuse original and deconfounded features, preserving non-confounded cues.
  • CFR Hard Negatives: We strictly filter out image augmentations from the same source and exclude semantically identical expressions. Crucially, original no-target pairs are treated as factual positives (label 1), not negatives. Cross-image mismatched pairs are used as negatives (label 0) enforced by a ranking margin.
  • Pseudocode: We have provided the forward-pass pseudocode for CADM:
    1. x=ModalityMap(xorig)x = ModalityMap(x_{orig})
    2. if training: Mmem=EMAUpdate(Mmem,MeanPool(xorig))M_{mem} = EMA_{Update}(M_{mem}, MeanPool(x_{orig}))
    3. q=Linearq(x)q = Linear_q(x)
    4. Fmed=CrossAttention(q,Mmem,mask)F_{med} = CrossAttention(q, M_{mem}, mask)\quad # MmemM_{mem} serves as both Key and Value
    5. lambda=Sigmoid(MLPa(Fmed)+MLPo(x))lambda = Sigmoid(MLP_a(F_{med}) + MLP_o(x))
    6. Fenh=LayerNorm(lambda∗Fmed+(1−lambda)∗x)F_{enh} = LayerNorm(lambda * F_{med} + (1 - lambda) * x)
    7. Ffinal=alpha∗xorig+(1−alpha)∗FenhF_{final} = alpha * x_{orig} + (1 - alpha) * F_{enh}

Correspondence to Eq. (1):

  • $ E_{x' \sim P(X)}[h(x')]$ is approximated by mapped features xx.

  • Em∼P(M∣X=x)[g(m)]E_{m \sim P(M \mid X=x)}[g(m)] is approximated by FmedF_{med} via cross-attention with memory.

  • P(X)P(X) uses the K-means memory as a proxy; P(M∣X=x)P(M \mid X=x)) uses softmax attention weights. Lines 5-6 gate the two front-door terms.

  • Variance We acknowledge the reviewer's valid concern. Due to computational resource constraints, our main results are based on single runs. However, we believe the robustness of our findings is strongly supported by two factors: (1) Consistent Trends: The performance gains are consistently observed across three distinct datasets and two model scales (B and L), indicating that the improvements are systematic rather than random noise. (2) Substantial Margins: The performance gaps, such as the +8.66% N-acc improvement on the OOD test set and the consistent ~1.5% F1score gains over strong baselines, far exceed the typical variance caused by random initialization.

Reviewer_h3kEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CIVG for generalized visual grounding with multiple or absent referents. Building on InstanceVG, it introduces a bimodal prototype-memory module, CADM, motivated by front-door adjustment, and a relevance regularizer, CFR, using mismatched image-text pairs and hard negatives. CIVG-Union additionally cascades B- and L-scale models. Experiments cover generalized referring expression comprehension and segmentation on gRefCOCO, Ref-ZOM, and the R-RefCOCO family. The reported results show improvements in grounding and no-target recognition, with component ablations supporting the predictive usefulness of the added modules.

Critical strengths
  1. The paper addresses the practically important problem of grounding when queries refer to multiple objects or no object. Evaluating both box grounding and segmentation, including no-target metrics across several benchmarks, makes the empirical contribution relevant to reliable grounding systems.

  2. The reported improvements are not solely an ensemble effect. In Table 1, CIVG-B improves validation gIoU from 72.74 to 74.01 over the reproduced InstanceVG, and testA/testB N-acc from 71.09/65.18 to 72.14/66.31. Table 3 also includes component ablations and a cascade without the new modules, providing useful evidence that CADM and CFR improve the underlying predictor.

Critical weaknesses
  1. The central claim of implementing front-door adjustment is not adequately established. Equation 1 gives the standard adjustment formula, but the transition to Eqs.2-5 is insufficiently justified. Under Eq.3's stated separable formulation, E[h(x')] over the marginal training distribution should not depend on the current input x, whereas Eq.4 uses the input-dependent query p(x) to determine its weights. The paper does not explain why this estimates the same quantity. Figure 3 and Eq.7 additionally retain original features through residual fusion without establishing the correspondence to the claimed estimator. Appendix B, p.16, lines 828-838, acknowledges insufficiently specified identification assumptions and that the module is not a fully identified causal estimator. This gap directly affects the paper's main theoretical contribution.

  2. The experiments establish predictive gains but do not validate the proposed deconfounding mechanism. There is no targeted test of the background, co-occurrence, or wording biases motivating the work, and no matched-capacity non-causal attention control. This matters because the encoder is frozen and CADM+CFR adds 12.62M parameters out of approximately 24.63M trainable parameters, based on Table 7. Ordinary representation enhancement remains a plausible explanation. Likewise, CFR's cross-image text pairing and BCE/ranking objectives are useful regularization operations but do not by themselves establish a structural counterfactual interpretation. The evidence therefore supports an effective adapter and relevance loss more directly than the central causal claims.

Questions for authors
  1. Clarification: Please provide forward-pass pseudocode linking the two expectations in Eq.1 to the actual operations in CADM. How are the input-dependent weights in Eq.4 and the residual fusion in Eq.7 consistent with the claimed estimator, and which identification assumptions are required? A concrete derivation would materially affect my assessment of the theoretical contribution.

  2. Validation of an existing claim: Can you provide a parameter- and training-matched non-causal attention control and a limited controlled test of one bias already identified in the paper? Such evidence would distinguish deconfounding from additional capacity or regularization and could change my recommendation. This concerns validation of the submitted mechanism rather than expansion to a new task.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.