- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CIVG for generalized visual grounding, covering both comprehension and segmentation with single, multiple, or absent targets. Built on InstanceVG, it introduces prototype-memory-based multimodal deconfounding and counterfactual relevance regularization. A dual-model variant combines target-existence prediction from a smaller model with localization from a larger model. Experiments report improvements across several grounding benchmarks.
Critical strengths
The framework addresses the practically relevant challenge of rejecting no-target expressions while maintaining localization accuracy. Experiments cover both grounding tasks and include a reproduced InstanceVG baseline under matched settings. Single-model improvements and component ablations provide evidence that the proposed additions offer benefits beyond model scaling or dual-model combination.
Critical weaknesses
The central causal interpretation is not sufficiently supported by the identification assumptions or the connection between the adjustment formula and its attention-based implementation. The experiments demonstrate predictive improvements but do not clearly attribute them to deconfounding or counterfactual reasoning. This gap limits the methodological contribution, while the dual-model results require stronger controls for decision calibration and computational cost.
Questions for authors
-
The manuscript does not establish that the proposed mediators satisfy the front-door identification conditions. In particular, the residual connection retains original input features, and its relationship to the assumed causal graph is unclear. The identification limitations acknowledged in the appendix should also be reflected in the main causal claims.
-
Equation 3 defines an expectation over the marginal training distribution, whereas Equation 4 uses input-dependent attention weights. These are not generally equivalent. The approximation requires further justification and a controlled comparison with ordinary prototype-memory attention to establish benefits beyond feature aggregation.
-
Expressions from different images are not necessarily invalid for the current image, so source-ID filtering cannot guarantee correct negative labels. Please clarify how false negatives are handled and whether original no-target pairs are treated as positives. A comparison with standard image–text matching supervision would also help assess CFR’s contribution.
-
Standard benchmark improvements do not directly demonstrate reduced reliance on background or linguistic shortcuts. Controlled background changes, meaning-preserving paraphrases, and shifted object co-occurrences would better test the claimed robustness and deconfounding effects.
-
CIVG-Union should be compared with the same cascade applied to baseline models and with threshold-calibrated alternatives. Please also report target-present recall alongside no-target accuracy, and provide the combined parameter count and measured inference cost of the two-model system.
3 comments
5. CIVG-Union: Recall and Cost
Recall: Target accuracy (T-acc) represents target-present recall. CIVG-Union maintains high recall while boosting no-target accuracy (Table 3), alongside strict precision (PR@7-9).
Table 3: CIVG-Union performance on gRefCOCO.
| Split | N-acc | T-acc (Recall) | gIoU | cIoU | [email protected] | [email protected] | [email protected] |
|---|---|---|---|---|---|---|---|
| val | 80.53 | 96.25 | 77.39 | 72.11 | 77.57 | 66.18 | 28.00 |
| testA | 75.87 | 96.19 | 77.08 | 76.16 | 80.57 | 70.18 | 34.04 |
| testB | 68.98 | 91.38 | 69.11 | 68.32 | 66.76 | 57.77 | 34.38 |
Cascade Comparison: As reported in Section 5.5, the pure baseline cascade (without CADM/CFR) yields 75.64 F1score and 75.21 N-acc, far inferior to our CIVG-Union (77.74 F1, 80.53 N-acc). This proves the gain comes from deconfounding, not the cascade.
Cost: Using an early-exit cascade, CIVG-Union adds marginal parameters (+45.65M) but achieves slightly faster inference (18.3 FPS vs. 17.18 FPS, Table 4).
Table 4: Parameter count and inference speed comparison.
| Model | Params (M) | Inference Speed (FPS) |
|---|---|---|
| InstanceVG-B | 170.89 | 73.99 |
| InstanceVG-L | 619.39 | 27.34 |
| InstanceVG-Union (Baseline) | 790.28 | 17.18 |
| CIVG-B | 195.52 | 71.48 |
| CIVG-L | 640.41 | 26.65 |
| CIVG-Union (Ours) | 835.93 | 18.30 |
[1]Shaowu Xu, et al. "Progressive Cross-Modal Causal Intervention for Long-Term Action Recognition." CVPR, 2026.
[2]Liuyi Wang, et al. "Vision-and-language navigation via causal learning." CVPR, 2024.
[3] Qiyi Wang, et al. "CausalVTG: Towards Robust Video Temporal Grounding via Causal Inference." NeurIPS, 2025.
Thank you for the additional explanations and experiments. The non-causal attention comparison, controlled bias test, and cascade evaluation address several of my previous concerns and improve the experimental evidence. However, some important issues remain:
- The front-door identification conditions are still not rigorously established. In particular, the role of residual connections and their potential impact on the assumed causal paths remain unclear.
- The kernel-regression explanation of Eq. (4) provides some intuition, but the theoretical connection to Eq. (3) is still insufficiently justified. The empirical comparison alone does not establish the validity of the causal approximation.
- The potential false negatives in CFR and the comparison with threshold-calibrated cascade alternatives have not been fully addressed.
The revisions are helpful, and I acknowledge the authors' efforts by increasing my score from 1 to 2. However, the fundamental methodological concerns, particularly regarding causal identification and theoretical justification, remain unresolved.
We thank the reviewer for the positive assessment and constructive feedback. The responses to your concerns, including the compute-matched comparisons and controlled Bias Test, have been fully incorporated into the revised manuscript. Below we directly address these points.
1. Mediators and Residual Connections
The front-door criterion requires the mediator M to intercept paths from X to Y, be d-separated from confounders, and directly affect Y. Our mediators Mv and Mt are built solely from encoded features via prototype-memory cross-attention. They intercept the path and feed directly into prediction heads, ensuring the X→M→Y path. Regarding residual connections, following GOAT[2], we retain them as an architectural choice to stabilize training and preserve non-confounded cues, not as a formal identification guarantee.
2. Approximation in Eq. (4) and Matched Non-causal Control
The strict theoretical formula of front-door adjustment is difficult to apply in practice, making approximation a common and accepted approach. Eq. (4) is an empirical query-based estimator rather than an exact equivalence.
Mathematically, the prototype memory banks (size K=256) act as a discrete empirical proxy for the marginal distribution P(X). The softmax-attention weights dynamically computed from the current input function as a non-parametric kernel regression, approximating the conditional distribution P(M∣X=x). This attention-based pooling is a standard, differentiable mechanism for estimating complex high-dimensional expectations in causal vision-language models (PCMCI[1], GOAT[2], CausalVTG[3]).
To verify that the gains are not merely from increased capacity, a parameter-matched non-causal attention control (using identical memory and attention mechanisms but without the front-door decomposition) yields significantly inferior results (Table 1), confirming the causal mechanism.
Table 1: Comparison with the non-causal prototype-attention control.
3. CFR Negative Sampling and False Negatives
To mitigate false negatives, we filter out same-source augmentations and identical expressions during pairing. Crucially, original
no-targetexpressions are factual positives (label 1), not negatives. Cross-image mismatched pairs serve as negatives, enforced by a ranking margin to focus on causal alignments.4. Controlled Bias Test
We constructed an out-of-distribution (OOD) test set (2,000 pairs, 600 no-target) by pasting
sheepinstances ontocatbackgrounds. This intentionally breaks visual co-occurrence (e.g., grass backgrounds).In this test, the baseline frequently hallucinates targets due to spurious background cues. In contrast, CIVG-B accurately rejects these no-target expressions, outperforming the baseline by +8.66% N-acc (Table 2). This directly proves the model has shifted from relying on background shortcuts to causal visual evidence.
Table 2: Performance on the controlled OOD bias test set.