- Rating
- 3(+1)
- Confidence
- 3(−1)
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces CounterGround3D, a counterfactual benchmark for diagnosing whether 3D large multimodal models ground their predictions in point-cloud evidence instead of language priors or dataset correlations. The benchmark covers six scene intervention and evaluates sensitivity to answer-critical changes together with invariance to irrelevant changes. The paper further proposes m2DPO, which combines counterfactual multimodal preference optimization with supervised training. Experiments across several 3D LMMs reveal substantial failures hidden by conventional benchmarks, while training on CounterGround3D improves counterfactual grounding, particularly for recoloring and when combined with PoseAlign.
Critical strengths
- The paired intervention idea provides a substantially more informative diagnostic than conventional task accuracy. A grounded model should change its answer when answer-critical visual evidence is modified while remaining invariant to irrelevant edits. The strong performance of visually blind models on conventional benchmarks but near-zero paired grounding further demonstrates why this diagnostic is useful.
- CounterGround3D goes beyond simple language perturbations by physically realizing edits in the point cloud. The object-level pipeline includes structured edit parsing, instance grounding, geometric realization, donor-bank filtering, and substantial manual refinement. Together with evaluation across multiple existing 3D LMMs, this makes the benchmark potentially useful for studying visual dependence and shortcut behavior in future 3D multimodal models.
Critical weaknesses
-
The central “paired counterfactual benchmark” claim is not consistently realized across the six intervention types. For recolor and flip, both the original and edited examples have ground-truth answers, enabling the proposed pair-level grounding metric . However, for move, add, replace, and remove, the paper states that Hypo3D does not provide pre-intervention answers, and consequently Table 2 reports only edited-scene LLM-as-Judge accuracy. Therefore, four of the six intervention types do not directly test whether predictions track a change from to , which weakens the paper’s broader claim of a paired causal evaluation across intervention types. This distinction should either be resolved with paired annotations or reflected more explicitly in the benchmark definition and claims.
-
The incremental contribution of m2DPO over straightforward counterfactual SFT is not yet convincingly established. Without PoseAlign, Ours-m2DPO is worse than Ours-composite on move, add, replace, remove, and multi-operation evaluation, while its main advantages are concentrated in recoloring and a small improvement on flip. The appendix ablation also shows that simpler configurations such as SP, CAP, or CPS are competitive with or outperform the reported CAPS configuration on the recolor experiment. Moreover, the exact default m2DPO objective is difficult to determine because the text identifies CAPS as the default while the subsequent hyperparameter table refers to CALPS. These results make it unclear how much improvement should be attributed specifically to m2DPO.
Questions for authors
-
For move, add, replace, and remove, could the authors clarify in what sense these subsets constitute paired counterfactual evaluation when no pre-intervention answer is available?
-
What exact loss configuration and weights are used for the Ours-m2DPO results in Tables 2–4? The appendix states that CAPS is used as the default, while Table 8 refers to a CALPS default. Please provide the complete final objective and clarify whether the language-DPO term is used in the main experiments. In addition, can the authors report a training budget-matched comparison against the straightforward SFT/composite baseline?
3.Could the authors provide additional validation for the benchmark evaluation protocol, including (i) whether training and validation sets are strictly scene-disjoint for every intervention source, and (ii) human agreement with the intervention-specific LLM-as-Judge metrics?
4 comments
Continued from Part 1/2.
- Comparison with budget-matched SFT. We re-train a budget-matched SFT model in PoseAlign setting with all training data in Table R2. The SP row is the budget-matched supervised finetuning (SFT) equivalent of m$^2$DPO, with matched training steps, learning rates, data, and batch size, which shows that SFT has a considerable performance gap from CAPS on the full data despite a fair training budget. We will incorporate this experiment and analysis in Table 2 of the revised version.
Table R2: Comparison with budget-matched SFT under PoseAlign using the full training data. SP denotes SFT and CAPS denotes m$^2$DPO, both start training from base.
| recolor G | flip G | move | add | replace | remove | multiop | |
|---|---|---|---|---|---|---|---|
| base (no m$^2$DPO) | 23.6 | 24.3 | 44.1 | 37.3 | 38.4 | 31.8 | 40.1 |
| CAPS (best) | 27.3 | 24.1 | 46.0 | 38.5 | 39.0 | 28.5 | 42.2 |
| SP (fair budget SFT) | 24.6 | 24.0 | 45.2 | 37.5 | 37.8 | 27.1 | 40.4 |
Q3(1): Are training and validation sets strictly scene-disjoint for every intervention source?
R6: The training and validation sets are strictly scene-disjoint. Specifically, we designate the training-validation-test split as exactly in official ScanNet splits.
Q3(2): What is the human agreement with the intervention-specific LLM-as-Judge metrics?
R7: Thank you for proposing this experiment. Following your great suggestion, we assessed the three evaluation judges against human annotations using outputs from six models on a shared random subset of 150 samples, stratified across recolor, flip, and general interventions. After excluding answers that cannot be judged due to referring to out-of-GT items, 868 of 900 responses remained. Table R3 reports accuracy (%) under judge and human scoring for each intervention group and overall. Correlations are calculated across the six models, with denoting squared Pearson correlation.
Table R3: Accuracy (%) under judge and human scoring, with squared Pearson correlations, Pearson p-values, and Kendall rank correlations (exact p-values) across six models.
| Model | Recolor judge | Recolor human | Flip judge | Flip human | General judge | General human | Overall judge | Overall human |
|---|---|---|---|---|---|---|---|---|
| Ours (m$^2$DPO + PoseAlign) | 64.0 | 64.0 | 52.2 | 56.5 | 38.8 | 42.9 | 51.7 | 54.5 |
| Ours (m$^2$DPO) | 44.9 | 42.9 | 35.7 | 40.5 | 40.0 | 42.0 | 40.4 | 41.8 |
| Composite SFT | 36.7 | 38.8 | 25.5 | 31.9 | 28.0 | 32.0 | 30.1 | 34.2 |
| 3D-LLaVA | 36.0 | 34.0 | 24.4 | 28.9 | 26.5 | 26.5 | 29.2 | 29.9 |
| LEO | 20.4 | 12.2 | 26.1 | 30.4 | 12.2 | 16.3 | 19.4 | 19.4 |
| LL3DA | 16.3 | 16.3 | 18.4 | 16.3 | 16.0 | 20.0 | 16.9 | 17.6 |
| R² | 0.970 | 0.960 | 0.978 | 0.991 | ||||
| p (Pearson) | 3.3e-4 | 6.1e-4 | 1.8e-4 | 2.9e-5 | ||||
| Kendall τ (exact p) | 0.87 (0.017) | 0.87 (0.017) | 0.87 (0.017) | 1.00 (0.003) |
All three judges closely track human assessments of model performance, with , , and for recolor, flip, and general interventions, respectively (all Pearson ). Each also shows strong ranking agreement (Kendall , ). Human scoring preserves the overall ranking of all six models, with () and (), while overall accuracy differs by at most 4.1 percentage points per model. These results support using the three judges for model comparison and show that the overall ordering is robust to replacing judge scores with human assessments.
Overall, we clarified how the existing paired subsets support our central finding (W1, Q1), explained our metric priority and CAPS design and supplied a full-data budget matched SFT (W2(1–2), Q2), specified the main objective and consistent exclusion of language DPO (W2(3), Q2), confirmed the scene-disjoint data splits (Q3(1)), and assessed human agreement with the three evaluation judges (Q3(2)).
Thank you for the detailed rebuttal and additional experiments. Most of my original concerns have been substantially addressed. Although the manual audit provides useful support for the validity of most CoPO negatives, the remaining ambiguous cases should be acknowledged as a limitation. Overall, the rebuttal has resolved most of my substantive concerns. Thus~, I‘d like to update my initial score to 3.
Thank you very much for the positive recognition of our work and the fruitful suggestions throughout the discussion period. We will clarify the limitation of the ambiguous CoPO negatives in the next revision. We would like to express our utmost gratitude for your feedback, which has helped us strengthen the evidence and improve the clarity of our paper.
Thank you very much for the fruitful and constructive review. It is uplifting to see that reviewer #mwDa recognizes the value of paired interventions for diagnosing visual dependence beyond conventional task accuracy, and appreciates the physically realized point cloud edits, the object intervention pipeline, and the substantial manual refinement.
R1: Thank you for the discussion on the realization of the diagnostic framework. We never meant to claim CounterGround3D as a benchmark of all paired counterfactual data according to L084, L189, L269, and L846. Section 3.4 introduces the object interventions as a secondary extension evaluated through edited-scene accuracy, while recolor and flip support paired grounding evaluation. We also did not claim to be able to verify whether models track the object changes. We have more broadly mentioned the definition of dataset subsets in the revision, especially in the abstract and introduction to clarify the scope of this work throughout the paper.
Besides, in our humble opinion, the current annotation already provides faithful, valuable, human-corrected object intervention data that already supports judging edited-scene VQA accuracy and invariance to irrelevant interventions. As agreed by all the reviewers, we have already contributed greatly through an object-level pipeline with substantial human refinement of 5,110 samples containing edited point clouds, questions, and answers, accompanied by 30k recolor and flip samples. Further annotating the original answers for the object subset requires substantial human effort, and the annotation of point clouds is very difficult for VLM automation due to the modality difference and object intersection gauging. Therefore, we conclude that additional annotation on object subset is an interesting yet unnecessarily costly process that is beyond the scope of this work. We have updated the revised version to match the precise claims.
R2: We thank the reviewer for better establishment of the contribution of m$^2$DPO preference training. As acknowledged by you (Reviewer #mwDa), our design excels in the unambiguous PoseAlign setting, because our design choice prioritizes paired visual grounding under clearly-defined problems over conventional metrics or settings. Relative to Ours-composite+PA, Ours-m$^2$DPO+PA improves recolor grounding from 23.6% to 27.3%, move accuracy from 44.1% to 46.0%, add from 37.3% to 38.5%, replace from 38.4% to 39.0%, and multi-op from 40.1% to 42.2% (Table 2). We select CAPS mainly to retain most complementary training objectives to avoid tailoring the method solely to the ranking on recolor data. Therefore, we attribute the performance of m$^2$DPO as a result of deliberate design choices and objective preferences rather than plain tradeoffs between equally-weighted performance axes.
R3: We thank the reviewer for examination of the loss ablation. The rationale for CAPS in Table 7 is to retain complementary training objectives and avoid tailoring the method solely to the ranking on recolor. It retains visual preference and anchoring from mDPO, while supervision on original and edited examples preserves existing capabilities better than raw mDPO. Please also see our response to Q2 for the budget-matched SFT comparison.
R4: As stated in L1011–L1013, we consistently use CAPS for the main experiments in Tables 2–4, and the language DPO term L is excluded. Table 8 uses CALPS as the reference configuration so that every loss weight, including that of L, can be varied. Within the wDPO sweep, setting wDPO=0 gives the highest grounding score, recovering the CAPS configuration used in the main experiments.
R5: Please see W2(2) for the component-selection rationale.
1. Final objective. Tables 2–4 consistently use CAPS, with the terms defined in Eqs. 9, 10, and 12:
LCAPS=wCoPO(LCoPO+LCoPOτ)+wAncPOLAncPO+wSFToLSFT+wSFTeLSFTτ.
Language DPO is excluded (wDPO=0). For object subsets without original answers, only wCoPOLCoPOτ+wSFTeLSFTτ applies (Appendix A.4).
We set wCoPO=wAncPO=1, wSFTo=wSFTe=0.5, δ=0, and β=0.1 as default parameters.
Our response continues in Part 2/2.