acceptodds
Under review as a conference paper at ICLR 2027

CounterGround3D: Probing and Improving 3D LMM Grounding with a Counterfactual Intervention Benchmark

Abstract

Reliable 3D grounding requires large multimodal models (LMMs) to base their predictions on the visual evidence present in the point cloud. However, existing 3D LMMs tend to be dominated by language priors and dataset-level correlations, while frequently neglecting visual input. We introduce CounterGround3D, a paired counterfactual benchmark that directly evaluates whether model predictions track controlled interventions in the point cloud. CounterGround3D comprises six types of interventions, namely, recolor, flip, add, remove, replace, and move, covering a wide range of factors in grounding tasks. Each edited scene is paired with its original counterpart, enabling evaluation of both sensitivity to answer-critical interventions and invariance to irrelevant ones. Leveraging CounterGround3D, we propose mDPO, a mixed multimodal direct preference optimization framework that employs counterfactual pairs as direct preference supervision and boosts 3D grounding with little catastrophic forgetting. Experiments across multiple widely adopted 3D LMMs show that CounterGround3D reveals substantial grounding failures that remain hidden under conventional benchmarks, while mDPO significantly improves 3D grounding capabilities.

Primary area:
datasets and benchmarks
Keywords:
point cloud, counterfactual grounding, large language model, multimodal

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mwDaOpenReview ↗
Rating
3(+1)
Confidence
3(−1)
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CounterGround3D, a counterfactual benchmark for diagnosing whether 3D large multimodal models ground their predictions in point-cloud evidence instead of language priors or dataset correlations. The benchmark covers six scene intervention and evaluates sensitivity to answer-critical changes together with invariance to irrelevant changes. The paper further proposes m2DPO, which combines counterfactual multimodal preference optimization with supervised training. Experiments across several 3D LMMs reveal substantial failures hidden by conventional benchmarks, while training on CounterGround3D improves counterfactual grounding, particularly for recoloring and when combined with PoseAlign.

Critical strengths
  1. The paired intervention idea provides a substantially more informative diagnostic than conventional task accuracy. A grounded model should change its answer when answer-critical visual evidence is modified while remaining invariant to irrelevant edits. The strong performance of visually blind models on conventional benchmarks but near-zero paired grounding further demonstrates why this diagnostic is useful.
  2. CounterGround3D goes beyond simple language perturbations by physically realizing edits in the point cloud. The object-level pipeline includes structured edit parsing, instance grounding, geometric realization, donor-bank filtering, and substantial manual refinement. Together with evaluation across multiple existing 3D LMMs, this makes the benchmark potentially useful for studying visual dependence and shortcut behavior in future 3D multimodal models.
Critical weaknesses
  1. The central “paired counterfactual benchmark” claim is not consistently realized across the six intervention types. For recolor and flip, both the original and edited examples have ground-truth answers, enabling the proposed pair-level grounding metric GG. However, for move, add, replace, and remove, the paper states that Hypo3D does not provide pre-intervention answers, and consequently Table 2 reports only edited-scene LLM-as-Judge accuracy. Therefore, four of the six intervention types do not directly test whether predictions track a change from aa to aτa^\tau, which weakens the paper’s broader claim of a paired causal evaluation across intervention types. This distinction should either be resolved with paired annotations or reflected more explicitly in the benchmark definition and claims.

  2. The incremental contribution of m2DPO over straightforward counterfactual SFT is not yet convincingly established. Without PoseAlign, Ours-m2DPO is worse than Ours-composite on move, add, replace, remove, and multi-operation evaluation, while its main advantages are concentrated in recoloring and a small improvement on flip. The appendix ablation also shows that simpler configurations such as SP, CAP, or CPS are competitive with or outperform the reported CAPS configuration on the recolor experiment. Moreover, the exact default m2DPO objective is difficult to determine because the text identifies CAPS as the default while the subsequent hyperparameter table refers to CALPS. These results make it unclear how much improvement should be attributed specifically to m2DPO.

Questions for authors
  1. For move, add, replace, and remove, could the authors clarify in what sense these subsets constitute paired counterfactual evaluation when no pre-intervention answer is available?

  2. What exact loss configuration and weights are used for the Ours-m2DPO results in Tables 2–4? The appendix states that CAPS is used as the default, while Table 8 refers to a CALPS default. Please provide the complete final objective and clarify whether the language-DPO term is used in the main experiments. In addition, can the authors report a training budget-matched comparison against the straightforward SFT/composite baseline?

3.Could the authors provide additional validation for the benchmark evaluation protocol, including (i) whether training and validation sets are strictly scene-disjoint for every intervention source, and (ii) human agreement with the intervention-specific LLM-as-Judge metrics?

4 comments

Thank you very much for the fruitful and constructive review. It is uplifting to see that reviewer #mwDa recognizes the value of paired interventions for diagnosing visual dependence beyond conventional task accuracy, and appreciates the physically realized point cloud edits, the object intervention pipeline, and the substantial manual refinement.

W1 & Q1: The central “paired counterfactual benchmark” claim is not consistently realized across the six intervention types.

R1: Thank you for the discussion on the realization of the diagnostic framework. We never meant to claim CounterGround3D as a benchmark of all paired counterfactual data according to L084, L189, L269, and L846. Section 3.4 introduces the object interventions as a secondary extension evaluated through edited-scene accuracy, while recolor and flip support paired grounding evaluation. We also did not claim to be able to verify whether models track the object changes. We have more broadly mentioned the definition of dataset subsets in the revision, especially in the abstract and introduction to clarify the scope of this work throughout the paper.

Besides, in our humble opinion, the current annotation already provides faithful, valuable, human-corrected object intervention data that already supports judging edited-scene VQA accuracy and invariance to irrelevant interventions. As agreed by all the reviewers, we have already contributed greatly through an object-level pipeline with substantial human refinement of 5,110 samples containing edited point clouds, questions, and answers, accompanied by 30k recolor and flip samples. Further annotating the original answers for the object subset requires substantial human effort, and the annotation of point clouds is very difficult for VLM automation due to the modality difference and object intersection gauging. Therefore, we conclude that additional annotation on object subset is an interesting yet unnecessarily costly process that is beyond the scope of this work. We have updated the revised version to match the precise claims.

W2(1): The incremental contribution of m$^2$DPO over straightforward counterfactual SFT is not yet convincingly established.

R2: We thank the reviewer for better establishment of the contribution of m$^2$DPO preference training. As acknowledged by you (Reviewer #mwDa), our design excels in the unambiguous PoseAlign setting, because our design choice prioritizes paired visual grounding under clearly-defined problems over conventional metrics or settings. Relative to Ours-composite+PA, Ours-m$^2$DPO+PA improves recolor grounding from 23.6% to 27.3%, move accuracy from 44.1% to 46.0%, add from 37.3% to 38.5%, replace from 38.4% to 39.0%, and multi-op from 40.1% to 42.2% (Table 2). We select CAPS mainly to retain most complementary training objectives to avoid tailoring the method solely to the ranking on recolor data. Therefore, we attribute the performance of m$^2$DPO as a result of deliberate design choices and objective preferences rather than plain tradeoffs between equally-weighted performance axes.

W2(2): Appendix ablation Table 7 confusing.

R3: We thank the reviewer for examination of the loss ablation. The rationale for CAPS in Table 7 is to retain complementary training objectives and avoid tailoring the method solely to the ranking on recolor. It retains visual preference and anchoring from mDPO, while supervision on original and edited examples preserves existing capabilities better than raw mDPO. Please also see our response to Q2 for the budget-matched SFT comparison.

W2(3): Why did hyperparameter table (Table 8) use CALPS while CAPS is used as the default?

R4: As stated in L1011–L1013, we consistently use CAPS for the main experiments in Tables 2–4, and the language DPO term L is excluded. Table 8 uses CALPS as the reference configuration so that every loss weight, including that of L, can be varied. Within the wDPOw_{DPO} sweep, setting wDPO=0w_{DPO}=0 gives the highest grounding score, recovering the CAPS configuration used in the main experiments.

Q2: What objective and weights are used in Tables 2–4, is language DPO included, and how about SFT with matched training budget?

R5: Please see W2(2) for the component-selection rationale.

1. Final objective. Tables 2–4 consistently use CAPS, with the terms defined in Eqs. 9, 10, and 12:

LCAPS=wCoPO(LCoPO+LCoPOτ)+wAncPOLAncPO+wSFToLSFT+wSFTeLSFTτ\mathcal{L}_{CAPS}=w_{CoPO}(\mathcal{L}_{CoPO}+\mathcal{L}_{CoPO}^{\tau})+w_{AncPO}\mathcal{L}_{AncPO}+w_{SFT_o}\mathcal{L}_{SFT}+w_{SFT_e}\mathcal{L}_{SFT}^{\tau}.

Language DPO is excluded (wDPO=0w_{DPO}=0). For object subsets without original answers, only wCoPOLCoPOτ+wSFTeLSFTτw_{CoPO}\mathcal{L}_{CoPO}^{\tau}+w_{SFT_e}\mathcal{L}_{SFT}^{\tau} applies (Appendix A.4).

We set wCoPO=wAncPO=1w_{CoPO}=w_{AncPO}=1, wSFTo=wSFTe=0.5w_{SFT_o}=w_{SFT_e}=0.5, δ=0\delta=0, and β=0.1\beta=0.1 as default parameters.


Our response continues in Part 2/2.

Continued from Part 1/2.


  1. Comparison with budget-matched SFT. We re-train a budget-matched SFT model in PoseAlign setting with all training data in Table R2. The SP row is the budget-matched supervised finetuning (SFT) equivalent of m$^2$DPO, with matched training steps, learning rates, data, and batch size, which shows that SFT has a considerable performance gap from CAPS on the full data despite a fair training budget. We will incorporate this experiment and analysis in Table 2 of the revised version.

Table R2: Comparison with budget-matched SFT under PoseAlign using the full training data. SP denotes SFT and CAPS denotes m$^2$DPO, both start training from base.

recolor Gflip Gmove L−AeL-A_eadd L−AeL-A_ereplace L−AeL-A_eremove L−AeL-A_emultiop L−AeL-A_e
base (no m$^2$DPO)23.624.344.137.338.431.840.1
CAPS (best)27.324.146.038.539.028.542.2
SP (fair budget SFT)24.624.045.237.537.827.140.4

Q3(1): Are training and validation sets strictly scene-disjoint for every intervention source?

R6: The training and validation sets are strictly scene-disjoint. Specifically, we designate the training-validation-test split as exactly in official ScanNet splits.

Q3(2): What is the human agreement with the intervention-specific LLM-as-Judge metrics?

R7: Thank you for proposing this experiment. Following your great suggestion, we assessed the three evaluation judges against human annotations using outputs from six models on a shared random subset of 150 samples, stratified across recolor, flip, and general interventions. After excluding answers that cannot be judged due to referring to out-of-GT items, 868 of 900 responses remained. Table R3 reports accuracy (%) under judge and human scoring for each intervention group and overall. Correlations are calculated across the six models, with R2R^2 denoting squared Pearson correlation.

Table R3: Accuracy (%) under judge and human scoring, with squared Pearson correlations, Pearson p-values, and Kendall rank correlations (exact p-values) across six models.

ModelRecolor judgeRecolor humanFlip judgeFlip humanGeneral judgeGeneral humanOverall judgeOverall human
Ours (m$^2$DPO + PoseAlign)64.064.052.256.538.842.951.754.5
Ours (m$^2$DPO)44.942.935.740.540.042.040.441.8
Composite SFT36.738.825.531.928.032.030.134.2
3D-LLaVA36.034.024.428.926.526.529.229.9
LEO20.412.226.130.412.216.319.419.4
LL3DA16.316.318.416.316.020.016.917.6
R²0.9700.9600.9780.991
p (Pearson)3.3e-46.1e-41.8e-42.9e-5
Kendall τ (exact p)0.87 (0.017)0.87 (0.017)0.87 (0.017)1.00 (0.003)

All three judges closely track human assessments of model performance, with R2=0.970R^2=0.970, 0.9600.960, and 0.9780.978 for recolor, flip, and general interventions, respectively (all Pearson p<0.001p<0.001). Each also shows strong ranking agreement (Kendall τ=0.87\tau=0.87, p=0.017p=0.017). Human scoring preserves the overall ranking of all six models, with R2=0.991R^2=0.991 (p=2.9×10−5p=2.9\times10^{-5}) and τ=1.00\tau=1.00 (p=0.003p=0.003), while overall accuracy differs by at most 4.1 percentage points per model. These results support using the three judges for model comparison and show that the overall ordering is robust to replacing judge scores with human assessments.


Overall, we clarified how the existing paired subsets support our central finding (W1, Q1), explained our metric priority and CAPS design and supplied a full-data budget matched SFT (W2(1–2), Q2), specified the main objective and consistent exclusion of language DPO (W2(3), Q2), confirmed the scene-disjoint data splits (Q3(1)), and assessed human agreement with the three evaluation judges (Q3(2)).

Thank you for the detailed rebuttal and additional experiments. Most of my original concerns have been substantially addressed. Although the manual audit provides useful support for the validity of most CoPO negatives, the remaining ambiguous cases should be acknowledged as a limitation. Overall, the rebuttal has resolved most of my substantive concerns. Thus~, I‘d like to update my initial score to 3.

Thank you very much for the positive recognition of our work and the fruitful suggestions throughout the discussion period. We will clarify the limitation of the ambiguous CoPO negatives in the next revision. We would like to express our utmost gratitude for your feedback, which has helped us strengthen the evidence and improve the clarity of our paper.

Reviewer_EDkkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper evaluates the response of 3D Large Multimodal Models (LMMs) to visual changes through six categories of point cloud editing: recolor, flip, move, add, replace, and remove. We propose m2DPO, a method that integrates a symmetry preference term with supervised learning. While recolor and flip tasks provide both pre- and post-edit answers, object-editing tasks offer only the post-edit results. Our experiments reveal a performance gap between traditional tasks and strict paired grounding; furthermore, we observe significant gains when training on recolor tasks and "PoseAlign" settings—the latter of which incorporates an explicit ego-frame.

Critical strengths
  1. The use of "recolor/flip" tasks—featuring fixed problem setups with variable answers—along with the "bilateral accuracy" metric (GG), provides a more direct assessment of visual dependency than unilateral accuracy alone.
  2. The "donor bank" is derived from the training data and refined through a combination of geometric filtering and extensive manual revision. A total of 5,110 object-editing instances have been retained, with detailed documentation provided regarding the types and proportions of the revisions made.
Critical weaknesses
  1. The diagnostic framework for paired causal attribution holds only partially. For four categories of object editing, the original ground-truth answers are unavailable; the main table reports only the post-edit L−AeL-A_e metric. Consequently, it is impossible to verify whether the model correctly tracks changes in the ground truth, let alone attribute these changes exclusively to linguistic priors. Furthermore, the CoPO (Causal-effect-based Prompt-Only) approach implicitly assumes that the original scene serves as a negative example for the post-edit answer—an assumption that lacks validation due to the absence of original ground-truth data (§3.4; Appendix A.4).
  2. The reference frame conditions for standard "flips" are not fully implemented. Lemma 1 correctly posits that a scene shift must be coupled with a reflection along the reference direction; simply reflecting the point cloud while keeping other axes fixed does not necessarily swap "left" and "right." It remains unclear how the implicit "ego-frame" transforms in non-PA (Pose-Aware) data. The significant performance gains observed with PA underscore the importance of reference-frame information, suggesting that the poor performance of standard flips cannot be attributed solely to a failure in visual grounding.
  3. Claims regarding the unique benefits and low-forgetting properties of m2DPO are not sufficiently substantiated. In Table 7, the EM-based score for CAPS (G=15.3G=15.3) is slightly lower than that of standard SFT (SP=15.5SP=15.5), and the advantage of CAP (15.815.8) is marginal. In Table 2, m2DPO underperforms the original model on "move" and "multi-op" tasks, with ScanQA CIDEr dropping from 96.0 to 87.0, and Scan2Cap performance declining from 76.3 to 70.4. These results reflect clear trade-offs; one cannot simply dismiss these performance losses as a "necessary cost" just because existing benchmarks may contain shortcuts.
  4. The quality of the "judge" model and the revised dataset lacks independent verification. The scoring provided by GPT-OSS-20B is not accompanied by an analysis of human-model agreement or error breakdowns by operation type. Templates that prioritize color or spatial orientation (left/right) may overlook other types of errors, while a strict, verbatim matching criterion (I3) might unfairly penalize semantically equivalent expressions. Although manual data cleaning is a strength, the extent of annotator disagreement and the error rate in the revised dataset remain unquantified.
Questions for authors
  1. Is it possible to replenish the original answer of the object editor and report the efficiency of matching G, answer maintenance/change ratio and CoPO negative example?
  2. How are the reference directions of each data source defined and transformed with reflection? Is it possible to manually audit common flip tags and compare models under the same explicit reference conditions?
  3. After matching the additional training rounds, data and parameter budget, is the default CAPS still better than pure SFT?
5 comments

Thank you very much for the fruitful and constructive review. We are grateful for your appreciation of paired recolor and flip evaluation for assessing visual dependence, and for praising the donor-bank construction, geometric filtering, and extensive manual refinement of the 5,110 object interventions.

W1 & Q1: The diagnostic framework for paired causal attribution holds only partially, limiting paired evaluation and validation of CoPO negatives.

R1: Thank you for the discussion on the realization of the diagnostic framework. We never meant to claim CounterGround3D as a benchmark of all paired counterfactual data according to L084, L189, L269, and L846. We also did not claim to be able to verify whether models track the object edits. Instead, we want to examine how the existing models behave under general geometric counterfactuals (L264). We have more broadly mentioned the definition of dataset subsets in the revision, especially in the abstract and introduction to pinpoint the scope of this work throughout the paper.

Besides, in our humble opinion, the current post-edit object subset already provides faithful, valuable, human-corrected label data that already support judging edited-scene VQA accuracy and bidirectional invariance metric to irrelevant interventions. As agreed by all the reviewers, we have already contributed greatly through combinational geometric filtering and manual annotation of 5,110 samples containing edited point clouds, questions, and answers. Further annotating the original answers for the object subset requires substantial human effort, beyond simple scalable agentic automation. Therefore, additional annotation on object subset is an interesting yet unnecessarily costly process that is beyond the scope of this work.

Finally, to comprehensively address the concern on CoPO validity, we manually annotated the original answers by selecting 155 random training samples in 4 object interventions using stratified sampling, covering 71, 34, 28, 22 random samples in move, add, replace, and remove, respectively. The edited answer is incorrect on the original scene in 124 cases (80%): 116 have different answers and eight have no valid answer on the original scene (Table R1). Besides, recolor and flip also provide 85% of the LCoPOτ\mathcal{L}^\tau_{CoPO} terms (24,639 of 28,956, Table 6) with valid negatives by construction. If all 4,317 object training pairs share the observed 80% rate, approximately 97% of all LCoPOτ\mathcal{L}^\tau_{CoPO} terms used valid negatives. Therefore, the effect of noisy CoPO negatives is insignificant, especially when considering the support of other complementary losses. We will add Table R1 and the above analysis to Section 3.4 and Appendix A.4.

Table R1: Answers on the original scene for 155 annotated object subset training pairs.

Answer on P\mathcal{P} compared with aτa^\tauPairsExample: qτq^\tau (answer on P\mathcal{P} → aτa^\tau)
Different116 (74.8%)Which object is closest to the whiteboard? (box → copier)
None, since qτq^\tau asks about the change8 (5.2%)What object is in front of the added chair on the table? (none → cabinet)
One answer refines the other1 (0.6%)Standing by the door, how are the cups positioned relative to the paper towel dispenser? (right → back right)
Same30 (19.4%)Which is positioned higher: the backpack or the mouse on the desk? (mouse → mouse)

W2 & Q2(1): The reference frame conditions for standard flips are not fully implemented.

R2: We thank the reviewer for noting a general problem of this field, which we have already overcome by leveraging the PoseAlign setting. Instead of being an incomplete implementation, our normal setting is a deliberate design choice to maximize comparability with existing benchmarks and models. As ScanQA and Scan2Cap do not provide explicit egocentric reference frames, their original formulations therefore require models to infer an implicit reference frame from scene geometry and linguistic cues in the question. In our experiments, the scene and question are transformed consistently under reflection, so the observable cues defining the implicit reference frame are transformed accordingly. Consequently, the ground-truth implicit reference frame follows the reflection without requiring an additional explicit pose specification. See the revised version appendix A.11 for some qualitative demonstrations of the flip protocol.


Our response continues in Part 2/3.

Continued from Part 1/3.


Training-free model compatibility is another solid motivation behind our design for the non-PoseAlign setting implementation. Following the existing benchmark conventions, vanilla LL3DA, 3D-LLaVA, Grounded-3D-LLM, and DEER-3D receive only a point cloud in the global coordinate frame and a question. Supplying an additional ego reference frame in any other way except transforming the input point cloud (i.e., PoseAlign) would inevitably require changing the inputs and retraining these models. Our PoseAlign experiments provide this explicit frame and evaluate the resulting models separately. LEO is the exception which natively accepts ego-pose encoding on SQA3D, and we did reflect that pose consistently with the scene along the global x axis.

Whether the model is able to decode such GT implicit reference frame, however, is a different topic, as correctly pointed out by you (Reviewer #EDkk). A model failing on flip in normal setting can be caused by failing to infer the reference frame, failing to ground the directional relations, or failing in both. The PoseAlign experiments allow us to examine the effect of directional grounding alone with an explicit reference frame. In Table 2, 3D-LLaVA's flip grounding rises from 1.9% to 9.9% with PoseAlign, showing the benefit of explicit alignment. Under the same PoseAlign setting, CounterGround3D training with m$^2$DPO raises it further to 24.1%. These results show that providing the frame alone still leaves substantial grounding failures unresolved, and that our training improves directional grounding when the frame is explicit. We will add the above clarification in the revision.

W3 & Q3: The additional benefit of m$^2$DPO and its preservation of existing capabilities need stronger support.

R3: We thank the reviewer for asking us to clarify the contribution of preference training. We select CAPS mainly to retain most complementary training objectives to avoid tailoring the method solely to the ranking on recolor data. Our design choice prioritizes paired visual grounding especially in the unambiguous PoseAlign setting, where improvements can be seen on most intervention types. Relative to Ours-composite+PA, Ours-m$^2$DPO+PA improves recolor grounding from 23.6% to 27.3%, move accuracy from 44.1% to 46.0%, add from 37.3% to 38.5%, replace from 38.4% to 39.0%, and multi-op from 40.1% to 42.2% (Table 2). We mainly attribute m$^2$DPO as being able to preserve existing capabilities in the sense that it preserves what's lost in raw mDPO: in Table 7, CAPS retains 31.0% original-scene EM against 32.4% for Ours-composite, while vanilla mDPO falls to 3.7%. Therefore, we attribute the performance of m$^2$DPO as a result of deliberate design choices and objective preferences rather than plain tradeoffs between equally weighted performance axes.

To comprehensively address Q3, we re-train a budget-matched SP model (SFT-only) identical to those of m$^2$DPO in PoseAlign setting with all training data in Table R2. The SP row is the budget-matched supervised finetuning (SFT) equivalent of m$^2$DPO, with 100% matched training steps, learning rates, data, and batch size, which shows that SFT has a considerable performance gap from CAPS on the full data despite a fair training budget. We will incorporate such experiment and analysis in Table 2 of the revised version.

Table R2: Comparison with budget-matched SFT under PoseAlign using the full training data. SP denotes SFT and CAPS denotes m$^2$DPO, both start training from base.

recolor Gflip Gmove L−AeL-A_eadd L−AeL-A_ereplace L−AeL-A_eremove L−AeL-A_emultiop L−AeL-A_e
base (no m$^2$DPO)23.624.344.137.338.431.840.1
CAPS (best)27.324.146.038.539.028.542.2
SP (fair budget SFT)24.624.045.237.537.827.140.4

Our response continues in Part 3/3.

Continued from Part 2/3.


W4: Human agreement with the judges.

R4: Thank you for proposing this experiment. Following your great suggestion, we assessed the three LLM-as-Judge prompt annotations by gpt-oss-20b against human annotations using outputs from six models on a shared random subset of 150 samples, stratified across recolor, flip, and general interventions. The human annotator is qualified for the task as an independent PhD holder with point cloud expertise. After excluding answers that cannot be judged due to referring to out-of-GT items, 868 of 900 responses remained. Table R3 reports accuracy (%) under judge and human scoring for each intervention group and overall. Correlations are calculated across the six models, with R2R^2 denoting squared Pearson correlation.

Table R3: Accuracy (%) under judge and human scoring, with squared Pearson correlations, Pearson p-values, and Kendall rank correlations (exact p-values) across six models.

ModelRecolor judgeRecolor humanFlip judgeFlip humanGeneral judgeGeneral humanOverall judgeOverall human
Ours (m$^2$DPO + PoseAlign)64.064.052.256.538.842.951.754.5
Ours (m$^2$DPO)44.942.935.740.540.042.040.441.8
Composite SFT36.738.825.531.928.032.030.134.2
3D-LLaVA36.034.024.428.926.526.529.229.9
LEO20.412.226.130.412.216.319.419.4
LL3DA16.316.318.416.316.020.016.917.6
R²0.9700.9600.9780.991
p (Pearson)3.3e-46.1e-41.8e-42.9e-5
Kendall τ (exact p)0.87 (0.017)0.87 (0.017)0.87 (0.017)1.00 (0.003)

All three judges closely track human assessments of model performance, with R2=0.970R^2=0.970, 0.9600.960, and 0.9780.978 for recolor, flip, and general interventions, respectively (all Pearson p<0.001p<0.001). Each also shows strong ranking agreement (Kendall τ=0.87\tau=0.87, p=0.017p=0.017). Human scoring preserves the overall ranking of all six models, with R2=0.991R^2=0.991 (p=2.9×10−5p=2.9\times10^{-5}) and τ=1.00\tau=1.00 (p=0.003p=0.003), while overall accuracy differs by at most 4.1 percentage points per model. These results support using the three judges for model comparison and show that the overall ordering is robust to replacing judge scores with human assessments.

Q2(2): Is it possible to manually audit common flip tags and compare models under the same explicit reference conditions?

R5: Thank you for asking for further clarification. During construction, we used GPT-5.4 to identify directional expressions in context and exclude nonspatial uses, such as “left” meaning “remaining” and “right” meaning “correct.” For the retained spatial relations, reflecting the scene together with its GT reference frame reverses the labels by construction (Lemma 1). We also manually inspected reflected scene pairs during development. See the revision appendix A.11 for some visualizations of the flipped twin of point cloud scenes, their ego reference frames proposed by PoseAlign, and the answer changes according to the flip.

Table 2 already compares models under the same explicit reference conditions in the PoseAlign setting. Any additional cue that specifies the reference direction would inevitably make that direction an explicit model input. We use PoseAlign because it supplies the frame by aligning the point cloud to the ego pose, while preserving the models' existing input format. This avoids adding textual hints or requiring a new pose encoder and training to use it, making the protocol applicable to existing models. This is also the generally accepted protocol as discussed in our reply to W2.


Overall, we clarified the evaluation scope and provided annotation evidence for CoPO negative validity (W1, Q1), explained why the reflection protocol preserves the original benchmarks' assumptions and comparability (W2, Q2), justified our priority on paired grounding and evaluated a fair SFT model under a matched training budget (W3, Q3), and assessed agreement between human judgments and the three evaluation judges (W4).

Reviewer_oBVHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CounterGround3D to evaluate whether 3D multimodal models follow controlled changes to point clouds. It contains approximately 31.9k examples covering recoloring, reflection, movement, addition, replacement, and removal. Recoloring and reflection support paired evaluation requiring correct answers before and after an intervention, while the four object-intervention subsets evaluate edited-scene accuracy. The paper also introduces m²DPO, combining multimodal preference objectives with supervised losses, and evaluates a separate PoseAlign setting with explicit egocentric alignment. Experiments show substantial failures of existing models on counterfactual inputs and clear improvements in recolor grounding from training on the new data.

Critical strengths
  1. The benchmark provides a useful diagnostic beyond conventional task accuracy. Table 2 shows that blind 3D-LLaVA retains substantial conventional performance while achieving zero recolor and flip grounding. Requiring correctness on both sides of a pair, together with invariance and invariant-correctness metrics, distinguishes sensitivity to relevant visual changes from mere answer stability. This is a meaningful contribution for evaluating 3D grounding.

  2. The curation effort and demonstrated value of the training data are substantial. The object subsets undergo exhaustive manual screening, with placement, question, and answer corrections documented in Table 5. Recolor paired grounding improves from 6.1% for 3D-LLaVA to 17.4% with composite training and 20.0% with m²DPO (Table 2), with supporting source-specific results in Table 4. The paper also transparently separates PoseAlign comparisons and documents the DEER-3D reimplementation. These results support the resource's practical value.

Critical weaknesses
  1. The scope and interpretation of the grounding diagnosis need stronger support. Section 3.4 states that the four object-intervention subsets lack pre-edit answers, so their scores measure edited-scene accuracy rather than correctness on both sides of a counterfactual pair. This affects about 16% of the examples and does not invalidate the paired recolor/flip evaluation; the limitation is also disclosed. However, without pre-edit answers, it remains unclear how the original scene is verified to be an incorrect condition for the edited answer used as a CoPO negative. Separately, low directional scores can reflect reference-frame ambiguity: Lemma 1 requires joint reflection of the scene and reference direction, and PoseAlign explicitly supplies an aligned frame. PoseAlign is appropriately ranked separately with corresponding baselines, so this is an interpretation concern, not an allegation of unfair comparison. The benchmark would be more convincing if answer-changing interventions and the observable reference frame were explicitly established for the conclusions being drawn.

  2. The additional benefit of m²DPO over matched supervised training is not established. m²DPO receives an extra training epoch after composite training (Appendix A.4, pp. 16–17). In the more controlled recolor ablation, default CAPS achieves 15.3% grounding versus 15.5% for supervised-only SP (Table 7). These are EM-based scores, distinct from Table 2's LLM-judge scores. Without PoseAlign, m²DPO also underperforms composite training on all four object operations, multi-operation evaluation, and several conventional metrics. This does not negate the dataset contribution, but it weakens the claimed independent benefit of the preference method and the claim of broadly preserved capabilities.

Overall, I lean toward weak rejection primarily because the diagnostic conclusions and intervention validity need clarification, with the method's uncertain additional benefit as a secondary concern. Focused evidence addressing these points could change my assessment.

Questions for authors
  1. Clarification of intervention/negative validity: How was it verified that the object interventions change the answer when pre-edit answers are unavailable? What prevents an edited answer that remains valid in the original scene from becoming a CoPO negative?

  2. Clarification of the training comparison: Are results available for an additional supervised epoch matched to m²DPO's initialization, data, and training budget? How should the default CAPS/SP comparison in Table 7 inform the claimed methodological contribution?

  3. Clarification of the directional protocol: What observable information determines the reference frame in each non-PoseAlign source, and how are that frame and any situated descriptions transformed during reflection? Which directional conclusions remain valid after accounting for reference-frame ambiguity?

6 comments

Thank you very much for the fruitful and constructive review. We are glad to reach an agreement with the reviewer that CounterGround3D provides "a useful diagnostic beyond conventional task accuracy", that paired correctness and the invariance metrics distinguish "sensitivity to relevant visual changes from mere answer stability", and that "the curation effort and demonstrated value of the training data are substantial", including the transparent separation of the PoseAlign comparisons and the documented DEER-3D reimplementation.

W1(1) & Q1: Concerns about scope of this work, and how is the original scene verified as the CoPO negative for the object interventions?

R1: We sincerely thank the reviewer for pointing this out. IMHO, instead of seeing this as a limitation, we treat this subset as a "secondary extension of the CounterGround3D dataset" (L269) which provides physically realized interventions on "object existence, identity, and spatial configuration" (L107) besides the paired recolor and flip data. We also highlight that all reviewers agree with the value of the already-labelled data, which represents substantial curation effort and practical value through full human annotation and correction of 5,110 object subset samples containing post-edit point clouds, questions, and answers. In the revision, we have mentioned the definition of dataset subsets in more locations, especially in the abstract and introduction to pinpoint the scope of this work throughout the paper.

Thanks to the manual annotation process, CoPO uses the original scene as a negative for the object subsets because the question is labelled to ask key changed attributes about the intervention: (1) move and add ask about spatial relations, (2) replace asks or implicitly requires object identity to answer, and (3) remove directly asks about the altered existence or counts. In theory, majority of these answers will change when the scene changes under such object interventions.

To further quantitatively measure how often the object edits change the answer, we manually annotated the original answers for 155 random training samples in 4 object interventions using stratified sampling, covering 71, 34, 28, 22 random samples in move, add, replace, and remove, respectively. The edited answer is wrong on the original scene in 124 pairs (80%), confirming CoPO negatives are mostly valid (Table R1). Besides, recolor and flip also provide 85% of the LCoPOτ\mathcal{L}^\tau_{CoPO} terms (24,639 of 28,956, Table 6), with valid negatives by construction. If all 4,317 object training samples share the observed 80% rate, approximately 97% of all LCoPOτ\mathcal{L}^\tau_{CoPO} terms used valid negatives. Therefore, the effect of noisy CoPO negatives is insignificant, especially when considering the support of other complementary losses. We will add Table R1 and this analysis to Section 3.4 and Appendix A.4.

Table R1: Answers on the original scene for 155 annotated object subset training pairs.

Answer on P\mathcal{P} compared with aτa^\tauPairsExample: qτq^\tau (answer on P\mathcal{P} → aτa^\tau)
Different116 (74.8%)Which object is closest to the whiteboard? (box → copier)
None, since qτq^\tau asks about the change8 (5.2%)What object is in front of the added chair on the table? (none → cabinet)
One answer refines the other1 (0.6%)Standing by the door, how are the cups positioned relative to the paper towel dispenser? (right → back right)
Same30 (19.4%)Which is positioned higher: the backpack or the mouse on the desk? (mouse → mouse)

W1(2): Low directional scores may reflect reference frame ambiguity, since Lemma 1 requires joint reflection of the scene and the reference direction, and PoseAlign supplies an aligned frame.

R2: We thank the reviewer for noting a general problem of this field, which we have already overcome by leveraging the PoseAlign setting. Reference frame ambiguity is already present in the original ScanQA and Scan2Cap settings, which provide no explicit egocentric reference frame, only a point cloud in global coordinate frame and a QA pair. Our experiments without PoseAlign preserve these input conventions to maximize comparability with existing benchmarks and baseline models. To neatly separate grounding failure from direction ambiguity, we leverage the explicit reference frame in the separately evaluated PoseAlign setting to overcome this problem.


Our response continues in Part 2/2.

Continued from Part 1/2.


Since ScanQA and Scan2Cap do not provide explicit ego-centric reference frames, they essentially require models to infer an implicit reference frame from the scene geometry and linguistic cues in the question. In our experiments, both the observable scene geometry and the question cues are flipped consistently according to Equations 4–5, so any ground truth (GT) implicit reference frame should also follow the flip without requiring any additional explicit pose specification. See the revised version appendix A.11 for some qualitative demonstrations of the flip protocol.

Whether the model is able to decode such GT reference frame, however, is a different topic. A model failing on flip in non-PoseAlign setting can be caused by failing to infer the reference frame, failing to ground the spatial relations, or failing in both. As discussed in L330, L362, L421, leveraging the PoseAlign setting, we successfully separate grounding capability from reference frame ambiguity in existing datasets. In Table 2, 3D-LLaVA's flip grounding rises from 1.9% to 9.9% with PoseAlign, showing the benefit of explicit alignment. Under the same PoseAlign setting, CounterGround3D training with m$^2$DPO raises it further to 24.1%. These results show that providing the frame alone leaves substantial grounding failures, and that our training improves directional grounding when the frame is explicit. We will add the above clarification in the revision.

W2 & Q2: What is the benefit of m$^2$DPO beyond supervised training with a matched budget, and how well does it preserve existing capabilities?

R3: We thank the reviewer for asking us to clarify the contribution of preference training. The main focus of m$^2$DPO lies in the PoseAlign setting (where the reference frame is clearly defined) with gains in almost all intervention types. Relative to Ours-composite+PA, Ours-m$^2$DPO+PA improves recolor grounding from 23.6% to 27.3%, move accuracy from 44.1% to 46.0%, add from 37.3% to 38.5%, replace from 38.4% to 39.0%, and multi-op from 40.1% to 42.2% (Table 2). We select CAPS mainly to retain most complementary training objectives to avoid tailoring the method solely to the ranking on recolor data.

To comprehensively address this concern, we re-train a budget-matched SFT model in PoseAlign setting with all training data in Table R2. The SP row is the budget-matched supervised finetuning (SFT) equivalent of CAPS (m$^2$DPO), with 100% matched training steps, learning rates, data, and batch size. An additional epoch brings a marginal increase in SFT performance, which still lags behind our m$^2$DPO despite a fair training budget.

Table R2: Comparison with budget-matched SFT under PoseAlign using the full training data. SP denotes SFT and CAPS denotes m$^2$DPO, both start training from base.

recolor Gflip Gmove L−AeL-A_eadd L−AeL-A_ereplace L−AeL-A_eremove L−AeL-A_emultiop L−AeL-A_e
base (no m$^2$DPO)23.624.344.137.338.431.840.1
CAPS (best)27.324.146.038.539.028.542.2
SP (fair budget SFT)24.624.045.237.537.827.140.4

Besides, we mainly attribute m$^2$DPO as being able to preserve existing capabilities in the sense that our m$^2$DPO preserves what's lost in raw mDPO: in Table 7, CAPS retains 31.0% original-scene EM against 32.4% for Ours-composite, while vanilla mDPO falls to 3.7%. We will incorporate such experiment, analysis, and clarification in Table 2 and appendix in the revised version.

Q3: Clarification of the directional protocol.

R4: We thank the reviewer for further querying the direction protocol details. Please first see our response to W1(2).

We follow common practice to add SQA3D situation descriptions to the questions, both of which are flipped together in Equation 5. The items with changed descriptions + questions are later filtered according to L255 to avoid language shortcuts. Please refer to the revision for some qualitative flip protocol visualizations.

All conclusions remain valid since we accounted for direction ambiguity from the start.


Overall, we provided annotation evidence supporting the validity of CoPO negatives (W1(1), Q1), clarified the inherited reference frame assumptions and the directional grounding gains under PoseAlign (W1(2), Q3), and evaluated the contribution of preference training through a budget-matched experiment (W2, Q2).

Thank you for the detailed response and additional experiments. The clearer distinction between paired recolor/flip evaluation and edited-scene evaluation for object interventions addresses my concern about the benchmark’s scope. I continue to appreciate the dataset construction and its diagnostic value.

The manual audit of 155 object-intervention pairs is useful. It supports the validity of most sampled negatives, while also identifying a nontrivial fraction for which the edited answer is unchanged or potentially compatible with the original scene. The estimated 97% overall validity helps contextualize this issue, but does not by itself establish that the training impact is negligible. Could you clarify whether these ambiguous or unchanged-answer cases are filtered during training? If not, I suggest qualifying the claim that their effect is insignificant.

The budget-matched SFT comparison also strengthens the evidence for m²DPO under PoseAlign, particularly on recolor grounding. The gains on several other metrics are smaller, so I would interpret this as a setting-specific benefit rather than a broad improvement across all capabilities. Could you confirm that CAPS and SP use the same initialization and checkpoint-selection criterion, particularly given the “CAPS (best)” label?

Finally, I appreciate the acknowledgement that non-PoseAlign directional failures can arise from reference-frame inference, spatial grounding, or both. The conclusions should preserve this distinction.

Overall, the rebuttal strengthens the case for the benchmark and partially resolves my methodological concerns. These remaining clarifications would help me finalize my assessment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Remarks by Authors

We sincerely thank all reviewers for their thorough evaluation and constructive feedback. We are encouraged that all reviewers recognize the value of CounterGround3D, and Reviewer #mwDa has increased their score to 3. The reviews have highlighted the following shared recognitions:

  • (All reviewers) Paired recolor/flip evaluation providing a more direct assessment of visual dependence than conventional task accuracy.
  • (All reviewers) Substantial dataset construction and curation, combining physical point-cloud edits, donor-bank and geometric filtering, and extensive manual correction with transparent documentation.
  • (Reviewers #EDkk, #mwDa) Significantly improved counterfactual grounding through training on CounterGround3D, with m$^2$DPO showing gains under PoseAlign.

Building on this positive reception, we are committed to further strengthening our contribution. All improvements promised in our rebuttal will be incorporated into the revised manuscript:

  • Enhanced documentation: Clearer dataset scope, detailed reference-frame protocols and flip visualizations, and a clearly defined total loss with default parameteres.
  • Expanded validation: The original-answer audit, budget-matched SFT comparison, and human evaluation supporting the reported LLM-as-Judge model rankings.
  • Refined interpretation: Explicit acknowledgment of ambiguous CoPO negatives and their unmeasured training impact, together with the distinction between reference-frame inference and spatial grounding.

We believe these enhancements will further strengthen CounterGround3D's contribution to the community as a benchmark and curated training resource for improving visual grounding in 3D LMMs. Thank you for your dedication throughout this review process.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.