acceptodds
Under review as a conference paper at ICLR 2027

Do 3D Large Language Models Really Use 3D Evidence?

31 views

Abstract

Recent 3D Large Language Models (3D-LLMs) have shown strong performance on scene question answering with extensive post-training. Yet, existing benchmarks rarely evaluate how 3D-LLMs respond to referential ambiguity, insufficient scene evidence, or false presuppositions, obscuring whether their responses are grounded in 3D evidence. In this work, we introduce UNICORNS, a benchmark for evaluating the reliability of 3D-LLM responses across four evidence conditions. UNICORNS contains 11,387 questions across 557 ScanNet scans, covering eight challenge categories alongside answerable controls. A response passes only if it satisfies the required response behavior and contains no scene claim unsupported by the available evidence. Evaluation of six open-sourced 3D-LLMs reveals a pronounced performance gap: even the strongest zero-shot model achieves 61.22% accuracy on answerable controls, but only 40.94% across the eight challenge categories. To address this gap, we further propose HORN, an obligation-aware post-training method combining group-relative task rewards, leaf-conditioned obligation rewards, and supervised control replay. Extensive experiments demonstrate that HORN reaches 75.1±1.0% across the challenge categories and 77.2±0.8% overall, outperforming all baselines without degrading answerable-control accuracy. Our findings enable a systematic evaluation and improvement of evidence-grounded 3D reasoning for 3D-LLMs(code).

Primary area:
datasets and benchmarks
Keywords:
3D large language models, point clouds, reliability, visual question answering, benchmark, post-training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_R6s8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper introduces UNICORNS, a benchmark for evaluating whether 3D-LLMs respond appropriately to ambiguity, unavailable information, and false premises without making unsupported claims. It also proposes HORN, a post-training approach combining task and obligation rewards with supervised control replay. Experiments evaluate existing models and examine improvements through ablations, controlled scene edits, and additional generalization tests.

Critical strengths
  1. The motivation is convincing. Reliable 3D question answering requires recognizing what the available observations can and cannot support. Evaluating clarification, uncertainty, and premise correction alongside ordinary answering addresses a meaningful gap in conventional benchmarks.

  2. The work is fairly complete, with paired answerable controls, scene-disjoint splits, training ablations, and controlled scene-edit experiments. In particular, the scene edits provide useful evidence that model responses follow changes in scene content. HORN also improves challenge performance while preserving answerable-control accuracy on the main backbone.

Critical weaknesses
  1. The validation of the underlying evidence needs more detail. Although the paper states that retained samples undergo human review, it is unclear whether reviewers independently inspect the original scenes to verify both the correctness and completeness of the structured evidence. This matters because the veto rule could reject a valid scene detail simply because it is missing from the evidence supplied to the judge. Agreement between human and LLM judges does not resolve this concern if both rely on the same evidence package.

  2. The paper does not report human performance on the benchmark itself. The 480-response audit evaluates human–judge agreement on model outputs, rather than humans answering the questions. A human reference under comparable observation conditions would help establish whether the questions and response requirements are reasonable and contextualize the remaining model errors. For a benchmark focused on evidence sufficiency, this is an important missing validation.

Questions for authors
  1. Could you clarify the human review procedure for dataset construction, including whether reviewers inspect the original scenes, how evidence omissions are identified, and how disagreements are resolved? Did the separate 480-response audit also allow access to the original scenes?

  2. Could you provide human answering performance on a representative subset, with access to comparable scene observations but without the ground-truth evidence or reference answers? An analysis of human failures would also help distinguish model limitations from ambiguity in the questions or scoring criteria.

Reviewer_Tcq5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether 3D-LLMs truly rely on 3D scene evidence when generating responses. The authors introduce the UNICORNS benchmark to evaluate response reliability under conditions such as ambiguity, unobserved information, and false premises. They further propose HORN, a post-training method that combines obligation-aware rewards with control replay to improve evidence-grounded response behavior. The experiments cover multiple 3D-LLMs and include ablation studies, controlled scene edits, Ego-Visible evaluation, text-only evaluation, and cross-domain testing.

Critical strengths
  1. The problem setting is meaningful. UNICORNS evaluates not only answer correctness, but also whether the response strategy is appropriate and whether the claims in the complete response are supported by scene evidence. Compared with conventional 3D QA benchmarks, this provides a more fine-grained evaluation of model reliability under insufficient evidence or false premises.

  2. The experimental analysis is relatively comprehensive. The authors include controlled scene edits, text-only evaluation, Ego-Visible experiments, and human validation, providing useful support for the reliability of the benchmark and evaluation protocol.

Critical weaknesses
  1. HORN is strongly coupled with the predefined taxonomy of UNICORNS. Its leaf-specific obligation rewards explicitly encode behaviors such as clarification, uncertainty, and correction. Therefore, part of the observed gain may come from learning response patterns associated with predefined question types rather than from more general 3D evidence reasoning. The current paper lacks unseen-template, paraphrase, or held-out-category experiments to rule out this possibility.

  2. The methodological contribution and evidence for generalization remain limited. The improvement of HORN over matched GRPO is relatively modest, and on KITTI-360 the Challenge score improves by only 2.74 percentage points while the Control score slightly decreases. This raises concerns about the significance of HORN as a general post-training method and its robustness under domain shift.

Questions for authors
  1. Could the authors provide experiments on unseen paraphrases, held-out question templates, or held-out categories to verify that HORN is not mainly learning response patterns associated with fixed challenge types?

  2. Do HORN and GRPO use strictly matched numbers of updates, sampling budgets, and hyperparameter tuning budgets? Please provide further details to clarify the fairness of the comparison.

2 comments
Reviewer_Tcq5OpenReview ↗

Minor Suggestions for Improving the Revised Manuscript

Thank you for the revised manuscript. The additional experiments and analyses have improved the paper. I have a few minor suggestions that could further strengthen its clarity and reproducibility.

  1. In Appendix F.3, it would be helpful to include some additional details on how the 13,998 question paraphrases were generated and how their semantic consistency was ensured. Reporting separate Challenge and Control results in Table 19 could also provide further insights.
  2. The related work discussion could be slightly broadened to include complementary approaches such as 3D-MoRe, which explores multimodal contextual reasoning for 3D question answering, and A0, which focuses on spatial affordance understanding for robotic manipulation. These works may provide useful perspectives on 3D scene understanding and embodied reasoning.
  3. In Appendix J, the authors may consider providing a brief clarification of the training and test splits used for KITTI-360 target-domain adaptation, particularly how overlap between training and evaluation scenes or crops was handled.

These are minor suggestions intended to improve the presentation and reproducibility of the work. I am inclined to accept this paper.

AuthorsOpenReview ↗

Thank You for Your Suggestions to Improve the Revised Manuscript

Thank you for your positive assessment of the revised manuscript and for these further suggestions. We address each point below.

1. Paraphrase generation, semantic consistency, and separate results

We used DeepSeek to generate question paraphrases, primarily varying wording and word order while preserving the original scene and the information being requested.

To check semantic consistency, the generation instructions required preserving the referent, attributes, relational meaning, and observation scope, while avoiding answer cues. After generation, the original–paraphrased question pairs underwent automated semantic screening, followed by AI-based verification against explicit logical and linguistic rules.

The separate results for Challenge and Control are as follows:

MethodChallengeControlOverall
SFT63.27%68.95%66.11%
GRPO64.24%69.98%67.11%
HORN69.31%74.88%72.10%

We will add the generation and verification details to Appendix F.3 and include the separate Challenge and Control results in Table 19.

2. Additional related work

Thank you for suggesting these complementary works. We will include 3D-MoRe and A₀ in the related work discussion.

3. Training/test splits for KITTI-360 target-domain adaptation

We split the data at the driving-sequence level, assigning all selected scene crops and their associated questions from a given sequence to the same subset.

We verified that the training and test sets have no shared question IDs, scene-crop IDs, or source PLY files. Using the saved crop coordinates, we also checked the 30 m × 30 m XY crop rectangles and found no positive-area overlap between crops across the two subsets.

We will add these split and overlap-verification details to Appendix J.

Reviewer_UE58OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper (UNICORNS) fills a gap by jointly testing evidence-appropriate response strategy and factual support for every scene claim in 3D-LLM domain. It proposes a benchmark UNICORNS that jointly evaluates evidence-appropriate response behavior and factual support for complete responses. It focus on the point-cloud-native 3D object. It includes 11,387 questions spanning 557 ScanNet scans and 353 object classes, covering four evidence states through eight challenge categories plus paired answerable controls. Scene-disjoint splits, executable inventory/geometry checks, and a three-LLM evidence-veto judge make evaluation diagnostic: any unsupported scene claim fails the response even if the strategy is appropriate.

The authors also propose HORN, a post-training recipe: it combines group-relative task rewards, leaf-conditioned obligation rewards, and supervised control replay. The reported gains over matched GRPO and SFT, together with ablations, scene edits, Ego-Visible tests, and text-only controls, give some support for its effectiveness. The evaluation of six 3D-LLMs is also informative. At the same time, the paper relies on LLM judges, covers only ScanNet and KITTI-360, and shows only modest gains on some backbones.

Critical strengths

UNICORNS contributes both a substantial dataset and a diagnostic evaluation design. The benchmark contains 11,387 questions across 557 ScanNet scans and 353 object classes, with eight challenge categories, paired answerable controls, scene-disjoint splits, and executable scene checks. It then scores response strategy and factual support for the complete response together, and the evidence-veto rule makes failures diagnosable.

Critical weaknesses

The major innovation from the view of benchmark, when comparing with AbstainEQA、Ambi3D、AQuA、Real-3DQA is not so outstanding. For example, when compared with AbstainEQA, UNICORNS’s Unobserved, Ambiguity, and False Presupposition categories overlap substantially in concept with AbstainEQA. It remains to be judged how much substantive innovation its 3D adaptation and finer subdivision of these issues actually bring.

Questions for authors
  1. UNICORNS uses agreement between LLM judges and human judges to support the reliability of its LLM judge. However, the human validation covers only 480 responses and is not reported separately by the eight challenge leaves, by model, or by pass/fail type. Please explain how the results from the two types of judges are aligned here.
  2. Could the authors report the stability of LLM-judge verdicts under prompt perturbations, human audits broken down by challenge category and model, and inter-annotator agreement?
2 comments
AuthorsOpenReview ↗

Q1: Alignment of human and LLM judgments

We thank for the question. Human and LLM judgments are aligned at the level of the same model response, using the final Pass/Fail label. The original manuscript reported an audit of 480 responses. Our supplementary model-level audit covers GPT4Scene, SceneCOT + GRPO, and SceneCOT + HORN, with 480 responses per model and 1,440 responses in total. Responses are sampled randomly, with equal Challenge and Control representation and balanced coverage of the eight challenge leaves. Three annotators independently assess each response, remain blind to the LLM verdicts, and may access the original 3D scene.

Both human and LLM judgments follow the corresponding scene evidence and response requirements: a response must use an appropriate strategy and satisfy factual-support requirements. For example, expressing uncertainty before inventing a count or attribute still results in Fail; an unwarranted refusal on an answerable Control item also fails.

Each of the three LLM judges casts a joint Pass vote only when both strategy and factual support pass. The ensemble verdict is the majority vote over these joint judgments. Each individual human label is then compared with the ensemble verdict for that response; human labels are not first aggregated by majority vote. This yields 4,320 human–LLM comparisons.

FP denotes an LLM Pass with a human Fail; FN denotes an LLM Fail with a human Pass.

ModelResponsesHuman–LLM comparisonsAgreementFPFN
GPT4Scene4801,44095.00%2052
SceneCOT + GRPO4801,44095.21%2148
SceneCOT + HORN4801,44095.42%2145
Overall (pooled)1,4404,32095.21%62145

The results by challenge leaf are:

Challenge leafMean agreementFPFN
U86.67%48
N87.78%38
I92.22%25
C96.67%12
O96.67%12
R85.56%58
N′90.00%36
S94.44%14
AuthorsOpenReview ↗

Q2: Prompt stability, grouped audits, and inter-annotator agreement

We thank for the question.

(a) Stability under prompt perturbations.

We added an evaluation of semantic rewrites of the scoring rules on 720 responses from 9 evaluated model configurations, comprising 360 Challenge and 360 Control responses. Each configuration contributes five responses per challenge leaf and 40 Control responses.

We hold the questions, scene evidence, complete model responses, reference answers, output JSON schema, and main section order fixed. We rewrite the global scoring rules and the 53 leaf-specific Pass/Fail criteria appearing in the sample, with the aim of preserving their meaning. The three global rewrites are authored separately and share the same rewritten leaf criteria. The baseline prompt and three rewrites yield 2,880 valid judgments.

Agreement with the baseline prompt is:

Scoring-rule rewriteVerdict agreementCohen’s κVerdict flipsChange in overall pass rate
Rewrite 199.86% (719/720)0.99721−0.14 pp
Rewrite 299.72% (718/720)0.994420.00 pp
Rewrite 399.44% (716/720)0.98894−0.28 pp

For these scoring-rule rewrites, verdict agreement ranges from 99.44% to 99.86%, and the net change in overall pass rate is at most 0.28 percentage points.

(b) Human audits by challenge category and model.

The human audits by challenge leaf and model are reported in our response to Q1 above.

(c) Inter-annotator agreement.

For the three human annotators on 480 responses, Fleiss’ κ is 0.9417, three-way unanimous agreement is 95.63%, and mean pairwise agreement is 97.08%.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revision Summary

We thank the reviewers for their feedback. Revisions are marked in blue.

  • UE58. The Introduction clarifies how UNICORNS constructs and verifies questions using 3D geometry. Appendix D reports agreement by model and challenge category (95.21% overall), inter-annotator agreement (Fleiss’ κ = 0.9417), and judge stability under prompt perturbations (99.44–99.86% agreement).
  • R6s8. Section 3.2 clarifies how reviewers verify each question against its original scene. Appendix D reports independent human answering on 480 questions (92.92% on Challenge and 98.06% on Control) and analyzes the main errors.
  • Tcq5. Appendices C, F.3, and J clarify the HORN–GRPO comparison, report HORN’s 4.99-point gain over GRPO on unseen paraphrases, and distinguish KITTI-360 target-domain adaptation from transfer. Generalization beyond the current taxonomy remains open. We improved the figure order on page 8 and placed Figure 7’s panels side by side. We will review all figures and tables for readability in the final version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.