acceptodds
Under review as a conference paper at ICLR 2027

Grounding Generative Verification in 3D Content

Abstract

Recent advances in multimodal large language models (MLLMs) and 3D generation have significantly expanded AI capabilities in understanding and synthesizing 3D content. However, beyond visual fidelity, task-specific correctness grounded in 3D evidence remains a major challenge. We argue that building reliable 3D systems requires not only stronger generation and reasoning, but also robust 3D verification, which offers a crucial mechanism for scaling inference by providing direct feedback for candidate selection and iterative refinement. We investigate the capabilities and supervision required for effective verification, which demands integrating multi-view partial observations and disentangling true 3D geometry from projection-dependent appearances. In this work, we make three contributions. First, we introduce 3DVerifierBench, a benchmark spanning 10 tasks that evaluates decisions, rejection explanations, and responses to controlled changes in specifications and candidates. A human study reveals a 21.6 percentage-point gap between humans and the strongest tested model, GPT-6 Astra, while explanation-aware analysis exposes further weaknesses in rejection reliability. Second, we analyze verification decisions across different fine-grained factual categories, with reviewed cases exposing incorrect factual values and target binding. Motivated by these findings, we develop a scalable data construction pipeline that links decisions to specific facts and contradictory evidence. It combines multi-view fact extraction, controlled 3D editing, and native annotations to produce a larger dataset with 14,126 samples, with explicit contradiction explanations and linked counterfactuals. Third, we examine the potential of verification under different requirements to support 3D generation through candidate selection and recursive refinement. In human-labeled sixteen-candidate pools, oracle selection improves complete factual success from 26.7% to 90.0%. In a separate eight-input study, human-guided recursive refinement satisfies all seven requirements for every input, compared with three under parallel Oracle@8. Together, our benchmark, analysis, and data pipeline establish a foundation for developing 3D verification as a capability in its own right and translating it into more reliable generative 3D systems.

Primary area:
datasets and benchmarks
Keywords:
3D Verification, Multimodal Large Language Models, 3D Reasoning, 3D Object Understanding, Benchmark, 3D Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_P5eLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the reliability of multimodal large language models (MLLMs) in verifying whether generated 3D content satisfies specific factual requirements. The authors introduce 3DVerifierBench, a benchmark comprising 2,581 examples across ten 3D verification tasks, evaluating binary decisions, contradiction explanations, and counterfactual consistency. They further develop a scalable data construction pipeline that combines multi-view fact extraction, controlled 3D editing, and expert review, producing 14,126 training examples. Experiments reveal substantial gaps between existing MLLMs and human performance, particularly in fine-grained geometry, identity preservation, and motion verification. Finally, the paper explores the potential of reliable verification for improving 3D generation through candidate selection and human-guided recursive refinement.

Critical strengths
  1. Meaningful benchmark design with fine-grained verification. The paper presents a valuable perspective on 3D generation evaluation by emphasizing requirement-specific factual correctness rather than overall visual quality. In particular, the specification-side and candidate-side counterfactual evaluations provide a useful mechanism for examining whether verification decisions consistently respond to changes in requirements and visual evidence. The inclusion of contradiction-aware explanation evaluation further strengthens the benchmark.

  2. Comprehensive evaluation and substantial data construction efforts. The benchmark covers ten heterogeneous 3D tasks and evaluates a broad range of MLLMs, providing useful insights into their limitations in 3D verification. The data construction pipeline integrates multiple 3D resources, controlled modifications, and expert validation. The human comparison and downstream generation experiments also help demonstrate the potential importance of reliable 3D verification.

Critical weaknesses
  1. Limited differentiation from existing 3D evaluation and verification research. My main concern is that several related studies have already investigated automated and fine-grained evaluation of generated 3D content, including Eval3D (CVPR 2025), Hi3DEval (NeurIPS 2025), and 3D-DefectBench (2026). In particular, 3D-DefectBench has explored fine-grained binary defect detection using VLMs, while OmniVerifier (ICLR 2026) has investigated visual verification and verification-guided generation refinement. Although the proposed benchmark introduces additional dimensions, particularly counterfactual consistency and contradiction-aware explanations, the paper could more clearly establish their independent contributions beyond existing evaluation frameworks. The comparisons with prior work are primarily conceptual, and further evidence demonstrating the distinct diagnostic value of the proposed benchmark would strengthen its novelty and significance.

  2. Insufficient validation of the practical utility of the proposed training resource and verification framework. Although the authors construct a training resource containing 14,126 examples, the paper does not demonstrate whether training a verifier on these samples actually improves verification performance. Moreover, the training resource covers only seven of the ten benchmark tasks, excluding challenging categories such as identity preservation, fine-grained geometry, and motion understanding. The downstream generation experiments are also somewhat limited: candidate selection relies on human oracle labels, while recursive refinement uses human-provided verification feedback and is evaluated on only eight inputs. Although these results demonstrate the potential value of reliable verification, they do not establish the effectiveness of a fully automated verification-guided 3D generation system. Further empirical validation would strengthen the practical significance of the proposed framework.

Questions for authors
  1. Compared with existing works such as Eval3D, Hi3DEval, and 3D-DefectBench, what additional verification capabilities does 3DVerifierBench uniquely measure? Could the authors provide more direct evidence that the proposed counterfactual and explanation-aware evaluations reveal limitations not adequately captured by existing 3D evaluation benchmarks?

  2. Could the authors provide preliminary experiments demonstrating the effectiveness of the 14,126-example training resource for improving a 3D verifier? Even a relatively simple fine-tuning experiment on an existing MLLM would help establish the practical value of the constructed data.

  3. The explanation-aware evaluation relies on a reference-semantic judge that compares generated explanations with reference explanations without accessing the underlying visual evidence. Could the authors further clarify the limitations of this evaluation protocol and provide additional evidence that the metric reliably reflects visually grounded verification?

  4. Since the recursive refinement experiments rely on human verification feedback, could the authors clarify how these results are expected to translate to an automated verification-guided generation system? Additional experiments using an automatic verifier, even on a smaller subset, would help substantiate the claimed downstream benefits.

3 comments

Dear Reviewer P5eL,

Since the authors have procvided their feedback, please discuss with the authors. Thanks.

Best regards,

AC

AuthorsOpenReview ↗

Follow-up to AC and Reviewer P5eL

Dear Reviewer P5eL,

We thank the Area Chair for facilitating this discussion, and thank you again for your careful review and constructive feedback.

For your convenience, here is a brief summary of how we addressed your main concerns:

  • Differentiation from existing work: Your main concern was whether 3DVerifierBench offers capabilities beyond existing 3D evaluation and defect-detection benchmarks. In response, we expanded the related work and added a capability comparison (Sec. 2, Table 1), clarifying that our benchmark focuses on verifying explicit requirements, checking contradiction-aware explanations, and testing consistency under controlled counterfactual changes, rather than simply evaluating asset quality or detecting predefined defects. We further demonstrated the diagnostic value of these capabilities: GPT-6 Astra achieves 69.4% OA but only 46.5% triplet success, revealing consistency failures that per-example accuracy alone misses. We also validated the explanation Judge against human assessments, achieving 91% agreement on 300 cases (Table 3).
  • Practical utility of the training set and verification framework: Your concern was that the usefulness of our training set and the feasibility of automated verification-guided refinement had not been demonstrated. Following your suggestions, we fine-tuned two open-source models on our 14,126-example training set, with Gemma-4-31B improving by 8.3 OA points to 69.1%, within 0.3 points of GPT-6 Astra (Sec. 5.2, Table 4). We also extended recursive refinement to 20 inputs and used GPT-6 Astra as an automatic verifier, achieving 14/20 full factual success compared with 11/20 for Parallel Oracle@8 (Sec. 5.5, Table 6). These results provide direct evidence that our training data improves verifier performance and that automatic verification can already benefit iterative 3D refinement.

As the discussion period is scheduled to close on November 18, we wanted to gently follow up, as we have not yet received your feedback on these responses. We understand that you may have a busy reviewing schedule and would greatly appreciate hearing whether these new results and clarifications address your concerns or if any questions remain.

Thank you again for your time and consideration.

Best regards,
The Authors

AuthorsOpenReview ↗

Brief Final Check-in Before Discussion Closes (Nov 18)

Dear Reviewer P5eL,

Apologies for the brief follow-up to our note yesterday. As the discussion period is scheduled to close on November 18, we wanted to send one final, gentle check-in regarding our rebuttal and newly added experiments—including LoRA fine-tuning on our 14k training set, 20-input automated recursive refinement, the benchmark capability comparison, and the visual audit of the explanation Judge.

Since several of your questions overlap with points raised earlier in the discussion, we also wanted to briefly share how those threads concluded: Reviewer qgso raised their initial rating after noting that the new fine-tuning results confirm the practical value of our training set and that the benchmark is now clearly described, while Reviewer 7bgA reaffirmed their positive assessment after our protocol clarifications, visual explanation audits, and new fine-tuning and automatic-refinement experiments addressed their main concerns.

We completely understand how busy your reviewing schedule is, so please do not feel obligated to write a lengthy reply. Even a brief note on whether these additions address your concerns would be greatly appreciated before the window closes.

Thank you once again for your time, constructive feedback, and consideration!

Best regards, The Authors

Reviewer_7bgAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces 3DVerifierBench, a benchmark with 2,581 examples across ten tasks for verifying explicit requirements against 3D evidence. It evaluates binary decisions, rejection explanations, and consistency under specification-side and candidate-side counterfactuals. On the same 400-example subset, humans achieve 87.8% overall balanced accuracy versus 66.2% for GPT-6 Astra; Astra scores 69.4% on the full benchmark. The paper also constructs a source-disjoint, 14,126-example training resource and studies automatic candidate selection, oracle selection, and human-guided refinement. The contribution is primarily a benchmark and empirical analysis, rather than a newly trained verifier.

Critical strengths
  1. The verification interface and linked counterfactuals provide a useful evaluation contribution. The benchmark tests explicit, falsifiable requirements across attributes, part relations, affordances, grasps, identity, geometry, and motion. Requiring correct decisions for an anchor and both specification-side and candidate-side negatives tests whether judgments track controlled changes, beyond isolated accuracy or overall visual quality (Section 3; Appendix C.3). The combination of decision, explanation, and consistency metrics provides complementary evidence about verifier reliability.

  2. The main empirical findings have meaningful controls and downstream relevance. The evaluation includes additional expert review, source-group exclusion, task-balanced aggregation, clustered confidence intervals, and human comparisons on matched examples. The 21.6-point human–Astra gap uses the same 400-example subset, rather than comparing different evaluation sets. Table 4 also demonstrates real automatic-selection benefits: full factual success increases from 26.7% under random selection to 55.0% with Astra. The 90.0% human-oracle result and the separate eight-input human-guided refinement study quantify further potential, while the paper acknowledges that these are not the performance of an existing automatic refinement system.

Critical weaknesses
  1. Several protocol details needed to audit the central findings are underspecified. Section 4 describes expert-established identity thresholds and geometry perturbations, but Appendix B.1 does not provide their numerical settings or boundary-case rules. Appendices B.4 and G.1 also leave unclear the retention procedure after annotation disagreements, the number of human-baseline participants, judgments per example, and aggregation. The construction-stage spot checks and six additional benchmark reviewers are meaningful safeguards; this concern is about reproducibility, not evidence of incorrect labels. The automatic-selection result is similarly worth documenting more fully: Appendix G.4 details oracle selection but does not comparably specify Astra's prompting, scoring, and ranking. These are focused reporting gaps that affect confidence in reproducing the human comparison and the demonstrated downstream gain.

  2. The explanation metric has a narrower diagnostic scope than general explanation correctness. The text-only Judge requires the same violated fact and contradiction as the reference, and explanations accompanying accepted claims are not checked (Appendices B.1 and C.2). The reported 91% agreement over 300 human-assessed explanations supports its use. Nevertheless, a valid alternative contradiction could fail this criterion. This is particularly relevant to the 48 correctly rejected FineGeo responses, pooled across ten models, whose explanations all fail the Judge (Table H6). Clarifying whether the human audit independently considered the visual evidence and alternative valid reasons would help establish whether these extreme failures reflect incorrect explanations or strict reference matching.

Overall, I lean toward weak acceptance. The benchmark structure and matched human-model evidence offer sufficient value, while these concerns mainly affect protocol transparency and the interpretation of explanation failures. The untrained resource and small human-guided refinement study should remain bounded contributions; neither is necessary to treat as proof of a trained automatic verifier.

Questions for authors
  1. Clarification of annotation and human evaluation: What numerical identity thresholds, geometry-perturbation settings, and disagreement/retention rules were used? How many participants contributed to the human baseline, how many judgments were collected per example, and how were they aggregated?

  2. Clarification of explanation validity: Did the explanation-validation participants inspect the original visual evidence independently of the reference explanation? How were alternative valid contradictions treated, and would allowing them change the FineGeo explanation-failure conclusion?

  3. Clarification of automatic selection: What exact prompting, scoring, aggregation, and tie-breaking procedure produces Astra's selected candidate? Please distinguish its full factual success of 55.0% from the oracle's 90.0% when characterizing the remaining gap; this distinction is important to assessing the practical benefit demonstrated by the paper.

4 comments

Dear Reviewer 7bgA,

Since the authors have procvided their feedback, please discuss with the authors. Thanks.

Best regards,

AC

AuthorsOpenReview ↗

Follow-up to AC and Reviewer 7bgA

Dear Reviewer 7bgA,

We thank the Area Chair for facilitating this discussion, and thank you again for your careful review and constructive feedback.

For your convenience, here is a brief summary of how we addressed your main concerns:

  • Protocol transparency and reproducibility: Your concern was that several protocol details were insufficiently specified to fully audit the central findings. In response, we documented the numerical settings for identity and geometry perturbations, including four condition-specific FaceScape identity-distance ranges ([1.2, 1.4], [1.4, 1.6], [1.6, 1.8], and [1.8, 2.0]) and CAD radial deformation parameters (A ≈ 3.67%, with area-weighted RMS displacement of 0.318 to 1.312 mm). We also clarified the expert review and sample-retention rules, together with the human-baseline protocol involving three independent participants, 400 examples, and 1,200 judgments with majority-vote aggregation (Appendices B.1, B.4, and G.1). Additionally, we specified Astra's prompting, scoring, five-key ranking procedure, and tie-breaking rules for automatic candidate selection (Appendix G.4).
  • Validity of explanation evaluation: Your concern was that the text-only Judge might reject valid alternative contradictions without examining the underlying visual evidence. To address this, we added a 300-case human validation in which three assessors independently inspected the visual evidence and model explanations without seeing the reference explanations, achieving 91% agreement with the Judge (Table 3). We further reviewed all 48 correctly rejected FineGeo responses against the original visual evidence and confirmed that all 48 explanations were genuinely invalid, rather than valid alternative contradictions (Appendices C.2 and H.7).

As the discussion period is scheduled to close on November 18, we wanted to gently follow up, as we have not yet received your feedback on these responses. We understand that you may have a busy reviewing schedule and would greatly appreciate hearing whether these clarifications address your concerns or if any questions remain.

Thank you again for your time and consideration.

Best regards,
The Authors

Thank you for the detailed response and additional analyses. The clarifications on annotation procedures, human evaluation, and automatic candidate selection address my main reproducibility concerns.

The explanation audit is particularly helpful: assessors examined the visual evidence without seeing the reference explanations and could accept valid alternative contradictions. The targeted review of the 48 FineGeo responses also addresses my concern about whether the reported failures arose from overly restrictive reference matching. The metric should still be interpreted within its stated scope, including that explanations accompanying accepted claims are not evaluated.

I also appreciate the clearer distinction between automatic selection and the human-oracle upper bound. The newly reported fine-tuning and automatic-refinement experiments provide additional support for the practical value of the resource, while the refinement study remains limited in scale.

Overall, the response strengthens my positive assessment of the benchmark and empirical contribution. I am maintaining my rating of Weak Acceptance.

AuthorsOpenReview ↗

Thank You for Your Feedback

Dear Reviewer 7bgA,

Thank you very much for your thoughtful follow-up and for taking the time to consider our responses and additional analyses. We are pleased that our clarifications on the evaluation protocols, automatic candidate selection, and explanation audits have addressed your main concerns. We also appreciate your positive assessment of the benchmark and its empirical contributions.

We will ensure that these points are clearly reflected in the final version of the manuscript. Specifically, we will retain the clarification that the explanation metric evaluates rejection explanations only, and clearly acknowledge the limited scale of the recursive-refinement study.

Thank you again for your careful review, constructive suggestions, and support throughout the discussion period.

Best regards, The Authors

Reviewer_qgsoOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This work proposes a method that verifies whether a 3D object satisfies a given text description. The use of such a method is that generated 3D objects could be accepted or rejected depending on whether they comply with the given geometric or functional requirements for which it was created. The method does not directly work on the 3D model, instead it uses mult-view 2D renderings of the 3D model but its not clearly described how that works. The paper runs a large number of experiments showing that there is still a large gap between state-of-the-art models like Astra, and human validators.

Critical strengths

The paper proposes an interesting task, evaluating the generated 3D objects against some specified requirements like geometry and functionality (see Fig. 3 for examples). The experiments highlight that even state-of-the-art MLLM still struggle with verifying whether a 3D model is consistent with these specified constraints, compared to human validators.

Critical weaknesses

A major weakness of this work is its writing clarity. Often, it’s really hard to follow due to vague language that is not clearly defined, unclear terminology, and missing technical descriptions.

  • Just as an example (and there are many more), line 288: “When native labels already define the relevant distinction, we use them directly instead of forcing every task through the same editing backend”. What are ‘native’ labels? What is a “relevant distinction”? “use them directly" for what? Unclear terminology “Training Resource” (line 295) - what does that even mean? Just call it a training dataset like everyone else? A key claimed contribution of the paper is the benchmark, however there are very few details that actually describe the benchmark, Section 3 (lines 158-187) are the only ones that describe how the benchmark works while using cryptic language and variable names that are hard to parse -- the fact that the variable symbols are defined on one page and only explained on another page without repeating the variable names just adds to the headache.

The paper describes and promises a 14k example training resource (?, is that a dataset?), but this dataset does not seem to be used anywhere in this work (unless I did not understand it since the paper is written so poorly). It would be interesting to see if the baselines that are compared on the benchmark would be finetuned on this dataset (is it a dataset?) and show potential improvements. This would support the quality and usefulness of the training dataset if it is indeed able to improve benchmark performance.

Questions for authors

Questions for Rebuttal

  • Can you show fune-tuning results on an open model that show that the provided dataset helps improving the performance of baselines on the benchmark? At this point the usefulness of the provided dataset is not demonstrated.

  • Can you more clearly describe how the benchmark actually works? Start from the input 3D model, and a requirement, then clearly describes step by step how the verifier comes to a binary answer with potential additional explanation (I was not able to find that in the paper), in particular how are the existing (close- and open-source) models used.

Remarks:

  • The abstract is terrible: it should at least clearly state the task, large parts are not understandable - it mentions things like “target binding” without explaining it. (Keep in mind that the purpose of the abstract is to decide whether to accept or reject the paper - this one is definitely not helpful for the former.)
  • The related work section seems very short -- usually it should contain some explanations how the proposed work differs from existing works.
1 comment

Dear Reviewer qgso,

Since the authors have procvided their feedback, please discuss with the authors. Thanks.

Best regards,

AC

Comments on the paper

1 comment
AuthorsOpenReview ↗

Discussion Wrap-up and Summary of Revisions

Dear Area Chair, Senior Area Chairs, Program Chairs, and Reviewers,

As the discussion period concludes, we sincerely thank the Reviewers, Area Chair (AC), Senior Area Chairs (SACs), and Program Chairs (PCs) for your dedicated time, careful evaluation, and constructive engagement, which have greatly strengthened our paper.

We are encouraged that all three reviewers see clear value in 3DVerifierBench and that our rebuttal addressed their core concerns. Below, we summarize the discussion outcomes, our primary contribution, and our revisions.


1. Summary of Reviewer Feedback, Addressed Concerns, and Discussion Outcomes

  • Reviewer qgso (Concerns Addressed; Raised Initial Rating):

    • Recognized Value: Highlighted our "interesting task" and that "even state-of-the-art MLLM still struggle with verifying whether a 3D model is consistent with these specified constraints."
    • Addressed & Rating: Confirmed our LoRA fine-tuning results (Sec. 5.2, Table 4) show the training set "indeed helps to improve performance" and our step-by-step pipeline (Sec. 3) makes the benchmark "more clearly described now." Reviewer qgso increased their initial rating and will discuss further with other reviewers.
  • Reviewer 7bgA (Concerns Addressed; Strengthened Positive Assessment):

    • Recognized Value: Highlighted that "the verification interface and linked counterfactuals provide a useful evaluation contribution," complementary metrics, and "meaningful controls and downstream relevance."
    • Addressed & Stance: Confirmed our documented perturbation settings, human-baseline and review protocols, selection rules (Appendices B.1, G.1, G.4), 300-case visual Judge validation (91.0% agreement, Table 3), and 48-case FineGeo audit (Appendix H.7) "address my main reproducibility concerns" and reference-matching questions, stating "the response strengthens my positive assessment" while maintaining their positive recommendation.
  • Reviewer P5eL (Main Concerns Addressed; Open to Increasing Initial Rating):

    • Recognized Value: Praised our "meaningful benchmark design with fine-grained verification," focus on requirement-specific correctness, "contradiction-aware explanation evaluation," and "comprehensive evaluation and substantial data construction efforts."
    • Addressed & Stance: Provided further comments only shortly before the discussion deadline, confirming our capability comparison (Sec. 2, Table 1), LoRA fine-tuning (Sec. 5.2, Table 4), visual Judge validation (Table 3), and 20-input automated GPT-6 Astra refinement (Sec. 5.5, Table 6, Appendix G.6) "address my previous concerns," and expressing openness to increasing their initial rating after discussing with other reviewers. Before the discussion closed, we promptly responded to their remaining question on the three uncovered tasks with quantitative and qualitative cross-task transfer analyses (Appendix D.5).

2. Summary of Main Contributions and Key Revisions in the Updated PDF

  • Primary Contribution & Scope: Our primary contribution is 3DVerifierBench and its empirical diagnosis of MLLM verification reliability across 2,581 examples and 10 tasks (revealing a 21.6-point human-model gap). The 14,126-example training set is a supporting resource validating our data pipeline, alongside downstream studies showing stronger verification improves 3D candidate selection and recursive refinement.
  • Experiments: Added LoRA fine-tuning on our 14,126-example training set (tuned Gemma-4-31B reaching 69.1% OA overall, within 0.3 points of GPT-6 Astra, and 76.1% OA on the 7 covered tasks), cross-task transfer analyses on the 3 uncovered tasks, and 20-input automated GPT-6 Astra recursive refinement.
  • Benchmark & Positioning: Added a step-by-step evaluation pipeline, standardized terms, clarified our primary benchmark contribution, and added a capability comparison against prior benchmarks (Abstract, Sec. 2-3, Table 1).
  • Protocols & Audits: Documented perturbation thresholds, review/retention rules, the human baseline, selection rules, the 300-case visual Judge validation (Table 3), and visual audits of FineGeo and Motion explanations (Appendices B.1, G.1, G.4, H.7).

Once again, we sincerely thank the Reviewers, ACs, SACs, and PCs for your time, insights, and service to the ICLR community. We believe the revised paper is substantially stronger and clearer, and we wish you all the best in the remainder of the review and decision process.

Yours sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.