- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates the reliability of multimodal large language models (MLLMs) in verifying whether generated 3D content satisfies specific factual requirements. The authors introduce 3DVerifierBench, a benchmark comprising 2,581 examples across ten 3D verification tasks, evaluating binary decisions, contradiction explanations, and counterfactual consistency. They further develop a scalable data construction pipeline that combines multi-view fact extraction, controlled 3D editing, and expert review, producing 14,126 training examples. Experiments reveal substantial gaps between existing MLLMs and human performance, particularly in fine-grained geometry, identity preservation, and motion verification. Finally, the paper explores the potential of reliable verification for improving 3D generation through candidate selection and human-guided recursive refinement.
Critical strengths
-
Meaningful benchmark design with fine-grained verification. The paper presents a valuable perspective on 3D generation evaluation by emphasizing requirement-specific factual correctness rather than overall visual quality. In particular, the specification-side and candidate-side counterfactual evaluations provide a useful mechanism for examining whether verification decisions consistently respond to changes in requirements and visual evidence. The inclusion of contradiction-aware explanation evaluation further strengthens the benchmark.
-
Comprehensive evaluation and substantial data construction efforts. The benchmark covers ten heterogeneous 3D tasks and evaluates a broad range of MLLMs, providing useful insights into their limitations in 3D verification. The data construction pipeline integrates multiple 3D resources, controlled modifications, and expert validation. The human comparison and downstream generation experiments also help demonstrate the potential importance of reliable 3D verification.
Critical weaknesses
-
Limited differentiation from existing 3D evaluation and verification research. My main concern is that several related studies have already investigated automated and fine-grained evaluation of generated 3D content, including Eval3D (CVPR 2025), Hi3DEval (NeurIPS 2025), and 3D-DefectBench (2026). In particular, 3D-DefectBench has explored fine-grained binary defect detection using VLMs, while OmniVerifier (ICLR 2026) has investigated visual verification and verification-guided generation refinement. Although the proposed benchmark introduces additional dimensions, particularly counterfactual consistency and contradiction-aware explanations, the paper could more clearly establish their independent contributions beyond existing evaluation frameworks. The comparisons with prior work are primarily conceptual, and further evidence demonstrating the distinct diagnostic value of the proposed benchmark would strengthen its novelty and significance.
-
Insufficient validation of the practical utility of the proposed training resource and verification framework. Although the authors construct a training resource containing 14,126 examples, the paper does not demonstrate whether training a verifier on these samples actually improves verification performance. Moreover, the training resource covers only seven of the ten benchmark tasks, excluding challenging categories such as identity preservation, fine-grained geometry, and motion understanding. The downstream generation experiments are also somewhat limited: candidate selection relies on human oracle labels, while recursive refinement uses human-provided verification feedback and is evaluated on only eight inputs. Although these results demonstrate the potential value of reliable verification, they do not establish the effectiveness of a fully automated verification-guided 3D generation system. Further empirical validation would strengthen the practical significance of the proposed framework.
Questions for authors
-
Compared with existing works such as Eval3D, Hi3DEval, and 3D-DefectBench, what additional verification capabilities does 3DVerifierBench uniquely measure? Could the authors provide more direct evidence that the proposed counterfactual and explanation-aware evaluations reveal limitations not adequately captured by existing 3D evaluation benchmarks?
-
Could the authors provide preliminary experiments demonstrating the effectiveness of the 14,126-example training resource for improving a 3D verifier? Even a relatively simple fine-tuning experiment on an existing MLLM would help establish the practical value of the constructed data.
-
The explanation-aware evaluation relies on a reference-semantic judge that compares generated explanations with reference explanations without accessing the underlying visual evidence. Could the authors further clarify the limitations of this evaluation protocol and provide additional evidence that the metric reliably reflects visually grounded verification?
-
Since the recursive refinement experiments rely on human verification feedback, could the authors clarify how these results are expected to translate to an automated verification-guided generation system? Additional experiments using an automatic verifier, even on a smaller subset, would help substantiate the claimed downstream benefits.
3 comments
Follow-up to AC and Reviewer P5eL
Dear Reviewer P5eL,
We thank the Area Chair for facilitating this discussion, and thank you again for your careful review and constructive feedback.
For your convenience, here is a brief summary of how we addressed your main concerns:
- Differentiation from existing work: Your main concern was whether 3DVerifierBench offers capabilities beyond existing 3D evaluation and defect-detection benchmarks. In response, we expanded the related work and added a capability comparison (Sec. 2, Table 1), clarifying that our benchmark focuses on verifying explicit requirements, checking contradiction-aware explanations, and testing consistency under controlled counterfactual changes, rather than simply evaluating asset quality or detecting predefined defects. We further demonstrated the diagnostic value of these capabilities: GPT-6 Astra achieves 69.4% OA but only 46.5% triplet success, revealing consistency failures that per-example accuracy alone misses. We also validated the explanation Judge against human assessments, achieving 91% agreement on 300 cases (Table 3).
- Practical utility of the training set and verification framework: Your concern was that the usefulness of our training set and the feasibility of automated verification-guided refinement had not been demonstrated. Following your suggestions, we fine-tuned two open-source models on our 14,126-example training set, with Gemma-4-31B improving by 8.3 OA points to 69.1%, within 0.3 points of GPT-6 Astra (Sec. 5.2, Table 4). We also extended recursive refinement to 20 inputs and used GPT-6 Astra as an automatic verifier, achieving 14/20 full factual success compared with 11/20 for Parallel Oracle@8 (Sec. 5.5, Table 6). These results provide direct evidence that our training data improves verifier performance and that automatic verification can already benefit iterative 3D refinement.
As the discussion period is scheduled to close on November 18, we wanted to gently follow up, as we have not yet received your feedback on these responses. We understand that you may have a busy reviewing schedule and would greatly appreciate hearing whether these new results and clarifications address your concerns or if any questions remain.
Thank you again for your time and consideration.
Best regards,
The Authors
Brief Final Check-in Before Discussion Closes (Nov 18)
Dear Reviewer P5eL,
Apologies for the brief follow-up to our note yesterday. As the discussion period is scheduled to close on November 18, we wanted to send one final, gentle check-in regarding our rebuttal and newly added experiments—including LoRA fine-tuning on our 14k training set, 20-input automated recursive refinement, the benchmark capability comparison, and the visual audit of the explanation Judge.
Since several of your questions overlap with points raised earlier in the discussion, we also wanted to briefly share how those threads concluded: Reviewer qgso raised their initial rating after noting that the new fine-tuning results confirm the practical value of our training set and that the benchmark is now clearly described, while Reviewer 7bgA reaffirmed their positive assessment after our protocol clarifications, visual explanation audits, and new fine-tuning and automatic-refinement experiments addressed their main concerns.
We completely understand how busy your reviewing schedule is, so please do not feel obligated to write a lengthy reply. Even a brief note on whether these additions address your concerns would be greatly appreciated before the window closes.
Thank you once again for your time, constructive feedback, and consideration!
Best regards, The Authors
Dear Reviewer P5eL,
Since the authors have procvided their feedback, please discuss with the authors. Thanks.
Best regards,
AC