acceptodds
Withdrawn from ICLR 2027

Who is a Better Dreamer: Subjective and Objective Quality Assessment for Text-to-3D Generated Contents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Text-to-3D content generation, often referred as “Dreamer," has emerged as a rapid solution for 3D modeling, yet current systems still suffer from pronounced quality deficiencies that undermine user experience. To systematically evaluate the quality of 3D generated contents (3DGCs) and to inform the future development of Dreamers, we present a comprehensive subjective and objective assessment. We benchmark 11 representative Dreamers using 484 prompts and collect 4,691 3DGCs, forming 3DGCQA-LS, the largest 3DGC quality assessment dataset to date. Through controlled multi-subject evaluation, we uncover substantial performance discrepancies across existing Dreamers and identify nine recurring distortion types. To further enable automated quality prediction, we introduce Hyper-VG, a novel 3DGC quality assessment framework that integrates the Huanyuan world model, persistent homotopy theory, and cube projection. Extensive experiments demonstrate that Hyper-VG achieves state-of-the-art (SOTA) performance across multiple benchmark datasets, providing a strong foundation for advancing reliable Dreamers. This work will be made open source to advance the field.

Primary area:
datasets and benchmarks
Keywords:
Quality assessment database, 3D AIGC, 3D quality assessment, world model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EDCKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the problem of subjective and objective quality assessment for Text-to-3D generated content (3DGCs), where generative models are referred to as "Dreamers". The authors curate 484 prompts across 10 semantic categories and generate 4,691 valid 3D meshes using 11 Text-to-3D models, including both open-source and closed-source models. To enable and accelerate automated 3D generation assessment, the authors propose Hyper-VG, a quality prediction model that leverages prior information from the Huanyuan world model and persistent homology to estimate different quality measurements.

Critical strengths
  • The authors construct a large-scale dataset of generated 3D content from both open-source and closed-source models, which is labeled by humans rather than non-3D knowledge injection Large Language Model, compared to previous research.
  • The quality assessment is detailed with 9 quality issues, which provides a comprehensive analysis of the factors that influence 3D generation content quality.
  • The writing is easy to follow, even though there are some issues with table references.
Critical weaknesses

Technical Approach:

  • I think the current dataset, collected and labeled in this research, has little impact on future development of 3D generation since the assets are generated by current and some outdated methods. As a result, to evaluate the quality of newer methods, we have to sample more generated content and rerun the whole protocol.
  • The use of persistent homology is not a strong feature since all the open-source tools used in the research leverage Marching Cubes [1] to extract an iso-level surface from the volume, which is not topology-aware, while closed-source methods like Tripo might have their own retopology module, which is a precondition for UV-texture unwarping; therefore, there might be a mismatch in quality assessment.

Experiments:

  • The ablation study in Table 3 is confusing. The authors should list feature combinations as checkboxes.
  • The authors need to include newer Text-to-3D methods [2][3][4][5] as well.

Writing:

  • The term "homotopy" used in the paper should be "homology". The term "homotopy" is used vastly in the paper, causing critical confusion for the readers.
  • There is a minor table link reference on line 130.

[1] Marching cubes: A high-resolution 3D surface construction algorithm

[2] ScaleDreamer: Scalable Text-to-3D Synthesis with Asynchronous Score Distillation

[3] JointDreamer: Ensuring Geometry Consistency and Text Congruence in Text-to-3D Generation via Joint Score Distillation

[4] Structured 3D Latents for Scalable and Versatile 3D Generation

[5] Consistent Flow Distillation for Text-to-3D Generation

[6] Text-to-3D Generation using Jensen-Shannon Score Distillation

Questions for authors
  • Can the author provide the protocol for how the participants evaluate or give a score for topology, triangle quality assessment? Does the author show the wireframe of the mesh or directly use some triangle quality metrics?
  • Can the author explain why persistent homology can benefit the assessment of the quality of 3D generated assets?
Reviewer_yyDyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Hyper-VG, a 3D metric for text-to-3D generated content. The authors first construct 3DGCQA-LS, a dataset containing 4,691 3D assets generated by 11 text-to-3D methods from 484 prompts, with human quality ratings and distortion annotations. Based on this dataset, the authors propose Hyper-VG, which combines features from the Hunyuan Voyager world model, persistent homology, geometric statistics, and cube projections to predict the overall perceptual quality of 3D assets. Experiments demonstrate improved correlation with human judgments compared with existing baselines.

Critical strengths
  1. Valuable dataset contribution. The construction of a large-scale 3D quality assessment dataset with human ratings and distortion annotations is valuable to the community and can facilitate future research on 3D generation evaluation.

  2. Strong quantitative performance. Hyper-VG achieves state-of-the-art performance on the evaluated benchmarks, demonstrating the effectiveness of proposed method.

Critical weaknesses
  1. Overstated dataset contribution. The paper repeatedly claims that 3DGCQA-LS is the largest 3D generated content quality assessment dataset. However, Hi3DEval (NeurIPS 2025) introduced Hi3DBench, containing approximately 15.3K generated 3D assets from 30 generative methods.

  2. Missing important related work and evaluation baselines. Several highly relevant 3D generation evaluation methods are neither adequately discussed nor compared, including GPTEval3D, T3Bench, Hi3DEval, and DreamReward. Comparisons with compatible methods, or at least a discussion of their differences, are necessary.

  3. Lack of multidimensional quality prediction. Hyper-VG predicts only a single overall quality score. However, 3D generation quality involves multiple distinct dimensions, such as geometric quality, texture fidelity, and text alignment. A single scalar score offers limited diagnostic value for identifying specific generation failures and guiding model improvements.

  4. Outdated selection of 3D generators. Most open-source Dreamers included in the dataset are relatively old methods, such as DreamFusion, Magic3D, and LatentNeRF. Recent advances in 3D generation are insufficiently represented. For example: 3DTopia-XL, TRELLIS and Hunyuan3D-2.1. The identified distortion patterns may not adequately reflect the current capabilities and failure modes of modern 3D generators.

  5. Insufficient evaluation of generalization to unseen generators. The experiments adopt five-fold cross-validation on the existing datasets. However, this does not necessarily demonstrate generalization to outputs from previously unseen 3D generation models.

Minor:

  • Line 130, the reference to "Table ??" is unresolved and should be corrected.

Hi3DEval: Advancing 3D Generation Evaluation with Hierarchical Validity

GPT-4V(ision) is a Human-Aligned Evaluator for Text-to-3D Generation

T3Bench: Benchmarking Current Progress in Text-to-3D Generation

DreamReward: Text-to-3D Generation with Human Preference

Questions for authors
  1. Multi-view feature aggregation. In Section 4.1, features from six cube projections are simply averaged. Would this discard important spatial relationships and cross-view inconsistencies, such as inconsistent geometry between the front and back views?

  2. Generalization. How well does Hyper-VG perform on assets generated by a completely unseen Dreamer?

Reviewer_RUWTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces 3DGCQA-LS, a benchmark containing 4,691 textured meshes generated by 11 text-to-3D methods from 484 prompts, with human quality scores and annotations of nine distortion types. It also proposes Hyper-VG, which combines multi-view visual features, pretrained multimodal representations, geometric statistics, and persistent-homology descriptors to predict mean opinion scores. Experiments report improvements over existing quality assessment methods, with ablations supporting the contribution of each feature group.

Critical strengths
  1. Substantial annotation effort. The dataset provides a sizable collection of human-rated 3D assets with explicit distortion annotations, making it potentially useful for studying perceptual quality and common generation failures.

  2. Complementary quality cues. Combining visual appearance with geometric and topological descriptors is reasonably motivated. The ablations show that these additional features improve prediction beyond the complete model without each corresponding branch.

  3. Broad evaluation against quality assessment methods. The comparisons include relevant approaches such as HyperScore, T23DAQA, and CAP. Hyper-VG achieves higher reported correlations with human judgments on both evaluated datasets.

Critical weaknesses
  1. Generalization to unseen generators remains unverified. The dataset includes many earlier optimization-based methods, such as DreamFusion, SJC, LatentNeRF, and HiFA. However, the reported five-fold cross-validation results do not provide sufficient evidence of generalization to unseen generators, particularly commercial systems such as Meshy and Tripo. Combining two datasets before cross-validation also does not constitute cross-dataset generalization. This is a central limitation for an evaluator intended to assess future generation methods: high aggregate correlation may partly reflect differences in average quality between familiar generators, without demonstrating reliable ranking within a generator or on unseen generation pipelines. Leave-one-generator-out evaluation and testing on a modern unseen generator, such as TRELLIS, would be particularly informative.

  2. A directly relevant human-preference baseline is missing. The paper neither discusses nor compares against Reward3D [1]. Reward3D learns human preferences over text-conditioned 3D content and is therefore directly relevant to the proposed evaluator. Although preference ranking differs from MOS regression, this distinction does not remove the need for comparison. Evaluating correlation with human scores and within-prompt preference accuracy would help establish whether Hyper-VG offers advantages over an existing 3D reward model. Any comparison should clearly distinguish pretrained evaluation from adaptation using the proposed dataset.

  3. The practical benefit beyond offline score prediction is unclear. The experiments do not establish whether improved MOS correlation translates into better sample selection or generation. For example, ITS3D [2] supports Reward3D as a verifier for inference-time search, providing a concrete setting in which to test Hyper-VG. Replacing the verifier while holding the generator and search procedure fixed would reveal whether the proposed metric guides generation toward outputs that humans actually prefer. Such an experiment should use independent evaluation and report the full computational cost, including rendering and geometric/topological preprocessing. A higher optimized Hyper-VG score alone would not demonstrate improved generation quality.

[1] Ye et al., Dreamreward: Text-to-3d generation with human preference. 2025. [2] Zhou et al., ITS3D: Inference-Time Scaling for Text-Guided 3D Diffusion Models. 2025.

Questions for authors

I appreciate the dataset construction effort and the consistent improvements reported in the quality prediction experiments. However, the current evidence does not establish that Hyper-VG reliably evaluates unseen generators. This issue limit confidence in the evaluator’s general applicability. A controlled downstream experiment would further strengthen its practical significance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.