- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the sensitivity of text-to-3D evaluation metrics to rendering and caption configurations. Using 300 fixed scenes generated by six methods across 50 prompts, the authors analyze 19 alignment evaluators under different evaluation settings. The paper examines score variability, ranking stability, and sensitivity to targeted scene degradations, highlighting how evaluation configurations can affect the reported performance of text-to-3D generators.
Critical strengths
-
The paper addresses an important issue concerning the reliability of text-to-3D evaluation. Understanding how evaluation configurations affect reported scores and rankings is relevant to the reproducibility of text-to-3D research.
-
The experiments cover a wide range of evaluation metrics and configuration factors, providing useful empirical observations about the sensitivity of existing evaluators.
Critical weaknesses
-
Limited novelty. The paper does not propose novel methods. Most of its analyses rely on existing evaluation and statistical techniques, and the unreliability of existing text-to-3D evaluation metrics is already well recognized in the literature. Consequently, demonstrating additional instances of evaluation instability, without providing substantially new insights or methodological advances, limits the novelty and significance of the contribution.
-
Limited experimental scale and significance of the findings. The paper evaluates only six generators using 50 prompts. Given the much larger number of existing text-to-3D generators, this scale is insufficient to support broad conclusions about the reliability of text-to-3D evaluation. Moreover, the observed ranking reversals may simply reflect the large number of comparisons, as none remains statistically significant after correction for multiple comparisons. Although this limitation is acknowledged in the paper, I do not find the observation that some rankings reverse among a large number of comparisons particularly interesting or compelling. This substantially weakens the significance of one of the paper's central findings.
Questions for authors
-
What are the main novel insights of this work beyond prior studies investigating evaluation sensitivity and ranking instability? Also, what interesting conclusions can be reliably drawn about ranking instability?
-
While numerous papers have proposed text-to-3D generation methods, how representative are the six selected generators and 50 prompts of the broader text-to-3D generation landscape? Would the main conclusions remain valid with a larger and more diverse set of generators and prompts?
2 comments
Response to Reviewer yW7d (2/2)
1. Reversals and the number of comparisons (W2)
the observed ranking reversals may simply reflect the large number of comparisons, as none remains statistically significant after correction
We agree, and that is why the paper labels every winner change as descriptive. The same reasoning applies to the orderings that papers report. A paper that reports one undeclared configuration gives the reader no way to tell whether it was fixed in advance or chosen after trying several. In the second case, its winner comes from the same family of comparisons that we search, and our band shows how little survives that correction: zero reversals, and one ordering out of 15 pairs × 19 evaluators. The card exists to make this search visible, by asking authors to declare the configuration set before they look at outcomes.
"Not confirmed" also does not mean "no effect". At 50 prompts, the simulated chance of confirming each of the four pre-registered contrasts is 0.35–0.57, and 75–100 prompts per contrast would reach 0.8 under the Holm plan (App. P, Table 25).
2. Six generators and 50 prompts (W2, Q2)
Would the main conclusions remain valid with a larger and more diverse set of generators and prompts?
For prompts, yes. We drew 50 new disjoint prompts with the same stratification, re-ran all six generators with every sweep, and scored all 300 new scenes with all 20 evaluators (App. R, Table 30).
| Measure (19 alignment evaluators) | Original 50 | New 50 |
|---|---|---|
| Evaluators changing winner, protocol factors | 18 | 19 |
| Evaluators changing winner, all swept configurations | 19 | 19 |
| Mean changed configurations, protocol factors | 60 | 48 |
| Mean changed configurations, all swept | 433 | 417 |
| Median changed configurations, all swept | 409 | 432 |
| Most common baseline winner | Scenethesis (16) | Scenethesis (17) |
One more evaluator changes its winner on the new prompts, while the average number of protocol settings with a changed winner falls from 60 to 48. The finding replicates; its size moves somewhat with the sample.
For generators, we did not add new ones in this period, and we do not claim the six represent the whole field. Two checks bear on it. Leaving out each generator in turn keeps peak ρ above 1 for all 19 evaluators, with the lowest value at 2.42 (Table 15). And the unit of analysis is the evaluator: the 19 we audit include the metrics that text-to-3D scene papers actually report, such as top-down CLIPScore (Holodeck) and BLIP-2, Long-CLIP and VQAScore (HSM). The six generators span six design families, from LLM procedural construction to reasoning-model planning, and all produce structured scenes, which the degradation probes need.
Thank you again. If these answers change your view of the contribution, we would appreciate it if you reflected that in your rating; if not, we would welcome a pointer to what remains unconvincing.
Response to Reviewer yW7d (1/2)
Thank you for the review. Both questions are fair, and the second, whether the reversals are just a product of making many comparisons, sits at the center of the paper. Some numbers in the submitted PDF changed after we repaired the HSM render set; item 1 of the first general comment lists them.
1. What is new beyond "evaluators are unreliable" (W1, Q1)
We agree, and the paper does not claim to discover it. It asks a narrower question: when does that unreliability change which generator wins? SceneCritic reports such reversals for one VLM judge; we measure them across 19 evaluators and eight factors on fixed scenes. Unreliable scores do not imply unreliable comparisons. If a setting shifts every generator by the same amount, the pairwise margins do not move (Eq. 1). The paper's contribution is to measure how much of the fragility reaches the decision, on fixed scenes, across 19 evaluators, with inference that accounts for the search over configurations. We are not proposing a new metric.
2. Conclusions about ranking instability that hold up (Q1)
These six hold at the inference level stated for each.
The first row is our main answer. The reliable conclusion is not that rankings reverse. It is that, at 50 prompts and with the configuration left open, almost no generator ordering in this field is established, while fixing and declaring the configuration in advance recovers about half of them at the pointwise level.