acceptodds
Under review as a conference paper at ICLR 2027

Frozen Scenes, Shifting Winners: Configuration Fragility in Text-to-3D Evaluation

Abstract

Can a text-to-3D leaderboard change when every generated scene stays fixed? We audit this question for rendered-image evaluation, where camera settings and caption wording become part of the measurement protocol. Across 300 frozen scenes from six generators, we vary eight render and caption factors for 19 alignment evaluators plus one perceptual-quality control, then test four targeted scene degradations. Peak configuration variance exceeds between-generator variance for 17/19 alignment evaluators, with prompt-bootstrap lower bounds above 1 for 11/19. Rankings are more stable than scores, yet 18/19 evaluators change their point-estimate winner under some configuration. Pairwise protocol margin envelopes show which comparisons keep their direction across the tested settings. Selected pairs have opposite pointwise intervals, but no reversal survives simultaneous inference over the full search. Thus the observed winner changes are descriptive, not confirmed changes in generator superiority. Sensitivity remains separate: no evaluator, even the prompt-free control, exceeds 67% tie-adjusted directional discrimination on layout scrambling, which is diagnostic rather than human-validated ground truth. The audit separates score stability, decision uncertainty, and targeted sensitivity, and recommends reporting (generator, score, card ID) with protocol-dependent comparisons and selection-aware uncertainty.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
evaluation reliability, benchmark auditing, vision-language models, 3D scene generation, configuration sensitivity, reproducibility, uncertainty quantification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yW7dOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the sensitivity of text-to-3D evaluation metrics to rendering and caption configurations. Using 300 fixed scenes generated by six methods across 50 prompts, the authors analyze 19 alignment evaluators under different evaluation settings. The paper examines score variability, ranking stability, and sensitivity to targeted scene degradations, highlighting how evaluation configurations can affect the reported performance of text-to-3D generators.

Critical strengths
  1. The paper addresses an important issue concerning the reliability of text-to-3D evaluation. Understanding how evaluation configurations affect reported scores and rankings is relevant to the reproducibility of text-to-3D research.

  2. The experiments cover a wide range of evaluation metrics and configuration factors, providing useful empirical observations about the sensitivity of existing evaluators.

Critical weaknesses
  1. Limited novelty. The paper does not propose novel methods. Most of its analyses rely on existing evaluation and statistical techniques, and the unreliability of existing text-to-3D evaluation metrics is already well recognized in the literature. Consequently, demonstrating additional instances of evaluation instability, without providing substantially new insights or methodological advances, limits the novelty and significance of the contribution.

  2. Limited experimental scale and significance of the findings. The paper evaluates only six generators using 50 prompts. Given the much larger number of existing text-to-3D generators, this scale is insufficient to support broad conclusions about the reliability of text-to-3D evaluation. Moreover, the observed ranking reversals may simply reflect the large number of comparisons, as none remains statistically significant after correction for multiple comparisons. Although this limitation is acknowledged in the paper, I do not find the observation that some rankings reverse among a large number of comparisons particularly interesting or compelling. This substantially weakens the significance of one of the paper's central findings.

Questions for authors
  1. What are the main novel insights of this work beyond prior studies investigating evaluation sensitivity and ranking instability? Also, what interesting conclusions can be reliably drawn about ranking instability?

  2. While numerous papers have proposed text-to-3D generation methods, how representative are the six selected generators and 50 prompts of the broader text-to-3D generation landscape? Would the main conclusions remain valid with a larger and more diverse set of generators and prompts?

2 comments
AuthorsOpenReview ↗

Response to Reviewer yW7d (1/2)

Thank you for the review. Both questions are fair, and the second, whether the reversals are just a product of making many comparisons, sits at the center of the paper. Some numbers in the submitted PDF changed after we repaired the HSM render set; item 1 of the first general comment lists them.

1. What is new beyond "evaluators are unreliable" (W1, Q1)

the unreliability of existing text-to-3D evaluation metrics is already well recognized in the literature

We agree, and the paper does not claim to discover it. It asks a narrower question: when does that unreliability change which generator wins? SceneCritic reports such reversals for one VLM judge; we measure them across 19 evaluators and eight factors on fixed scenes. Unreliable scores do not imply unreliable comparisons. If a setting shifts every generator by the same amount, the pairwise margins do not move (Eq. 1). The paper's contribution is to measure how much of the fragility reaches the decision, on fixed scenes, across 19 evaluators, with inference that accounts for the search over configurations. We are not proposing a new metric.

2. Conclusions about ranking instability that hold up (Q1)

what interesting conclusions can be reliably drawn about ranking instability?

These six hold at the inference level stated for each.

ConclusionEvidence
Once the configuration is treated as a choice, almost no ordering in this field is resolvedUnder the full-search band, exactly one strict ordering survives (Scenethesis over Holodeck, for one evaluator); with one configuration fixed in advance, a median of 8 of 15 pairs are pointwise separable
Score fragility is large, but most of it cancels before the decisionPeak ρ > 1 for 19 of 19 evaluators (minimum 3.48); the generator-by-level component that can flip a pair is at most 0.98% of variance (median) for every factor except pitch (5.33%), and flips happen because the leading generators are close
The bias has a direction, so it can be exploitedPicking the best tested view raises median scores by 1.03–1.18× for CLIP encoders and 1.29× for VQAScore; the best caption beats the default by a median 17% (up to 39%)
Averaging views is not a fixPre-registered: the 15-view mean still flips 5 of 15 and 2 of 15 pair signs
Camera-driven margins are partly metric artifactBlinded human study: ties on 14 of 25 camera pairs whose median metric margin (1.73) exceeds the degradation median (1.38)
Some settings are safe to ignore90.5% of resolution shifts are practically negligible under equivalence testing, against 37.9% of pitch shifts

The first row is our main answer. The reliable conclusion is not that rankings reverse. It is that, at 50 prompts and with the configuration left open, almost no generator ordering in this field is established, while fixing and declaring the configuration in advance recovers about half of them at the pointwise level.

AuthorsOpenReview ↗

Response to Reviewer yW7d (2/2)

1. Reversals and the number of comparisons (W2)

the observed ranking reversals may simply reflect the large number of comparisons, as none remains statistically significant after correction

We agree, and that is why the paper labels every winner change as descriptive. The same reasoning applies to the orderings that papers report. A paper that reports one undeclared configuration gives the reader no way to tell whether it was fixed in advance or chosen after trying several. In the second case, its winner comes from the same family of comparisons that we search, and our band shows how little survives that correction: zero reversals, and one ordering out of 15 pairs × 19 evaluators. The card exists to make this search visible, by asking authors to declare the configuration set before they look at outcomes.

"Not confirmed" also does not mean "no effect". At 50 prompts, the simulated chance of confirming each of the four pre-registered contrasts is 0.35–0.57, and 75–100 prompts per contrast would reach 0.8 under the Holm plan (App. P, Table 25).

2. Six generators and 50 prompts (W2, Q2)

Would the main conclusions remain valid with a larger and more diverse set of generators and prompts?

For prompts, yes. We drew 50 new disjoint prompts with the same stratification, re-ran all six generators with every sweep, and scored all 300 new scenes with all 20 evaluators (App. R, Table 30).

Measure (19 alignment evaluators)Original 50New 50
Evaluators changing winner, protocol factors1819
Evaluators changing winner, all swept configurations1919
Mean changed configurations, protocol factors6048
Mean changed configurations, all swept433417
Median changed configurations, all swept409432
Most common baseline winnerScenethesis (16)Scenethesis (17)

One more evaluator changes its winner on the new prompts, while the average number of protocol settings with a changed winner falls from 60 to 48. The finding replicates; its size moves somewhat with the sample.

For generators, we did not add new ones in this period, and we do not claim the six represent the whole field. Two checks bear on it. Leaving out each generator in turn keeps peak ρ above 1 for all 19 evaluators, with the lowest value at 2.42 (Table 15). And the unit of analysis is the evaluator: the 19 we audit include the metrics that text-to-3D scene papers actually report, such as top-down CLIPScore (Holodeck) and BLIP-2, Long-CLIP and VQAScore (HSM). The six generators span six design families, from LLM procedural construction to reasoning-model planning, and all produce structured scenes, which the degradation probes need.

Thank you again. If these answers change your view of the contribution, we would appreciate it if you reflected that in your rating; if not, we would welcome a pointer to what remains unconvincing.

Reviewer_cFNjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the sensitivity of text-to-3D evaluation to rendering and caption configurations. The authors conduct a systematic audit involving 300 fixed scenes from six generation methods, examining 19 text-image alignment evaluators and one perceptual-quality control under different rendering and linguistic settings. The results reveal substantial score variations and changes in point-estimate rankings across evaluation configurations. The paper further investigates evaluator sensitivity through controlled scene degradations and proposes an Evaluator Configuration Card to improve the transparency and reproducibility of evaluation protocols.

Critical strengths
  1. Practical relevance. The paper addresses an important issue in text-to-3D evaluation, highlighting how seemingly minor choices in rendering and caption formulation can influence reported performance.

  2. Broad experimental coverage. The evaluation includes multiple scene generators, diverse alignment metrics, and several configuration factors, providing useful empirical observations about evaluation stability.

  3. Careful statistical analysis. The distinction between score variation, ranking stability, and statistically confirmed ranking reversals is valuable. The use of bootstrap uncertainty analysis also improves the reliability of the reported observations.

Critical weaknesses
  1. The methodological novelty is relatively limited. The central observation that vision-language evaluators are sensitive to viewpoints and prompt formulations has been explored in previous studies, including SceneCritic and 3D-DefectBench. Although this paper provides a broader empirical analysis, the authors should better clarify the conceptual advances beyond existing evaluation audits.

  2. The evidence for meaningful ranking instability is not sufficiently strong. While 18 of 19 evaluators exhibit changes in their point-estimate winners, none of the examined reversals remains statistically confirmed after full-search correction. Moreover, the overall rankings show relatively high concordance, making the practical significance of the observed winner changes less clear.

  3. The absence of human evaluation limits the validity of the conclusions. Different camera configurations can genuinely change the visibility of scene content, while caption modifications may alter the semantic requirements. Without human judgments or independently verified scene-quality references, it remains difficult to distinguish evaluator unreliability from legitimate changes in the available visual evidence.

  4. The experimental design limits generalizability. The study uses only 50 prompts, six generators, and a single rendering pipeline. Furthermore, the one-factor-at-a-time design does not capture interactions among evaluation settings, such as camera viewpoint, lighting, and caption formulation. A broader evaluation would better support the general conclusions.

  5. The reliability and reproducibility of the evaluation require improvement. The paraphrase audit indicates that approximately half of the generated rewrites may not preserve the original semantics, introducing uncertainty into the corresponding sensitivity analysis. Additionally, the manuscript acknowledges that the complete raw evaluation records and verified manifest are not yet available in the provided repository checkout, limiting independent reproduction of the reported findings.

Questions for authors
  1. Can the authors provide human-validated comparisons to distinguish genuine evaluation failures from changes in scene visibility or caption semantics?

  2. Would the main conclusions remain consistent under joint configuration changes, additional scene generators, and an independent set of evaluation prompts?

  3. Could the authors further clarify the practical advantages of the proposed configuration reporting framework over existing evaluation documentation approaches?

3 comments
AuthorsOpenReview ↗

Response to Reviewer cFNj (1/3)

Thank you for the detailed review. Several of your points led to new experiments (Apps. M–S of the revised PDF), and where you asked for one, the full table is in this thread. Some numbers in the submitted PDF changed after we repaired the HSM render set; item 1 of the first general comment lists them.

1. Novelty relative to SceneCritic and 3D-DefectBench (W1)

The central observation that vision-language evaluators are sensitive to viewpoints and prompt formulations has been explored in previous studies ...

We agree that the sensitivity itself is not new, and the paper builds directly on both works (Secs. 1–2). The studies ask different questions (App. B, Table 5):

SceneCritic3D-DefectBenchThis paper
TaskJudging generated indoor scenesDetecting fine-grained 3D defectsText–scene alignment scoring
EvaluatorsOne VLM judge (Gemini-2.5-Pro)VLM pipelines, 84 inference designs19 alignment evaluators + 1 control
What variesViewpoint, repeated callsJudge, camera, visual input, prompt schema (crossed)6 render factors, 337 caption templates, paraphrases
Outcome studiedView-dependent method-ranking reversalsDetection vs. human defect labelsEvery pairwise generator margin and the winner
OutputA human-validated symbolic criticReleased data and metadataA configuration card, applied to two published protocols

3D-DefectBench is stronger than our study on human grounding and on crossing factors. What we add is a way to tell when evaluator sensitivity changes a generator comparison. We measure it against the size of the generator differences (ρ). We trace it to pairwise margins, where shifts shared by all generators cancel (Eq. 1). And we treat the choice of configuration as a search, with a simultaneous band over all 110,295 searched contrasts. The last step is what lets us say that none of the observed reversals, and only one ordering, holds up.

2. Is the instability practically meaningful? (W2)

none of the examined reversals remains statistically confirmed ... the overall rankings show relatively high concordance

We agree that no reversal is confirmed, and the paper claims none. The practical point is different. A published comparison typically fixes one configuration, and the two we carded in App. Q report no uncertainty at all. At the pointwise standard, 9 of 19 evaluators give significant results in opposite directions at two levels of one factor (8 under a second bootstrap seed). Two examples from Table 4:

Evaluator, factorPairLevelGap [pointwise 95%]Simultaneous 95%
CLIP ViT-B/32, lightingScenethesis − I-Designcity+0.91 [+0.12, +1.72][−0.87, +2.68]
night−0.91 [−1.62, −0.20][−2.54, +0.72]
MetaCLIP ViT-L/14, pitchScenethesis − LayoutVLM0°+2.31 [+1.03, +3.67][−0.61, +5.24]
75°−1.57 [−2.88, −0.22][−4.52, +1.39]

Two papers that each picked one of these settings would report opposite, pointwise-significant orderings. On concordance, the revised tables give a median minimum Kendall's W of 0.58 across the 20 evaluators, usually reached under pitch. A high W can still coexist with a changed winner when the leading methods are close.

AuthorsOpenReview ↗

Response to Reviewer cFNj (2/3)

1. Human evaluation (W3, Q1)

Can the authors provide human-validated comparisons to distinguish genuine evaluation failures from changes in scene visibility or caption semantics?

A blinded pairwise study built for this question is complete. Its analysis was fixed before any data were collected.

ArmTrialsRaters compareWhat it decides
Layout25Intact vs. scrambled scene, same cameraWhether people see the layout change, on which evaluators score only 55–71%
Camera25One frozen scene at two camera settings (pitch or yaw)Whether the view changes how well the render matches the prompt

Agreement is reported as Krippendorff's α. Under the pinned rule, if raters judge two views as equally faithful to the prompt, any evaluator margin between them counts as an artifact. A shift counts as evidence-driven only when the raters' preference between views reaches the threshold set by the layout arm. The 20 participants who finished every trial detect the layout change at 82.3% [77.6, 86.7] (α 0.34) but tie on 14 of the 25 camera pairs (α 0.25), and the pinned evaluator agrees with the human majority at chance level (52–56%). The pinned rule therefore returns its artifact verdict: the median evaluator margin on the human-tie camera pairs (1.73) exceeds the median on the layout pairs humans detect (1.38), so part of the configuration-driven score movement is metric artifact rather than visible evidence. Full numbers are in App. S. On caption semantics, the paraphrase re-analysis in item 1 of the next comment is the closest additional evidence.

2. Joint changes, more generators, independent prompts (W4, Q2)

Would the main conclusions remain consistent under joint configuration changes, additional scene generators, and an independent set of evaluation prompts?

2.1 Independent prompts

Yes. We drew 50 new disjoint prompts with the same stratification, re-ran all six generators with every sweep, and scored them with all 20 evaluators (App. R, Table 30).

Measure (19 alignment evaluators)Original 50New 50
Evaluators changing winner, protocol factors1819
Evaluators changing winner, all swept configurations1919
Mean changed configurations, protocol factors6048
Mean changed configurations, all swept433417
Median changed configurations, all swept409432
Most common baseline winnerScenethesis (16)Scenethesis (17)

The first row matches the abstract's definition. Three evaluators pick a different baseline winner on the new prompts, which is itself prompt dependence of the kind we audit.

2.2 Joint changes

We crossed the largest text factor with the largest render factor, caption and pitch, in a pre-registered 2×2 factorial on all 300 scenes (App. M, Table 19).

EvaluatorCaption effectPitch effectCaption × pitch
SigLIP (pinned)3.938 [3.433, 4.444]0.142 [−0.157, 0.436]−0.523 [−0.670, −0.383]
CLIP ViT-L/142.441 [2.031, 2.858]0.251 [−0.147, 0.665]−0.174 [−0.315, −0.026]
GPT-4o judge (0–100)3.350 [2.283, 4.500]−0.683 [−3.700, 2.084]−0.033 [−3.067, 2.783]

The interaction is about 13% of the caption effect for the pinned evaluator and its interval spans zero for the judge, so direction holds and size is evaluator dependent. Lighting × caption was not crossed and remains unmeasured.

2.3 More generators

We did not add generators in this period. Leaving out each generator in turn keeps peak ρ above 1 for all 19 evaluators, with the lowest value at 2.42 (Table 15), but this does not cover a wholly different field. The six systems span six design families and include the two whose published protocols we audit in App. Q.

AuthorsOpenReview ↗

Response to Reviewer cFNj (3/3)

1. Paraphrase audit (W5)

approximately half of the generated rewrites may not preserve the original semantics

That is correct, which is why paraphrases are kept out of the 18 of 19, the rank-stability analysis, and the simultaneous band. To check whether meaning drift drives paraphrase dispersion, we re-computed it on the 180 rewrites both auditors accept and the 196 both reject (App. O).

EvaluatorAll rewritesJointly acceptedJointly rejected
GPT-4o judge0.6300.426 [0.319, 0.562]0.388
CLIP ViT-L/140.180.0920.148
CLIP ViT-B/320.130.0750.099

The judge moves more on meaning-preserving rewrites, so drift does not explain its instability. CLIP moves more on meaning-changing rewrites, which is the behavior one would want. The labels are machine verdicts, so these are sensitivity cuts rather than cleaned data.

2. Raw records and reproduction (W5)

The gap you point out is closed in the revision. The updated supplementary archive contains the complete frozen raw records for all 20 evaluators on both prompt sets, the evaluation harness with its vendored LAVIS dependency, the verified intake manifest, and the follow-up records. All 20 SHA-256 hashes match the committed manifest. Only the original renders and the full hosted request logs are left out because of size, and they will be released at camera-ready.

3. What the card adds over existing documentation (Q3)

Could the authors further clarify the practical advantages of the proposed configuration reporting framework over existing evaluation documentation approaches?

The card is narrower than Evaluation Cards, Eval Factsheets, and BenchmarkCards, and is meant to sit under them (App. C). Those describe an evaluation as a whole. Our card records the fields that decide a rendered-image 3D score: render protocol, caption protocol, view policy, evaluator snapshot, the comparison family behind any uncertainty claim, and how ranks behave across the declared configurations. A validator gives each card a deterministic ID and a readiness verdict, so a score can be reported as (generator, score, card ID).

In practice, App. Q fills cards for two published protocols from their text alone. Holodeck (CVPR 2024) fully discloses 15 of 43 fields and HSM (3DV 2026) 11, and neither reports any uncertainty or a second configuration. When we re-ran Holodeck's protocol on our scenes, neighboring settings flipped 11 of 15 pair margins, and 6 of the 10 pairs that do not involve Reason-3D. The paper alone gives a reader no way to see this. A filled card shows which settings were fixed and whether any neighboring setting was tested.

Thank you again for the careful review. If these answers resolve your concerns, we would be grateful if you took them into account in your final assessment.

Reviewer_6KWHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether text-to-3D leaderboard conclusions depend on evaluator configuration. It freezes 300 scenes from six generators and sweeps eight render and caption factors across 19 alignment evaluators, plus four targeted scene degradations. Peak configuration variance exceeds between-generator variance for 17 of 19 evaluators, and point-estimate winners change for 18 of 19. However, no reversal survives simultaneous inference. The proposed remedy is an Evaluator Configuration Card for reporting.

Critical strengths
  1. Freezing the generated scenes isolates the measurement pipeline as the object of study. The design also separates reliability from sensitivity.

  2. The paper keeps score drift, ranking change, and decision change apart. It also reports openly that its strongest test, a simultaneous band over 110,295 contrasts, leaves no winner reversal standing.

  3. The paper shows which knobs matter most. The caption template dominates for 18 of 19 evaluators, pitch and lighting dominate among render factors, and best-view selection inflates scores. Practitioners can act on these findings directly.

Critical weaknesses
  1. Pitch and lighting change what is actually visible, so part of the variance is a legitimate response to new evidence. There is no human study to separate that part from metric artifact. As a result, the paper cannot say which configurations to trust.

  2. No reversal survives simultaneous inference, and strict orderings hold for only 4–11 of 15 generator pairs. The design is also one-factor-at-a-time, with 50 prompts, a single Blender pipeline, and only indoor scenes. This supports a descriptive fragility result, not a claim that any published comparison is wrong. It also leaves factor interactions unmeasured.

  3. The configuration card is a reporting schema that is never tested on a published comparison, and no robust protocol is validated. Some sub-audits are also noisy: paraphrase auditors accept only about half of the rewrites, and 6,758 of 120,270 GPT-4o scores are zeros that are bounded rather than resolved.

Questions for authors
  1. Could a small blinded human study separate evidence-driven score shifts from metric artifacts?

  2. Could you tone the title claim down to match the evidence, and add a case study where the configuration card changes a published conclusion?

  3. How large are interactions such as pitch × template under a fractional factorial design? Will the raw records be released so that the tables can be regenerated?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 6KWH (1/3)

Thank you for the careful review. We agree with the scope you set out in W2: the evidence supports a descriptive fragility result, not a claim that any published comparison is wrong. Some numbers you quote changed after we repaired the HSM render set, and item 1 of the first general comment lists every change. The first two comments and item 1 of the third comment answer your three questions. Items 2–5 of the third comment cover the remaining weaknesses.

1. Separating evidence-driven shifts from metric artifacts (Q1, W1)

Could a small blinded human study separate evidence-driven score shifts from metric artifacts?

Yes. We designed this study, fixed its analysis before collecting any data, and it is now complete: 20 participants finished every trial, consented, and passed an attention check placed between the two blocks (1,000 judgments).

ArmTrialsRaters compareWhat it decides
Layout25Intact vs. scrambled scene, same cameraWhether people see the layout change, on which evaluators score only 55–71%
Camera25One frozen scene at two camera settings (pitch or yaw)Whether the view changes how well the render matches the prompt

Agreement is reported as Krippendorff's α. The reading rule is fixed in the pin file in the supplement: if raters judge two views as equally faithful to the prompt, any evaluator margin between those views counts as an artifact. An evaluator shift counts as evidence-driven only when the raters' preference between the views reaches the threshold set by the layout arm.

The results are in. Participants detect the layout change at 82.3% [77.6, 86.7] (Krippendorff's α 0.34), so the degradation arm has a clear signal. On camera trials they mark a tie on 41.6% of judgments and answer "uncertain" on another 10.2%, and 14 of the 25 camera pairs show no significant majority preference (α 0.25). The pinned evaluator (App. S) prefers the intact scene on only 16 of the 25 layout pairs and agrees with the human majority at 52% on layout and 56% on camera pairs, which is chance level. Under the pinned rule the verdict is artifact: the median evaluator margin on the 14 human-tie camera pairs (1.73 score points) exceeds the median margin on the layout pairs that humans detect (1.38). The evaluator responds to view changes that people judge equivalent at least as strongly as to scene changes that they see plainly.

Two further points frame this result. First, not every factor needs a human check: equivalence testing classifies 90.5% of resolution shifts as practically negligible, against 37.9% of pitch shifts (Sec. 5). Second, even if a pitch effect turns out to be legitimate evidence, a comparison made at an undeclared pitch still cannot be reproduced, because its winner depends on a choice the reader cannot see. The human study tells us which views people actually prefer. It does not remove the need to declare the view, which is what the card asks for.

2. A case study on published work (Q2, W3)

... add a case study where the configuration card changes a published conclusion?

We added a case study, but it does not overturn a published conclusion, and we would rather say so up front. We first filled observed cards for two published evaluations, using only their text (App. Q, Table 27).

CardDisclosedPartialUndisclosed (of 43 fields)
Holodeck (CVPR 2024)15523
HSM (3DV 2026)11626

Neither paper reports any score uncertainty, tests a second configuration, or releases its scored renders, so both cards fail the readiness gate. The card does what it is meant to do here: it turns under-reporting into a count. The second half of this case study, a re-run of Holodeck's protocol, continues in the next comment.

AuthorsOpenReview ↗

Response to Reviewer 6KWH (2/3)

1. (continued) Re-running the Holodeck protocol

We re-ran Holodeck's evaluation protocol on our 300 frozen scenes: OpenCLIP ViT-L/14 trained on LAION-2B, a top-down view, and its template "a top-down view of {prompt}" (Table 28).

ScenethesisHSMI-DesignLayoutVLMHolodeckReason-3D
Anchor mean CLIPScore28.4427.3225.8425.6025.0922.31

We then moved only to neighboring settings that already exist in our sweep, namely the eight realistic caption forms and the seven shared pitch levels. This flips the sign of 11 of the 15 pair margins (Table 29). Five of the 11 involve Reason-3D, whose renderer uses an inverted pitch convention, so here are the other ten pairs on their own.

PairAnchor marginFlips sign under
Scenethesis − I-Design+2.59pitch 75°
Scenethesis − LayoutVLM+2.84pitch 75° and 90°
Holodeck − HSM−2.22pitch 75° and 90°
HSM − I-Design+1.48pitch 60°–90°
HSM − LayoutVLM+1.72pitch 60°–90°
I-Design − LayoutVLM+0.25caption change; pitch 75° and 90°
Scenethesis − Holodeck, Scenethesis − HSM, Holodeck − I-Design, Holodeck − LayoutVLMno flip

Six of these ten pairs flip, and the overall winner moves from Scenethesis to LayoutVLM at pitch 75° and 90°. These margins describe saved scores and carry no inferential claim. What the exercise shows is that the ordering Holodeck's protocol produces on a shared scene set depends on settings the protocol leaves open. It is not a re-test of Holodeck's own comparison, which used its own baselines and scenes.

2. The title (Q2)

Could you tone the title claim down to match the evidence ...

We would like to keep the title, and we think it stays within the evidence. "Shifting winners" describes what we observe: the point-estimate winner changes for 18 of 19 evaluators, and for 19 of 19 on the new prompts, while every scene stays fixed. The subtitle says fragility, not invalidity. The abstract and Sec. 1 define a winner change as a change in the reported point-estimate winner and say right away that none is statistically confirmed. If you still find the title too strong, we are open to a more literal version such as "Frozen Scenes, Shifting Point-Estimate Winners".

3. Interactions (Q3, W2)

How large are interactions such as pitch × template under a fractional factorial design?

For the pair we expected to interact most, the interaction is real but much smaller than the main effect. We crossed two captions (bare prompt vs. "a top-down view of {prompt}") with two semantic pitches (0° vs. 45°) on all 300 frozen scenes, giving 1,200 scored cells per evaluator. A rule declared before scoring pinned the evaluator, and it selected SigLIP (App. M, Table 19).

EvaluatorCaption effectPitch effectCaption × pitch
SigLIP (pinned)3.938 [3.433, 4.444]0.142 [−0.157, 0.436]−0.523 [−0.670, −0.383]
CLIP ViT-L/14 (descriptive)2.441 [2.031, 2.858]0.251 [−0.147, 0.665]−0.174 [−0.315, −0.026]
GPT-4o judge (descriptive, 0–100 scale)3.350 [2.283, 4.500]−0.683 [−3.700, 2.084]−0.033 [−3.067, 2.783]

For SigLIP the top-down caption still helps at 45°, and the interaction is about 13% of the size of the caption effect. The interaction is smaller for CLIP ViT-L/14, and its interval spans zero for the hosted judge. So for this pair the one-factor-at-a-time estimates do not mislead about direction, while the size of the interaction depends on the evaluator. Table 20 breaks it down by generator. We did not run a fractional design over all eight factors, and lighting × caption in particular remains unmeasured (App. C).

AuthorsOpenReview ↗

Response to Reviewer 6KWH (3/3)

1. Raw records (Q3)

Yes, they are released. The updated supplementary archive contains the complete frozen raw records for all 20 evaluators on both prompt sets, the evaluation harness with its vendored LAVIS dependency, the verified intake manifest, and the records of every follow-up experiment. All 20 SHA-256 hashes in the released manifest match the committed manifest, so every table can be regenerated from the frozen records. The original renders and the full hosted request logs are too large for the archive; we will release them at camera-ready.

2. Prompt sample and scope (W2)

We drew 50 new prompts, disjoint from the original ones and stratified the same way, ran all six generators with every variant and sweep, and scored the resulting 300 scenes with all 20 evaluators (App. R, Table 30).

Measure (19 alignment evaluators)Original 50 promptsNew 50 prompts
Evaluators changing winner, protocol factors1819
Evaluators changing winner, all swept configurations1919
Mean changed configurations, protocol factors6048
Mean changed configurations, all swept configurations433417
Median changed configurations, all swept configurations409432
Most common baseline winnerScenethesis (16)Scenethesis (17)

The first row uses the definition behind the abstract's 18 of 19. The second scope also counts paraphrases and the four degradations, so its counts are not comparable with the first. The single Blender pipeline and indoor-only scenes remain limitations, and the paper states them as such.

3. No validated robust protocol (W3)

We agree, and we do not propose one. We tested the most natural candidate, averaging over views, in a pre-registered comparison at a fixed budget of 15 poses (App. N, Table 21).

EvaluatorMean vs. single viewMax vs. single viewMean vs. max
SigLIP5/154/151/15
BLIP-VQA2/155/153/15

Averaging changes which pairs flip rather than removing flips, so the aggregation rule becomes one more setting to declare.

4. GPT-4o zeros (W3)

After the HSM repair, 2,333 of the 120,270 GPT-4o cells are stored as zero (6,758 before). We re-scored every one of them at temperature 0 with up to five retries (App. O).

Re-score outcomeCellsShare
Resolved to nonzero (mean 15.25)1,84679.1%
Zero on every attempt48720.9%

Every persistent zero comes with a written justification naming requested objects that are missing from the render. The template result holds under each zero policy: full-grammar dispersion is 96.9% with zeros kept, 85.8% with zero cells dropped, and 87.5% with re-scored values (Table 24). On the original prompts the judge is also the one evaluator whose winner never changes under protocol factors, so its zeros cannot inflate the 18 of 19.

5. Paraphrase auditors (W3)

Paraphrases do not enter the 18 of 19, the rank-stability analysis, or the simultaneous band; we treat them as a stress test. The two auditors accept 49.0% and 47.6% of the 500 rewrites. Re-computing dispersion on the 180 rewrites both accept and the 196 both reject gives:

EvaluatorAll rewritesJointly acceptedJointly rejected
GPT-4o judge0.6300.426 [0.319, 0.562]0.388
CLIP ViT-L/140.180.0920.148
CLIP ViT-B/320.130.0750.099

The judge moves more on meaning-preserving rewrites than on meaning-changing ones, so meaning drift does not explain its paraphrase instability. For CLIP the order reverses, which is what a metric that tracks meaning should do. These labels are machine verdicts, so we report the subsets as sensitivity cuts.

We hope these answers and the new experiments address your concerns. If any point still seems unconvincing, please tell us and we will try to resolve it during the discussion.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response (1/2): a correction, and what is new in the revision

We thank the three reviewers for specific and careful reviews. The reviews share four concerns: there is no human check of whether a score shift reflects the scene or the metric, factor interactions are not measured, 50 prompts and six generators may be too few, and the configuration card is never applied to published work. We answer each of them in the reviewer threads. First, we need to report a correction, because it changes several numbers that the reviews quote.

1. Correction to the submitted numbers

After submission we found that our rendering step silently dropped each object's stored dimensions when exporting HSM scenes, so the furniture was missing from all 50 HSM renders and the scenes showed largely empty rooms. We repaired the HSM renders with the same Blender pipeline, re-scored them with all 20 evaluators, and regenerated every table. The generated scenes themselves did not change. The quantities quoted in the reviews move as follows.

QuantitySubmitted PDFRevised PDF
Evaluators whose point-estimate winner changes (of 19)1818
Reversals confirmed under the full-search band00
Peak ρ > 1 (of 19)1719
Peak ρ with bootstrap lower bound > 1 (of 19)1119
Caption template is the largest factor (of 19)1817
Evaluators with opposite pointwise intervals49
Strictly ordered pairs per evaluator (of 15)4–110–7
Pairs separable at baseline, median (of 15)118
Best tie-adjusted scramble discrimination67%71%
Stored GPT-4o zero cells (of 120,270)6,7582,333
GPT-4o template dispersion, full grammar / common forms390% / 83%97% / 61%

The two decision-level results in the first two rows are unchanged. The large rise in ρ is mainly a change in its denominator: with the defective renders, HSM scored well below the other five generators, which inflated the between-generator variance and made pairs involving HSM look strictly ordered. With the renders repaired, the six generators sit closer together, so ρ goes up and fewer pairs keep one sign. Two changes go against us. The best scramble discrimination rose from 67% to 71%, and the caption template is now the largest factor for 17 evaluators rather than 18. The revised PDF uses the repaired numbers everywhere.

AuthorsOpenReview ↗

General response (2/2)

1. New experiments and where they are

Concern (reviewers)What we ranMain resultWhere
No human check (6KWH, cFNj)Blinded pairwise human study, 50 trials, analysis fixed before data collectionMetric-artifact verdict: 82.3% degradation detection vs. ties on 14 of 25 camera pairs, whose median metric margin (1.73) exceeds the degradation median (1.38)Ethics statement; App. S
Interactions (6KWH, cFNj)Pre-registered 2×2 caption × pitch factorial, all 300 scenesInteraction −0.523 [−0.670, −0.383] for the pinned evaluator, about 13% of its caption effect; interval spans zero for the hosted judgeApp. M, Tables 19–20
Only 50 prompts (all three)50 new disjoint prompts; all generators, sweeps and evaluators re-run19 of 19 evaluators change their point-estimate winnerApp. R, Table 30
Card never applied (6KWH, cFNj)Observed cards for Holodeck and HSM; Holodeck's protocol re-run on our scenesBoth cards fail the readiness gate; 11 of 15 pair margins flip under neighboring settings, 6 of the 10 pairs not involving Reason-3DApp. Q, Tables 26–29
Is view averaging a fix?Pre-registered aggregation comparison at 15 posesThe mean still flips 5 of 15 (SigLIP) and 2 of 15 (BLIP-VQA) pair signsApp. N, Tables 21–22
GPT-4o zeros (6KWH)Re-scored all 2,333 stored zero cells79.1% resolve; template result holds under all three zero policiesApp. O, Tables 23–24
Paraphrase drift (6KWH, cFNj)Re-analysis on jointly accepted vs. jointly rejected rewritesThe judge disperses more on accepted rewrites (0.426 vs. 0.388)App. O
Raw records (6KWH, cFNj)Updated supplementary archiveRaw records for all 20 evaluators on both prompt sets, harness, manifest; all 20 hashes matchReproducibility statement
Sample size for confirmationPower analysis for four pinned contrastsPower 0.35–0.57 at 50 prompts; 75–100 prompts reach 0.8App. P, Table 25

2. What the new evidence does not show

No winner reversal is confirmed. None survives the simultaneous band over the 110,295 searched contrasts, and under the same band only one strict ordering survives (Scenethesis over Holodeck, for one evaluator). The Holodeck case study applies Holodeck's evaluation protocol to our scenes; it does not re-test Holodeck's published comparison, and we do not claim that any published conclusion is wrong. The human study is complete and classifies part of the configuration-driven score movement as metric artifact, not visible evidence. We did not add generators, a second renderer, or non-indoor scenes, and interactions other than caption × pitch remain unmeasured.

Each reviewer thread gives the full table for every experiment that reviewer asked for.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.