acceptodds
Withdrawn from ICLR 2027

GREP: Grounding Generated Room Evaluation in Principles

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent LLM-centric advances can generate attractive furnished rooms by selecting and arranging assets. Yet evaluation remains limited to specific perspectives or predefined questions, failing to reflect overall scene quality. To address this pressing need, we introduce GREP. Its core is a focus shift from defining specific questions to establishing a three-tier evaluation philosophy: basic physical validity checks whether a scene obeys physical constraints; visual coherence concerns compatibility in object scale, style, and composition; and, at the highest level, Human-use logic requires object arrangements to align with everyday human habits. Guided by these principles, an evaluation agent autonomously and adaptively identifies issues from scene context, gathers evidence, and produces scores and reports. This supports scene-specific issue discovery and reliable judgments. Experiments show that (1) GREP achieves 87% agreement with human majority preferences, substantially outperforming baseline evaluators. (2) GREP reports improve scene quality through refinement, whereas score-only and no-feedback baselines show slight declines. (3) GREP suggests that as planning ability improves, recent models such as GPT-5.6 Sol can move from relying on pipelines and harnesses to challenging them, but still lack the ability to resolve the conflicts. Stronger models are therefore expected not only to plan well, but also to adapt any external guidance or constraints more flexibly.

Primary area:
datasets and benchmarks
Keywords:
3D Scene Evaluation, Spatial Reasoning, Vision-Language Models, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WaBXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a three-axes evaluation scheme for room-furnishing agentic pipelines including 1) physical validity, 2) visual coherence, and 3) human-use logic. By following this scheme, an evaluation agent can autonomously identify failures, gather evidence, and provide detailed reports. The proposed protocol shows a high-agreement with human judgement, and incorporating it into existing room-furnishing pipeline help improve the performance through refinement.

Critical strengths
  • The paper tackles spatial reasoning through room furnishing, one of the most challenging tasks for LLMs.
Critical weaknesses
  1. There are not sufficient information about the evaluation benchmark that was used in 4.1. For instance, what type of rooms, layouts, furniture, and levels of difficulty that were included? Without these information, it is difficult to judge the impact of the proposed scheme.

  2. Results in Table 1 show that vanilla LLMs can perform well with several models surpassing 80% across different evaluation axes. The results are improved further with the agentic approaches in Table 3 even for weaker LLM such as HY4. These results raises concerns on either the proposed evaluation protocol has any added value to the development of this task when LLMs can already perform very well.

  3. The inconsistent use of base LLM throughout experiments makes it difficult to draw conclusion. More specifically, section 4.1 uses ChatGPT 5.4 for baseline evaluation, while section 4.2 uses only ChatGPT-4.1 for refinement. To be able to tell if the proposed evaluation scheme can benefit different baselines through refinement, the baselines need to be the same between section 4.1 and 4.2.

  4. In general, I find the number of scene used at different experiments low (e.g. 30 in 4.1 and 5 in 4.2). To be able to derive reliable and statistically significant outcomes, a larger and diverse scenes must be evaluated.

Minor:

  1. Figure 1 is quite confusing. It is not clear what a check or cross mark means. Moreover, evaluation approaches are not consistent across A, B, and C. Given that the paper starts with this figure, the clarity and accuracy are critical.
Questions for authors
  1. In the scoring formula at line 225, why is α\alpha and γ\gamma set to 0.4 and 4 respectively? Would the performance of the models and alignment with humans change if these values were changed? A sensitivity analysis is needed to answer these questions.

  2. How are metrics weights in Table 6 determined?

  3. What is the purpose of FrozenAssets* setup that is reported in the paper?

Reviewer_9vJ9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GREP, an agent-based framework for evaluating generated 3D indoor scenes. Guided by physical validity, visual coherence, and human-use logic, the framework combines scene-specific checks, geometric measurements, and adaptive view acquisition to produce multidimensional scores and object-level diagnoses. The authors apply GREP to several scene generation systems and evaluate it through human preference comparisons, repeatability tests, and report-guided refinement experiments.

Critical strengths
  1. Comprehensive evaluation dimensions with practical value. The framework integrates physical validity, visual coherence, and functional usability, providing a useful approach to assessing scene quality and diagnosing problems in generated scenes.
  2. Broad applicability. GREP can evaluate scenes produced through native model planning, specialized pipelines, and agent-based systems. Its object-level diagnoses also help identify and address specific problems.
  3. Clear presentation. The paper is well organized and generally easy to follow. The figures and appendices help explain the evaluation workflow, system design, and experimental settings.
Critical weaknesses
  1. Limited scoring stability for some scenes. Although the authors provide repeatability tests, identical layouts can still receive substantially different scores, weakening confidence in model comparisons and reported refinement gains.
  2. Small validation samples. The human preference study includes only six scenes, and the refinement study includes only five. Both use wall-free scenes, providing limited evidence of reliability and generalization.
  3. Insufficient controlled ablations. The experiments do not adequately isolate the contributions of the framework design, additional evidence, and increased computational budgets.
Questions for authors
  1. Expand the validation experiments. I recommend increasing the number of independent scenes in both the human preference and report-guided refinement studies, covering scenes with walls, multiple rooms, and varying layout complexity. Confidence intervals or error bars would strengthen the main conclusions.
  2. Independently validate refinement gains. According to Table 10, the S102 layout remains unchanged throughout all three rounds in the No-feedback branch, yet its score drops from 0.608 to 0.403. This indicates that scoring variability may affect the interpretation of refinement gains. I recommend repeated evaluations of identical layouts and cross-validation through blinded human assessment or alternative judge models, such as Astra. Relying solely on the same evaluation framework to assess the benefits of its own feedback is insufficient to establish genuine improvements in scene quality.
  3. Provide ablations across different budgets. Using the same model and scenes, please vary the number of views, evidence-acquisition rounds, and model-call budgets, and report evaluation accuracy, stability, and computational cost. Comparing fixed-view evaluation with adaptive evidence acquisition under matched budgets would help isolate the contribution of the acquisition strategy.
  4. Clarify scoring when evidence is insufficient. Please specify the conditions that trigger forced verdicts and default-valid outcomes, how these outcomes contribute to scores, and how frequently they occur. An analysis of how different handling policies affect final scores and model rankings would also be helpful.
Reviewer_CBnSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GREP evaluates generated indoor scenes through three principles: physical validity, visual coherence, and human-use logic. It combines geometric measurements, VLM-driven discovery of scene-specific checks, and adaptive view acquisition to produce scores and diagnostic reports. Validation includes benchmarking scene generators, a human-preference study, repeatability analysis, and experiments using diagnostic feedback for scene refinement.

Critical strengths
  1. The paper addresses an important evaluation gap. Discovering scene-specific usability and coherence problems beyond explicitly requested relationships is valuable for assessing whether generated environments make sense as rooms.
  2. The inspection process produces useful, traceable diagnoses. Combining geometric measurements with targeted views and recording the reasons behind judgments makes the output more actionable than a single holistic score.
Critical weaknesses
  1. The metric’s advantage is insufficiently validated. Human agreement is measured on only six scenes (15 pairs) rated by five participants, all computer graphics researchers (Appendix D.2), while comparisons with existing evaluators are mainly illustrative (Fig. 1). The small scene and participant pools, together with the lack of systematic quantitative comparisons, provide limited evidence that GREP more accurately reflects scene quality or consistently corrects errors made by previous metrics.
  2. Benchmark reusability is unclear. Appendix C specifies task counts and settings, but the manuscript provides no artifact link or explicit release commitment for the exact tasks, asset inventories, and evaluator implementation. This leaves independent reproduction and evaluation of new methods uncertain
Questions for authors
  1. Can the authors quantitatively compare GREP with SceneEval and LEGO-Eval on the same scenes and shared evaluation criteria, using independent human judgments to assess which evaluator is more accurate?
  2. How were the six scenes and five participants selected? Is additional validation with more scenes and a broader participant pool available to support the reported human agreement?
  3. How often do unresolved checks receive forced verdicts or default-valid outcomes, and how sensitive are scene rankings to these choices and the inspection budget?
  4. Do the refinement gains persist under the main GREP scoring protocol and independent human evaluation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.