- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces a three-axes evaluation scheme for room-furnishing agentic pipelines including 1) physical validity, 2) visual coherence, and 3) human-use logic. By following this scheme, an evaluation agent can autonomously identify failures, gather evidence, and provide detailed reports. The proposed protocol shows a high-agreement with human judgement, and incorporating it into existing room-furnishing pipeline help improve the performance through refinement.
Critical strengths
- The paper tackles spatial reasoning through room furnishing, one of the most challenging tasks for LLMs.
Critical weaknesses
-
There are not sufficient information about the evaluation benchmark that was used in 4.1. For instance, what type of rooms, layouts, furniture, and levels of difficulty that were included? Without these information, it is difficult to judge the impact of the proposed scheme.
-
Results in Table 1 show that vanilla LLMs can perform well with several models surpassing 80% across different evaluation axes. The results are improved further with the agentic approaches in Table 3 even for weaker LLM such as HY4. These results raises concerns on either the proposed evaluation protocol has any added value to the development of this task when LLMs can already perform very well.
-
The inconsistent use of base LLM throughout experiments makes it difficult to draw conclusion. More specifically, section 4.1 uses ChatGPT 5.4 for baseline evaluation, while section 4.2 uses only ChatGPT-4.1 for refinement. To be able to tell if the proposed evaluation scheme can benefit different baselines through refinement, the baselines need to be the same between section 4.1 and 4.2.
-
In general, I find the number of scene used at different experiments low (e.g. 30 in 4.1 and 5 in 4.2). To be able to derive reliable and statistically significant outcomes, a larger and diverse scenes must be evaluated.
Minor:
- Figure 1 is quite confusing. It is not clear what a check or cross mark means. Moreover, evaluation approaches are not consistent across A, B, and C. Given that the paper starts with this figure, the clarity and accuracy are critical.
Questions for authors
-
In the scoring formula at line 225, why is and set to 0.4 and 4 respectively? Would the performance of the models and alignment with humans change if these values were changed? A sensitivity analysis is needed to answer these questions.
-
How are metrics weights in Table 6 determined?
-
What is the purpose of FrozenAssets* setup that is reported in the paper?