- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper builds partial indoor 3D scenes (LLM -> constraint solver -> TexVerse assets -> 8192-point cloud and scene graph) and derives multiple-choice questions over the graph as executable programs. And then propose a dynamic evaluation system that selects the next batch of questions similarly to failed samples. On six point-cloud MLLMs, the fraction of wrong answers among 1,000 queries rises from 65.8% (uniform sampling) to 83.7%. The failure profiles are read as architectural diagnoses. PointLLM is fine-tuned on retrieved synthetic data and evaluated on GLUE3D as a real-world transfer test.
Critical strengths
- Clear problem statement and the scene and question generation pipeline. The generator could be usefull for community.
- Two disjoint 1m task pools, a documented task grammar and a clear evaluation protocol.
- Appendices reasonable complete on the algorithmic side.
Critical weaknesses
- Authors cited GreenPLM (Zeng et al., 2022), but that paper is about NLP. GreenPLM: Cross-Lingual Transfer of Monolingual Pre-Trained Language Models at Almost No Cost. Authors likely wanted to cite "More Text, Less Point: Towards 3D Data-Efficient Point-Language Understanding" by Tang et al. AAAI 2025.
- All six tested models were trained on single objects. Feeding them a room with objects in the same 8192 points is a distribution shift that the models were never trained for. The models that are actually designed for scenes are cited and never tested, e.g., 3D-LLM, LL3DA, Chat-Scene, SpatialLM, 3D-R1, and other public models.
- Figure 5 is the only scene the reader ever sees, and it is a polished illustration, not a model input as I got. Beyond that single figure, there is no gallery, no table of generated questions, no statistics on how often each relation appears, and no released data.
- "procedural engines (Deitke et al., 2022) enforce physical validity only through rigid rules that restrict diversity": the proposed framework uses basically the same "rigid rules". AnyPoint generates the relation graph with an LLM and then resolves it with rules. Holodeck already does essentially this: an LLM proposes objects and relations, a constraint solver places them. Same as SceneSmith and Architect-Ant.
- The whole idea of the benchmark is questionable. The paper argues that “benchmarks rapidly saturate as models improve,” but the proposed benchmark seems too difficult for the tested models. Under uniform sampling, five of the six models perform near "the chance floor". The failure rate rising from 66% to 84% under the proposed policy doesn't necessarily indicate that more weaknesses have been discovered. It may simply mean that the same failed questions are sampled repeatedly. The exploration term penalizes only questions from the correct set C, not those from the failure set E. Figure 4 supports this concern: for five models, "Counting" accounts for 52–76% of the budget. This also contradicts the claim in §3.4 that the method “traces diverse vulnerabilities without collapsing into duplicate clusters”.
Minors:
- The bounding boxes for tables and chairs are not required to have zero overlap. Some overlap is acceptable, as scenes often depict chairs tucked under tables. I would tone down the claims about “realistic indoor scene”.
- Closest related 3D benchmarks never cited, like CLEVR3D, ObjVariantEnsemble, and etc.
Questions for authors
Could be statistics on the dataset published with more examples? Can some more powerful models be tested?