acceptodds
Withdrawn from ICLR 2027

AnyPoint: Dynamic Evaluation for Point-Cloud Multimodal Large Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Point-cloud multimodal large language models (MLLMs) are advancing rapidly in 3D scene understanding, yet their evaluation relies almost exclusively on static benchmarks that quickly saturate as models improve and fail to locate individual model weaknesses. Transitioning to dynamic evaluation requires an expansive pool of physically plausible 3D scenes with verified ground-truth answers, a dual challenge that neither manual annotation nor pure language model generation can satisfy. To resolve this dilemma, we present AnyPoint, a dynamic evaluation framework that couples programmatic 3D scene generation with adaptive failure discovery for point-cloud MLLMs. AnyPoint decouples scene creation into semantic layout generation via a language model and spatial coordinate solving via a deterministic constraint solver, producing physically valid 3D scenes alongside grounded 3D scene graphs. To avert saturation, evaluation tasks are derived compositionally as programs over each graph, allowing ground-truth answers to be computed directly from the geometry without human annotation. Within this generated task space, a budget-aware dynamic evaluation algorithm adaptively probes each model's failure boundary by balancing error exploitation and novelty exploration. Across six representative point-cloud MLLMs, AnyPoint raises the failure discovery rate within a 1,000-query budget from 65.80% under uniform sampling to 83.73%. The recovered failure profiles reveal distinct model-specific weaknesses alongside shared capability bottlenecks that single aggregate scores obscure. Finally, we demonstrate that discovered failures can drive targeted model improvement, while preserving task diversity is essential for overall generalization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Point-Cloud Multimodal Large Language Models, Model Evaluation, Task Generation, Data-Centric AI

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_iA49OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds partial indoor 3D scenes (LLM -> constraint solver -> TexVerse assets -> 8192-point cloud and scene graph) and derives multiple-choice questions over the graph as executable programs. And then propose a dynamic evaluation system that selects the next batch of questions similarly to failed samples. On six point-cloud MLLMs, the fraction of wrong answers among 1,000 queries rises from 65.8% (uniform sampling) to 83.7%. The failure profiles are read as architectural diagnoses. PointLLM is fine-tuned on retrieved synthetic data and evaluated on GLUE3D as a real-world transfer test.

Critical strengths
  1. Clear problem statement and the scene and question generation pipeline. The generator could be usefull for community.
  2. Two disjoint 1m task pools, a documented task grammar and a clear evaluation protocol.
  3. Appendices reasonable complete on the algorithmic side.
Critical weaknesses
  1. Authors cited GreenPLM (Zeng et al., 2022), but that paper is about NLP. GreenPLM: Cross-Lingual Transfer of Monolingual Pre-Trained Language Models at Almost No Cost. Authors likely wanted to cite "More Text, Less Point: Towards 3D Data-Efficient Point-Language Understanding" by Tang et al. AAAI 2025.
  2. All six tested models were trained on single objects. Feeding them a room with objects in the same 8192 points is a distribution shift that the models were never trained for. The models that are actually designed for scenes are cited and never tested, e.g., 3D-LLM, LL3DA, Chat-Scene, SpatialLM, 3D-R1, and other public models.
  3. Figure 5 is the only scene the reader ever sees, and it is a polished illustration, not a model input as I got. Beyond that single figure, there is no gallery, no table of generated questions, no statistics on how often each relation appears, and no released data.
  4. "procedural engines (Deitke et al., 2022) enforce physical validity only through rigid rules that restrict diversity": the proposed framework uses basically the same "rigid rules". AnyPoint generates the relation graph with an LLM and then resolves it with rules. Holodeck already does essentially this: an LLM proposes objects and relations, a constraint solver places them. Same as SceneSmith and Architect-Ant.
  5. The whole idea of the benchmark is questionable. The paper argues that “benchmarks rapidly saturate as models improve,” but the proposed benchmark seems too difficult for the tested models. Under uniform sampling, five of the six models perform near "the chance floor". The failure rate rising from 66% to 84% under the proposed policy doesn't necessarily indicate that more weaknesses have been discovered. It may simply mean that the same failed questions are sampled repeatedly. The exploration term penalizes only questions from the correct set C, not those from the failure set E. Figure 4 supports this concern: for five models, "Counting" accounts for 52–76% of the budget. This also contradicts the claim in §3.4 that the method “traces diverse vulnerabilities without collapsing into duplicate clusters”.

Minors:

  • The bounding boxes for tables and chairs are not required to have zero overlap. Some overlap is acceptable, as scenes often depict chairs tucked under tables. I would tone down the claims about “realistic indoor scene”.
  • Closest related 3D benchmarks never cited, like CLEVR3D, ObjVariantEnsemble, and etc.
Questions for authors

Could be statistics on the dataset published with more examples? Can some more powerful models be tested?

Reviewer_8vUeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a new evaluation framework for 3D scene understanding with point clouds for point-cloud MLLMs. The motivation is that static evaluation protocols cannot mine hard cases for the evaluated models, while dynamic evaluation is commonly heavy. The authors extract 3D models from TexVerse and use LLM-prompted layouts to generate 3D scenes with rule-based geometrical locations, so that the questions can be directly verified by the rules (including colors, distances, counting, etc.). Then, the authors propose a hard-case mining algorithm that uses Qwen3 to embed the semantics of the questions and scenes, then uses the embeddings to dynamically select questions whose embeddings are close to those of wrong cases and far from those of correct cases. The results show that different models may have different patterns and weaknesses in understanding the scene. Additional training on the selected wrong cases shows improvements when using PointLLM as the baseline.

Critical strengths
  1. The authors propose a valuable framework to dynamically identify and diagnose the true weaknesses of different point-cloud LLMs.
  2. The adaptive selection algorithm is simple but reasonable. It is effective under the 1,000-query evaluation budget.
Critical weaknesses
  1. The automated geometry checks and the independent interpreter check are helpful. But they may not fully cover the actual point clouds paired with natural-language questions. The paper should report human validation of the final generated questions (at least a sampled set of them). Otherwise, ambiguous or even wrong questions may be mined by the adaptive selection algorithm, and the model may be judged wrong not because it is actually wrong, but because the question has a wrong answer or the question itself is wrong.
  2. On PointLLM, uniform training achieves 58.68%, while failure-guided training achieves 60.35%. On real-world GLUE3D, the results are 34.77% vs. 35.16%. It seems to me that there is little benefit, or at least only a marginal benefit, even when we have failure cases to guide us?
Questions for authors

Could the authors test another embedding model (other than Qwen) to see whether the failure discovery rate and the main diagnostic findings remain consistent? This would help clarify how sensitive the framework is to the choice of task representation.

Reviewer_KcN9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AnyPoint, a dynamic evaluation framework for point-cloud multimodal large language models (MLLMs), aiming to discover model-specific failure cases under a limited evaluation budget. The framework consists of three main components: Programmatic 3D scene synthesis, Programmatic task generation and Adaptive failure discovery. Experiments on six point-cloud MLLMs show that AnyPoint achieves an average failure discovery rate of 83.73% under a 1,000-query budget, compared with 65.80% for uniform sampling. The paper also investigates failure-guided fine-tuning and its generalization to real-world scanned scenes. While the proposed framework demonstrates promising improvements in evaluation efficiency, the realism of its synthesized scenes and the breadth of its evaluation tasks raise concerns about whether the framework adequately measures general-purpose 3D scene understanding.

Critical strengths
  1. Efficient and scalable generation of evaluation tasks with verifiable ground truth. Firstly, the combination of LLM-based semantic layout generation, deterministic geometric constraint solving, and programmatic question generation provides a systematic way to create large numbers of evaluation instances without manually annotating every question. Secondly, this design is valuable for reducing benchmark saturation and enabling reproducible evaluation at scale.

  2. Adaptive evaluation improves failure discovery under a constrained query budget. The proposed exploit–explore strategy achieves an average failure discovery rate of 83.73%, outperforming uniform sampling and the evaluated adaptive baselines on average. Moreover, the experiments also demonstrate that discovered failures can inform targeted fine-tuning, suggesting potential practical value beyond conventional aggregate benchmark scores.

Critical weaknesses
  1. The synthesized 3D scenes have limited realism and may not adequately represent real-world environments.
  • The proposed scene synthesis mainly combines existing 3D object assets according to generated spatial relationships. Although the geometric constraint solver enforces collision-free placement and spatial consistency, these properties alone do not ensure realistic scene composition.
  • In particular, the synthesized scenes appear to lack realistic environmental backgrounds and contextual structures, such as walls, floors, room layouts, and other surrounding elements that are important for understanding real-world 3D scenes. The absence of such context may substantially simplify the evaluation compared with real-world point clouds.
  • Consequently, it remains unclear whether the discovered failure patterns reflect genuine limitations in real-world 3D scene understanding or are specific to the synthetic object arrangements. The limited and uneven transfer results on the real-world GLUE3D benchmark further highlight the need to establish this connection more convincingly.
  1. The evaluation tasks primarily assess multi-object localization and relational reasoning rather than comprehensive 3D scene understanding.
  • The generated evaluation tasks are executable programs composed of a localization selector and a query operator. As described in the paper, these operations primarily identify target objects based on categories, spatial relationships, regions, or attributes, and subsequently query their properties or cardinalities.
  • This formulation mainly evaluates whether a model can locate objects, identify spatial relationships, count objects, and retrieve object attributes. These are useful capabilities, but they do not necessarily cover broader aspects of 3D scene understanding, such as holistic scene interpretation, contextual reasoning, environmental structure understanding, and interactions between objects and their surroundings.
  • Therefore, the benchmark may be better characterized as a dynamic evaluation framework for multi-object perception and compositional spatial reasoning than as a general framework for evaluating 3D scene understanding. This limitation affects the scope of the paper's central claims and the generalizability of its conclusions.
Questions for authors
  1. How representative are the synthesized scenes of real-world 3D environments? Do the generated scenes contain environmental structures, such as room layouts, walls, floors, and contextual background geometry, or do they primarily consist of individual object assets placed in otherwise empty space?

  2. How well does AnyPoint evaluate general 3D scene understanding beyond object localization and relational queries? Can the authors clarify which capabilities are covered by the localization selectors and query operators, and which important scene-understanding capabilities are outside the current task grammar?

  3. How should the limited real-world transfer results be interpreted? The reported experiments show that synthetic-data fine-tuning improves overall GLUE3D accuracy from 32.81% to 34.77% after the first round, but performance subsequently declines to approximately 30% after repeated training. How do the authors distinguish the benefits of the proposed failure-guided evaluation from the effects of synthetic-data distribution and domain mismatch?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.