acceptodds
Under review as a conference paper at ICLR 2027

VERA: Visual Evidence Refinement Agent for Training-Free 3D Referring Segmentation

Abstract

Recent advances in multimodal large language models (MLLMs) have substantially improved visual understanding and reasoning in 2D domains, opening new possibilities for 3D scene understanding without task-specific training. However, extending these capabilities to 3D referring segmentation (3D-RES) remains challenging, as it requires not only fine-grained understanding of appearance and spatial relationships, but also accurate alignment between language semantics and point-level information in 3D scenes. Consequently, achieving reliable 3D-RES without task-specific supervision remains difficult. To address these challenges, we propose VERA, a training-free framework for 3D-RES that integrates a pretrained MLLM with complementary 2D and 3D perception tools. Specifically, a 3D Candidate Generation module first constructs a query-relevant instance candidate space while preserving the corresponding 3D mask for each candidate. Building upon this candidate space, we introduce a Visual Evidence Self-Refinement module that formulates target selection as a visual evidence-driven process of hypothesis verification and self-refinement. Through this process, the module jointly leverages global scene views, local contextual observations, and object-level crops to assess the semantic and spatial consistency between candidate hypotheses and the referring expression. When the initial hypothesis lacks sufficient visual support, verification feedback is used to reassess competing candidates and refine the current hypothesis through direct visual comparison. The point-level mask associated with the selected candidate is returned as the final segmentation. Extensive experiments on multiple benchmarks demonstrate the effectiveness of VERA for training-free 3D-RES without task-specific supervision.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Referring Segmentation, Vision-Language Models, Training-Free Learning, Visual Evidence Refinement

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gFsROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose VERA, a training-free framework for 3D referring segmentation. It constructs a query-relevant shortlist based on OneFormer3D instance proposals, with optional Grounding DINO recovery and preserves each point-level mask. Subsequently, a frozen MLLM validates the candidate schemes through global, contextual, and object-level visual evidence, and may revise the initial selection through pairwise comparisons. The final selected proposal mask is returned as the segmentation result. Experiments are conducted on the ScanRefer, Nr3D, and Multi3DRefer datasets.

Critical strengths

1.The paper addresses a useful problem: reducing dependence on expensive annotations between language semantics and point-level information. Separating 3D candidate region generation from MLLM-based semantic selection makes sense, as the perceptual model provides point-level masks, while MLLM handles language understanding and instance disambiguation. 2.The use of global views, RGB context, and object crops is well motivated. These cues cover scene layout, spatial relations, and appearance, while self-refinement provides a reasonable alternative that allows for direct acceptance of top-ranked proposals. 3.The evaluation covers ScanRefer, Nr3D, and Multi3DRefer and reports competitive training-free results. In particular, the self-refinement improves ScanRefer Multiple mIoU from 29.8 to 41.0, and the ablations support the value of the proposed visual inputs.

Critical weaknesses

1.The difference in methodology between VERA and recently untrained 3D grounding agents is not yet fully clear. Especially, SPAZER also utilizes global and local visual evidence to validate 3D candidates. VERA preserves a point-level mask for each candidate and adds conditional refinement for 3D referring segmentation, but the paper does not clearly show whether its validation mechanism itself is new or mainly based on improvements to existing paradigms. 2.Table 3 supports the overall benefit of self-refinement and its visual evidence sources. However, it does not isolate candidate verification, alternative candidate search, and pairwise comparison, so the contribution of each decision stage is not yet unclear. 3.VERA directly returns masks from OneFormer3D, thus setting strict performance limits on candidate region recall and mask quality. Although the paper cites OneFormer3D, it does not identify the exact checkpoint or the data used to train that checkpoint, which is crucial for understanding the statement 'no training required'. In addition, all three benchmark tests used the ScanNet scenario, so the cross dataset generalization ability is not clear. 4.The actual benefits of the Candidate Filter have not yet been fully determined. Table 4 only reports the improvement of the oracle candidate metrics and does not show whether the Candidate Filter has improved the final performance [email protected] Or mIoU. Figure 3 reports per-query inference time for different MLLMs, but does not separately list the Candidate Filter or additional costs for self-improvement.

Questions for authors

1.What specific changes does VERA make to candidate verification and refinement compared with SPAZER, beyond retaining point-level masks for 3D referring segmentation? 2.Could the authors isolate the contributions of candidate verification, alternative search, and pairwise comparison? Table 3 establishes the overall gain, but does not distinguish these decision stages. 3.Which OneFormer3D checkpoint and training data were used? Could the authors assess whether VERA's results hold with a different proposal backbone or on scenes outside ScanNet? 4.How much does the Candidate Filter change final [email protected] and mIoU, and how much additional inference time does it require?

Reviewer_gz92OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents VERA, a training-free framework for 3D referring segmentation that combines pretrained 3D instance segmentation with visual evidence and vision-language model (VLM) verification. Given a referring expression, VERA first generates and ranks a set of 3D object candidates using category, appearance, spatial, and proposal-level cues. It then constructs complementary visual evidence, including global scene views, registered RGB observations, and object crops, and employs a VLM to verify the candidates and refine the initial selection. The framework further adopts different verification strategies for single-object and multi-object referring segmentation. Experiments on ScanRefer, Nr3D, and Multi3DRefer demonstrate that the proposed visual evidence refinement can improve segmentation performance, particularly for ambiguous and multiple-instance cases. Overall, VERA provides a complete training-free pipeline for leveraging multimodal visual evidence to improve 3D referring segmentation.

Critical strengths
  1. Meaningful problem and practical motivation. The paper addresses training-free 3D referring segmentation, where ambiguous references and multiple instances of the same category remain challenging. The motivation of leveraging additional visual evidence to disambiguate 3D candidates is intuitive and practically relevant.

  2. Well-designed and coherent framework. VERA provides a clear pipeline that combines 3D candidate generation, complementary visual evidence construction, and VLM-based candidate verification. In particular, preserving the original 3D instance masks throughout the pipeline avoids introducing an additional point-level segmentation stage after candidate selection.

  3. Comprehensive experimental evaluation. The method is evaluated on three established benchmarks, including ScanRefer, Nr3D, and Multi3DRefer, with comparisons and ablation studies covering the visual evidence sources and refinement components. The results show consistent benefits from visual evidence refinement, particularly in more challenging multiple-instance settings.

Critical weaknesses
  1. Limited methodological novelty. The overall framework appears to primarily combine existing components, including off-the-shelf 3D instance segmentation for candidate generation, heuristic candidate ranking, multi-view visual grounding, and VLM-based candidate verification. Although this combination is practically useful, the paper does not sufficiently demonstrate a substantially new algorithmic insight beyond this composition. In particular, the relationship to recent agentic and evidence-based 3D understanding approaches such as Agent3D-Zero, SeeGround, VLM-Grounder, and SPAZER should be discussed and experimentally distinguished more carefully.

  2. The proposed “self-refinement” is not clearly distinguished from candidate verification and reranking. The refinement procedure mainly verifies the initial candidate, inspects alternatives when necessary, and switches to a better candidate through pairwise comparison. This appears closer to VLM-based candidate verification/reranking than a genuinely iterative self-refinement process involving explicit error identification and hypothesis generation or modification. The paper should clarify what constitutes the novel refinement mechanism and why it is fundamentally different from existing verification-based pipelines.

  3. The contribution of the proposed reasoning strategy is not sufficiently isolated from the benefit of additional visual evidence. The ablations demonstrate that removing visual evidence significantly degrades performance, but this does not necessarily establish that the proposed refinement strategy itself is responsible for the gains. A stronger comparison would use the same visual evidence while replacing the proposed refinement procedure with a simpler candidate reranking or direct VLM selection strategy, thereby isolating the contribution of the “self-refinement” mechanism.

  4. The dependence on candidate-generation recall is insufficiently analyzed. Since VERA ultimately selects and returns an existing 3D proposal mask, it cannot recover a target that is absent from the candidate set. The Candidate Filter improves proposal coverage, but the paper should more clearly report candidate recall/oracle performance as a function of shortlist size and quantify how many failures originate from missing proposals versus incorrect VLM verification. This is particularly important for understanding the actual upper bound of the proposed training-free pipeline.

  5. The computational cost of the VLM-based pipeline is not sufficiently characterized. VERA involves global viewpoint selection, candidate-specific visual evidence construction, verification of up to 12 candidates, and potentially additional pairwise comparisons. Although the paper reports inference time, a more detailed breakdown of the number of VLM calls, average number of candidates actually verified, rendering/projection cost, VLM inference time, and computational resources would be important for evaluating the practical advantages of a training-free approach.

  6. The robustness of query decomposition and heuristic candidate ranking is underexplored. Candidate generation relies on decomposing the referring expression into category, appearance, and spatial relations, followed by a manually designed weighted score. The fixed coefficients are not learned from annotations, but the sensitivity of the pipeline to parsing errors and the choice of these weights is not sufficiently investigated. In particular, errors in spatial or appearance interpretation could affect the initial shortlist and subsequently limit the VLM's ability to recover the correct target.

Questions for authors
  1. Candidate verification: What is the average number of candidates actually verified by the VLM per query, compared with the maximum budget of 12 candidates? How much does the selective verification strategy reduce the number of VLM calls in practice?

  2. Candidate ranking: How sensitive is the performance to the manually specified weights in the initial candidate scoring function? Have the authors evaluated whether different reasonable weight settings lead to substantially different results?

Reviewer_1AAQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work proposes a training-free 3D referring segmentation pipeline that takes a 3D point-cloud scene and a referring expression as input. It first uses off-the-shelf 3D perception models to construct and rank candidate object instances while preserving their point-level masks. It then renders multi-granularity visual evidence, including global scene views, contextual RGB observations, and local object crops, and asks a frozen MLLM to verify and refine the candidate selection. The mask associated with the final selected candidate is returned as the segmentation output. The method is evaluated on ScanRefer, Nr3D, and Multi3DRefer, where it shows competitive zero-shot performance, particularly compared with existing LLM-based approaches.

Critical strengths
  1. This work addresses a meaningful problem, i.e., training-free 3D referring segmentation by proposing a simple and intuitive decomposition that preserves point-level masks while using an MLLM for semantic and spatial disambiguation. The proposed pipeline is easy to understand and follow, meanwhile efficient.

  2. This paper is well written and it clearly introduce the structure of the proposed pipeline. Experiments settigs and parameters and LLMs utilized in the experiments are introduced sufficiently. Authors make it clear for reviewers to understand this work's contribution and high-level philosophy.

Critical weaknesses
  1. Lack of novelty. Most of this work's core capability is inherited from pretrained components such as OneFormer3D, Grounding DINO, and Qwen3-VL. The proposed Visual Evidence Self-Refinement is largely an inference-time prompting and context-engineering procedure, where progressively richer visual evidence is presented to a frozen MLLM for discrete candidate verification and comparison. Consequently, I hereby view the work as a strong system-engineering contribution but relatively limited in methodological novelty.

  2. Missing technical details. Several important implementation details, particularly the query parsing function (line 160, equation 3), are underspecified, authors did not explain how this function works with a clear detailed formula. The current ablations do not sufficiently disentangle the benefit of richer visual evidence from the proposed self-refinement procedure itself.

Questions for authors
  1. Reproducibility. Please clarify how the underspecified functions, especially the query parsing function (line 160, equation 3), and the semantic/spatial matching scores used for candidate ranking, are implemented, and provide the exact configuration of the pretrained proposal generator.

  2. Stability. Please provide evidence that the complete multi-stage pipeline is stable with respect to reasonable variations in candidate generation, candidate/view ordering, prompts, or repeated inference, rather than reporting only single-run performance.

  3. Ablation. Please provide controlled experiments that disentangle the benefit of richer multi-granularity visual evidence from the benefit of the proposed sequential self-refinement mechanism itself, and further isolate the contributions of query parsing, candidate ranking, verification, alternative search, and pairwise comparison.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.