acceptodds
Under review as a conference paper at ICLR 2027

Entity3R: An End-to-End Entity-Aware Geometric Foundation Model

Abstract

Class-agnostic 3D entity understanding aims to recover open-world entity identities with basic scene geometry from multi-view images or videos. Recent state-of-the-art methods built on instance-grounded geometric foundation models rely on hyperparameter-sensitive clustering to decode objects, which makes entity mask inference unreliable and costly. They also inherit the low native resolution of geometry transformers, limiting fine-grained entity perception. We present Entity3R, an end-to-end, entity-aware 3D/4D geometric foundation model. Entity3R extracts high-resolution geometry-instance representations with a dual-branch backbone, and decodes 3D/4D-consistent entity masks through learnable global object queries that interact with frame-level instance features. This design enables accurate and efficient 3D entity segmentation without post-hoc clustering. We further introduce a training-free, chunk-wise decoding strategy that allows Entity3R to decode more entities than its query budget and to process longer input sequences. To train Entity3R, we build EntityWorld, a large-scale corpus for joint geometry and instance learning, containing 360K sequences and 84.8M frames drawn from existing geometric datasets and newly collected photorealistic game data. Entity masklets are generated at scale by an automatic annotation pipeline. Trained on EntityWorld, Entity3R achieves state-of-the-art performance on both entity segmentation and geometric reconstruction benchmarks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Reconstruction, Entity Segmentation, Geometry Transformer

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_W5ZuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Entity3R, a model that jointly predicts scene geometry and class-agnostic entity identities from unposed multi-view images or videos. It employs a low-resolution geometry Transformer alongside a frame-wise high-resolution encoder to construct features that balance cross-view consistency with local detail. Furthermore, it incorporates a patch-based entity decoding mechanism—which requires no retraining—and associates entities across different patches through feature-center matching. The authors also present EntityWorld, a dataset comprising approximately 360K sequences and 84.8M frames.

Critical strengths
  1. Instance segmentation performance is significantly improved, while the computational cost for object decoding remains low.
  2. The system design, which unifies geometric and instance perception, offers strong practical utility.
Critical weaknesses
  1. The independent contributions of the core methods have not been sufficiently disentangled: Entity3R incorporates a combination of stronger VGGT-Ω initialization, a high-resolution DINOv3 encoder, a novel query-based decoder, and the large-scale EntityWorld dataset. While the provided ablations, covering low-resolution variants and query position configurations, demonstrate the efficacy of certain design choices, they fail to clarify the source of performance gains relative to clustering-based approaches: specifically, how much improvement stems from the query-based decoder versus changes to the backbone or the training data. In particular, although the EntityWorld dataset is highlighted as a primary contribution, the paper lacks ablations that isolate its impact, such as experiments involving the removal of the newly added game data, controlling the scale of the dataset, or comparing data from different annotation sources. Given that the key components are built upon existing methods, controlled experiments are essential to rigorously assess the significance of the contributions presented in this paper.
  2. The validation of cross-segment identity consistency and the ability to overcome query-count limitations remains inadequate: The segmentation-based approach relies on cosine similarity between the centers of masked-pooled features to establish associations; however, the paper provides insufficient detail regarding the matching thresholds, conflict resolution strategies, and the rules for updating historical centers. While the overall metrics in Table 3 and the qualitative examples in Figure 6 suggest the method’s effectiveness, they are insufficient to evaluate its stability in scenarios involving prolonged occlusion, the re-emergence of entities, or the presence of entities with similar appearances.
Questions for authors

1.Is it possible to compare query-based decoding with clustering-based decoding while keeping the pre-trained backbone, input resolution, training data, and training budget constant? Furthermore, could you provide results obtained by excluding the EntityWorld-Game dataset or by controlling the scale of the training data, in order to disentangle the contributions of the architecture from those of the data? 2. What matching thresholds, matching constraints, and historical centroid update mechanisms are employed for cross-segment association? Could you report entity recall and identity association metrics specifically for sequences containing more than 200 distinct entities, as well as for challenging scenarios such as re-emergence after occlusion or the presence of entities with similar appearances? 3. EntityWorld incorporates data sources, such as Waymo, Hypersim, and Co3Dv2, that are also used for evaluation. Please specify the training/validation/test splits for each data source, as well as the methods used for deduplication at the scene or sequence level. The objective here is to clarify the evaluation isolation protocol, rather than to suggest that using the training split of a given dataset is inherently problematic.

Reviewer_BaY4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Entity3R, a feed-forward model capable of jointly predicting scene geometry and class-agnostic entity masks from unposed images or videos. The model combines two encoding branches, a low-resolution geometry transformer (initialized from vggt-omega), and a high-resolution per-frame encoder (initialized from DINOv3). The high-res tokens which are only "locally aware", receive global context via a dedicate cross attention adapter to the low-res tokens, which underwent global attention as part of the geometry transformer. The refined high-res tokens are then used by a mask2former style decoder to produce entity mask, alongside depths per frame. camera parameters are decoded from the camera tokens taken from the low-res branch to complete the decoding. The authors also suggests a chunk-wise association for longer sequences to, and assemble a datasets collection "EntityWorld" for joint geometry and instance training. Entity3R demonstrates strong results across segmentation and reconstruction tasks.

Critical strengths
  • Strong demonstrated segmentation results: Entity3R achieves substantial improvements over the compared methods across a broad range of datasets. Table 1 reports 78.9 DAVIS J&F and 71.2 VIPSeg VEQ, compared with 71.4 and 61.4 for EntitySAM. Improvements also extend to the video panoptic metrics, supporting accurate masks and consistent entity identities while jointly predicting scene geometry.

  • Effective adaptation of the dual-resolution architecture: Repurposing DAGE’s low-resolution global branch and high-resolution per-frame branch enables detailed, consistent segmentation with substantially lower memory requirements. The design combines cross-frame context with fine spatial detail while keeping high-resolution tokens outside global geometry attention. For the 100-frame sequence at width 1024 in Table 8, peak memory decreases from 76.9 GB with direct high-resolution geometry processing to 27.3 GB with the dual-resolution design.

Critical weaknesses
  • The contribution beyond closely related architectures is not clearly isolated. DAGE already provides the low-resolution global/high-resolution per-frame fusion design, while PanSt3R combines multi-view geometry with shared-query mask decoding but is not discussed. Entity3R extends these ideas to joint geometry and class-agnostic entity prediction in dynamic scenes, but comparisons change architecture, backbone, and training data together. Some dedicated design ablation are missing. Moreover, its reconstruction gains over DAGE cannot be attributed to the architecture alone. The initialization from VGGT-Omega checkpoint further complicates interpretation, given the maintainers' potential-contamination notice and recommendation to benchmark with retrained weights (https://github.com/facebookresearch/vggt-omega/blob/main/reproduction.md).

  • Table 3 leaves the role of joint cross-frame entity decoding insufficiently explained. S=1 achieves better results compared to full-context decoding. All frames still enter the geometry backbone jointly, so these results support frame-wise decoding followed by association on globally contextualized features. They do not establish an advantage from cross-frame attention within the entity decoder


Minor:

  • EntityWorld's total size does not establish the contribution of its new data. ASE supplies 58.1M of 84.8M stored frames (~70%), with existing masks. Dataset sampling probabilities and a data-source ablation are missing (if applicable).
  • Figure 8 contains "Oours" twice. in Table 8 some of the rows don't sum to their "total".
Questions for authors
  1. The entire network is trained e2e. As the architecture is quite modulated and the roles are well-established, wouldn't it be sufficient to freeze the encoders, and train only the new fusion adapter, and the decoder heads?
  2. Does the adapter use any explicit spatial alignment or positional-encoding adjustment between the LR and HR tokens, given their different patch sizes (14 versus 16) and different positional encodings (DINOv2’s additive embeddings vs DINOv3’s RoPE)?
  3. Which exact VGGT-Omega checkpoint is used for initialization and the baseline?
  4. Given the effectiveness of encoder-based queries in works such as EoMT (https://arxiv.org/abs/2503.19108), the large gap in Table 3:a-b is somewhat surprising. Could you provide further insight into whether it stems from query placement itself or from differences in feature representations or supervision?
Reviewer_S4xbOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Entity3R for joint class-agnostic entity segmentation and geometric reconstruction from multi-view images or videos. It combines a dual-resolution backbone with query-based entity decoding and chunk-wise object decoding. The authors also construct EntityWorld, containing approximately 84.8M frames with geometry and entity annotations. Experiments show strong entity segmentation and downstream panoptic segmentation results, alongside competitive geometry estimation. Overall, the results are promising, but annotation quality and the separate contributions of the dataset and architecture remain insufficiently established.

Critical strengths
  • EntityWorld combines existing geometric datasets with newly collected game sequences across real and synthetic, indoor and outdoor, and static and dynamic settings. Its scale and coverage make it a potentially useful resource for joint entity segmentation and reconstruction.
  • Entity3R leads the evaluated baselines on all segmentation metrics in Table 1 while retaining competitive geometry estimation. These results support the practical value of the unified model.
Critical weaknesses
  • EntityWorld’s validation is limited. Fig. 7 provides qualitative annotation comparisons, and Appendix A.5 acknowledges under-segmentation of stuff regions, but no quantitative annotation assessment or systematic manual audit is reported. Data ablations and experiments training other methods on EntityWorld are also absent, leaving its contribution to performance and broader utility unclear.
  • I find the methodological novelty limited, given the use of a DAGE-inspired backbone and a Mask2Former-style decoder. The HR-branch, query-placement, and chunking ablations support these design choices. However, the comparisons with prior methods do not isolate architectural gains from differences in training data and backbone initialization.
Questions for authors
  • Beyond Fig. 7 and the limitation discussed in A.5, was annotation quality quantitatively evaluated or systematically checked by human annotators? If so, please report the protocol and results.
  • Do you have controlled results that isolate EntityWorld’s contribution, such as training an existing method on it, or comparing architectures while controlling for training data and backbone differences?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.