- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Entity3R, a model that jointly predicts scene geometry and class-agnostic entity identities from unposed multi-view images or videos. It employs a low-resolution geometry Transformer alongside a frame-wise high-resolution encoder to construct features that balance cross-view consistency with local detail. Furthermore, it incorporates a patch-based entity decoding mechanism—which requires no retraining—and associates entities across different patches through feature-center matching. The authors also present EntityWorld, a dataset comprising approximately 360K sequences and 84.8M frames.
Critical strengths
- Instance segmentation performance is significantly improved, while the computational cost for object decoding remains low.
- The system design, which unifies geometric and instance perception, offers strong practical utility.
Critical weaknesses
- The independent contributions of the core methods have not been sufficiently disentangled: Entity3R incorporates a combination of stronger VGGT-Ω initialization, a high-resolution DINOv3 encoder, a novel query-based decoder, and the large-scale EntityWorld dataset. While the provided ablations, covering low-resolution variants and query position configurations, demonstrate the efficacy of certain design choices, they fail to clarify the source of performance gains relative to clustering-based approaches: specifically, how much improvement stems from the query-based decoder versus changes to the backbone or the training data. In particular, although the EntityWorld dataset is highlighted as a primary contribution, the paper lacks ablations that isolate its impact, such as experiments involving the removal of the newly added game data, controlling the scale of the dataset, or comparing data from different annotation sources. Given that the key components are built upon existing methods, controlled experiments are essential to rigorously assess the significance of the contributions presented in this paper.
- The validation of cross-segment identity consistency and the ability to overcome query-count limitations remains inadequate: The segmentation-based approach relies on cosine similarity between the centers of masked-pooled features to establish associations; however, the paper provides insufficient detail regarding the matching thresholds, conflict resolution strategies, and the rules for updating historical centers. While the overall metrics in Table 3 and the qualitative examples in Figure 6 suggest the method’s effectiveness, they are insufficient to evaluate its stability in scenarios involving prolonged occlusion, the re-emergence of entities, or the presence of entities with similar appearances.
Questions for authors
1.Is it possible to compare query-based decoding with clustering-based decoding while keeping the pre-trained backbone, input resolution, training data, and training budget constant? Furthermore, could you provide results obtained by excluding the EntityWorld-Game dataset or by controlling the scale of the training data, in order to disentangle the contributions of the architecture from those of the data? 2. What matching thresholds, matching constraints, and historical centroid update mechanisms are employed for cross-segment association? Could you report entity recall and identity association metrics specifically for sequences containing more than 200 distinct entities, as well as for challenging scenarios such as re-emergence after occlusion or the presence of entities with similar appearances? 3. EntityWorld incorporates data sources, such as Waymo, Hypersim, and Co3Dv2, that are also used for evaluation. Please specify the training/validation/test splits for each data source, as well as the methods used for deduplication at the scene or sequence level. The objective here is to clarify the evaluation isolation protocol, rather than to suggest that using the training split of a given dataset is inherently problematic.