- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper presents different evaluations of dynamic object identification, location, and pose in indoor, and generated, scenes, using a new way to reason about sameness. One such problem is the case where an object has undergone a hidden relocation, and then observed again; should it get a new ID, i.e., it is seen as a new object, or reuse the id of an already seen object, i.e., it has moved. The paper shows that the established metric HOTA will increase when using new ID on new objects, and when using old ID on moved objects, and otherwise decrease the score. In some cases it is impossible by observations to know which one case it is, and therefore using a metric susceptible to this in a benchmark may misrepresent the method under study. LIMBO determines how much information is available and what a method may know about the environment, which could be used to evaluate methods in a more fair way.
Also, using the grammar and the compatible groups, two new metrics are defined that are used together in an evidential F1 score. The results show that a quite small part of 3RScan contain ambiguities.
Critical strengths
A way to determine how much information is actually available in a given moment seems necessary to better judge how much can be expected from a method in this moment. In this paper it is accomplished by formalizing what has happened with an object over time by a grammar to express evidence contracts.
An evaluation seems to be done on the rate of scenes/worlds that shows ambiguousness which can influence the well established metrics like HOTA.
Critical weaknesses
It is difficult from the Results section to understand how LIMBO best is used in helping to evaluate trackers, mappers, etc. There are some numbers presented which may indicate that ambiguous interpretation of the environment may happen in around 5% of the scenes. Does this number indicate that methods having similar scores cannot determine a “winner” because the methods could have been “lucky”/“unlucky” to various degrees in these ambiguous scenes?
The presentation of the paper could be improved. More examples would be useful to introduce the problem. Describing use cases of LIMBO early in the paper and then using these use cases in the results section would improve the reading of the paper. Some things are introduced without proper definitions like evaluation contracts (why contracts, what makes them contracts. A contract guarantees something), typed evaluation protocol (why typed, what does it mean, and why protocol, what does it mean here). finite claim language (what means finite claim here). typed map claims needs to be explained (what means typed here. What means claims here).
The evidence ladder needs to be described in a clearer way. An introductory paragraph in section 5 describing what can be expected from the experiments would be helpful.
Figure 3 is not referenced anywhere in the text.
Questions for authors
Following my "Critical Weaknesses" assessment, the main question that the authors should clarify in the paper is: how is LIMBO used in helping to evaluate trackers, mappers, etc.? Additionally, I recommend the authors to carefully look at the "Critical Weaknesses" and address them.
4 comments
Thank you again for your constructive feedback. We would like to kindly follow up on our response and the revised manuscript.
Regarding your main concern about LIMBO's practical use, we have clarified three concrete evaluation workflows in Section 5. In particular, our new experiments show that reference selection can reverse method rankings in 29 of 120 comparisons under HOTA, illustrating why accounting for ambiguous histories matters in practice.
We have also substantially revised the presentation, including clearer definitions, additional explanations of practical use cases, and a reorganized experimental section to improve readability.
We hope these revisions help address your concerns. We would greatly appreciate any further feedback, especially whether our clarifications have resolved your main questions.
Thank you for your time and consideration.
Thanks to the authors for the revision of the document and consideration of the feedback.
It is still unclear if the possibility of metric flips (e.g., HOTA) will have a chance of changing the order of evaluated methods and if evaluating under this new way will affect the order. E.g., is there a chance that a list of state of the art methods is "wrong" and an evaluation under LIMBO changes the order?
To improve the readability it would be good to visualize how the different terms relate to each other and how they are used in the method, e.g., by extending Figure 1 or by introducing a new figure. Such a visualization could relate the terms via examples.
A new careful read through shows that line 298 says “We also evaluate the authors’ released GLASSES and …”, which could be interpreted as giving away the authors of the paper, but perhaps the sentence is misunderstood by the reviewer.
We sincerely thank you for re-reading our revised manuscript so carefully and for your constructive follow-up questions! Below we address all three points directly.
1. Can HOTA/accuracy be "wrong" and does evaluating under LIMBO actually change the ranking order of SOTA methods? Yes—evaluating under LIMBO directly reverses the ranking order of state-of-the-art mechanisms and released systems in both benchmarks:
- Example A — SOTA Mechanism Ranking Reversal (Table 9, Page 16 & Table 13, Page 18): Consider comparing the SOTA active/missing object library mechanism (GLASSES, Bogenberger et al., 2026) against Track-continuity and Hard replace on held-out ProcTHOR groups:
- Under conventional HOTA (wrongly favoring overreach):
Hard replace(0.977) andTrack-continuity(0.977) outrankGLASSES(0.864). Why? Because single-reference HOTA rewards guessing new IDs on replacement histories, ignoring thatHard replacecommits 38.1% unsupported history errors. - Under LIMBO Evidential F1 (order reversed): The ranking flips completely.
GLASSES(0.895, rising to 0.974 withLIMBO Scopeat 0.0% unsupported error) decisively outranks bothTrack-continuity(0.811) andHard replace(0.603), because LIMBO credits GLASSES for evidence-supported free-space clearing (+0.809 F1 gain on ray contracts; Table 13) and penalizes unsupported identity guessing.
- Under conventional HOTA (wrongly favoring overreach):
- Example B — Released 3RScan System Ranking Reversal (Table 4, Page 9 & Table 45, Page 33): Comparing the released ConceptGraphs (Gu et al., 2024) and GLASSES RGB-D pipelines across 2,789 targets:
- Under standard association accuracy:
ConceptGraphsranks 1st (100.0%) andGLASSESranks 2nd (78.0%–81.6%), because ConceptGraphs emits only 3 ultra-conservative stationary matches (0.1% recall). - Under LIMBO Evidential F1 (order reversed):
GLASSESranks 1st (0.082, and 0.083 withGLASSES + LIMBO Scope, retaining all 237 locations with 0 unsupported claims) whileConceptGraphsdrops to last (0.001).
- Under standard association accuracy:
2. Visualizing How the Core Terms Relate via the Figure 1 Chair Example Thank you for this great suggestion! We have extended Figure 1 (and its caption walkthrough) to explicitly map each core term onto the concrete chair example:
- Evaluation Contract [Blue box, Fig. 1]: Pairs the public sensor input (calibrated depth C, mask D, pose, opaque handle
mfor the chair) with the admitted physical grammar (which allows both hidden relocation and twin replacement). - Compatibility Group [Green & Orange boxes, Fig. 1]: All physical worlds matching under —here, World 1 (same chair
mmoved) and World 2 (chairmremoved, identical chairotherborn at destination). - Finite Claim Language [Predicates inside boxes]: The finite set of candidate ground atoms instantiated for this chair:
{AT(dest), SAME_LINEAGE(m), SAME_LINEAGE(other), REMOVED(m), BORN(other)}. - Typed Evaluation Protocol & Supported Typed Map Claims [Yellow "Intersection" box, Fig. 1]: Compiles mapper outputs into spatial realization claims (
AT(dest)) vs. entity lifecycle claims (SAME_LINEAGE,BORN,REMOVED), and intersects World 1 and World 2 truths. OnlyAT(dest)survives in the intersection (guaranteed true in every compatible history), while lineage/birth/removal remain unresolved (?).
3. Clarification on Line 298 ("the authors' released GLASSES...") Thank you for catching this phrasing! We are NOT the authors of GLASSES or ConceptGraphs—by "the authors' released GLASSES and ConceptGraphs...", we meant "the official open-source implementations publicly released by the respective authors of those cited papers (Bogenberger et al., 2026; Gu et al., 2024)." To prevent any ambiguity regarding double-blind review, we have updated Line 298 to read: "We also evaluate the official publicly released GLASSES and ConceptGraphs RGB-D mapping components ..."
We thank you for your positive assessment of our soundness (Rating: 3) and core motivation, and for your constructive feedback on improving the presentation. In the uploaded revision, we have restructured the text, added upfront definitions, and clarified the practical workflows to address Q1-Q4 from your review of the original submission.
Q1 (Main Question & W1): How is LIMBO used in helping to evaluate trackers and mappers, and what does the ~5% ambiguity rate imply for determining a "winner"? We reorganized the opening of Section 5 (Page 6) around three concrete workflows showing how practitioners use LIMBO:
GLASSES + LIMBO Scope, Table 4) strips out 100% of unsupported lineage claims (31 to 0) while preserving 100% of matched 3D object locations (237/237) (see also Table 9 across 16 tracker configurations).Q2 (W2): Clear Upfront Definitions of Core Terminology In revised Sec. 1 (Page 1) and Sec. 3.1 (Pages 3-4), we added bolded definitions for all four terms that lacked clear introductions in the original submission:
AT(r,p),ABSENT_AT(r,p)—where an observed footprint is located or cleared now) and persistent entity lifecycle predicates (SAME_LINEAGE(r,e),BORN(e),REMOVED(e)—which physical lineage an observation belongs to and whether an entity entered or left the map).Q3 (W3): Clearer Evidence Ladder and Section 5 Introductory Paragraph
Q4 (W4): Missing Reference to Figure 3 Fixed in the revision: Figure 3 is now explicitly referenced and discussed in Sec. 5.4 (Line 418) and Sec. 6 (Line 517).