- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduced a query-independent long-term visual memory based on the evidence per opportunity. It formulate retrospective open-vocabulary memory as censored inference over long-term object prevalence, explicitly separating semantic evidence from observation opportunity. And the paper proposed a benchmark called LOOP-Bench to test whether a robot can infer where an object usually occurs from repeated, uneven observations.
Critical strengths
-
The modeling of long-term object occurrence under uneven observation is motivated. A robot deployed over weeks or months should learn where a object usually occurs. Distinguishing between an object being absent and the robot lacking an opportunity to observe it is a good point.
-
LOOP-Bench provides a useful evaluation setting and helps study the effects of uneven observation. The real-robot experiments provide additional evidence of practical applicability, and the video in supplementary material helps illustrate the system’s behavior.
Critical weaknesses
- Retrospective querying currently incurs substantial latency. In the runtime analysis in appendix E, processing the stored 1,719 frames takes approximately 80 seconds per query on an RTX 5090. The appendix reports useful speed–accuracy trade-offs, but reducing the number of processed frames also reduces performance. Further analysis of how query latency scales with accumulated history would help assess its practicality during extended deployment.
Questions for authors
- Could authors extend the reported frame-reduction experiments to provide a more complete latency–performance curve and analyze query cost as the historical memory grows? It may also be useful to discuss whether frame selection or memory compression could keep latency manageable while preserving the benefits.
3 comments
Thank you for the detailed response and additional analyses. The comparison between opportunity recorded during mapping and opportunity recomputed from selected keyframes is particularly interesting, as it directly concerns how observation opportunity is estimated. It is the central part of the paper.
Overall, I like the paper’s perspective and its focus on learning long-term object occurrence under uneven observation. The additional analyses address my concerns, and I am inclined to maintain my positive score.
Thank you for your encouraging feedback. We appreciate your thoughtful comments and are glad the additional analyses addressed your concerns.
Extended runtime analysis
We thank the reviewer for this suggestion. We ran the three analyses and report them in the Runtime part of Appendix E, with the new Table 4 and Figure 6, and the main text summarizes them in Section 6.5. In brief, scoring only the 20% of the keyframes most similar to the query reduces the query time from 80 s to 16 s at the same AP, the query time grows much more slowly than the history, and caching the image tokens of the detector during mapping answers a query in 13 ms with a similar AP and search success.
Latency-performance curve. We extended the frame-reduction experiment from two settings to shares between 5% and 100% of the stored keyframes, selected by SigLIP similarity to the query, and we evaluated estimation and search at each share. In the protocol of the submission, which recomputes the observation opportunity from the depth of the selected keyframes, AP is .191 at 50% of the keyframes and .182 at 20%, compared with .216 for exhaustive scoring. We additionally tested a setting that keeps the opportunity recorded during mapping instead of recomputing it. This opportunity is measured from the depth of all mapping frames, so it does not decrease when fewer keyframes are scored. At the same query time, this setting gives a higher AP: .217 at 20% of the keyframes (16 s instead of 80 s) and .203 at 5% (4.2 s).
Query cost as the memory grows. Query time is proportional to the number of stored keyframes, and this number grows much more slowly than the history, because CROSS stores a keyframe only when its view is not already matched by the map. The first traversal of a home stores 437 keyframes on average (49% of its frames), whereas the traversals after the tenth store about 12 each. Exhaustive query time is therefore 20 s after one traversal, 62 s after five and 80 s after twenty, and each of these later traversals adds about 0.6 s. AP saturates in the same way (.063, .190 and .216, Figure 6c).
Frame selection and memory compression. Frame selection by retrieval works as described above. We additionally test the idea to compress the memory before any query is known: keeping the 20% of the keyframes that cover the most support surface leaves AP at .217, with SR .540. The largest gain comes from caching. OWLv2 computes the class embedding and the box of each image token from the image alone, and the query enters only through its text embedding. We therefore run the image side once, when a keyframe is stored (44 ms), and answer queries from the stored tokens. Given the same text embedding, the cached logits are identical to those of the detector for all 713 million token-prompt pairs of the benchmark. A query then takes 28 ms with all tokens, and 13 ms when we keep the 256 most object-like tokens per keyframe. This budget reduces the token store from 14 GB to 0.7 GB per home, with the same AP and a similar search success (SR .567, compared with .560 for exhaustive scoring on the same GPU). With cached tokens, each further traversal adds less than a millisecond to a query.
Query times are for one query on an RTX 5090, averaged over the ten homes. The last row repeats exhaustive scoring on the GPU that computed the cached tokens, because bfloat16 detector scores differ in their last bits between GPU models. This rounding alone changes AP from .216 to .214 and SR from .587 to .560, so we regard SR differences of two to three points as within the variation of the search.