acceptodds
Under review as a conference paper at ICLR 2027

Never Look Back: Understanding Persistence in 3D Object Memory From Egocentric Videos

Abstract

As we move through the world and carry out everyday tasks, we encounter objects that may become relevant only later. We are capable of recalling where we left something or what was inside a container, even without knowing we would need it later. Here, we study how an embodied assistant can build a similar memory from egocentric videos, by observing a person’s day-to-day activities. We present LEDGER, a persistent 3D object memory that combines object locations, their his- tories, and contextual descriptions. It associates observations across the recording and retains objects after they leave the view, including those the person never touches. It clusters each object’s observations by resting locations and records a move only after repeated evidence, reducing the effect of localization noise. Short descriptions preserve details such as an object’s contents or supported surface. It saves these records to later answer spatial questions without having to access the original images or video. Our memory raises HD-EPIC accuracy from 29.7% to 42.6%, UCS-Bench accuracy from 33.8% to 38.5% and localizes Ego4D objects with a 0.99 m median error on returned predictions. Our analyses identify complementary roles for temporal persistence, contextual descriptions, and retrieval. Our study on 100 stitched streams of multiple scenes each further exposes failures in both retrieval and construction. Per-scene construction partially recovers the performance lost across scene changes compared to that of single scene streams.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied Agents, Video Understanding, Scene Graphs, Object Memory

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ZXFyOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper has a reasonable setting, and the proposed persistent object memory also takes into account challenges such as uninteracted objects, motion changes, and contextual information, and has certain research value. However, the quality of the paper's presentation is poor, which seriously affects understanding. As a conference paper, it lacks condensation of the main core content of the article and instead scatters various details. Meanwhile, the main text heavily relies on supplementary materials, and the boundary between method innovation and engineering combination is also unclear.

Critical strengths

The problem has clear research value. The paper focuses on a very crucial but not yet fully resolved issue in embodied intelligence: how intelligent agents can retain the information of objects that are currently not important but may be queried in the future.

The motivation and module design are largely consistent. The retention of objects that have not been touched corresponds to the problem of memory coverage; repeated displacement evidence corresponds to positioning noise; context description makes up for the deficiency of only saving coordinates; historical records support cross-time querying.

Cross-task and cross-dataset validation is relatively sufficient. HD-EPIC, UCS-Bench, and VQ3D cover different issues such as position, motion, relationship, and positioning, which can reflect the applicable scope of this memory representation.

Critical weaknesses

The article structure lacks conciseness. As a conference paper, it fails to condense the core content of the article. Instead, it simply scatters various details. The main text heavily relies on supplementary materials, and readers need to consult the supplementary materials to fully understand the methods and experimental logic.

The innovation and engineering aspects of the method are confused. Detection, three-dimensional enhancement, object association, language description, and language model reading are mostly existing components. The current version has not fully explained the new methodological contributions of LEDGER or whether performance improvement is mainly achieved through engineering combination.

The experimental content is too scattered. Multiple datasets, scene stitching, answerers, and construction variants all have certain value, but currently they are more like a collection of separate experimental projects. We suggest organizing the main text around a core conclusion and moving the remaining results to the supplementary materials.

Questions for authors
  1. What is the paper's single central claim, and which parts of LEDGER constitute the novel methodological contribution according to the video-reasoning setting this paper concerns?

  2. Which experiments are intended to support the main claim, and which are exploratory analyses? In particular, how should readers interpret the roles of the multiple datasets, scene-stitching experiments, answering models, and construction variants in relation to the paper's central contribution?

  3. The main text should contain sufficient information for readers to grasp the paper’s core idea, reasoning, and conclusions without relying heavily on supplementary material.

5 comments
AuthorsOpenReview ↗

Response to reviewer ZXFy

We thank the reviewer for the feedback on the paper's organization. The weaknesses and the questions raise the same three points, so we answer them in pairs: Weakness 2 with Question 1, Weakness 3 with Question 2, and Weakness 1 with Question 3. We also describe the changes made in the revision.

Question 1 and Weakness 2. "What is the paper's single central claim, and which parts of LEDGER constitute the novel methodological contribution"

Central claim. A persistent 3D object memory lets a language model answer spatial questions that were not known in advance, without going back to the video. The memory needs three properties: (1) it keeps every object it sees, including after the object leaves view, (2) it keeps a history of where each object rested, and records a move only after repeated evidence, and (3) it stores a short description of the object's context at each resting place. With this memory and no images, GPT-5.4's accuracy improves from 29.7% to 42.6% on HD-EPIC and from 33.8% to 38.5% on UCS-Bench (Tables 2 and 3). The revised introduction now states this claim in one sentence.

What is new and what is existing. The reviewer is right that the detector, 3D lifter, tracker and language model are existing components. Our contribution is the object record these components fill: which objects are kept, how a move is told apart from localization noise, and what is stored for each resting place. The revised Figure 2 caption and the opening of Section 3 now state this division.

Weakness 2. "whether performance improvement is mainly achieved through engineering combination"

No. Most of the improvement comes from the record, and we measured this in three ways.

  • Removing a part of the record (Table 12, GPT-5.4). HD-EPIC accuracy is 42.8% with the full memory. It falls to 37.3% without the repeated-evidence rule for moves and to 37.9% without descriptions.
  • Keeping the components and changing only the record (new, Appendix I.4, Table 47, Qwen3.5-9B). With the same detections and the same retrieved objects, accuracy is 23.0% with no memory and 27.7% with a basic memory that only detects objects, places them in 3D and links them over time. Adding our record raises it to 35.3%.
  • Replacing the existing components (Table 48). Replacing the depth estimator, the retrieval method, the detector or the tracker changes accuracy by at most 2 points. Keeping only each object's last-seen position, without its history, lowers it by 7.4 points.

The full results, including the step-by-step table and the table of component replacements, are in our responses to Reviewer Qqob (comments 2 and 3).

Question 2 and Weakness 3. "Which experiments are intended to support the main claim, and which are exploratory analyses?"

Each experiment has one role, and the revised Section 4 now opens with a paragraph that gives this map:

ExperimentRoleWhere
HD-EPIC, 2,400 questionsMain test of the claim, and ablation of each part of the recordRQ1 to RQ3
UCS-Bench, 2,771 questionsTests the claim with estimated camera posesRQ1
VQ3D, 164 queriesTests that the stored 3D positions are accurateRQ3
Other memories rebuilt on the same dataCompares our record with other representationsTables 2 to 4
Several answering models and a shared readerControl: the gain does not depend on one answering modelRQ1, RQ4
Recording length and multi-scene streamsScope: shows where the claim stops holdingRQ5
Construction variantsSensitivity to component choicesAppendix B.6

Weakness 3. "organizing the main text around a core conclusion and moving the remaining results to the supplementary materials"

Section 4 is now organized around the claim. It has two parts. Section 4.1 tests the claim (RQ1 to RQ3). Section 4.2 gives its scope (RQ4 and RQ5). We kept these two scope analyses in the main text, in one paragraph each, because they state the limits of the claim: comparisons depend on the reader, and accuracy drops when a stream changes scene. Construction variants, results per question type, cost and failure analyses are in the appendices.

Weakness 1 and Question 3. "The main text heavily relies on supplementary materials" / "The main text should contain sufficient information for readers to grasp the paper's core idea, reasoning, and conclusions"

The revised main text can be read without the appendices.

  • The introduction states the central claim.
  • Section 3 states which parts are new and gives the rule for recording a move with its settings: a move is recorded after three consecutive observations more than 0.3 m from the resting place.
  • Section 4 opens with the map above, and each research question begins with a one-sentence answer.
  • The appendices now hold only details: protocols, per-question-type tables and additional analyses.

All changes are highlighted in yellow in the revised PDF.

AuthorsOpenReview ↗

Following up

We thank the reviewer again for their time and thorough review. We wanted to check whether our response sufficiently addressed their concerns. If there are any remaining questions, we would be happy to answer them until the discussion deadline on 18th November.

Sincerely,

Authors

AuthorsOpenReview ↗

Following up

We thank the reviewer once again for their feedback on the organisation of our paper. We have addressed all the points raised and clarified the key takeaways and conclusions drawn from our experiments to further strengthen the paper. With only about a day remaining in the discussion period, we would like to kindly ask whether the reviewer has any remaining concerns that we could address before the deadline.

Sincerely, Authors

Reviewer_QqobOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents LEDGER, a persistent 3D object memory built from long egocentric videos. The system stores object locations, movement histories, and contextual descriptions, including objects that are only passively observed, and later answers spatial questions without revisiting the original video. The paper evaluates the approach on HD-EPIC, Ego4D VQ3D, and UCS-Bench, and provides extensive analyses of persistence, retrieval, geometric noise, recording length, and scene changes. Overall, the paper studies not only whether memory helps, but also what information should be preserved for long-horizon spatial reasoning.

Critical strengths

A meaningful and well-motivated problem. Persistent spatial memory is important for embodied and egocentric assistants, especially when future queries are unknown during observation.

Thorough empirical evaluation. The paper evaluates on multiple benchmarks and compares against several relevant memory-based methods, including ReMEmbR, AMEGO, OSNOM, and DirectMe.

Strong diagnostic analysis. I particularly liked the analysis of localization noise versus actual object movement, as well as the study of failures across scene changes. These analyses provide useful insights beyond reporting final QA accuracy.

Clear decomposition of memory components. The ablations on persistence, contextual descriptions, geometry, and retrieval help explain where the gains come from.

Critical weaknesses

The conceptual novelty is somewhat incremental. Persistent object/spatial memory from long egocentric video has already been explored by several prior works. LEDGER mainly combines and systematically studies several previously investigated memory attributes rather than introducing an entirely new problem.

The method relies on a relatively complex offline pipeline. Detection, 3D lifting, tracking, pose estimation, descriptions, and LLM-based answering introduce many components, making it somewhat difficult to separate the contribution of the representation itself from improvements in the overall pipeline.

The evaluation is not fully online. Although timestamp filtering prevents explicit future observations from reaching the answerer, memory construction itself still uses the full recording. Therefore, the experiments do not fully establish causal streaming memory performance.

Questions for authors
  • Given closely related works such as OSNOM, AMEGO, ReMEmbR, DirectMe, and R4DSG, what do the authors consider the most fundamental new problem introduced by LEDGER, rather than a combination of previously explored memory attributes?

  • How much of the observed improvement comes specifically from the proposed persistent memory representation, versus the stronger detection, 3D localization, contextual description, and retrieval components used in the overall pipeline?

8 comments
AuthorsOpenReview ↗

Response to reviewer Qqob's comment 1

We thank the reviewer for the careful review and are happy to see that the reviewer finds our problem meaningful and our analysis and ablations rigorous. We answer each point in a separate comment below.

"the most fundamental new problem introduced by LEDGER"

To our knowledge, LEDGER is the first memory from long egocentric video to keep an explicit 3D record of moves for objects regardless of hand contact, including objects that leave view and later reappear. This raises two problems. First, an object must be recognized when it reappears, without continuous tracking. Second, localization noise is as large as many real moves: half of annotated moves are under 0.8 m, while the 95th percentile of localization error is 0.84 m, so a distance threshold cannot separate the two (RQ2, Section 4). No prior memory addresses both, or either one for objects the wearer never touches.

We address the first with re-identification in two stages. SAM 3 is the strongest appearance matcher we tested, but running it on every detection is costly: even when run only on fragments, it takes 68% of the online build time (Appendix I.3). We therefore first link observations cheaply by label and 3D distance, then use SAM 3 to merge the fragments that step leaves (Section 3.1, Appendix A.3). We address the second with persistence: a move is recorded only after repeated evidence.

How prior memories fall short.

  • OSNOM, AMEGO and Whareformer re-identify objects, but only objects in hand contact. Untouched objects never enter memory, so problem 1 is never solved for them. Moves are tied to contact, so problem 2 never arises.
  • Embodied VideoAgent keeps untouched objects, but each holds one current box, smoothed by a moving average. A move is flagged from a single view when the object is not where its box predicts. No history of resting positions is kept, so it can tell that an object moved, but not where it rested before or how often it moved (problem 2).
  • DirectMe follows objects with SAM 2 tracks; re-identification across gaps is an optional check whose criteria are not given (problem 1). It optionally pools each object's positions into one location to reduce jitter, with no rule that separates noise from a move (problem 2).
  • R4DSG records moves only relative to anchor objects, and MEMORA only as text. Neither keeps 3D positions over time, so neither can filter noise from moves.

What LEDGER adds:

  • Re-identification: SAM 3 beats the DINOv2 features OSNOM uses, AllTracker and MARCO (Appendix I.6, Table 49).
  • Persistence: it adds 5.5 points on HD-EPIC (Table 12) and nearly halves the error in counted moves (Appendix I.5).
  • Keeping untouched objects: it adds about 3 points overall, almost all on fixture-location questions (Appendix I.4).
AuthorsOpenReview ↗

Response to Reviewer Qqob's comment 2

"How much of the observed improvement comes specifically from the proposed persistent memory representation"

The improvement comes mainly from the two parts of the representation that we propose: persistence and descriptions. Persistence means that a move is recorded only after repeated evidence, so each object keeps a history of its resting places. Descriptions are short texts attached to each resting place, such as the object's contents or the surface it rests on. We show this with the ablations already in the paper and with a new experiment added in the revision.

1. Ablations in the paper (Appendix B.5, Tables 11 and 12). With GPT-5.4 as the answering model, HD-EPIC accuracy is 29.5% with no memory and 42.8% with the full LEDGER memory. (These ablations use 2,317 of the 2,400 questions, so the figures differ slightly from the 29.7% and 42.6% in Table 2.) To find which parts of the memory produce this gain, Table 12 removes one part at a time and answers the same questions again:

  • Persistence: accuracy is 37.3% without it and 42.8% with it, a gain of 5.5 points.
  • Descriptions: accuracy is 37.9% without them and 42.8% with them, a gain of 4.9 points.
  • For comparison, more precise 3D positions (multi-view triangulation): accuracy is 40.5% without it and 42.8% with it, a gain of 2.2 points.

The effect of persistence is largest where we expect it. On questions that ask how many times an object was moved, accuracy is 29.5% without persistence and 45.0% with it (Table 11).

2. New experiment that isolates the representation (Appendix I.4, Tables 47 and 48). The ablations above remove one part from the full system. To measure what the representation adds on top of the existing components, we also built the memory in steps, with Qwen3.5-9B as the answering model (1,899 questions from 123 videos). Every step uses the same detections and gives the answering model the same retrieved objects, so only what the memory stores changes (Table 47):

Memory given to the answering modelAccuracy
No memory23.0%
Basic memory: objects detected, placed in 3D and linked over time, without persistence or descriptions27.7%
Basic memory + persistence30.9%
Basic memory + persistence + descriptions35.3%

The existing components alone raise accuracy from 23.0% to 27.7%. Adding our representation raises it further, from 27.7% to 35.3%. Section 4 (RQ2) now reports this result in the main text.

We also ran the reverse test (Table 48). We kept the final memory (35.2%) but replaced each object's history with only its last-seen position. Accuracy falls by 7.4 points, to about 27.8%, which is back at the level of the basic memory.

3. The choice of perception component matters less (Table 48). In the same experiment, replacing the depth estimator, the retrieval method, the detector or the tracker with an alternative changes accuracy by at most 2 points. Only estimated camera poses cost more, 2.5 to 4.8 points. Table 13 shows where this cost falls: with GPT-5.4, it is mainly on fixture-location questions. These questions ask for a direction relative to the camera, so the answer depends on the camera's heading (RQ3, Section 4).

This does not mean perception is solved. Using ground-truth objects, positions and resting fixtures in place of the ones estimated by the perception components, accuracy rises by 15.2 points, from 35.2% to 50.4% (Table 48). Perception remains a bottleneck, but the components available today perform alike, so the gain we report comes from what the memory stores.

The 95% confidence intervals for all of these results are in Tables 12, 47 and 48.

AuthorsOpenReview ↗

Response to Reviewer Qqob's comment 3

"difficult to separate the contribution of the representation itself"

We separate the contribution of the representation from the contribution of the rest of the pipeline components with two experiments. In the first, we keep the representation and replace one pipeline component at a time (detector, depth, tracking, camera poses, retrieval, answering model). In the second, we keep the pipeline and change only the representation. Replacing a pipeline component changes accuracy little, with one exception (camera poses). Changing the representation changes it much more.

1. Same representation, different pipeline components (Appendix B.6, Table 13). The final system reaches 42.6% on HD-EPIC with GPT-5.4 as the answering model (all 2,400 questions). Table 13 replaces one component at a time and answers the same questions again:

Component replacedAlternatives testedAccuracy (final system: 42.6%)
Depth used for 3D liftingthree alternatives42.0% to 44.7%
Retrieval of objects for a questionfour alternatives39.6% to 42.0%
Camera posesestimated by FastVGGT, gravity-levelled40.8%
Camera posesestimated by FastVGGT, not levelled39.1%
Answering modelGPT-5.645.0% (27.2% without memory)
  • Depth and retrieval: no replacement changes accuracy significantly after correction for multiple comparisons.
  • Answering model: the memory helps both models. GPT-5.4 rises from 29.7% to 42.6% and GPT-5.6 from 27.2% to 45.0%.
  • Detector and tagger: on the 418 calibration questions, where the final system scores 33.4%, a SAM 3 detector gives 36.3% and two other taggers give 32.8% and 34.9%.
  • Tracking: removing the SAM 3 step that merges duplicate tracks changes accuracy by 1.1 points, with no difference detected (Qwen3.5-9B; Appendix I.4, Table 48).
  • Camera poses are the exception. With estimated poses, accuracy on fixture-location questions falls from 56.0% to 44.6%, and to 26.6% without gravity levelling. These questions ask for a direction, which depends on the camera's heading (RQ3, Section 4). All other HD-EPIC results use the poses supplied with the dataset.

2. Same pipeline, different representation (Tables 12, 47 and 48). Descriptions and persistence are the parts of the representation we propose, so we test them by removing them and keeping every other component fixed:

  • With GPT-5.4 (Table 12): removing persistence lowers accuracy from 42.8% to 37.3%, and removing descriptions lowers it from 42.8% to 37.9%.
  • With Qwen3.5-9B (Appendix I.4, Table 47): using the same detections and the same retrieved objects, a memory without persistence or descriptions reaches 27.7%. Adding both raises it to 35.3%.
  • Keeping only each object's last-seen position lowers the final memory's accuracy by 7.4 points (Table 48). This is more than any component replacement in the same experiment, where the largest is 4.8 points, for estimated poses.

A clarification on Table 13. The retrieval row previously labelled "3D IoU" ranks objects by 3D box-centre distance on 140 of the 150 videos, because 3D box files exist for only 10. On those 10 videos, ranking by true 3D IoU gives no detectable change in accuracy. The revision relabels the row.

The 95% confidence intervals and significance tests are in Appendix B.6 (Table 13) and Appendix I.4 (Tables 47 and 48). The caption of Figure 2 and the opening of Section 3 in the revision now state which parts are existing components (the detector, 3D lifter, tracker and language model) and which part is new (the object record with its rest segments and descriptions).

Reviewer_w9GwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LEDGER, a persistent 3D object memory for VLMs. Given an egocentric video, LEDGER combines object locations, histories, and contextual descriptions into the memory, and in the inference time, model retrieve the memory. LEDGER raises HD-EPIC accuracy from 29.7% to 42.6%, UCS-Bench from 33.8% to 38.5%.

Critical strengths
  1. Separate memory construction and query-time reasoning. They define the problem well and eliminate the "look back" short cut in many VLM evaluations.
  2. Analysis is thorough, very detailed analysis on which component might affect which score.
Critical weaknesses
  1. The method overall sounds like a online construction + rag-like framework. And the online construction is unlimited, which means if the videos is very long, they don't have a mechanism to avoid storage overflow.
  2. Object identity association is still weak. Persistent object memory needs to recognize the same object across different space, but in LEDGER, it's basically relying on label, spatial distance, and tracker consolidation.
Questions for authors

See weakness.

6 comments
AuthorsOpenReview ↗

Response to Reviewer w9Gw

We thank the reviewer for their feedback and are happy to see that the reviewer finds our problem impactful and our analysis rigorous. We address the concerns below.

"The method overall sounds like a online construction + rag-like framework."

While the overall pipeline does resemble a RAG-based language model answering based on retrieval of relevant records from online constructed memory (Sec 3.2), we would like to clarify that our key methodological contribution here is how the memory is built and what each record stores (Sec 3.1).

  • Objects (and their 3D positions over time) enter memory whether or not the wearer touches them since they’re discovered with an open vocabulary, and each detection is lifted to 3D in the scene frame using the camera poses.
  • Records persist after objects leave view. Observations are linked by label and 3D distance, and the tracker pass re-joins objects that were split into several object instances.
  • Each record is a history of rest segments. A move is recorded only after repeated evidence of displacement (persistence), so 3D noise is not mistaken for motion, and resting positions are refined by multi-view triangulation.
  • Each rest segment carries a description of the object in its surroundings, such as its contents, supporting surface and interactions.

What the record holds matters more than how it is fetched. Specifically, on HD-EPIC, we see that removing our persistence design or the descriptions costs about 5 points each, while swapping the retrieval method costs at most 3 (Tables 12 and 13). This is the motivation behind our focus on memory construction.

Section 3 now opens by stating which parts are existing components and what is new. Our responses to Reviewer Qqob give the full evidence for this point (comments 2 and 3) and describe the online construction in detail (comment 4).

"they don't have a mechanism to avoid storage overflow"

We find that the memory storage growth is small and linear. We evaluated memory growth per hour of video across HD-EPIC, UCS-Bench subsets (EgoLife, TeleEgo, HourVideo) and the stitched streams (482 recordings, 136 hours), counting every memory storage file the LLM answerer reads from: object records, camera path, relations and places. Here, the mean growth is 11.4 ± 3.9 MB per hour as stored, 6.9 ± 1.8 in compact form and 1.0 ± 0.4 gzipped (Appendix I.1, Table 43). To check that growth does not speed up on longer videos, we took 8 EgoLife recordings of about 12 minutes each and rebuilt each memory with up to 2 hours of the wearer's earlier footage from the same day placed before it. Memory size grows at a steady rate, about 14 MB per hour, from about 3 MB to about 30 MB.

While the proposed method has no explicit cap on storage, our new experiment (Appendix I.2, Table 45) evaluates the performance under a storage budget for very long videos. Note that when the memory exceeds its budget, the tracking does not stop. Instead, we (1) compact it, dropping whitespace and fields the reader never reads; (2) for observations made more than 12 minutes before the latest one, keep only the first, middle and last observation of each consecutive run within a rest segment, so every segment keeps its position, time span and description, and keep one camera pose every 6 seconds; and (3) if it is still over budget, delete objects not seen in the last 12 minutes (configurable), least-observed first. This uses only stored data, so it can run during a stream. On the 2-hour streams above, shrinking the memory from about 17 MB to as little as 2.4 MB lowers Qwen3.5-27B accuracy by at most 2.5 points, and none of these changes is statistically significant; Qwen3.5-9B shows no drop either.

AuthorsOpenReview ↗

Continuing response to Reviewer w9Gw

"Persistent object memory needs to recognize the same object across different space"

We agree that this is a hard case, and it contains a basic ambiguity on which even humans can make errors. When an object is seen at place A and later at place B, and the two sightings never appear in the same frame, appearance alone cannot tell whether one object was moved or a second, identical object stands at B. LEDGER resolves this conservatively. A co-occurrence check keeps apart objects seen in the same frame, and a distance check keeps two sightings apart if they were more than 1 m apart at the moment they were seen closest in time, treating the second as a possible duplicate (Section 3.1, Appendix A.3). This check is a tunable setting, not a limit of the representation. Appendix D.2 shows that removing it on 12 UCS-Bench recordings does not lower accuracy. Without it, objects are matched across any distance by appearance alone, so the quality of the appearance matcher becomes decisive.

We therefore tested SAM 3 as an appearance matcher in a new experiment (Appendix I.6, Table 49). We took views of handled objects from HD-EPIC, whose annotations say which views show the same physical object, shuffled them so that no time order was available, kept similar-looking objects in, and gave every method only the images, with no labels or 3D. Each tracking/re-id method scores every pair of views, the same grouping step (average-link clustering) turns these scores into objects, and we score the grouping by pairwise F1.

  • SAM 3 is the strongest appearance matcher we tested. It reaches an F1 of 48, against 35 for DINOv2 (the appearance features OSNOM uses), 33 for AllTracker (on the 38 videos it could process, with time order given) and 18 for MARCO. It is also the most accurate on the hard cases: views at different fixtures, after the object moved more than 1 m, more than 5 minutes apart, and among lookalike objects. All of these differences are statistically significant. DINOv2 and MARCO are slightly more often right about the single closest view.
  • Removing the distance check works with SAM 3. Without the check, SAM 3 joins 19% of view pairs of objects that moved more than 1 m, against almost none with it, and F1 rises from 47 to 48 (Appendix I.6, Table 50). The cost is slightly more merges between lookalike objects (11% of such pairs, against 9%).

The revision adds this comparison and the distance-check ablation in Appendix I.6, and Section 3.1 now states that the 1 m check is a tunable setting.

"basically relying on label, spatial distance, and tracker consolidation"

We would like to clarify that our tracker consolidation, in fact, uses the SAM 3 video tracker, which re-identifies duplicate instances of an object in the memory by appearance. Specifically, it seeds one instance's mask and propagates it over one frame from each other instance, so fragments merge even when those frames are far apart in time. Co-occurrence and 3D checks prevent merging distinct objects (Section 3.1, Appendix A.3).

AuthorsOpenReview ↗

Following up

We thank the reviewer again for their time and careful review. We wanted to check whether our response addressed their concerns. If there are any remaining questions, we would be happy to answer them until the discussion deadline on 18th November.

Sincerely,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.