- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper formulates Memory-Grounded Affordance Prediction and introduces MemAff-Bench, where a system must retrieve a target observation from long visual memory based on a natural-language intent and output an object mask. The authors also propose MemAff, a training-free pipeline combining multi-branch retrieval, visual reranking, and open-vocabulary segmentation. The contributions are evaluated across multiple memory sizes and real-world data sources using long-context and affordance baselines, larger-model comparisons, ablations, and error decomposition. The results show that MemAff achieves competitive end-to-end performance with substantially lower online context usage and identify candidate retrieval as the main bottleneck at scale.
Critical strengths
-
The task and benchmark address a meaningful gap. The paper jointly evaluates retrieval from long-term visual memory and pixel-level affordance grounding, bridging the gap between single-image affordance prediction and long-video tasks that return only timestamps, clips, or textual answers.
-
The experiments provide strong diagnostic value. Beyond comparing aggregate performance, the paper uses model scaling, retrieval ablations, and error decomposition to show that context capacity is not the only limitation and that candidate recall is the dominant bottleneck in large visual memories. This conclusion is more informative for future work than a benchmark improvement alone.
Critical weaknesses
-
The strict exact-frame metric does not fully capture semantically valid task success. The same object may appear in several adjacent frames or from multiple usable viewpoints, yet the current metric assigns credit only when the uniquely annotated frame is selected. The authors also note that larger memories may contain a second plausible referent. The metric is therefore useful as a strict retrieval diagnostic but may underestimate effective performance in realistic embodied settings. A complementary evaluation that accepts the same object instance, a temporal neighborhood, or multiple valid targets would substantially improve the benchmark’s interpretability.
-
MemAff does not yet resolve the scalability challenge identified by the paper. The system primarily combines existing frozen components. Its simplicity, transparency, and training-free design make it a useful reference baseline, but performance still declines markedly as memory grows, with many targets already lost during coarse retrieval. MemAff should therefore be viewed as a baseline rather than a complete solution to memory-grounded affordance prediction. The paper’s strongest contributions lie in the task formulation, benchmark, and diagnostic analysis, while the technical novelty of the method itself is more limited.
Questions for authors
-
Could the authors provide a complementary evaluation on a representative subset that accepts other clear views of the same object instance, a temporal window around the annotated target frame, or multiple human-verified valid referents? Does the relative ordering of the evaluated systems remain unchanged under such a protocol?
-
The reported online token cost excludes the offline construction of affordance inventories and visual representations. Could the authors report the full indexing cost, cache requirements, and amortized cost under different numbers of queries to clarify the settings in which MemAff offers an overall efficiency advantage?
Response to Reviewer yNJj
Thank you for highlighting the task formulation and the separation of retrieval and localization errors.
Weakness 1 and Q1 · Alternative answers and ranking. We audited the final frames returned by five systems on 35 questions at 120/240/480/960 frames (15 PAP, 10 EPIC, 10 Ego4D). The annotator credited frames that visibly satisfy the instruction, including alternate views or suitable objects. Uncertain selections received zero. System identities were hidden initially and visible for a few later checks. Five PAP questions were the first completed items from a 20-question extension.
MemAff’s exact hits are 9/8/3/3 and its human-confirmed hits are 27/30/27/29, out of 35 at each size. SigLIP with the same reranker goes from 8/5/5/4 to 24/26/25/27. Their relative order reverses at 480 and 960 on the audited questions. At 480 the confirmed gap is two questions (exploratory exact McNemar p = 0.69). One-shot Qwen leads confirmed frame hits at 120 by one (28 vs. 27). MemAff has the most confirmed hits among systems with outputs at 240, 480, and 960.
With human-checked reference masks for valid alternative frames, end-to-end mIoU leads change from Qwen/Qwen/SigLIP/SigLIP under strict scoring to MemAff/SigLIP/MemAff/MemAff. The MemAff–SigLIP gap is 0.28 percentage points at 480. Some methods lack outputs at larger sizes. The confirmed hit counts and mIoU of MemAff and SigLIP fluctuate as memory grows on these 35 questions (Appendix F, Tables 13–14 and Figure 10).
Weakness 2 · Contribution and scaling. MemAff is a training-free reference baseline assembled from frozen components. Our contribution is the task, the benchmark, and the diagnosis of context capacity, frame selection, and localization. Across the full fixed cohorts, scale remains a challenge for recovering the exact annotated frame. The human audit gives complementary evidence about instruction satisfaction on a limited set of questions.
Q2 · Indexing and reuse. On 3,597 nonempty 120-frame questions on H100s, MemAff averages 62.456 s to index and 0.439 s per online query. SigLIP with the same reranker averages 1.707 s and 0.401 s. Persistent index sizes are 1.628 and 0.234 MiB, respectively. Amortizing those measured times over 1/10/100 queries yields 62.895/6.684/1.063 s per question for MemAff and 2.108/0.572/0.419 for SigLIP. The other per-question times are 2.642 s for one-shot Qwen, 4.544 s for A4-Agent, and 18.851 s for R1. Under this measured-means model, MemAff becomes faster from 29, 16, and 4 uses, respectively. SigLIP remains faster in this model. Measured generation-token and index-storage details are in Appendix I, Table 17 and Figure 11.