acceptodds
Under review as a conference paper at ICLR 2027

MemAff: Retrieving and Grounding Affordances from Visual Memory

11 views

Abstract

Affordances bridge high-level task reasoning and low-level physical action by identifying what objects enable and where an agent can interact with them. However, existing affordance prediction methods typically assume that the task-relevant object is visible in the current image. This assumption breaks for embodied agents with persistent visual experience: an instruction may refer to an object seen much earlier, requiring the agent to search its visual memory before determining where to act. We formulate this setting as Memory-Grounded Affordance Prediction and introduce MemAff-Bench, a benchmark with 3,628 tasks drawn from egocentric videos and indoor panoramas. Given an intent-style instruction and a sequence of past observations, an agent must retrieve the relevant observation and localize the target object at the pixel level. By systematically increasing the amount of visual memory, MemAff-Bench reveals that existing systems struggle to identify task-relevant observations as memory grows, even when long-context models can technically process the full sequence. To address this challenge, we further develop MemAff, a training-free framework that retrieves promising observations, visually reranks them, and grounds the target object. MemAff matches or exceeds the frame accuracy of a larger Qwen3-VL-32B single-pass model while using one to two orders of magnitude fewer prompt tokens per query. Our analysis also identifies that reliable retrieval among many visually similar observations is still a central challenge, establishing memory-grounded affordance prediction as an important direction for embodied agents that must reason and act over long-term visual experience.

Primary area:
datasets and benchmarks
Keywords:
benchmark, embodied agents, affordance, visual memory, retrieval, egocentric vision

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_yNJjOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper formulates Memory-Grounded Affordance Prediction and introduces MemAff-Bench, where a system must retrieve a target observation from long visual memory based on a natural-language intent and output an object mask. The authors also propose MemAff, a training-free pipeline combining multi-branch retrieval, visual reranking, and open-vocabulary segmentation. The contributions are evaluated across multiple memory sizes and real-world data sources using long-context and affordance baselines, larger-model comparisons, ablations, and error decomposition. The results show that MemAff achieves competitive end-to-end performance with substantially lower online context usage and identify candidate retrieval as the main bottleneck at scale.

Critical strengths
  1. The task and benchmark address a meaningful gap. The paper jointly evaluates retrieval from long-term visual memory and pixel-level affordance grounding, bridging the gap between single-image affordance prediction and long-video tasks that return only timestamps, clips, or textual answers.

  2. The experiments provide strong diagnostic value. Beyond comparing aggregate performance, the paper uses model scaling, retrieval ablations, and error decomposition to show that context capacity is not the only limitation and that candidate recall is the dominant bottleneck in large visual memories. This conclusion is more informative for future work than a benchmark improvement alone.

Critical weaknesses
  1. The strict exact-frame metric does not fully capture semantically valid task success. The same object may appear in several adjacent frames or from multiple usable viewpoints, yet the current metric assigns credit only when the uniquely annotated frame is selected. The authors also note that larger memories may contain a second plausible referent. The metric is therefore useful as a strict retrieval diagnostic but may underestimate effective performance in realistic embodied settings. A complementary evaluation that accepts the same object instance, a temporal neighborhood, or multiple valid targets would substantially improve the benchmark’s interpretability.

  2. MemAff does not yet resolve the scalability challenge identified by the paper. The system primarily combines existing frozen components. Its simplicity, transparency, and training-free design make it a useful reference baseline, but performance still declines markedly as memory grows, with many targets already lost during coarse retrieval. MemAff should therefore be viewed as a baseline rather than a complete solution to memory-grounded affordance prediction. The paper’s strongest contributions lie in the task formulation, benchmark, and diagnostic analysis, while the technical novelty of the method itself is more limited.

Questions for authors
  1. Could the authors provide a complementary evaluation on a representative subset that accepts other clear views of the same object instance, a temporal window around the annotated target frame, or multiple human-verified valid referents? Does the relative ordering of the evaluated systems remain unchanged under such a protocol?

  2. The reported online token cost excludes the offline construction of affordance inventories and visual representations. Could the authors report the full indexing cost, cache requirements, and amortized cost under different numbers of queries to clarify the settings in which MemAff offers an overall efficiency advantage?

1 comment
AuthorsOpenReview ↗

Response to Reviewer yNJj

Thank you for highlighting the task formulation and the separation of retrieval and localization errors.

Weakness 1 and Q1 · Alternative answers and ranking. We audited the final frames returned by five systems on 35 questions at 120/240/480/960 frames (15 PAP, 10 EPIC, 10 Ego4D). The annotator credited frames that visibly satisfy the instruction, including alternate views or suitable objects. Uncertain selections received zero. System identities were hidden initially and visible for a few later checks. Five PAP questions were the first completed items from a 20-question extension.

MemAff’s exact hits are 9/8/3/3 and its human-confirmed hits are 27/30/27/29, out of 35 at each size. SigLIP with the same reranker goes from 8/5/5/4 to 24/26/25/27. Their relative order reverses at 480 and 960 on the audited questions. At 480 the confirmed gap is two questions (exploratory exact McNemar p = 0.69). One-shot Qwen leads confirmed frame hits at 120 by one (28 vs. 27). MemAff has the most confirmed hits among systems with outputs at 240, 480, and 960.

With human-checked reference masks for valid alternative frames, end-to-end mIoU leads change from Qwen/Qwen/SigLIP/SigLIP under strict scoring to MemAff/SigLIP/MemAff/MemAff. The MemAff–SigLIP gap is 0.28 percentage points at 480. Some methods lack outputs at larger sizes. The confirmed hit counts and mIoU of MemAff and SigLIP fluctuate as memory grows on these 35 questions (Appendix F, Tables 13–14 and Figure 10).

Weakness 2 · Contribution and scaling. MemAff is a training-free reference baseline assembled from frozen components. Our contribution is the task, the benchmark, and the diagnosis of context capacity, frame selection, and localization. Across the full fixed cohorts, scale remains a challenge for recovering the exact annotated frame. The human audit gives complementary evidence about instruction satisfaction on a limited set of questions.

Q2 · Indexing and reuse. On 3,597 nonempty 120-frame questions on H100s, MemAff averages 62.456 s to index and 0.439 s per online query. SigLIP with the same reranker averages 1.707 s and 0.401 s. Persistent index sizes are 1.628 and 0.234 MiB, respectively. Amortizing those measured times over 1/10/100 queries yields 62.895/6.684/1.063 s per question for MemAff and 2.108/0.572/0.419 for SigLIP. The other per-question times are 2.642 s for one-shot Qwen, 4.544 s for A4-Agent, and 18.851 s for R1. Under this measured-means model, MemAff becomes faster from 29, 16, and 4 uses, respectively. SigLIP remains faster in this model. Measured generation-token and index-storage details are in Appendix I, Table 17 and Figure 11.

Reviewer_67G2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose Memory-Grounded Affordance Prediction, a task in which an agent receives an instruction that does not name the target object (e.g., “something to tighten a screw”) and must retrieve the past frame containing that object from a long visual memory and segment the object. The authors argue that most affordance methods assume that the object is visible in the given image, while long-video benchmarks typically output timestamps, clips, or text rather than masks.

To evaluate this task, the authors build MemAff-Bench, which consists of 3,628 tasks from EPIC-KITCHENS, Ego4D, and PAP indoor panoramas. Each visual memory consists of frames sampled from a single video or of perspective views rendered from indoor rooms, and its size is increased from 120 to 960 frames while the instruction and target are kept fixed. To ensure that each instruction has a single answer, the authors use an LLM to remove tasks in which more than one object satisfies the instruction. Each task is evaluated with the strict end-to-end mIoU (E2E), which gives the mask IoU only when the single annotated frame is selected.

The authors also propose MemAff, a training-free retrieve-then-rerank system. MemAff first describes each frame with Qwen3-VL-8B and embeds it with SigLIP. Given an instruction, it infers the name of the target object with a language model, retrieves eight candidate frames using the instruction, the object name, and the frame descriptions, selects one frame with Qwen3-VL-8B, and segments the object with Grounding-DINO and SAM 2.1. Experiments show that MemAff achieves the best E2E among the compared systems (one-shot Qwen3-VL-8B, Affordance-R1, and A4-Agent) and frame accuracy comparable to that of one-shot Qwen3-VL-32B with 14 to 116 times fewer prompt tokens per query. Based on an analysis of where the E2E score is lost, the authors conclude that longer context alone does not solve the task and that retrieving candidate frames is the main bottleneck.

Critical strengths

S1. The authors build a benchmark that keeps the instruction and target fixed while increasing the memory size, which enables an analysis of how performance changes with memory size.

S2. The authors report detailed analyses, such as the decomposition of E2E into frame accuracy and mask IoU, which clarify where the errors come from.

Critical weaknesses

W1. The success criterion of MemAff-Bench and the analysis across memory sizes are not sufficiently validated.

W1.1 The strict E2E accepts only the annotated frame, although the instruction does not specify which frame to select when the target appears in several frames, as is likely in memories sampled from a single video or rendered as eight views per room. The authors also state that the metric “does not credit another view of the target or another suitable object” (Section 3.1). For example, frame #124 selected by MemAff in Figure 9 appears to show the same potted plant as the annotated frame #123 but is counted as a miss. The authors should report, on a human-verified subset, the results that also accept other frames satisfying the instruction.

W1.2 The performance drop at larger memories may partly come from the success criterion. The authors attribute the drop mainly to retrieval failures (Section 5.3 and Table 11), but larger EPIC-KITCHENS memories are built by sampling the same video more densely (Section 3.1), so more frames may show the target as the memory grows. The authors should report how many frames show the target at each memory size, including Ego4D after its filtering, and repeat the analysis with the evaluation suggested in W1.1.

W1.3 The comparison across memory sizes is not fully controlled. The number of tasks decreases from 3,628 at 120 frames to 3,543 at 960 frames, and from 439 to 370 for Ego4D (Table 1), and the significance tests are not corrected for tasks that share source videos or recur across sizes, as the authors note in Section 5.1. The authors should report the results on the tasks shared by all memory sizes, with confidence intervals that account for this dependence.

W1.4 The uniqueness of the target is not re-checked when the memory is enlarged (Section 3.3). A lenient automatic check flags a possible second referent in 15% (PAP) and 28% (EPIC-KITCHENS) of the sampled 960-frame tasks (Appendix D). The authors should manually verify these flagged tasks and report whether the results change.

W2. The experiments do not show the added value of the proposed task and of MemAff over existing approaches.

W2.1 The difference from existing episodic-memory tasks is not demonstrated. The visual query task of Ego4D (VQ2D) [1] already localizes a queried object with bounding boxes in past frames, although Section 2 states that episodic-memory tasks usually output timestamps, clips, or text. The proposed task differs in its instruction and mask output, but the paper does not show that these create new difficulties, and the mask IoU in Table 7 is computed on different subsets of correctly selected frames (Appendix A.3). The authors should discuss VQ2D and report the mask quality of each system when the annotated frame is given.

W2.2 The amount of functional reasoning required by the instructions is unclear. For EPIC-KITCHENS, which accounts for 71% of the tasks, the instructions are generated by rewriting the annotated verb and noun (Section 3.2), and retrieving candidates with the inferred object name, followed by the same instruction-conditioned reranker, gives frame accuracy similar to that of the full system (Table 8). The authors should separately analyze the tasks that require reasoning about the state, relation, or function of the object beyond its name.

W2.3 The benefit of the retrieval and reranking stages of MemAff is not isolated, and a similar retrieve-rerank-segment pipeline has been proposed for object localization from visual memory [2]. The authors should compare MemAff with simpler alternatives, such as scoring every frame with the same Grounding-DINO and SAM 2.1 using the inferred object name or with SAM 3 [3], and with a training-free frame selection method such as AKS [4], BOLT [5], or T* [6], reporting the accuracy together with the cost.

W2.4 Two experimental settings may favor MemAff. For Ego4D, the non-target frames whose Qwen3-VL-8B description mentions the target object are removed (Section 3.2), and MemAff retrieves over the same descriptions (Appendix D). In addition, Affordance-R1, which is a single-image model, and A4-Agent receive the whole memory in a single request and run out of GPU memory at larger memory sizes (Appendix D). The authors should use a filtering model independent of MemAff and apply these baselines frame by frame with a defined rule for selecting the final frame.

[1] Grauman, Kristen, et al. “Ego4D: Around the world in 3,000 hours of egocentric video.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2022. [2] Zhou, Rui, et al. “Memory Over Maps: 3D Object Localization Without Reconstruction.” arXiv preprint arXiv:2603.20530 (2026). [3] Carion, Nicolas, et al. “SAM 3: Segment Anything with Concepts.” International Conference on Learning Representations (ICLR). 2026. [4] Tang, Xi, et al. “Adaptive keyframe sampling for long video understanding.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2025. [5] Liu, Shuming, et al. “BOLT: Boost large vision-language model without training for long-form video understanding.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2025. [6] Ye, Jinhui, et al. “Re-thinking temporal search for long-form video understanding.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2025.

W3. The scope of the claims and several experimental details should be clarified.

W3.1 Section 1 motivates the task with a robot that fetches an object seen earlier in another room, but the output is a mask in a past frame without camera poses, and the PAP memories concatenate rooms in an order that “need not correspond to an agent’s trajectory” (Section 3.2). The authors should clarify the scope of the claims and the additional steps, such as localization or re-identification, that an agent needs to use the output.

W3.2 The per-query token counts exclude the memory construction, as stated in Section 4. The authors should state in the abstract that the savings refer to the online cost, and report the average cost per query including the construction cost divided by the number of queries that reuse the memory.

W3.3 Section 5.2 states that A4-Agent ingests all 960 EPIC-KITCHENS frames, while Appendix D states that it runs out of memory at 960 frames on an 80 GB GPU. The authors should specify the GPU, the number of tokens, and whether the run succeeded for each source.

W3.4 The authors should clarify whether Affordance-R1 and A4-Agent in Table 2 were run on all 2,589 EPIC-KITCHENS tasks, since Appendix A.2 states that all five systems were run on a 497-task subset.

W3.5 Appendices A.4, B.1, and B.2 have headings without content, and the text under Appendix B.3 does not match its heading.

Questions for authors

Q1. How do the ranking of the systems and the main conclusions change on a human-verified subset where other frames satisfying the instruction are also accepted? (Ref W1.1 and W1.2)

Q2. Appendix B states that the benchmark uses 101 PAP panoramas, while a 960-frame PAP memory contains 120 rooms. Where do the additional rooms come from, and can the same room appear more than once in one memory? (Ref W1.1 and W1.4)

Q3. How are the memory frames sampled from each video (e.g., uniformly or randomly, and at what interval), and how many frames show the target at each memory size? (Ref W1.2)

Q4. Which failures of the evaluated systems are specific to the instruction that does not name the object or to the mask output, compared with VQ2D? (Ref W2.1 and W2.2)

Q5. How many queries reuse the cached frames of one memory, and what is the average cost per query including the memory construction? (Ref W3.2)

Please also refer to W2.3, W2.4, and W3.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 67G2 (1/3)

Thank you for identifying where the success criterion affects the interpretation of our results. We added a human audit and narrowed the scaling claim to retrieval of the annotated frame. We present MemAff as a training-free reference baseline.

Q1 · How rankings and conclusions change. We scored the same 35 questions and five systems at 120/240/480/960 frames. The table gives strict → human-confirmed end-to-end mIoU (%). We retained the original frame choices, used human-checked SAM-assisted reference masks for valid alternative frames, and averaged over all 35 questions. Invalid, uncertain, or missing selections contribute zero. A dash marks no available outputs.

FramesMemAffSigLIP + rerankQwen3-VL-8BA4-AgentAffordance-R1
12010.15 → 49.5015.56 → 46.6721.53 → 42.9513.34 → 29.190.00† → 0.18
24010.25 → 51.5012.81 → 53.5113.69 → 41.000.00 → 23.350.00 → 0.00
4802.49 → 45.149.96 → 44.865.33 → 34.810.00 → 2.54—
9605.26 → 57.689.96 → 47.02—0.00 → 0.97—

† At 120 frames, a same-frame replay filled one missing R1 prediction mask and scored zero IoU. The original-run strict aggregate is bounded by 0.00–2.86%. A4-Agent returned frames on 10/35 questions at 480 and 960, and R1 on 33/35 and 18/35 at 120 and 240. Among systems with outputs, the strict leaders across sizes are Qwen/Qwen/SigLIP/SigLIP. With reviewed alternatives they are MemAff/SigLIP/MemAff/MemAff, with a 0.28-point MemAff–SigLIP gap at 480. The human-confirmed mIoU of both methods fluctuates across scales. These are results on the audited subset (Appendix F, Tables 13–14 and Figure 10).

W1.1 · Other valid views. We agree that frame 124 can satisfy the instruction if it shows the same usable plant as frame 123 (Figure 9). For 15 PAP, 10 EPIC, and 10 Ego4D questions, we pooled the five systems’ final selections and accepted frames that visibly satisfied the instruction. Method identities were hidden initially and visible for a few later checks. Five PAP questions were the first completed items of a 20-question extension.

W1.2 · Scaling under two scoring rules. At 120/240/480/960 frames, MemAff’s exact hits are 9/8/3/3 versus 27/30/27/29 human-confirmed hits. For SigLIP they are 8/5/5/4 versus 24/26/25/27, all out of 35. The strict decline does not carry over to this subset under human scoring. Both the size and composition of some EPIC and Ego4D memories vary across scales. We limit the interpretation to the measured setups. Q3 reports how many views of the same object we confirmed in a separate audit.

W1.3 · Fixed tasks and dependence. Holding questions fixed at all four scales (2,557 EPIC, 370 Ego4D, 585 PAP), we jointly resampled clusters across methods and scales 10,000 times. From 120 to 960, MemAff’s exact-frame accuracy changed by −12.32/−14.59/−23.42 percentage points (95% intervals [−13.49, −11.10], [−18.26, −10.87], [−27.18, −19.58]); SigLIP’s changes were −11.07/−14.32/−17.78. Clusters were source videos in EPIC and Ego4D. PAP base episodes are scene proxies and may share panoramas (Appendix H).

W1.4 · Second referents. At 960 frames, we manually audited the 18 PAP and 33 EPIC questions flagged by the automatic screen. We confirmed a distinct valid object in 4 PAP and 13 EPIC questions. With the reference frame and reviewed valid alternatives credited on these flagged sets, MemAff and SigLIP went from 4→12 and 5→11 hits on PAP, and from 2→27 and 1→21 on EPIC. Uncertain alternatives received zero extra credit. These checks cover the flagged questions and return frames.

AuthorsOpenReview ↗

Response to Reviewer 67G2 (2/3)

W2.1 · VQ2D and oracle-frame segmentation. We discuss Ego4D VQ2D in Sec. 2. VQ2D takes an image crop of an object instance and localizes its most recent occurrence with a track of 2D boxes. MemAff-Bench takes an implicit functional instruction and asks for a frame and a pixel mask. Comparing the difficulty of these formulations would require controlled changes to the query and output on matched examples.

Table 7 averages mask IoU over MemAff's exact-frame hits, so its subset varies by source and memory size. For the matched comparison, we gave all five systems the same original annotated frame and instruction for each of the 35 audited questions (15 PAP, 10 EPIC-KITCHENS, 10 Ego4D). With failed predictions scored as zero, mask mIoU was 54.93% for MemAff and SigLIP (a single shared Grounding-DINO + SAM 2.1 run), 53.34% for one-shot Qwen3-VL-8B, 47.31% for Affordance-R1, and 40.90% for A4-Agent. These scores measure mask quality on the same inputs with frame retrieval bypassed.

W2.2 · Functional reasoning. Keeping the same instruction-conditioned reranker on EPIC, noun-only retrieval gives 19.2/13.8/10.8/7.0% strict frame accuracy versus 18.4/14.3/9.5/6.2% for fusion. In an exploratory set of 55 prompts selected by explicit state, relation, or function cues, the annotated frame appears in the top eight on 18/13/9/6 questions for noun-only and 18/14/12/8 for fusion. Final hits are 9/3/2/2 and 8/5/4/1, respectively. This comparison changes retrieval while keeping full instruction access in both rerankers. Quantifying which cues actually rule out same-category candidates requires further annotation.

W2.3 · Simpler baselines. We cite Memory Over Maps as related retrieve–rerank localization work. On one fixed question per source and all four sizes (5,400 candidate frames), we ran Grounding-DINO on every frame using MemAff’s cached noun, then ran SAM 2.1 on the highest-confidence frame. On an H100, warmed detection took 10.52–17.04 s at 120 frames and 88.02–135.93 s at 960. Selected-frame segmentation added 0.06–0.13 s. Strict hits on these 12 task–size pairs were 0 for exhaustive detection, 1 for MemAff, and 0 for SigLIP. This pilot illustrates computational cost. Benchmark-scale accuracy and the suggested SAM 3 and training-free selector controls remain open.

W2.4 · Shared filtering and native per-frame evaluation. Replacing Qwen-inventory filtering with Grounding-DINO on 427/416/384/315 paired Ego4D questions changes MemAff’s strict mIoU margins over SigLIP from +2.87/+4.19/+5.00/+2.62 to +1.20/+1.79/+0.20/−0.26 percentage points. Its Qwen margins through 480 change from +2.10/+4.61/+5.77 to −0.58/−0.18/+1.67. Grounding-DINO is also used downstream (Table 15).

Per-frame control. We ran native A4-Agent and Affordance-R1 on every frame of 15 fixed questions (5 per source, 120 frames per question), drawn by task ID from the 35-question audit before the native runs. We ranked frames with nonempty native masks by instruction-to-image SigLIP similarity. The same Qwen visual reranker used by MemAff then selected from the top eight using the full instruction. The submitted mask is the selected frame's native prediction. Selection uses no reference labels.

System and inputExact hitsStrict E2E mIoUE2E mIoU accepting reviewed alternatives
MemAff, archived3/155.81%52.67%
A4-Agent, sequence2/1512.37%35.08%
A4-Agent, per-frame + rerank2/153.62%13.57%
Affordance-R1, sequence0/150.00%0.41%
Affordance-R1, per-frame + rerank2/159.38%27.40%

Without visual reranking, both per-frame systems obtain 1/15 exact hits and 0.00% strict mIoU using SigLIP's highest-scoring eligible frame. R1 improves under per-frame reranking and exceeds MemAff's strict mIoU on these questions. MemAff retrieves the annotated frame into the Top 8 on 9/15 questions, compared with 5/15 for both per-frame baselines. We manually reviewed the four newly selected frames (three from A4 and one from R1). All four failed to satisfy the instruction and received zero under the reviewed-alternative rule. MemAff remains higher under that rule on this 120-frame subset.

Q4 · Which failures are specific to the output or query? With the frame and predictions fixed in the 600-task PAP oracle check, replacing both prediction and reference masks with tight boxes changes which outputs pass IoU 0.50. Box-only passes number 22 for A4 and 18 for R1. Mask-only passes number 10 and 21, respectively. This isolates the scoring representation. A matched text-versus-visual-query experiment would be needed to isolate the query effect.

AuthorsOpenReview ↗

Response to Reviewer 67G2 (3/3)

W3.1 · Embodied scope. The evaluated output is a mask in a past frame. PAP combines room views into a memory. Navigation, camera-pose recovery, and current-world re-identification are outside the evaluation.

W3.2 · Offline and online costs. The abstract’s token claim concerns online processing. On H100s, across 3,597 nonempty 120-frame questions, MemAff takes 62.456 s to index a memory and 0.439 s per query. The SigLIP control takes 1.707 s and 0.401 s. MemAff averages 26,114 input / 8,242 output generation tokens to index and 1,540 / 11 online. Q5 gives the amortized comparison (Appendix I, Table 17).

W3.3 · A4-Agent at 960 frames. Eight A800 replay shards submitted 960 images per processed EPIC request and yielded 2,091 frame selections and 2,086 recorded masks. With the earlier 497-task subset, EPIC totals 2,458 selections and 2,453 masks out of 2,588. Per-result GPU provenance of the earlier subset is unresolved. The archived Ego4D and PAP runs returned no frames (HTTP 500). Actual EPIC-960 token counts were unlogged. The 93k–124k range refers to the 480-frame prefill configuration.

W3.4 · Full-source coverage. A4’s archived EPIC selections are 2,589/2,589, 2,589/2,589, 2,459/2,589, and 2,458/2,588. R1 has 2,568/2,589 and 2,384/2,589 at 120/240. The high-scale R1 OOM control used the 497-question EPIC subset. Table 2 uses the full evaluation denominators and identifies resource-limit cells.

W3.5 · Appendix layout. We put the tables and explanatory text after their headings in A.4, B.1 and B.2 and placed the error analysis with its heading in B.3. We checked the compiled revision.

Q2 · Repeated PAP rooms. The 960-frame construction appends 840 rendered views from other episodes to 120 base views. Episodes draw from 101 panoramas and may share images. Ordered image hashes show repeated eight-view blocks in all 148 inspected memories, with a median of 64 distinct blocks out of 120 (range 60–68).

Q3 · Sampling and target visibility. We deterministically take approximately equally spaced positions in the chronologically ordered eligible EPIC and Ego4D frames, with one slot reserved for the annotated frame. Elapsed time varies with video, eligible frames, and memory size. For one EPIC example the median interval was 7.04 s at 120 frames and 0.96 s at 960. We reviewed 12 candidate images per question for five questions from each source. Matching confirmed views to each actual memory gives mean lower bounds of 1.2/2.2/3.2/5.4 visible frames in EPIC and 1.0/2.0/2.0/3.0 in filtered Ego4D at 120/240/480/960. These counts include the annotated frame and cover the reviewed images (Appendix F).

Q5 · How much reuse is measured? PAP has 600 questions on 155 memories (3.87 per memory); the 439 Ego4D and 2,558 EPIC cost-cohort records each have one question. For 1/10/100 queries per memory, dividing measured indexing latency by the assumed reuse count and adding measured online latency gives 62.895/6.684/1.063 s per question for MemAff and 2.108/0.572/0.419 for SigLIP. Qwen, A4 and R1 take 2.642, 4.544 and 18.851 s per query with no separate index. Persistent indexes average 1.628 MiB for MemAff and 0.234 MiB for SigLIP. The three reuse levels are amortization scenarios.

Reviewer_SnZsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Memory-Grounded Affordance Prediction and constructs 3,628 evaluation tasks, each comprising 120 frames, from EPIC-KITCHENS/VISOR, Ego4D, and PAP, with extensions to up to 960 frames. Given an implicit functional intent and visual memory, the task is to predict an object mask in a specified historical frame. The method is evaluated on multiple datasets.

Critical strengths
  • Joint evaluation of retrieval and pixel-level localization: The benchmark connects retrieval of past frames with object mask prediction, covers three image sources and multiple memory sizes, and reports both frame accuracy and mask quality conditional on a successful retrieval. Compared with reporting only an overall score, this helps distinguish between failure to retrieve the correct frame and poor segmentation after successful retrieval.
  • The bottleneck analysis goes beyond a simple leaderboard: Even when the 32B single-pass model can accommodate 960 frames, its exact-frame accuracy remains low; replacing only the 8B reranker with a 32B model does not consistently improve performance. The paper also includes controls for three-way recall, candidate order, resolution, alternative aggregation methods, and segmentation thresholds.
Critical weaknesses
  • Exact-frame metrics may count reasonable answers as failures: Instructions typically describe only the task goal and do not specify that a particular point in time must be retrieved. Even if the source memory contains only one suitable object, that object may appear in multiple frames; strict E2E assigns zero credit to the correct object in any other frame. Expanding memory also introduces more similar frames or usable objects, so the performance decline may conflate genuine retrieval difficulty with an annotation effect in which more answers become valid but only one is accepted. The authors acknowledge that uniqueness was not rechecked for larger memories, and permissive automatic screening suggests a possible second candidate in 15% of PAP samples and approximately 28% of EPIC samples at 960 frames.
  • Benchmark construction and the evaluated methods are not sufficiently independent: For Ego4D, object lists generated by Qwen3-VL-8B are used to remove non-target frames that mention the target object, while MemAff’s text retrieval branch uses those same lists. Thus, the tool that determines which distractors remain is directly coupled with the features used by an evaluated method, potentially altering relative difficulty. Shared SAM-family boundary refinement may also affect segmentation evaluation. The paper candidly discloses these issues but has not quantified their impact on the rankings.
Questions for authors
  • Sensitivity to valid answers. Could the authors sample a manually verified subset from each source and memory size, annotate acceptable answers—including other usable frames of the same object and other objects that genuinely satisfy the task—and recompute object-/track-level or multiple-answer metrics? On this subset, do the model rankings and the declining performance trend as memory size increases persist?
  • Independence of benchmark construction. After removing the 31 empty instructions, does the advantage of three-way fusion over single-branch retrieval and single-pass VLMs persist when Ego4D deduplication uses independent manual annotations or tracking information, or at least avoids filtering with the lists used by MemAff’s retrieval branch? How large is the effect of shared SAM boundary refinement on a subset with original manual masks? Reannotating the entire dataset is unnecessary, but controls sufficient to estimate the impact are needed.
  • Independent statistical units. Could the authors use cluster resampling by video/scene and recompute confidence intervals and differences on the intersection of tasks across memory sizes? Treating tasks from the same video, or tasks repeated across memory sizes, as independent observations may overstate nominal statistical significance.
  • Full usage costs. Please report the total runtime and token usage for offline visual inventory generation, storage, noun inference, retrieval, reranking, and segmentation, and provide the amortized cost of answering 1, 10, and 100 questions per memory.
2 comments
AuthorsOpenReview ↗

Response to Reviewer SnZs

Thank you for highlighting the benchmark’s joint evaluation of historical retrieval and pixel-level localization.

Weakness 1 and Q1 · Alternative answers and scale. We pooled the final frames returned by five systems at four memory sizes on the same 35 questions (15 PAP, 10 EPIC, 10 Ego4D). We credit a frame that visibly satisfies the instruction. Uncertain selections receive zero. We initially hid system identities and revisited a few labels with identities visible. Five PAP questions were the first completed items from a 20-question extension.

At 120/240/480/960 frames, MemAff has 9/8/3/3 exact hits and 27/30/27/29 confirmed hits. The SigLIP control has 8/5/5/4 and 24/26/25/27. Their relative order changes at 480 and 960 on these questions. The resulting 480-frame gap is two hits (exploratory exact McNemar p = 0.69). Confirmed hits for Qwen are 28/24/23/—, A4-Agent 18/13/4/3, and R1 6/8/—/—. A dash marks no outputs. A4 returns frames on 10/35 at 480 and 960, and R1 on 33/35 and 18/35 at 120 and 240.

With human-checked alternative reference masks, the leading system by end-to-end mIoU changes from Qwen/Qwen/SigLIP/SigLIP under strict scoring to MemAff/SigLIP/MemAff/MemAff. The 480-frame MemAff–SigLIP gap is 0.28 percentage points. Human-confirmed frame hits and mIoU for both methods fluctuate across sizes on the audited subset. The full fixed-cohort decline concerns strict scoring (Appendix F, Tables 13–14 and Figure 10).

Weakness 2 and Q2 · Construction and mask references. Replacing the Qwen-inventory filter with Grounding-DINO gives 427/416/384/315 paired Ego4D questions. MemAff’s strict mIoU margins over SigLIP shrink from +2.87/+4.19/+5.00/+2.62 to +1.20/+1.79/+0.20/−0.26 points. Its margins over one-shot Qwen through 480 change from +2.10/+4.61/+5.77 to −0.58/−0.18/+1.67. This removes reuse of the Qwen inventories in filtering. Grounding-DINO is also used downstream. In EPIC, rescoring the same predictions with the original human VISOR masks reduces MemAff’s margin over SigLIP by 0.06–0.34 points. That margin remains positive at all four sizes on the 2,557 questions shared across sizes. This control covers EPIC (Appendix G).

Q3 · Dependence across questions and scales. We held the 2,557 EPIC, 370 Ego4D, and 585 PAP questions fixed and used 10,000 paired cluster bootstrap replicates, jointly across models and sizes. Source videos define EPIC and Ego4D clusters. PAP base episodes serve as scene proxies and may share panoramas. From 120 to 960 frames, MemAff’s exact-frame accuracy falls by 12.32/14.59/23.42 points by source, and SigLIP’s by 11.07/14.32/17.78. Strict end-to-end mIoU also declines on each cohort (Appendix H, Table 16).

Q4 · Indexing, querying, and reuse. On H100s with 3,597 nonempty 120-frame questions, MemAff’s indexing averages 62.456 s: 59.801 for inventories and 1.707/0.948 for visual/text embeddings. The 0.439 s online average breaks down as 0.055/0.027/0.200/0.113/0.044 for noun inference, retrieval, reranking, detection, and masks. Index/query input and output generation tokens average 26,114/8,242 and 1,540/11; the persistent index averages 1.628 MiB (SigLIP: 0.234 MiB). At 1/10/100 uses, the amortized seconds per question are 62.895/6.684/1.063 for MemAff and 2.108/0.572/0.419 for SigLIP. One-shot Qwen, A4-Agent, and R1 take 2.642, 4.544, and 18.851 s per question, respectively. We divide measured construction cost by the assumed reuse count and add measured online time (Appendix I, Table 17 and Figure 11).

I thank the authors for their response, which addresses most of my concerns. I have decided to maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.