acceptodds
Under review as a conference paper at ICLR 2027

See Once, Query Many: 3D Scene Affordance Segmentation via Agentic Memory

Abstract

Scene-level affordance segmentation is fundamental for embodied agents to perceive interaction possibilities in complex 3D environments. Existing MLLM-based methods face two key limitations: repetitive visual processing causes substantial redundancy, while insufficient modeling of remote functional dependencies and complex spatial relations limits their ability to ground intricate instructions. We present MemAff, a training-free agentic framework that converts a single scan into an evolving Agentic Scene Memory. Through incremental editing and on-demand visual recall, a vision-language agent anchors objects to frames and cross-links spatially separated functional pairs. For each instruction, MemAff traverses this memory to select the view best exposing the actionable region, followed by temporal propagation and cross-view 3D lifting. This agentic memory construction and retrieval enables “See Once, Query Many” without repeated visual processing. Experiments on SceneFun3D and our 3,149-instruction FunEnhance benchmark show state-of-the-art performance, reaching 30.5% AP50 and 24.9% mIoU while reducing per-instruction latency by over 20×.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
affordance, 3D scene understanding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KR2uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MemAff is a training-free framework for repeated-query 3D affordance segmentation. It builds a persistent, frame-linked scene memory through VLM-driven editing and visual recall, then performs instruction-conditioned retrieval, point localization, temporal segmentation, and multi-view fusion. Evaluation uses SceneFun3D and its instruction-level extension, FunEnhance.

Critical strengths
  1. The paper provides a repeated-query formulation. Reusing frame-linked scene knowledge addresses redundant perception and provides an inspectable interface for grounding.
  2. The Experiment evidence supports memory utility. Matched-frame comparisons across three VLMs, together with editing and recall ablations, support the usefulness of persistent memory beyond changes in visual inputs or backbone.
Critical weaknesses
  1. The reported 20× speedup is difficult to assess without comparable baseline timings. Section 5.1 states that prior methods, including TASA, take over 20 minutes per instruction, but TASA reports 37.61 s in its own setting. The appendix discusses amortization, yet it remains unclear whether these comparisons account for the same processing steps and reusable computation.

  2. The experiments do not show how much functional cross-links contribute beyond the rest of the memory. Removing the section component also changes descriptions and frame access, while similarity-based retrieval changes several reasoning steps. Neither comparison directly tests whether the links help resolve remote functional dependencies.

  3. Equation (9) leaves the normalization unclear: are votes divided by all frames or only those with valid observations? This matters for partially visible points at the 0.5 threshold, though it does not establish an implementation error. Also, removing propagation and fusion together does not show how much of the improvement comes from voting itself.

  4. It is unclear whether the reported AP follows the official SceneFun3D protocol. Fun3DU/AffordBot instead threshold point-level precision, which can assign high scores to small, incomplete regions. Results from these two definitions are not directly comparable.

Questions for authors

Please see the weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer KR2u

We sincerely thank the reviewer for recognizing the repeated-query formulation and the evidence supporting persistent memory. We address each concern below.

w1: Timing boundaries and cumulative efficiency

Timing interpretation. TASA reports per-instruction time without specifying within-scene concurrency. For example, 20 instructions completed in 10 min average 30 s each, but overlapping requests can have longer individual latency. We evaluate all three methods sequentially, submitting each instruction only after the preceding 3D prediction completes. TASA’s default configuration averages 299.67 s from submission to prediction.

Measurement boundaries and cumulative time. Our original timing included local VLM setup and weight loading, obscuring comparison with baselines. We now use a third-party API for all VLM calls, including response waits but excluding local model loading and data preprocessing. For each scene, we sum the recorded times of the first Q queries and count MemAff’s setup once (272.22 s); mean online query time is 22.95 s.

QMemAff total (s)TASA total (s)Speedup vs TASAFun3DU total (s)Speedup vs Fun3DU
1293.16299.671.02×734.922.51×
2318.12599.341.88×1459.924.59×
10502.323000.705.97×7385.2714.70×
20731.226003.408.21×14717.3220.13×

MemAff and TASA have comparable first-query costs because both analyze the scene. Thereafter, MemAff reuses memory while TASA repeats view search, yielding a 1.88× cumulative speedup at Q=2, including construction. At Q=20, speedups reach 8.21× over TASA and 20.13× over Fun3DU. We will correct our original “20× per-instruction latency reduction” claim to reflect cumulative speedup.

w2: Isolating the contribution of functional cross-links

We isolate functional links by deleting only their assertions from existing GPT-6.1Sol and Qwen3.5-derived memories, rather than removing entire sections or changing retrieval. Other descriptions, frame anchors, retrieval, the downstream pipeline and inference limits remain fixed.

We separately evaluate 72 cross-link candidates (16.18%) and 373 ordinary part-level tasks (83.82%) among 445 original SceneFun3D instructions. Δ = ablated − full mIoU; negative values indicate loss after link removal. The 95% CIs use paired scene-cluster bootstrap resampling.

MemoryTask groupFull mIoUAblated mIoUΔ (pp)95% CI for Δ
GPTCross-link candidates (72)22.3219.13−3.19[−4.05, −2.79]
QwenCross-link candidates (72)22.1018.04−4.06[−4.64, −3.56]
GPTOrdinary part-level (373)25.8624.61−1.25[−1.74, −0.98]
QwenOrdinary part-level (373)25.5424.42−1.12[−1.46, −0.92]

Link removal reduces mIoU by 3.19/4.06 on cross-link candidates versus 1.25/1.12 on ordinary part-level tasks (GPT/Qwen); all four 95% CIs lie below zero. The larger performance drops on cross-link candidates suggest that explicit functional links are particularly beneficial for tasks requiring functional relationships, while also contributing to ordinary part-level grounding.

w3: Vote normalization and the independent contribution of voting

Normalization in Eq. (9). We divide each point’s foreground votes by its valid observation count among fusion views, not all scene frames. Valid observations require the point to project inside the image and pass visibility and depth-consistency checks; zero foreground votes still count in the denominator. Additional foreground filtering affects only the numerator. Foreground requires ≥1 valid observation and a vote fraction ≥0.50. We added this clarification after Eq. (9).

Independent voting ablation. We vary only the final fusion rule, keeping memories, selected views, contact points, propagated SAM3 masks and geometric filtering fixed. With temporal propagation and all upstream outputs fixed, this comparison isolates the contribution of the final fusion rule. All three conditions use the same evaluator on 445 original instructions from 30 SceneFun3D scenes.

Fusion rulemAPAP50AP25mIoU
Union of valid projected masks9.7120.6740.2222.29
Voting normalized by all processed views13.1522.9236.1817.17
Voting normalized by point-wise valid observations17.430.548.424.9

Point-wise valid-observation voting improves mAP/AP50/AP25/mIoU over union by 7.69/9.83/8.18/2.61 and mIoU over all-view normalization by 7.73. These results favor point-wise valid-observation voting among the three fusion rules compared.

w4: Evaluation protocol and comparability

Our metrics follow the evaluation code of Fun3DU, AffordBot and TASA, using point-level precision thresholds for AP rather than official SceneFun3D instance-IoU matching. We agree that results under these definitions are not directly comparable. We clarify the distinction between our shared protocol and the official evaluator in Appendix C.1.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Additional Results

Dear Reviewer KR2u,

As the discussion phase proceeds, we would like to kindly follow up on our response and check if our rebuttal along with the additional results have adequately resolved your concerns.

We remain fully available if you have any further questions or if any additional clarification would be helpful. If you find our response satisfactory, we would deeply appreciate it if you could consider reflecting this in your score and assessment.

Thank you very much for your time, valuable feedback, and dedication!

Thank you for the additional experiments. They address parts of my earlier concerns. However, the evaluation concern remains: the reported AP uses point-level precision thresholds, while the official SceneFun3D evaluator uses instance IoU. Appendix C.1 discusses differences in inputs and models but does not explain why the official evaluator was not used. Since the methods ultimately output 3D masks, Table 8 cannot replace a comparison under the official metric. I therefore maintain my original score.

Evaluation protocol and comparability. We clarify that our evaluation follows the publicly released Fun3DU evaluator, consistent with the protocol adopted by the compared methods, including TASA, AffordBot, FunGraph, and UniFunc3D. We adopt this protocol to ensure consistent metric definitions and enable direct comparisons with prior work. Moreover, our evaluation jointly reports AP, AR, and mIoU, which respectively assess prediction precision, ground-truth coverage, and overall mask overlap. These complementary metrics provide a balanced assessment of segmentation quality rather than relying solely on precision. We acknowledge that this protocol differs from the official SceneFun3D instance-IoU evaluation and will explicitly clarify this distinction in the revised manuscript.

We hope this clarification helps address your concerns regarding evaluation comparability and would greatly appreciate your reconsideration of our work in light of these clarifications.

Reviewer_Kx7iOpenReview ↗
Rating
2(−1)
Confidence
4
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose MemAff as a training-free framework for 3D scene affordance segmentation by proposing persistent agentic memory from a single scan. Then they use that across multiple instructions without visual processing. The memory is constructed by a VLM agent using multiple atomic skills. During query, the generated memory is searched to invoke relevant entities and frames, then a VLM critic selects the best view, followed by SAM3.

Critical strengths
  • The idea that scene understanding could be reframed as a persistent memory representation rather than multiple re-query is underexplored, novel and a simple solution. The authors did an excellent job to tap this limitation of the existing literature. Moreover, this paradigm could be utilized beyond affordance understanding, especially in agentic embodied AI.

  • The cross-view voting with depth-gating for 2D-to-3D lifting is geometrically well-motivated

  • Table 10 is surprising and interesting where the authors had a finding that markdown memory is better than graph memory. The authors also had a convincing justification and provides future direction

  • By giving the no-memory baseline access to the exact same frames as MemAff and measuring the performance gap, the authors identified the contribution of memory organization from the contribution of visual evidence.

  • FunEnhance is a useful community contribution, and could lead to useful future research

  • Ablation experiments are comprehensive across multiple dimensions

Critical weaknesses
  • The entire paper reports single-run results with no random seeds, # of runs or confidence intervals. We know tht VLM is stochastic so memory construction involves 60+ VLM calls per scene, each potentially sampling differently. This memory construction error could propagate to all downstream queries for that scene. The variance across multiple memory constructions for the same scene could be huge, but never measured.

  • UniFun3D and OpenFunGraph are cited but not evaluated. AFFORDMEM, a memory-based affordance method similar to MemAff is presented only in appendix, that too using a non-default VLM backend. These omissions make the SOTA comparison incomplete.

  • All quantitative affordance results are on SceneFun3D exclusively. The ScanNet++ evaluation is only qualitative. Object-level segmentation are provided but no affordance grounding. Moreover, it is not benchmarked against any baseline.

  • Missing Reproducibility, Ethics, and AI Use statements and only filled with placeholders. This is a potential policy violation.

  • 51 second online latency is not suitable for real-world deployment.

Questions for authors

Please address the concerns raised in the weakness section.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Kx7i

We sincerely thank the reviewer for recognizing the value of persistent scene memory and for the constructive comments. We address each concern below.

w1: Variability across independent memory constructions

We ran three independent reconstruction-and-query trials on the same 30 SceneFun3D validation scenes and 445 original instructions (90 memory constructions; 1,335 queries), with random seed 20261110. Each trial rebuilt all memories from scratch and reran downstream VLM inference for all associated instructions, using identical scene inputs, prompts and settings without reusing memories or API responses across trials. This captures memory-construction variability propagated to final predictions. All metrics use our original evaluation script and are reported in percent; SD is the sample standard deviation across the three trials.

Metric (%)Run 1Run 2Run 3Mean ± SD
mAP17.4917.0017.8717.45 ± 0.43
AP5030.6929.8430.9730.5 ± 0.59
AP2548.1946.3948.9747.85 ± 1.32
mIoU24.8123.9525.1124.62 ± 0.60

mIoU ranges from 23.95% to 25.11%, with a sample SD of 0.60, indicating stable overall performance across three independent memory reconstructions and downstream inference runs.

w2: Coverage of relevant baselines and comparison settings

We added AFFORDMEM to Table 1 using results reported in its paper. UniFunc3D was not included because it had not yet been formally published and its full inference code was unavailable. The table reports SceneFun3D Split 0 results.

MethodmAPAP50AP25mARAR50AR25mIoU
AFFORDMEM—20.1341.66—44.0450.6217.32
UniFunc3D14.4731.2451.0132.7646.9758.8824.30
MemAff (ours)17.4030.5048.4036.8051.1060.8024.90

AFFORDMEM. We cite its published results (arXiv:2605.11616) since its inference code is unavailable. For a fairer comparison, our appendix uses the same Qwen3-VL-32B backend and SAM3. MemAff outperforms AFFORDMEM with both the matched and default VLM backends (mIoU: 20.37/24.90 vs. 17.32).

UniFunc3D. Its mAP/mAR come from the authors’ released evaluation code; the other five metrics come from their Table 1. MemAff achieves superior or competitive results across metrics: higher mAP/mAR (+2.93/+4.04) and AR50/AR25/mIoU (+4.13/+1.92/+0.60), with competitive AP50/AP25 (−0.74/−2.61).

MemAff reduces per-query visual computation by reusing scene memory and inspecting only relevant linked views; however, UniFunc3D repeats scene-wide search for each instruction, processing four groups of 64 frames for view selection, followed by frame-wise localization and candidate-mask verification. This makes MemAff more efficient for repeated queries in familiar scenes. We will quantify compute and latency under matched conditions once UniFunc3D’s full inference code is available.

OpenFunGraph. Its native evaluation focuses on object/interactive-part nodes and functional relations; it reports no quantitative instruction-conditioned affordance segmentation results. Because no compatible evaluation setup exists for this task, we did not conduct a direct quantitative comparison. However, inspired by this work, we instead compare graph-based and Markdown-based memories within our framework (Appendix Table 10).

w3: Dataset coverage and the ScanNet++ evaluation

To our knowledge, SceneFun3D is the only public dataset designed specifically for our task, scene-level 3D affordance segmentation. We therefore use it for quantitative evaluation and welcome additional benchmarks for evaluating MemAff as they become available.

Appendix E.2 uses ScanNet++ to qualitatively demonstrate 3D object-level grounding with the same framework and no task-specific retraining. This illustrates applicability beyond affordance grounding, not a quantitative cross-dataset affordance evaluation or baseline comparison.

w4: Reproducibility, Ethics, and AI Use statements

We sincerely thank the reviewer for pointing out the missing statements. We have completed the Reproducibility, Ethics, and AI Use statements in the updated PDF.

w5: Online latency and real-world deployment

We agree that 51 s is not suitable for real-time deployment, but MemAff is the fastest among the compared methods. This self-hosted timing included model loading and depended on concurrency. With a third-party API and sequential queries, mean submission-to-3D latency is 22.95 s with memory built, including API waits, or 293.16 s for the first query including construction. Over 10 queries, memory reuse yields cumulative speedups of 5.97× over TASA and 14.70× over Fun3DU. The 22.95 s latency still limits real-time deployment; we will work to reduce it.

Reviewer_Kx7iOpenReview ↗

Rebuttal Response

Thank you for the detailed response and the additional experiments. After considering the rebuttal, some of my concerns remain:

Variance vs. claimed gains (W1/W2). The three-trial study is welcome, but it covers 30 validation scenes rather than the full evaluation, uses a single seed for all trials (please clarify what varies between runs), and reports no variance for any baseline. More importantly, the reported mIoU gain over UniFunc3D (+0.60) equals MemAff's own run-to-run standard deviation (0.60), and MemAff is lower on AP50 (−0.74) and AP25 (−2.61). On this evidence, the improvement over UniFunc3D is within noise, and the claim of superior or competitive performance is not established.

Comparison conditions (W2). The UniFunc3D numbers combine its released evaluation code (mAP/mAR) with values copied from its paper, and the AFFORDMEM numbers are copied from its paper without mAP/mAR. Neither baseline is run under matched conditions, and the efficiency advantage over UniFunc3D is argued but not measured.

Scope (W3, W5). I accept that SceneFun3D is the only suitable benchmark, but the generality claims should then be limited to it. The latency concern remains, as the authors acknowledge.

I still find the persistent-memory formulation valuable. However, given that the main quantitative advantage is not separable from run-to-run variance, I no longer think the empirical claims are sufficiently supported.

AuthorsOpenReview ↗

Response to Reviewer Kx7i

We sincerely thank you for your prompt feedback and recognition of persistent scene memory. We hope the clarifications and results below address your concerns.

Performance, variability, and comparison conditions

Evaluation coverage. Our new experiments all use the complete SceneFun3D validation split, not a selected subset.

Evaluation protocol and baseline variance. We acknowledge that our previous comparison mixed recomputed metrics with paper-reported values, and apologize for this inconsistency. We have now re-evaluated all seven UniFunc3D metrics using its released predictions and the metric definitions of the SceneFun3D benchmark. The recomputed results are consistent with those previously reported. As its full inference pipeline is unavailable, we cannot estimate its run-to-run variance; the recomputed scores serve as fixed references without assuming zero variance.

Experimental results. We apologize for the limitations of our earlier three-trial study, which used the same seed (20261110) and provided limited evidence of variability. Even with a fixed seed, VLM sampling, GPU nondeterminism, and inference frameworks may affect memory construction and downstream predictions. We therefore conducted two more experiments: (1) ten independent runs with different seeds (20261111–20261120), and (2) ten repeated runs with the same seed (20261110).

Each run reconstructs scene memory from scratch and repeats downstream inference under identical inputs, prompts, model settings, and evaluation procedures. This lets us assess variability across different seeds and repeated runs under a fixed seed. For each metric, we exclude the maximum and minimum and report the mean and sample SD of the remaining eight values. Δ = Trimmed Mean - the UniFunc3D score.

Table 1. Performance Variability Across Different Seeds

MetricRun 1Run 2Run 3Run 4Run 5Run 6Run 7Run 8Run 9Run 10Trimmed Mean ± SD(n=8)Δ
mAP17.8517.4017.3217.4217.6617.5217.4117.6317.4917.4817.50 ± 0.10+3.03
AP5030.2430.1430.8730.9330.6329.9930.6630.3430.5230.4330.48 ± 0.24−0.76
AP2549.2549.2349.4649.2749.3750.0449.6350.0149.5449.8949.55 ± 0.28−1.46
mAR36.9537.1236.5237.2036.6937.2936.4337.0336.6036.8036.86 ± 0.25+4.10
AR5050.8351.4350.5751.2650.7451.3450.6651.1750.7451.1950.99 ± 0.27+4.02
AR2560.7460.1561.6060.9361.1560.9660.3760.7160.9761.3860.90 ± 0.30+2.02
mIoU24.7325.1624.3925.0824.5625.2524.6824.9925.6025.1924.96 ± 0.26+0.66

Table 2. Performance Variability Under a Fixed Seed

MetricRun 1Run 2Run 3Run 4Run 5Run 6Run 7Run 8Run 9Run 10Trimmed Mean ± SD(n=8)Δ
mAP17.4917.0017.8717.3917.4217.5517.2117.8717.3617.5717.48 ± 0.19+3.01
AP5030.6929.8430.9730.5430.2030.7930.3030.1231.1130.6330.53 ± 0.30−0.71
AP2548.1946.3948.9748.1948.4048.9048.2248.1148.3948.8048.40 ± 0.30−2.61
mAR36.4336.7336.7037.1936.6436.5236.3336.9736.6937.1036.72 ± 0.22+3.96
AR5050.7950.7251.0751.0750.9850.9751.6850.9050.6450.6850.90 ± 0.15+3.93
AR2561.1461.1060.6260.7860.4360.4060.9560.9861.4461.0660.88 ± 0.25+2.00
mIoU24.8123.9525.1124.8324.7824.9824.8624.8724.7725.4124.88 ± 0.12+0.58

Table 3. Validation comparison

MethodmAPAP50AP25mARAR50AR25mIoU
UniFunc3D14.4731.2451.0132.7646.9758.8824.30
MemAff (our)17.430.548.436.851.160.824.9

mIoU performance and variability. In Tables 1 and 2, 10/10 and 9/10 runs, respectively, exceed UniFunc3D’s mIoU of 24.30. Across all ten untrimmed runs per batch, the 95% Student-t confidence intervals for the mean are [24.70, 25.23] and [24.57, 25.10], both above this reference. Under the independent-run t-model assumptions, this supports a stable mean mIoU advantage over the fixed UniFunc3D reference, accounting for MemAff’s observed run-to-run variability. The advantage persists after trimming, with means of 24.96 and 24.88.

Precision and recall. Across both batches, MemAff achieves higher mAP, mAR, AR50, and AR25 than UniFunc3D. We acknowledge that AP50 and AP25 are lower. However, unlike these single-threshold metrics, mAP averages the fractions of instructions meeting point-wise precision thresholds from 0.50 to 0.95 in steps of 0.05. This indicates better average performance across stricter precision thresholds, despite the lower AP50.

Efficiency. MemAff's see-once-query-many design amortizes scene understanding and memory construction across multiple queries, avoiding repeated scene processing and offering a structural efficiency advantage. However, UniFunc3D's full inference pipeline is unavailable, preventing a direct end-to-end latency comparison. We will clarify this distinction in the revised manuscript and conduct a matched efficiency evaluation once its implementation becomes available.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Additional Results

Dear Reviewer Kx7i, As the discussion period is approaching its end, we would like to kindly follow up on our rebuttal and the additional results we provided. We would greatly appreciate hearing whether these have addressed your concerns.

We remain fully available if you have any further questions or if any additional clarification would be helpful. If you find our response satisfactory, we would deeply appreciate it if you could consider reflecting this in your score and assessment.

Thank you again for your time and thoughtful feedback. We look forward to hearing from you!

Reviewer_D7fzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MemAff, a method requiring no task-specific training that converts scene scans into persistent Markdown memory containing a scene overview, a walkthrough narrative, and an object inventory. A vision-language agent updates the records through insertion, replacement, and image-revisiting tools, links them to image frames, and stores spatial or functional relationships. At query time, it first parses the target, host object, and tool to be used, then selects views with better visibility and predicts contact points. Finally, it uses SAM3 temporal propagation and cross-view voting to produce a 3D affordance mask. The author evaluate the method on SceneFun3D, demonstrating its effectiveness.

Critical strengths

The benefits of memory are supported by strong paired comparisons. Appendix Table 8 compares direct frame-by-frame querying with the memory-based method using the same sampled frames and corresponding VLMs. Across the three backbones, memory improves mAP by approximately 11.98, 11.68, and 11.95 percentage points, respectively; for the main backbone, mIoU increases from 10.70 to 24.90. This provides stronger support for the usefulness of organizing observations into reusable memory than simply placing systems based on different large models on the same leaderboard. The main experiments also achieve strong results in both AP50 and mIoU.

Critical weaknesses
  • The efficiency lacks a consistent measurement: Although the authors have clarified that the 20$\times$ speedup mainly refers to online querying after memory construction, the current evidence remains insufficient to verify a complete efficiency comparison under matched conditions. Thus, there is evidence that reuse reduces costs, but establishing the speedup achieved at a given number of queries, under specific caching and hardware configurations, still requires consistent logging and baseline measurements.
  • The attribution of functional-relation capabilities remains limited: The author emphasize long-range functional connections, such as those between remote controls and televisions or switches and lights, but provides neither a controlled comparison that removes only functional cross-links while retaining the other records, nor subgroup results and relation accuracy for such tasks. FunEnhance uses the same scenes and masks, while ScanNet++ provides only object-level qualitative visualizations; neither therefore independently validates the reliability of functional relations across a broader range of scenes.
Questions for authors
  • Unified cost accounting. Could the authors clarify how the 20 memory-construction inferences and the 61.2 calls are counted, and report time distributions and input/output token counts for memory construction, caching, and individual queries on the same set of scenes? For the direct-frame control using the same backbone and baselines that support caching, what are the total costs at Q = 1, 10, and 20? These figures would determine the strength of the efficiency claims; this request does not require deployment on new hardware.
  • Contribution of functional links. When object descriptions, frame anchors, models, and budgets are held fixed, how does removing functional cross-links affect “distant target–tool” tasks and ordinary part-level tasks? What proportion of instructions in the existing test set fall into the former category? Could the authors audit the correctness of these relations and distinguish observed connections from commonsense inferences?
  • Evaluation and reproducibility details. Please specify the result sources or rerun settings for each baseline in the main table, the numbers of validation scenes and instructions, and uncertainty aggregated by scene. Please also clarify whether the denominator of Normalize in Eq. 9 includes all views or only views that are visible and depth-consistent, and provide the complete prompts for view selection and contact-point localization. Appendix G.3 currently provides only decision summaries and explicitly states that these two complete prompts are not included.
2 comments
AuthorsOpenReview ↗

Response to Reviewer D7fz

We sincerely thank the reviewer for recognizing our work. We address each concern below.

W1 & Q1. End-to-end efficiency and cost accounting

We agree that online latency alone cannot establish cumulative speedup. We added a matched-backbone comparison on 6 SceneFun3D scenes (20 original instructions each), with identical sampled frames (80 per scene), localization, SAM3 segmentation and 3D lifting.

Measurement boundaries and caching. Self-hosted timings depend on configuration and load. On the same third-party API, we timed setup per scene and 120 queries per method from instruction to final 3D prediction. Token counts are means; input totals include provider-reported cached tokens. Memory reuse reduced both mean query time and input tokens.

StageMean (s)Median (s)Input tokensOutput tokensCached input tokens
MemAff setup / scene272.22263.12519,07619,181105,974
MemAff online / query22.9522.4015,5429135,901
Direct-frame online / query320.52308.66269,84710,63077,601

Construction rounds and VLM calls. The 20 rounds are batch-level steps for building one scene’s memory. The 61.2 calls are the mean total VLM requests per scene, including visual-input requests, tool-feedback follow-ups and final consolidation. Each round may involve multiple requests; local operations do not count as VLM calls.

Cumulative time. Following Eq. (10), we count setup once per scene and sum the recorded times for queries 1 through Q.

QMemAff time (s)Direct-frame time (s)Speedup
1293.16320.521.09×
2318.12641.042.02×
10502.323203.136.38×
20731.276413.418.77×

Both methods initially observe the scene, giving comparable first-query costs including setup. MemAff then reuses memory while the control searches anew, yielding increasing cumulative speedups: 2.02× at Q=2 and 8.77× at Q=20.

W2 & Q2. Contribution and reliability of functional cross-links

We deleted only functional-link assertions from GPT-6.1Sol and Qwen3.5-derived memories, keeping other descriptions, frame anchors, the query pipeline and inference limits fixed.

Task coverage and subgroup results. Of 445 original SceneFun3D instructions, 72 (16.18%) were cross-link candidates and 373 (83.82%) were ordinary part-level tasks. Δ = ablated − full mIoU.

MemoryTask groupFull mIoUAblated mIoUΔ (pp)95% CI for Δ
GPTCross-link candidates (72)22.3219.13−3.19[−4.05, −2.79]
QwenCross-link candidates (72)22.1018.04−4.06[−4.64, −3.56]
GPTOrdinary part-level (373)25.8624.61−1.25[−1.74, −0.98]
QwenOrdinary part-level (373)25.5424.42−1.12[−1.46, −0.92]

Link removal reduced mIoU by 3.19/4.06 on cross-link candidates versus 1.25/1.12 on ordinary part-level tasks (GPT/Qwen). All four paired 95% scene-bootstrap intervals for Δ are below zero. Explicit links benefit both groups, with larger gains on the evaluated cross-link candidates.

Relation evidence audit. We used GPT-6-astra to audit 477 occurrences of cross-object functional-relation statements in the scene memories built by MemAff, checking each against up to eight linked images: 391 (81.97%) had direct visual support; 73 (15.30%) had inferential support, e.g., a single TV and remote in a room; 4 (0.84%) were visually contradicted and 9 (1.89%) unverifiable.

Q3. Evaluation and reproducibility

Baseline sources. We reran Fun3DU, AffordBot and TASA in their original configurations because they localize or segment 3D functional regions. We used published scores for general open-vocabulary/object-level methods (OpenMask3D, LERF, OpenIns3D), methods without fully open-source implementations (AFFORDMEM, UniFunc3D).

Scene-level uncertainty. For 445 instructions across 30 SceneFun3D validation scenes, we bootstrap scenes with replacement 2,000 times, keeping predictions fixed, retaining each scene’s instructions together, and recomputing task-weighted metrics. The 2.5th/97.5th percentiles define 95% CIs. Paired comparisons, including link ablation, share scene draws. This preserves within-scene dependence and measures sensitivity to scene composition.

Normalization in Eq. (9). Each point’s foreground votes are divided by its valid observation count across temporal fusion views. Observations count if the point projects inside the image and passes visibility and depth-consistency checks, even with zero foreground votes. Additional foreground filtering affects only the numerator. Foreground requires ≥1 valid observation and a vote fraction ≥0.50. We clarified this after Eq. (9).

Complete prompts. We add the complete system prompts and user-message templates for view selection and contact-point localization to Appendix G.3, including input formatting, coordinate conventions and output schemas.

I thank the authors for their rebuttal, which addresses my concerns. After considering the other reviewers’ assessments, I have decided to maintain my current score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.