- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MemAff is a training-free framework for repeated-query 3D affordance segmentation. It builds a persistent, frame-linked scene memory through VLM-driven editing and visual recall, then performs instruction-conditioned retrieval, point localization, temporal segmentation, and multi-view fusion. Evaluation uses SceneFun3D and its instruction-level extension, FunEnhance.
Critical strengths
- The paper provides a repeated-query formulation. Reusing frame-linked scene knowledge addresses redundant perception and provides an inspectable interface for grounding.
- The Experiment evidence supports memory utility. Matched-frame comparisons across three VLMs, together with editing and recall ablations, support the usefulness of persistent memory beyond changes in visual inputs or backbone.
Critical weaknesses
-
The reported 20× speedup is difficult to assess without comparable baseline timings. Section 5.1 states that prior methods, including TASA, take over 20 minutes per instruction, but TASA reports 37.61 s in its own setting. The appendix discusses amortization, yet it remains unclear whether these comparisons account for the same processing steps and reusable computation.
-
The experiments do not show how much functional cross-links contribute beyond the rest of the memory. Removing the section component also changes descriptions and frame access, while similarity-based retrieval changes several reasoning steps. Neither comparison directly tests whether the links help resolve remote functional dependencies.
-
Equation (9) leaves the normalization unclear: are votes divided by all frames or only those with valid observations? This matters for partially visible points at the 0.5 threshold, though it does not establish an implementation error. Also, removing propagation and fusion together does not show how much of the improvement comes from voting itself.
-
It is unclear whether the reported AP follows the official SceneFun3D protocol. Fun3DU/AffordBot instead threshold point-level precision, which can assign high scores to small, incomplete regions. Results from these two definitions are not directly comparable.
Questions for authors
Please see the weakness.
4 comments
Follow-up on Our Rebuttal and Additional Results
Dear Reviewer KR2u,
As the discussion phase proceeds, we would like to kindly follow up on our response and check if our rebuttal along with the additional results have adequately resolved your concerns.
We remain fully available if you have any further questions or if any additional clarification would be helpful. If you find our response satisfactory, we would deeply appreciate it if you could consider reflecting this in your score and assessment.
Thank you very much for your time, valuable feedback, and dedication!
Thank you for the additional experiments. They address parts of my earlier concerns. However, the evaluation concern remains: the reported AP uses point-level precision thresholds, while the official SceneFun3D evaluator uses instance IoU. Appendix C.1 discusses differences in inputs and models but does not explain why the official evaluator was not used. Since the methods ultimately output 3D masks, Table 8 cannot replace a comparison under the official metric. I therefore maintain my original score.
Evaluation protocol and comparability. We clarify that our evaluation follows the publicly released Fun3DU evaluator, consistent with the protocol adopted by the compared methods, including TASA, AffordBot, FunGraph, and UniFunc3D. We adopt this protocol to ensure consistent metric definitions and enable direct comparisons with prior work. Moreover, our evaluation jointly reports AP, AR, and mIoU, which respectively assess prediction precision, ground-truth coverage, and overall mask overlap. These complementary metrics provide a balanced assessment of segmentation quality rather than relying solely on precision. We acknowledge that this protocol differs from the official SceneFun3D instance-IoU evaluation and will explicitly clarify this distinction in the revised manuscript.
We hope this clarification helps address your concerns regarding evaluation comparability and would greatly appreciate your reconsideration of our work in light of these clarifications.
Response to Reviewer KR2u
We sincerely thank the reviewer for recognizing the repeated-query formulation and the evidence supporting persistent memory. We address each concern below.
w1: Timing boundaries and cumulative efficiency
Timing interpretation. TASA reports per-instruction time without specifying within-scene concurrency. For example, 20 instructions completed in 10 min average 30 s each, but overlapping requests can have longer individual latency. We evaluate all three methods sequentially, submitting each instruction only after the preceding 3D prediction completes. TASA’s default configuration averages 299.67 s from submission to prediction.
Measurement boundaries and cumulative time. Our original timing included local VLM setup and weight loading, obscuring comparison with baselines. We now use a third-party API for all VLM calls, including response waits but excluding local model loading and data preprocessing. For each scene, we sum the recorded times of the first Q queries and count MemAff’s setup once (272.22 s); mean online query time is 22.95 s.
MemAff and TASA have comparable first-query costs because both analyze the scene. Thereafter, MemAff reuses memory while TASA repeats view search, yielding a 1.88× cumulative speedup at Q=2, including construction. At Q=20, speedups reach 8.21× over TASA and 20.13× over Fun3DU. We will correct our original “20× per-instruction latency reduction” claim to reflect cumulative speedup.
w2: Isolating the contribution of functional cross-links
We isolate functional links by deleting only their assertions from existing GPT-6.1Sol and Qwen3.5-derived memories, rather than removing entire sections or changing retrieval. Other descriptions, frame anchors, retrieval, the downstream pipeline and inference limits remain fixed.
We separately evaluate 72 cross-link candidates (16.18%) and 373 ordinary part-level tasks (83.82%) among 445 original SceneFun3D instructions. Δ = ablated − full mIoU; negative values indicate loss after link removal. The 95% CIs use paired scene-cluster bootstrap resampling.
Link removal reduces mIoU by 3.19/4.06 on cross-link candidates versus 1.25/1.12 on ordinary part-level tasks (GPT/Qwen); all four 95% CIs lie below zero. The larger performance drops on cross-link candidates suggest that explicit functional links are particularly beneficial for tasks requiring functional relationships, while also contributing to ordinary part-level grounding.
w3: Vote normalization and the independent contribution of voting
Normalization in Eq. (9). We divide each point’s foreground votes by its valid observation count among fusion views, not all scene frames. Valid observations require the point to project inside the image and pass visibility and depth-consistency checks; zero foreground votes still count in the denominator. Additional foreground filtering affects only the numerator. Foreground requires ≥1 valid observation and a vote fraction ≥0.50. We added this clarification after Eq. (9).
Independent voting ablation. We vary only the final fusion rule, keeping memories, selected views, contact points, propagated SAM3 masks and geometric filtering fixed. With temporal propagation and all upstream outputs fixed, this comparison isolates the contribution of the final fusion rule. All three conditions use the same evaluator on 445 original instructions from 30 SceneFun3D scenes.
Point-wise valid-observation voting improves mAP/AP50/AP25/mIoU over union by 7.69/9.83/8.18/2.61 and mIoU over all-view normalization by 7.73. These results favor point-wise valid-observation voting among the three fusion rules compared.
w4: Evaluation protocol and comparability
Our metrics follow the evaluation code of Fun3DU, AffordBot and TASA, using point-level precision thresholds for AP rather than official SceneFun3D instance-IoU matching. We agree that results under these definitions are not directly comparable. We clarify the distinction between our shared protocol and the official evaluator in Appendix C.1.