- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
CausalMem is a training-free memory framework for frozen visual reasoning VLMs that writes memory only after counterfactual validation: in a wake phase the frozen VLM answers while retrieving validated semantic rules, hard negatives, and anchor episodes; in a sleep phase (every 10 tasks) it proposes causal/nuisance factors, edits stored images with a frozen image editor (environment edit vs object edit), re-answers the edits, and promotes a rule only if it is invariant to nuisance edits and sensitive to causal edits, subject to fidelity filtering (tau_q=0.70) and a conflict check. A finite-sample bound (Prop. 1: Pr[accept] <= exp(-2n(tau-p_s)^2)+neta) formalizes how edit fidelity and validation diversity control false acceptance. On seven benchmarks, CausalMem improves frozen Qwen2.5-VL-7B by +4.18 and Qwen3-VL-8B by +3.29 average points over vanilla (best in 6/7 and 5/7 columns), with the largest gains on fine-grained visual search (V +8.90/+8.38); ablations show passive episodic RAG is 9.40 pts below vanilla, whereas counterfactual validation turns memory into a net gain (+15.65 over passive RAG).
Critical strengths
- The core idea - treat "which rules deserve to be remembered" as an interventional falsification problem rather than a retrieval problem - is a genuinely novel and well-articulated framing for agent memory; the shortcut-amplification failure mode it targets is real and rarely addressed. 2. Evaluation discipline is strong: transductive online protocol with no label leakage, stream-order and inductive-split controls, cross-benchmark transfer, alternative editor and VLM families, plus the striking ablation that passive RAG hurts while validated memory helps.
Critical weaknesses
- The evaluation protocol is transductive and online (memory updated during evaluation, same fixed stream for all methods) - the gains could in principle depend on stream order and the specific 10-task sleep cadence; the inductive-split control numbers were not visible in the read portion, so the robustness claim needs verification. 2. Cost: sleep-time editing + re-answering runs per hypothesis (K_e=K_o=4, M=3) adds real compute; the paper mentions cost measurements, but the numbers were not in the read portion - this matters for a "training-free" claim that is really "inference-time-costly."
Questions for authors
- How much does memory growth matter: does performance saturate as S grows, and how does the pruning budget B interact with the 0.80 retrieval cutoff? 2. What fraction of proposed hypotheses are actually accepted (selectivity), and how often does a rule get falsified after promotion (churn)? 3. In the inductive-split control, are the gains maintained when memory is frozen before evaluation - and is the gap to transductive evaluation quantified?
3 comments
Response to Reviewer qBJ9 (2/3)
Table E. Accuracy and cost on V* (Qwen3-VL-8B).
| Method | Acc. (%) | User-facing latency / question | Total time / question | Extra prompt tokens / question |
|---|---|---|---|---|
| Vanilla | 78.01 | 0.71 s | 0.71 s | 0 |
| M$^2$Note | 80.10 | 3.68 s† | 3.68 s | 14.0k |
| HtT | 79.06 | 8.46 s† | 8.46 s | 14.3k |
| CausalMem, InstructPix2Pix | 84.82 | 1.32 s | 7.71 s (1.32 wake + 6.39 sleep) | 16.8k |
| CausalMem, Qwen-Image-Edit | 86.39 | 1.32 s | 153.16 s (1.32 wake + 151.84 sleep) | 65.7k |
†HtT and M$^2$Note ran their update steps inline with answering in our measurement, so their per-question cost appears in both time columns; CausalMem's sleep is asynchronous and batched offline.
With the lightweight editor, CausalMem takes less time per question than HtT and is 5.76 points more accurate; against M$^2$Note it uses 2.1× the time for +4.72 points. Qwen-Image-Edit adds a further 1.57 points for about 20× the time, so editor quality is a knob with diminishing returns. CausalMem uses 16.8k extra prompt tokens per question with InstructPix2Pix (65.7k with Qwen-Image-Edit), against about 14k for the baselines. Because sleep is asynchronous and batched offline, the quantity on the user-facing critical path is the wake latency: 1.32 s, against 3.68 s for M$^2$Note and 8.46 s for HtT under the same measurement (V*, Qwen3-VL-8B). The trade-off plots are added to the experiments section of the Rebuttal Revision PDF and in our anonymous repository: and (click to the anonymous repository). Across all seven benchmarks and both backbones, these two baselines change the vanilla average by −1.99 to +0.65 points, against +3.29 / +4.18 for CausalMem, and CausalMem is ahead of both in all 14 benchmark–backbone cells. This is further support for the observation the reviewer highlighted: what makes the memory useful is the validation, not the retrieval.
4. The sleep cost is amortized. It is paid once per consolidated episode and not per query. The inductive split (Table C) keeps 82–98% of the gain with no sleep on any evaluated query, and one shared memory built from 300 unlabeled samples raises pooled accuracy over the full V*, MMStar and MMBench sets from 78.69 to 80.90 (Table 2).
In the Rebuttal Revision PDF the cost table moves into the main text, together with the cost of the two new verified-memory baselines and the trade-off plots, and the consolidation cost is stated next to the training-free claim in the introduction.
Does performance saturate as S grows, and how does the pruning budget B interact with the 0.80 retrieval cutoff?
Growth and saturation. More memory is not better. Without pruning, the store keeps growing (559 rules on MMStar, Fig. 4b), and accuracy on the V*/MM-Vet average drops by 4.55 points (69.84 vs. 74.39, Fig. 3a), because redundant and conflicting entries compete for the prompt. With pruning, S itself saturates: the active MMStar store ends at 97 rules (82.6% pruned, Fig. 4b), and the advantage over vanilla persists to the end of the stream (+8.3 points on the V* moving average, Fig. 3b). Returns also diminish in the number of consolidated episodes: a memory built from half of the data keeps 82–98% of the gain (Table C), and a memory built from 300 samples already lifts pooled accuracy on three full benchmarks by 2.21 points.
and the cutoff act at different stages. is a store-level budget enforced during sleep; it bounds how many items exist (97 rules on MMStar at tokens). The cutoff and the top-k limits act per query at wake time: at most 4 rules, 2 hard negatives and 2 anchors enter a prompt, which is why the wake context averages 2,234 tokens although is 12,000. therefore controls the size of the candidate pool and not the length of the prompt. The cutoff is a cosine distance of 0.80, i.e., a similarity of at least 0.20. It is a loose relevance floor: it matters mainly when the store has few relevant rules (early in the stream, or for queries outside the stored scopes), where it keeps unrelated rules out of the prompt. Among the items that pass it, selection is made by the ranking score of Eq. 3. The interaction is thus one-directional: a larger lets more low-value rules pass the cutoff and compete for the four slots.
Table F. × cutoff on V* (Qwen3-VL-8B), accuracy (%).
| \ cutoff (cosine distance) | 0.70 | 0.80 (default) | 0.90 |
|---|---|---|---|
| 4,000 tokens | 78.01 | 81.15 | 80.10 |
| 12,000 tokens (default) | 85.34 | 86.39 | 84.29 |
| No pruning | 78.53 | 79.58 | 78.01 |
Response to Reviewer qBJ9 (3/3)
The default sits at the grid optimum, and each direction alone costs 1.05–5.24 points (budget 12k → 4k: −5.24; cutoff 0.80 → 0.90: −2.10; 0.80 → 0.70: −1.05). The cutoff matters more when the budget is tight: the spread across cutoffs is 3.14 points at 4,000 tokens against 2.10 at 12,000, since a small store needs a permissive floor to be used at all. Removing the budget hurts at every cutoff, including a tighter one (best unpruned cell 79.58, against 86.39 at the default), which mirrors the −4.55-point no-pruning ablation on the two-task average (Fig. 3a).
What fraction of proposed hypotheses is accepted, and how often is a rule falsified after promotion?
Selectivity. The gate accepts 45.5% of the candidates on V* (87 accepted, 104 rejected) and 37.3% on MMStar (559 accepted, 941 rejected); see Fig. 4a. On V*, 94.23% (98/104) of the rejected hypotheses match one of five shortcut categories aligned with MM-SpuBench, mostly background/scene (79/104); this keyword audit is a coarse proxy and is described with its caveats in Appendix F.
Churn. After promotion, a rule can leave the active set in three ways: it fails a later sleep re-validation (a contradiction event, which also lowers its LCB), it loses a conflict to a rule with a higher LCB and becomes dormant, or it is evicted by the budget as redundant or low-value. The breakdown by cause is:
| Promoted | Failed a later re-validation | Dormant after a conflict | Evicted by budget / redundancy | Active at the end | |
|---|---|---|---|---|---|
| V* | 87 | 2 | 16 | 0 | 69 |
| MMStar | 559 | 13 | 57 | 392 | 97 |
Falsification after promotion is rare (2 of 87 on V*, 13 of 559 on MMStar), and a rule that loses a conflict simply becomes dormant (16 and 57). Nearly all removals on MMStar are budget evictions of redundant or low-value rules (392), which is how the active store stays bounded as the stream grows; on V* the store never reaches the budget.
In the inductive split, are the gains maintained when memory is frozen, and is the gap to transductive evaluation quantified?
Yes to both; the numbers are in Table C above. With memory built on one half, frozen, and the other half answered with wake only (Qwen2.5-VL-7B, both directions averaged), CausalMem reaches 81.15 on V* and 71.71 on HallusionBench, against 73.82 and 66.14 for vanilla: gains of +7.33 and +5.57. The transductive results on the same benchmarks are 82.72 and 71.82 (+8.90 and +5.68). The gap is therefore 1.57 points on V* and 0.11 points on HallusionBench, i.e., the frozen memory keeps 82% and 98% of the transductive gain while seeing only half of the data and none of the evaluated images. The larger gap on V* is consistent with its small size (191 questions), where halving the data available for consolidation matters more. In the submission we report this gap explicitly in the main text (Sec. 4.4).
Summary of revisions for Reviewer qBJ9. (i) The transductive–inductive gap and the worst-order result are stated in the main text of the submission (Sec. 4.4). (ii) The cost table and the trade-off plots move to the main text of the Rebuttal Revision PDF, together with the cost of the two new verified-memory baselines. (iii) Selectivity and churn statistics are reported in Sec. 4.5. We thank the reviewer again and are happy to provide further details.
Response to Reviewer qBJ9 (1/3)
Dear Reviewer qBJ9,
Thank you for the careful reading and the positive assessment of the framing and of the evaluation protocol. The two weaknesses concern numbers that are in the submission but outside the read portion (the cost table in Appendix F). We reproduce both below with new measurements, and the Rebuttal Revision PDF places them in the main text so they are visible without the appendix.
The inductive rows are the submission's Table 2 (p. 9).
Table C. Protocol controls (Qwen2.5-VL-7B; from Tables 1 and 2).
The numbers are in Table 5 (Appendix F). We moved them into the main text of the Rebuttal Revision PDF:
Table D. Cost per sample on V* (Qwen3-VL-8B; Table 5).
Four points put these numbers in context.
1. The user-facing cost is one VLM call. Wake takes 1.32 s against 0.71 s (1.86×) because of 2,234 extra prompt tokens. This context does not grow with the stream, since pruning keeps the active store near constant (Fig. 4b).
2. The editor dominates the sleep time and is replaceable. The editor accounts for 130.87 of the 151.84 s (86%). With InstructPix2Pix instead of Qwen-Image-Edit (Table 2), the whole sleep phase, VLM and editor together, takes 6.39 s per sample, and accuracy is 84.82 on V* and 61.47 on MM-Vet (86.39 and 62.39 with Qwen-Image-Edit).
3. Comparison with two verified-memory baselines. During the rebuttal we ran HtT (arXiv:2310.07064) and M$^2$Note (arXiv:2607.00685) under the protocol of Table 1, label-free, and timed them in the same environment. Time is model-call time per question for the entire pipeline (answering plus memory update).