- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies repair of derived memories after source corrections, distinguishing structural descendants, update-sensitive descendants, and stale stored states. In controlled experiments, LLM provenance reaches nearly all stale memories despite incomplete structural coverage, outperforming matched-random graphs. Value-agreement support reproduces this behavior and admits a reachability guarantee under stated assumptions. MemoryAgentBench experiments further show that stale intermediates harm subsequent answers.
Critical strengths
- The paper makes a useful distinction between provenance fidelity and repair adequacy. Equations (1)–(3) separate historical lineage, update sensitivity, and actual stored-state error. Accounting for reader errors is particularly valuable: a sensitive descendant may already contain the updated answer, while an insensitive descendant may remain incorrect. This provides a more precise evaluation framework than measuring structural edge recovery alone.
- Several complementary controls strengthen the interpretation. The value-agreement baseline offers a concrete explanation for the observed selectivity. The second attributor improves structural precision and recall while increasing recomputation without improving stale coverage. The content-removal and length-matched replacement experiments also help attribute behavioral degradation to stale content rather than merely additional context.
Critical weaknesses
- The controlled environment is narrow in ways that favor the proposed support criterion. All rules are unate, answers are binary, task rules are explicitly supplied, retrieval prioritizes required parents, and the depth sweep uses small chains. This is an effective diagnostic environment, but it does not reveal how robust the result is to ambiguous support, imperfect retrieval, larger branching graphs, or memories containing multiple claims. In particular, agreement of the final Boolean answer can conceal obsolete reasoning or supporting details, as the limitations section recognizes.
- Reduced re-derivation work is not yet evidence of reduced total cost. The reduction from roughly 8.2 to 5.9–6.0 re-derivations per episode is meaningful under the fixed operator, but the LLM provenance approach also requires attribution calls. Reporting attribution and repair tokens, latency, and total calls would clarify the practical tradeoff. The current results support savings in repair work specifically.
Questions for authors
- Does the value-agreement baseline retain near-oracle post-repair accuracy when run end to end with fresh generation and refreshed attribution?
- Can the authors establish sufficient conditions under which formation-time value-agreement edges remain adequate across sequential corrections?
- Can LLM attribution be evaluated on the public multi-hop setting, including stale coverage, structural coverage, and repair work?
- How were public evaluation units selected, and why do original and reversed orderings yield different unit counts? Are confidence intervals clustered sufficiently to account for any shared source streams?
Response to Reviewer oR6d
We thank Reviewer
oR6dfor the thoughtful review of our three-set formulation and controls, and have added closed-loop runs, sequential-update theory, public LLM attribution, unit accounting, and cost analyses.1. Value-agreement provenance under live regeneration and re-attribution
Yes. Paired values below are reported as Luna/DeepSeek. In closed-loop runs across D∈{1,3,5} (50 episodes/reader) regenerating answers and refreshing provenance at every repair, value-agreement provenance reached 0.97/1.00 stale coverage and 0.05/0.00 final probe errors (oracle: 0.02/0.00) while cutting re-derivations from 10.8/10.8 to 5.7/5.8. Its selectivity holds when repairs alter subsequent memory state (§5.5, Fig. 3, Table 16).
2. Sufficient conditions for formation-time edges across sequential updates
Proposition 2 (§3.4) proves this on a fixed structural DAG and fixed deterministic local rules over finite parent domains: (i) at formation, each node computes its value accurately from current parents; (ii) recorded edges contain a certificate forcing that stored value; and (iii) at each correction, reachable active nodes are invalidated, re-derived in topological order from the latest values of all required structural parents, and given a fresh certificate.
While certificate parents remain uninvalidated, they keep their formation values and force the stored output regardless of omitted parents; if any certificate parent changes, the node is reached, re-derived, and given a new certificate. By induction, across arbitrary sequential or simultaneous updates, all active memories match the task values of the current root assignment. On unate rules, formation-time value-agreement parents form a certificate (proof and 20,000-system checks/family: App. C, Table 8).
3. LLM attribution coverage and repair work on public multi-hop data
Replacing oracle path provenance on
MemoryAgentBenchwith frozendeepseek-flash(§5.7), final accuracy over all evaluated units matched oracle repair in 7 of 8 conditions and was +0.015 higher (95% CI [0.000, 0.040]) in 1 (0.42–0.56 re-derivations/unit vs. 0.46–0.61 for oracle). Because 94–100% of structurally linked intermediates on these lookup paths change value after an upstream update, structural and stale coverage nearly coincided (0.75–0.98), showing the boundary case without redundant support (Fig. 5, Table 11).4. Evaluation-unit selection, ordering denominators, and shared-fact dependence
A unit is a multi-hop question with a unique relation path whose pre-correction path is complete at the intermediate query step and differs from the post-correction path on at least one hop. Input order changes which questions have a complete pre-correction window before the update: in
mh_6k, 4 questions lack a resolved path in both orders, while 29 (original) vs. 34 (reversed) lack a pre-correction window, giving 67 vs. 62 units (65 vs. 52 inmh_32k; App. D Table 10).To handle shared facts within a stream, we group questions sharing any subject–relation key into connected components (30–46/split) and report component-clustered bootstrap 95% CIs alongside unit CIs (Table 10). Under component clustering, the write-back × validity interaction upper bound is < 0 in 7 of 8 conditions (primary
mh_6kreversed: Luna [-0.62, -0.21], DeepSeek [-0.38, -0.09]); onlymh_32kreversed DeepSeek (with a small point effect) touches zero at [-0.19, 0.00], noted in App. D.5. Do fewer re-derivations mean lower total cost?
App. H (Table 20, pooled over D=1,…,5) separates attribution and repair calls, tokens, and latency. Separate thinking-enabled
deepseek-flashattribution uses around 102k/101k tokens/ep. (~12.5k for low-effort Luna), exceeding tokens saved on short Boolean repairs (2,104/2,537). Thus §§5.5 and 6 scope efficiency claims strictly to re-derivation work reduction (44–57%). Meanwhile, value-agreement and minimal-certificate provenance cut re-derivations without LLM attribution calls: minimal certificates lowered repair time/ep. from 20s to 9s (Luna) and 15s to 7s (DeepSeek), while edge logging added 1.8 ms CPU time and 384 bytes/ep.6. Incomplete retrieval, wider graphs, and multi-claim memories
We added closed-loop runs on retrieval misses and wide branching graphs, and explained multi-claim handling in App. C. Dropping required parents at 25% degraded all methods including oracle repair (1.76/1.48 errors) and lowered LLM stale coverage to 0.80, confirming complete parent retrieval is needed (Table 17). On non-unate rules, value agreement dropped to 0.42/0.41 while minimal certificates and LLM provenance reached 0.99/1.00; on wide branching graphs (max fan-out 10), LLM provenance kept 1.00/1.00 stale coverage (Table 19). For multi-claim memories, App. C outlines per-claim certificates or retaining the union of component-claim certificates.