- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies prompt-injection attacks against memory-augmented LLM agents, focusing on how injected content propagates through memory updates, retrieval, context construction, and downstream decisions. The authors propose a Lifecycle Red Teaming framework consisting of LifecycleTracer, which reconstructs injection paths and identifies lifecycle stages associated with attack outcomes, and LifecycleRefiner, which uses attribution feedback to guide attack revisions. The framework is evaluated across four memory platforms (Qdrant, LangMem, Graphiti, and Mem0), covering lifecycle reconstruction, stage localization, interaction attribution, and attack refinement.
The problem is relevant, and the paper presents a substantial experimental effort. However, I have significant concerns about the methodological novelty, the strength of the evaluation protocol, and whether the reported results adequately support the broader claims. In particular, the reconstruction and attribution results rely heavily on controlled operational references and replay assumptions, while the practical improvement from attribution-guided refinement over the strongest baseline is relatively small. Overall, I find the contribution interesting but not sufficiently convincing for acceptance in its current form.
Critical strengths
1-The paper addresses an important problem in the security of memory-augmented agents. Distinguishing between injection placement, retrieval, context inclusion, and downstream influence is a useful perspective that is often obscured by end-to-end attack success metrics. 2-The authors make a considerable implementation effort, evaluating the framework across four memory platforms and several attack families. The inclusion of matched execution controls and multiple diagnostic settings is appreciated. 3- The treatment of incomplete evidence is a positive aspect of the work. In particular, retaining unresolved judgments rather than forcing unsupported conclusions is a sensible design choice. 4- The paper provides extensive methodological and experimental details in the appendix, including execution controls, evaluation metrics, and computational costs.
Critical weaknesses
1- The methodological novelty is not sufficiently clear: My main concern is that the proposed framework appears to combine several relatively established ideas, including event provenance, state comparison, controlled replay, and feedback-guided attack selection. While applying these techniques to persistent agent memory is useful, the paper does not clearly establish what new methodological insight is introduced beyond organizing these components into a unified framework. For example, LifecycleTracer reconstructs memory paths using identity, derivation, and observation checks, while the attribution procedures compare outcomes under matched interventions. LifecycleRefiner then selects candidate revisions based on the resulting feedback. These are reasonable design choices, but it remains unclear which aspects are fundamentally different from conventional provenance tracking and intervention-based debugging. The paper would benefit from a sharper distinction between the conceptual contribution of lifecycle-aware analysis and the algorithmic novelty of the proposed procedures. As written, the contribution appears primarily to be a systems integration and evaluation framework rather than a substantial new learning or inference method.
2- The reconstruction evaluation raises concerns about the independence of the ground truth: The paper reports strong lifecycle reconstruction performance, including 94.6% complete-case reconstruction and 85.5% signed F1. However, the reference judgments are constructed using canonical memory executions and operational evidence, as described in Appendix B.6. This raises a concern about the extent to which the evaluation independently validates the proposed reconstruction mechanism. The reference labels and LifecycleTracer both rely on closely related notions of memory identity, derivation, and observed lifecycle operations. Therefore, agreement with these references does not necessarily demonstrate that the framework can recover the underlying causal history of an injection in a more general setting. I do not think this invalidates the reported results, but it substantially narrows their interpretation. The experiments demonstrate agreement with a specific operational reference under controlled instrumentation. They provide weaker evidence for general lifecycle reconstruction when observations are incomplete or when the true causal relationships are not directly represented by the available memory operations. An evaluation with independently constructed ground truth or known causal dependencies would make this contribution considerably more convincing.
3- The attribution claims are stronger than the available experimental evidence: The paper emphasizes the ability to identify interactions between injections and determine how one injection changes another's downstream effect. However, the proposed attribution procedure is based on comparisons between matched executions under selected interventions. Such comparisons can establish conditional effects under the tested configurations, but they do not necessarily identify a complete or unique causal explanation. The results depend on the selected intervention set, the available observations, and the replay assumptions. Furthermore, the interaction-attribution evaluation uses a relatively small 30-case controlled holdout, including 16 directed-interaction cases. The reported 79.2% overall F1 is promising, but the sample is limited given the complexity of the problem and the centrality of interaction attribution to the paper. I would like to see a substantially larger evaluation involving more diverse interaction structures, including higher-order dependencies and cases where relevant intermediate observations are unavailable. Without this, it is difficult to assess how reliably the method generalizes beyond the constructed scenarios.
4- The refinement results provide limited evidence of a substantial advantage over a strong baseline: The attack-refinement results are interesting, but I find their interpretation somewhat overstated. In Table 4, the pooled attack success rate increases from 41.4% for the original attacks to 93.9% for fixed-joint revision and 95.9% for LifecycleRefiner. While the improvement over the original attacks is large, the more relevant comparison is with fixed-joint revision, which already achieves nearly the same performance. The resulting 2.0-percentage-point improvement is relatively modest considering the additional complexity introduced by lifecycle reconstruction, attribution, and revision selection. The reported uncertainty estimates suggest that the gain is consistent within the evaluated setting, but its practical significance remains unclear. I would have liked to see stronger evidence that attribution-guided refinement provides advantages that simpler revision strategies cannot achieve. In particular, harder attack settings where the baseline is not already close to saturation would be more informative. This issue is especially important because refinement is presented as one of the paper's main contributions.
5- The localization experiments do not always distinguish the proposed method from simpler alternatives: The paper reports strong results for stage localization, but several controlled conditions appear relatively easy once sufficient evidence is available. For example, the additional results in Appendix B.7 show that simpler methods can achieve perfect F1 in multiple conditions. This makes it difficult to determine how much of the performance is attributable to the proposed lifecycle reasoning rather than the availability of informative observations. The experiments involving evidence redaction and abstention are more compelling, as they better reflect the difficulty of diagnosis under partial observability. However, the evaluation would be stronger if it focused more directly on these challenging conditions and compared methods at matched coverage and abstention rates. As a result, I am not fully convinced that the localization results establish a clear methodological advantage.
6- The evaluation remains highly controlled, and generalization is uncertain: Although the paper evaluates four memory platforms, the experiments rely on constructed applications, canonical records, platform-specific instrumentation, and matched execution protocols. These controls are useful for reproducibility, but they also limit the conclusions that can be drawn about realistic memory-augmented agents. In practical deployments, memory states may evolve over long interaction histories, retrieval behavior may vary between executions, and exact restoration or replay may not always be possible. The paper does not sufficiently investigate how these factors affect reconstruction reliability or attribution quality. Moreover, evaluating multiple memory backends under a shared controlled protocol does not necessarily establish generalization to substantially different agent architectures. A more realistic end-to-end evaluation with incomplete observability and less constrained agent behavior would significantly strengthen the paper.
7- The computational overhead is not adequately justified by the demonstrated gains: The framework involves multiple replay branches, memory restoration, additional retrieval operations, and repeated model evaluations. The appendix provides useful runtime and complexity analyses, but the cost-benefit tradeoff is not sufficiently emphasized. This matters particularly for LifecycleRefiner, where the improvement over fixed-joint revision is small. The paper would be more convincing if it demonstrated that the additional diagnostic computation produces meaningful improvements under comparable execution budgets. For instance, reporting attack success as a function of replay budget, model calls, or total runtime for all revision strategies would help clarify whether the proposed approach is practically worthwhile.
Questions for authors
1- What is the main algorithmic contribution of LifecycleTracer beyond conventional provenance tracking and controlled replay? Could the authors provide a clearer comparison with simpler alternatives? 2- How independent are the reconstruction reference labels from the operational assumptions used by LifecycleTracer? Can the authors evaluate against ground truth established separately from the proposed lifecycle representation? 3- How robust is the attribution procedure when memory restoration is imperfect, retrieval is nondeterministic, or intermediate observations are unavailable? 4- Why is the interaction-attribution holdout limited to 30 cases? Would the reported performance remain stable on a larger and more diverse collection of interaction scenarios? 5- Given that fixed-joint revision already achieves 93.9% attack success, what practical advantage justifies the additional complexity of LifecycleRefiner? 6- Can the authors provide stronger cost-matched baselines and evaluate harder attack settings where the current revision baseline is not close to saturation?
Thank you for your detailed reviews. Here are our responses:
W1/Q1&W5 · What does Tracer add to conventional provenance and replay, and does it outperform simpler alternatives?
Tracer prevents over-attribution by formally specifying when evidence links a stage-specific value to a specific source and memory version. While conventional replay logs execution, agent memory requires tracing mutating objects by checking read versions, alternate suppliers, and actual messages sent. At matched coverage, simple replay equaled Tracer. However, in exclusive origin tests, native binding falsely asserted exclusivity in 12/12 dual-supplier cases. By actively checking what remained after omitting the source, Tracer safely withheld these 12 ambiguous claims while preserving all 20 identifiable correct decisions.
W2/Q2 · How independent are the original reference labels, and what does a separately established reference show?
Evaluation against an independent, human-adjudicated readout confirms perfect 24/24 stage agreement. We acknowledge RQ1 measures agreement with an operational reference. To test independence, a human blindly evaluated 24 new native Graphiti cases.
W3/Q4 · Why use 30 cases, and do performance and causal attribution hold on a larger/diverse interaction set?
Expanding to 60 cases shows our revised algorithm perfectly recovers minimal conditions given dense observations, while fresh questions confirm natural higher-order dependencies. We expanded the original 30-case baseline to 60 generated cases (4, 6, or 8 inputs). Our revised algorithm recovered every minimal condition across all 51 fully observed cases, reaching 83.7% exact-edge F1 on dense observations (6,720 subsets). Separately, one of three fresh public questions strictly required all five annotated sessions to produce the correct answer (verified across 64 subsets), confirming that complex, higher-order dependencies naturally exist in agent workflows.
W4/Q5 · What practical gain remains over fixed joint at 93.9% success?
Selection eliminated 33.2% of the residual failures from the fixed joint baseline, recovering 65 of the 81 theoretically recoverable opportunities. Fixed joint succeeded on 2,997 of 3,193 applications, while selection succeeded on 3,062 (an absolute gain of +2.04 percentage points). This represents 65 additional successes out of the 196 residual failures. Since 81 of those failures had a successful evaluated alternative, LifecycleRefiner captured 80.2% of the recoverable opportunities. This measures the benefit of choosing among the evaluated revisions.
W6/Q3 · How robust is the attribution procedure under imperfect restoration, nondeterministic retrieval, or missing observations?
Tracer's validity gating robustly handles imperfect restoration and retrieval variability. By checking read versions and monitoring commits, it successfully prevented all 12 stale-copy acceptances and rejected all 24 observed mixed reads (costing only 8 abstentions on coherent reads).
W7/Q6 · Do cost comparisons demonstrate equal-cost gains on harder, unsaturated attacks?
On fresh, unsaturated QA cohorts, an efficient 6-call selector captured the recoverable benefit without exhausting the branch budget. On a 32-question inventory QA cohort, a 6-call check recovered 13 of 16 fixed-view errors. Increasing to a 15-call full check yielded no additional answers. Similarly, on 24 public dialogue questions, both 6-call and 15-call selectors recovered the same answers, proving that massive diagnostic branch exhaustion is unnecessary for cost-efficient deployment.