acceptodds
Under review as a conference paper at ICLR 2027

Tracing and Refining Injection Attacks in Agent Memory via Lifecycle Red Teaming

Abstract

Large language model agents increasingly suffer from untrusted data that can contaminate memory and pose major security risks. While recent red teaming research explores various memory injection attacks, current evaluations primarily measure final success rates. They overlook the lifecycle of injections and fail to capture the causal chain of attacks, making it impossible to identify the precise stage where an injection fails. Furthermore, current studies often overlook that injected memories might require interactions with each other to take effect. Tracing these hidden dynamics is difficult in standard systems but becomes exceptionally severe in emerging graph memory environments, where injected inputs undergo complex merging and structural evolution. To address these challenges across diverse platforms, we propose a Lifecycle Red Teaming framework to reconstruct the lifecycle path and refine injected memory attacks. Specifically, we first propose LifecycleTracer to reconstruct fragmented lifecycle paths and expose injection interactions. As the foundation, we establish a formal memory lifecycle chain and a validity gating mechanism to standardize evaluations across diverse memory platforms. To pinpoint precise failure stages for individual injections, we incorporate Intra-Lifecycle Replay Attribution to bridge causal gaps within native logs. To uncover complex memory interactions, we also design Cross-Lifecycle Dynamic Attribution to capture interdependencies among multiple injected inputs. Finally, we leverage this feedback to refine injection attacks. Extensive experiments confirm that our framework effectively traces the lifecycle chain and enhances attack performance.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
memory injection attack, llm agent memory, lifecycle tracing, red teaming

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6TP7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies prompt-injection attacks against memory-augmented LLM agents, focusing on how injected content propagates through memory updates, retrieval, context construction, and downstream decisions. The authors propose a Lifecycle Red Teaming framework consisting of LifecycleTracer, which reconstructs injection paths and identifies lifecycle stages associated with attack outcomes, and LifecycleRefiner, which uses attribution feedback to guide attack revisions. The framework is evaluated across four memory platforms (Qdrant, LangMem, Graphiti, and Mem0), covering lifecycle reconstruction, stage localization, interaction attribution, and attack refinement.

The problem is relevant, and the paper presents a substantial experimental effort. However, I have significant concerns about the methodological novelty, the strength of the evaluation protocol, and whether the reported results adequately support the broader claims. In particular, the reconstruction and attribution results rely heavily on controlled operational references and replay assumptions, while the practical improvement from attribution-guided refinement over the strongest baseline is relatively small. Overall, I find the contribution interesting but not sufficiently convincing for acceptance in its current form.

Critical strengths

1-The paper addresses an important problem in the security of memory-augmented agents. Distinguishing between injection placement, retrieval, context inclusion, and downstream influence is a useful perspective that is often obscured by end-to-end attack success metrics. 2-The authors make a considerable implementation effort, evaluating the framework across four memory platforms and several attack families. The inclusion of matched execution controls and multiple diagnostic settings is appreciated. 3- The treatment of incomplete evidence is a positive aspect of the work. In particular, retaining unresolved judgments rather than forcing unsupported conclusions is a sensible design choice. 4- The paper provides extensive methodological and experimental details in the appendix, including execution controls, evaluation metrics, and computational costs.

Critical weaknesses

1- The methodological novelty is not sufficiently clear: My main concern is that the proposed framework appears to combine several relatively established ideas, including event provenance, state comparison, controlled replay, and feedback-guided attack selection. While applying these techniques to persistent agent memory is useful, the paper does not clearly establish what new methodological insight is introduced beyond organizing these components into a unified framework. For example, LifecycleTracer reconstructs memory paths using identity, derivation, and observation checks, while the attribution procedures compare outcomes under matched interventions. LifecycleRefiner then selects candidate revisions based on the resulting feedback. These are reasonable design choices, but it remains unclear which aspects are fundamentally different from conventional provenance tracking and intervention-based debugging. The paper would benefit from a sharper distinction between the conceptual contribution of lifecycle-aware analysis and the algorithmic novelty of the proposed procedures. As written, the contribution appears primarily to be a systems integration and evaluation framework rather than a substantial new learning or inference method.

2- The reconstruction evaluation raises concerns about the independence of the ground truth: The paper reports strong lifecycle reconstruction performance, including 94.6% complete-case reconstruction and 85.5% signed F1. However, the reference judgments are constructed using canonical memory executions and operational evidence, as described in Appendix B.6. This raises a concern about the extent to which the evaluation independently validates the proposed reconstruction mechanism. The reference labels and LifecycleTracer both rely on closely related notions of memory identity, derivation, and observed lifecycle operations. Therefore, agreement with these references does not necessarily demonstrate that the framework can recover the underlying causal history of an injection in a more general setting. I do not think this invalidates the reported results, but it substantially narrows their interpretation. The experiments demonstrate agreement with a specific operational reference under controlled instrumentation. They provide weaker evidence for general lifecycle reconstruction when observations are incomplete or when the true causal relationships are not directly represented by the available memory operations. An evaluation with independently constructed ground truth or known causal dependencies would make this contribution considerably more convincing.

3- The attribution claims are stronger than the available experimental evidence: The paper emphasizes the ability to identify interactions between injections and determine how one injection changes another's downstream effect. However, the proposed attribution procedure is based on comparisons between matched executions under selected interventions. Such comparisons can establish conditional effects under the tested configurations, but they do not necessarily identify a complete or unique causal explanation. The results depend on the selected intervention set, the available observations, and the replay assumptions. Furthermore, the interaction-attribution evaluation uses a relatively small 30-case controlled holdout, including 16 directed-interaction cases. The reported 79.2% overall F1 is promising, but the sample is limited given the complexity of the problem and the centrality of interaction attribution to the paper. I would like to see a substantially larger evaluation involving more diverse interaction structures, including higher-order dependencies and cases where relevant intermediate observations are unavailable. Without this, it is difficult to assess how reliably the method generalizes beyond the constructed scenarios.

4- The refinement results provide limited evidence of a substantial advantage over a strong baseline: The attack-refinement results are interesting, but I find their interpretation somewhat overstated. In Table 4, the pooled attack success rate increases from 41.4% for the original attacks to 93.9% for fixed-joint revision and 95.9% for LifecycleRefiner. While the improvement over the original attacks is large, the more relevant comparison is with fixed-joint revision, which already achieves nearly the same performance. The resulting 2.0-percentage-point improvement is relatively modest considering the additional complexity introduced by lifecycle reconstruction, attribution, and revision selection. The reported uncertainty estimates suggest that the gain is consistent within the evaluated setting, but its practical significance remains unclear. I would have liked to see stronger evidence that attribution-guided refinement provides advantages that simpler revision strategies cannot achieve. In particular, harder attack settings where the baseline is not already close to saturation would be more informative. This issue is especially important because refinement is presented as one of the paper's main contributions.

5- The localization experiments do not always distinguish the proposed method from simpler alternatives: The paper reports strong results for stage localization, but several controlled conditions appear relatively easy once sufficient evidence is available. For example, the additional results in Appendix B.7 show that simpler methods can achieve perfect F1 in multiple conditions. This makes it difficult to determine how much of the performance is attributable to the proposed lifecycle reasoning rather than the availability of informative observations. The experiments involving evidence redaction and abstention are more compelling, as they better reflect the difficulty of diagnosis under partial observability. However, the evaluation would be stronger if it focused more directly on these challenging conditions and compared methods at matched coverage and abstention rates. As a result, I am not fully convinced that the localization results establish a clear methodological advantage.

6- The evaluation remains highly controlled, and generalization is uncertain: Although the paper evaluates four memory platforms, the experiments rely on constructed applications, canonical records, platform-specific instrumentation, and matched execution protocols. These controls are useful for reproducibility, but they also limit the conclusions that can be drawn about realistic memory-augmented agents. In practical deployments, memory states may evolve over long interaction histories, retrieval behavior may vary between executions, and exact restoration or replay may not always be possible. The paper does not sufficiently investigate how these factors affect reconstruction reliability or attribution quality. Moreover, evaluating multiple memory backends under a shared controlled protocol does not necessarily establish generalization to substantially different agent architectures. A more realistic end-to-end evaluation with incomplete observability and less constrained agent behavior would significantly strengthen the paper.

7- The computational overhead is not adequately justified by the demonstrated gains: The framework involves multiple replay branches, memory restoration, additional retrieval operations, and repeated model evaluations. The appendix provides useful runtime and complexity analyses, but the cost-benefit tradeoff is not sufficiently emphasized. This matters particularly for LifecycleRefiner, where the improvement over fixed-joint revision is small. The paper would be more convincing if it demonstrated that the additional diagnostic computation produces meaningful improvements under comparable execution budgets. For instance, reporting attack success as a function of replay budget, model calls, or total runtime for all revision strategies would help clarify whether the proposed approach is practically worthwhile.

Questions for authors

1- What is the main algorithmic contribution of LifecycleTracer beyond conventional provenance tracking and controlled replay? Could the authors provide a clearer comparison with simpler alternatives? 2- How independent are the reconstruction reference labels from the operational assumptions used by LifecycleTracer? Can the authors evaluate against ground truth established separately from the proposed lifecycle representation? 3- How robust is the attribution procedure when memory restoration is imperfect, retrieval is nondeterministic, or intermediate observations are unavailable? 4- Why is the interaction-attribution holdout limited to 30 cases? Would the reported performance remain stable on a larger and more diverse collection of interaction scenarios? 5- Given that fixed-joint revision already achieves 93.9% attack success, what practical advantage justifies the additional complexity of LifecycleRefiner? 6- Can the authors provide stronger cost-matched baselines and evaluate harder attack settings where the current revision baseline is not close to saturation?

1 comment

Thank you for your detailed reviews. Here are our responses:

W1/Q1&W5 · What does Tracer add to conventional provenance and replay, and does it outperform simpler alternatives?

Tracer prevents over-attribution by formally specifying when evidence links a stage-specific value to a specific source and memory version. While conventional replay logs execution, agent memory requires tracing mutating objects by checking read versions, alternate suppliers, and actual messages sent. At matched coverage, simple replay equaled Tracer. However, in exclusive origin tests, native binding falsely asserted exclusivity in 12/12 dual-supplier cases. By actively checking what remained after omitting the source, Tracer safely withheld these 12 ambiguous claims while preserving all 20 identifiable correct decisions.

W2/Q2 · How independent are the original reference labels, and what does a separately established reference show?

Evaluation against an independent, human-adjudicated readout confirms perfect 24/24 stage agreement. We acknowledge RQ1 measures agreement with an operational reference. To test independence, a human blindly evaluated 24 new native Graphiti cases.

Independent readoutCases where source inclusion changed target visibility or match, out of 24
Current facts9
Retrieved facts12
Actual model input8
Answer match7
Tracer perfectly matched this independent semantic ground truth on all 24/24 cases, accurately capturing nuanced dynamics like 5 upstream changes with fixed endpoints and 3 answers repeating expired facts. (Note: As in W1, Tracer also avoided over-attribution in exclusive origin tests where native binding failed).

W3/Q4 · Why use 30 cases, and do performance and causal attribution hold on a larger/diverse interaction set?

Expanding to 60 cases shows our revised algorithm perfectly recovers minimal conditions given dense observations, while fresh questions confirm natural higher-order dependencies. We expanded the original 30-case baseline to 60 generated cases (4, 6, or 8 inputs). Our revised algorithm recovered every minimal condition across all 51 fully observed cases, reaching 83.7% exact-edge F1 on dense observations (6,720 subsets). Separately, one of three fresh public questions strictly required all five annotated sessions to produce the correct answer (verified across 64 subsets), confirming that complex, higher-order dependencies naturally exist in agent workflows.

W4/Q5 · What practical gain remains over fixed joint at 93.9% success?

Selection eliminated 33.2% of the residual failures from the fixed joint baseline, recovering 65 of the 81 theoretically recoverable opportunities. Fixed joint succeeded on 2,997 of 3,193 applications, while selection succeeded on 3,062 (an absolute gain of +2.04 percentage points). This represents 65 additional successes out of the 196 residual failures. Since 81 of those failures had a successful evaluated alternative, LifecycleRefiner captured 80.2% of the recoverable opportunities. This measures the benefit of choosing among the evaluated revisions.

W6/Q3 · How robust is the attribution procedure under imperfect restoration, nondeterministic retrieval, or missing observations?

Tracer's validity gating robustly handles imperfect restoration and retrieval variability. By checking read versions and monitoring commits, it successfully prevented all 12 stale-copy acceptances and rejected all 24 observed mixed reads (costing only 8 abstentions on coherent reads).

Graphiti conditionEvidence and coverage
Pinned version52/76 known-valid pairs certified (156/156 correct stage judgments). All 24 known-invalid and 12 unknown-validity pairs safely withheld.
Mutable databaseOf 56 first reads, 24 mixed versions occurred. Commit monitoring accepted 24/32 coherent reads and strictly rejected all 24 mixed reads.
Under nondeterministic retrieval (144 native searches, 120 adjacent comparisons), rankings remained completely fixed. In 80 comparisons where hits could exchange ranks, Tracer made zero false calls and perfectly identified changes under actual source omission. As designed, missing intermediate observations appropriately triggered abstentions.

W7/Q6 · Do cost comparisons demonstrate equal-cost gains on harder, unsaturated attacks?

On fresh, unsaturated QA cohorts, an efficient 6-call selector captured the recoverable benefit without exhausting the branch budget. On a 32-question inventory QA cohort, a 6-call check recovered 13 of 16 fixed-view errors. Increasing to a 15-call full check yielded no additional answers. Similarly, on 24 public dialogue questions, both 6-call and 15-call selectors recovered the same answers, proving that massive diagnostic branch exhaustion is unnecessary for cost-efficient deployment.

Fresh QA cohortFixed viewSix-call source checkFifteen-call full check
Complementary inventory, 32 questions16/3229/19229/480
Public dialogue, 24 questions1/242/1442/360
Reviewer_tcLDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Lifecycle Red Teaming, a framework for diagnosing memory injection attacks. It traces injected content through memory updates, retrieval, model context, and final responses. Matched replays with different injection combinations help locate failures and identify interactions between injections. The framework also selects among fixed attack revisions using component feedback. Experiments across four memory platforms use adapted attacks and controlled cases to evaluate lifecycle reconstruction, failure localization, interaction attribution, and attack success. The results show high agreement with operational lifecycle references and improved attack success after revision.

Critical strengths
  1. The paper looks at a useful problem: understanding where a memory injection fails and how multiple injections affect each other. This can help developers debug attacks and defenses.

  2. The method distinguishes missing evidence from confirmed failure. It also uses matched replays to test an injection’s effect, rather than relying only on logs.

  3. The evaluation includes useful controls. In particular, the fixed-joint baseline helps separate the gains from adding attack records from those of the selection rule.

Critical weaknesses

Major weaknesses

  1. The tracing framework’s role in improving attacks is unclear. Fixed templates already raise ASR from 41.4% to 93.9%. The selector adds 2.0 points, and a simple rule using only success flags makes the same choices in the authors’ same-platform analysis. This does not show that detailed lifecycle feedback is needed.

  2. The experiments simplify the hardest part of the problem. Many cases directly write prepared records or graph relations, bypassing native memory extraction and consolidation. The attacks are also substantially adapted. For example, MINJA no longer uses its query-only injection process. It is unclear how well the framework handles real attacks whose content gets rewritten or merged.

  3. The results do not clearly show an advantage over simpler diagnostics. In RQ2, native-stage logging already matches replay at 100% decided accuracy. In RQ1, the reference uses the same operational rules as the tracer, while the baselines cannot issue negative judgments. Stronger comparisons are needed to establish what the framework adds.

  4. The interaction results are limited to small, controlled cases. The holdout has only 30 cases, each with four injections. The method executes 448 of the 480 possible subsets, so it saves little over exhaustive testing here. Its accuracy and cost on larger, naturally occurring interactions remain unclear.

Minor weakness

The evaluation reuses a relatively small set of questions. The 3,193 refinement applications come from 65 LongMemEval questions and 89 LoCoMo questions. The reported variation across five question subsets does not establish stability across independent conversations or runs.

Questions for authors
  1. Appendix B.9.2 shows that success flags alone reproduce the selector’s choices in the same-platform analysis. Which lifecycle information, beyond these flags, affects refinement? Is there existing evidence that it improves selection?

  2. Beyond the three focused checks in Appendix B.6.5, which results test tracing through native memory extraction, rewriting, or merging? Please clarify how much of the reported performance comes from these operations versus canonical writes.

  3. In RQ1, how much of the gain comes from better tracing versus the ability to issue negative judgments, which the baselines lack? Can the existing results separate these two effects?

3 comments

Thank you for your thoughtful comments and review. Here are our responses:

W1 / Q1 · How does the tracing framework improve attacks?

Lifecycle tracing drove the design of the components responsible for the massive 41.4% to 93.9% jump, while selection eliminates one-third of the remaining failures. The fixed joint is not an unguided baseline: it applies both components explicitly designed from Tracer's bottleneck analysis. Thus, the 41.4% to 93.9% jump inherently validates this tracing-guided design. The additional 2.04% selection gain recovers 65 of 81 cases where combining components caused interference, effectively eliminating 33.2% of residual failures.

W1 / Q1 · Which lifecycle information is absent from success flags, and does richer feedback improve selection?

Success flags suffice for online selection between pre-designed components, but only lifecycle tracing provides the stage-level feedback necessary to design them. Success flags only measure outcomes, missing exactly where attacks fail. For example, in 4 of 11 native Graphiti pairs, tracing revealed revisions successfully improved upstream stages (e.g., reaching the prompt) even when the final answer remained unchanged. Distinguishing "failed to reach prompt" from "failed to flip decision" was the critical feedback that drove our component design. Therefore, rich lifecycle feedback is indispensable for offline diagnostics and design, while outcome flags suffice for online same-platform selection.

W2 / Q2 · Which experiments actually exercise native extraction, and how much of pooled performance reflects these operations?

Graphiti experiments directly observed object updates, strict relation merges, and new source bindings. To standardize cross-platform evaluation, pooled applications bypassed the initial LLM-based memory extraction phase. However, this does not bypass native consolidation.

EvidenceCounting unitObserved native operation
Original Graphiti–ShadowMerge stratum100 applications29 updated a native object, including 7 strict same-relation merges
New raw-text Graphiti studyEvents across history and updates10,070 node-summary changes and 1,032 extensions of an existing fact’s source bindings
In the 100 original cases, the backend natively rewrote the payload in 29 cases and performed a strict same-relation merge in 7 cases. Furthermore, our new raw-text study triggered massive native consolidation. In one instance, a retrieved fact natively linked to two source episodes was followed into the actual message sent to the model.

W3 / Q3 · How much of the RQ1 advantage comes from negative judgments?

Restoring negative judgments adds 8.6 to 13.8 percentage points of signed F1 across platforms, driving Tracer's advantage over NaiveUnion. We rescored the 1,600 judgments per platform, holding positive outputs fixed and withholding negative outputs as unresolved.

PlatformNaiveUnionTracer, negatives withheldFull Tracer
Qdrant91.3%91.3%100.0%
LangMem90.8%91.4%100.0%
Graphiti82.2%85.0%98.2%
Mem076.0%80.7%94.6%
With negatives withheld, remaining margins over NaiveUnion drop to 0.0–4.7 points, isolating the massive contribution of negative certification.

W3 · Why did native logging tie replay in RQ2?

Native logging matches replay only when passive logs are perfectly intact (100% decided accuracy at 83.3% coverage). When we hid passive source bindings in 12 controls, native logging abstained. Given the same paired executions, Tracer utilizing replay successfully recovered all 12 paths and all 36 stage judgments.

W4 · How do accuracy and cost change on larger interaction problems?

The revised algorithm improves exact-edge F1 on larger interactions given dense observations, while runtimes remain efficient. The original test executed 448/480 subsets (saving 6.7%). On 60 new cases with four, six, and eight inputs, we fed identical observations to the original algorithm (which had a condition minimization flaw) and the revised algorithm.

Observed subsets across 60 casesOriginal F1Revised F1
1,32955.3%49.4%
6,72064.9%83.7%
With denser observations, the revised algorithm reaches 83.7% exact-edge F1. Separately, at four inputs (8 observations), subset-size ordering and uniform sampling both scored 57.1% F1 in near-identical time (237 vs 235 ms). Thus, the framework remains computationally practical.

W4 · Do higher-order interactions occur on natural questions?

Yes. One of three fresh public questions strictly required all five annotated sessions for the reader to produce the correct answer. On ten earlier three- or four-source questions, direct answers passed on 0/10, while a two-step procedure passed on 2/10. For the fresh question, the correct answer appeared exactly when all five annotated sessions were present across 64 subsets. This confirms that higher-order dependencies exist naturally in agent memory workflows.

Thank you for the detailed response. The new controls show where replay helps when logs are incomplete. The negative-judgment ablation also clarifies the RQ1 gains. Two concerns remain:

  1. The most distinctive tracing capability is still not well validated. One important part is tracing an injection after its content is rewritten or merged with other memories. The new study shows that many such updates occur, but it is unclear whether the framework correctly reconstructs their sources and paths.

  2. The refiner still does not fully illustrate the value of detailed tracing. The revisions are predefined. On the same platform, a simple rule using only attack success or failure selects exactly the same revisions as the full method. Tracing may have inspired the templates, but their effectiveness does not establish that tracing improves their design. The selection rule also needs more explanation: Appendix B.9.2 reports 12 cases where both individual components succeed, but the rule selects their failing combination. Why choose the combination when successful alternatives have already been evaluated?

I appreciate this memory-specific diagnostic framework. However, the current evidence is stronger for controlled stage checks and fixed attack templates than for reliable tracing through memory evolution or tracing-guided refinement.

Thank you for the thoughtful follow-up and for acknowledging our replay controls and negative-judgment ablation. Below is our concise response to your two remaining points, along with how we will sharpen the paper's.

1. Does LifecycleTracer accurately reconstruct sources and paths after native updates and merges?

To directly verify path and source accuracy under native updates and merges where ground truth is known, we rely on three scored evaluations:

  • Path accuracy after native Graphiti updates (24 blind-scored cases): An independent evaluator scored raw Graphiti facts, retrieval returns, outbound prompts, and answers without seeing diagnostic outputs. LifecycleTracer matched all four stages on 24/24 cases, including 3 cases where answers repeated an expired retrieved fact.
Independent readoutCases where source inclusion changed target visibility or match, out of 24
Current facts9
Retrieved facts12
Actual model input8
Answer match7
  • Source accuracy after multi-source merging (dual-supplier tests): When a fact linked to two source episodes, native binding falsely asserted exclusive origin in 12/12 cases. By checking post-omission state, LifecycleTracer recovered 20/20 identifiable single-source cases and safely withheld all 12/12 ambiguous merged claims (and similarly withheld 8/8 ambiguous claims when alternate suppliers ranked 3rd–4th).
  • Path accuracy on strict relation merges (Table 8, Appendix B.6.5): Across the 100 Graphiti–ShadowMerge cases (29 native object updates; 7 strict same-relation merges with version advancement and payload change), LifecycleTracer reaches 99.2% F1 / 98.0% complete-case agreement (vs. 82.5% / 37.0% for NaiveUnion), with manual inspection confirming both audited strict-merge cases.

2. Role of tracing in refinement and why Eq. 6 picked the joint branch in the 12 tie cases

  • On tracing-guided design and selection: The refinement study establishes two complementary results: offline stage tracing pinpoints pipeline bottlenecks to inform revision components, while individual online evaluation resolves component interference efficiently. Specifically, stage-level diagnostics exposed where attacks stalled (e.g., failing to reach the prompt vs. failing to flip the decision), directly motivating the design of d1d_1 and d2d_2. Online, testing candidates individually prevents destructive interference from blindly combining components. Our ablation further demonstrates practical deployment efficiency. Once candidate revisions are fixed on the same platform, simple branch sufficiency flags reproduce all 3,193 selections, capturing this interference-resolution gain without requiring full stage traces at selection time.
  • On the 12 cases where both components succeed: Eq. 6 selected the joint branch in those 12 cases because it only switches away from the default joint action when exactly one component succeeds. Simply choosing the smaller successful component (d2d_2) whenever both succeed immediately recovers all 12 cases (raising pooled ASR to 96.3%). Specifically, Eq. 6 depends only on the singleton certificates (φr(d1),φr(d2))(\varphi_r(d_1), \varphi_r(d_2)) without inspecting the joint outcome, so when both components succeed (φr(d1)=φr(d2)=1\varphi_r(d_1)=\varphi_r(d_2)=1), it retains the default {d1,d2}\{d_1, d_2\}. Selecting d2d_2 instead (3 records vs. 4 for d1d_1 and 7 for joint) when both succeed guarantees zero regressions on the recorded same-platform branches (since φr(d2)=1\varphi_r(d_2)=1 confirms d2d_2 succeeds), increases pooled ASR from 3,062/3,193 (95.9%) to 3,074/3,193 (96.3%), and avoids running the joint branch in those cases. We will clarify this design choice and include this simple rule update in Section 4.3 and Appendix B.9.2.

3. Aligning the paper's framing with your overall assessment

We appreciate your recognition of our memory-specific diagnostic framework. In the revision, we will sharpen the Abstract, Introduction, and Section 5.5 to foreground our core validated contributions—LifecycleTracer (stage-level memory diagnostics, validity gating with certified negatives and abstention, and conditional interaction attribution)—while positioning LifecycleRefiner crisply as a downstream application demonstrating interference-aware selection over candidate revisions, ensuring every claim aligns tightly with our empirical evidence.

Do the responses above address your remaining concerns and clear up any ambiguities? We look forward to your further feedback.

Reviewer_QcpMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Lifecycle Red Teaming, a framework for analyzing and improving memory injection attacks against LLM agents. The framework consists of two main components: LifecycleTracer and LifecycleRefiner. LifecycleTracer reconstructs the propagation of injected content through different memory stages, including storage, retrieval, context inclusion, and final response generation. It further employs controlled replay and cross-lifecycle attribution to identify failure stages and interactions among multiple injected memories. LifecycleRefiner leverages attribution feedback to select among predefined attack revisions.

Experiments across four memory platforms (Qdrant, LangMem, Graphiti, and Mem0) demonstrate improvements in lifecycle reconstruction, failure localization, and attack success rates. The proposed framework achieves an average attack success rate of 95.9%, compared with 41.4% for the original attacks and 93.9% for a fixed joint revision baseline.

Critical strengths
  1. This paper is well-written and easy to follow.

  2. The paper highlights an important limitation of existing memory injection evaluations, which primarily focus on final attack success rates without examining how malicious information propagates through different memory stages.

  3. The proposed LifecycleTracer provides a structured approach to identifying injection failure stages and analyzing interactions between multiple injected memories. The use of controlled replay offers a useful mechanism for understanding attack behavior beyond final outcomes.

  4. The experiments cover four different memory platforms and multiple attack methods. The paper also includes ablation studies, controlled attribution experiments, and cross-platform transfer evaluations, providing relatively comprehensive empirical evidence.

Critical weaknesses
  1. The effectiveness of lifecycle attribution in improving attacks is not sufficiently demonstrated. Although the overall attack success rate increases substantially from 41.4% to 95.9%, the fixed joint revision already achieves 93.9%. Thus, only a 2.0 percentage-point improvement can be attributed to the proposed selection mechanism over this baseline. This raises questions about how much of the overall improvement actually comes from lifecycle-aware feedback rather than simply adding more malicious memory records.
Questions for authors

see weakness

3 comments

Thank you for your insightful feedback and careful assessment of our work. We present our response as follows:

W1-Part1 · How much does selection add over fixed joint?

Per-application selection adds 65 successes over fixed joint on the same 3,193 applications (+2.04 percentage points), successfully eliminating one-third of the remaining failures.

Both policies utilize the same diagnosis-guided revision components. While Fixed joint always applies them blindly together, LifecycleRefiner adaptively chooses among the evaluated revisions for each instance.

PolicySuccesses / 3,193Pooled success rateAdditional successes
Original input1,32241.4%—
Fixed joint2,99793.9%+1,675
Per-application selection3,06295.9%+65

The +2.04% absolute gain represents a substantial relative improvement near the performance ceiling. Furthermore, as reported in the appendix, there were 81 applications where the fixed joint failed, but another evaluated revision succeeded. Our selection mechanism successfully recovered 65 of these 81 opportunities (80.2%).

W1-Part2 · How much of the larger gain can be attributed to lifecycle feedback rather than record count?

We conduct a new matched-budget controls experiment to prove that merely increasing the record count yields no exact answers, confirming that the lifecycle-guided design is the actual driver of success.

The two fixed components were explicitly designed to address distinct bottlenecks identified by our tracing: failure of source information to reach the prompt, and failure to change the answer after reaching the prompt. Because the historical fixed revision changed both content and quantity, we introduced two matched-budget controls in native Graphiti, each scheduled to add exactly seven records.

ControlComplete pairsSource entered prompt only with joint / only with controlExact answers with joint / control
Repeat original records103 / 10 / 0
Add ordinary QA records91 / 00 / 0

Even with record counts matched, simply repeating original records or adding ordinary QA records failed to improve exact answers, and our joint revision was more successful at forcing the source into the prompt.

To further isolate structural design, a separate 16-question ordinary-QA test held the four source sentences, two records, total added tokens, and one-hit prompt budget strictly fixed within each pair. Grouping sentences by collection versus by attribute yielded different input and exact-answer outcomes on nine questions (five favored collection, four favored attribute). This confirms that attacks are highly sensitive to record organization and lifecycle-targeted design, not just the sheer volume of injected records.

Reviewer_QcpMOpenReview ↗

Response to rebuttal

Thanks for your detailed rebuttal! It has already adressd my original concerns. I lean to accept this paper, so I still maintain my original score

Thank you for your detailed reviews and recognition of our paper!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.