- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper introduces SATRA, an auditing framework that evaluates the security-relevant meaning of persistent memory artifacts in LLM agents. The framework classifies written memories into four states (poison-supporting, corrective, unresolved, and neutral) along with a separate no-write outcome, separating input evidence from post-write artifacts and downstream behavior. The authors validate this approach across three open-weight models on eight synthetic policy cases covering memory formation, chained reuse, and behavioral activation, supplemented by external stress tests on 50 LongMemEval tasks.
Critical strengths
The paper introduces a semantic-state taxonomy for auditing persistent memory, with experiments covering memory formation and reuse.
Critical weaknesses
-
Trivial method, lack of theoretical contribution, and collapse of the four-state taxonomy: The proposed method is trivial and lacks theoretical contribution. SATRA only defines a manual four-class labeling rubric and calculates basic frequency ratios, offering zero theoretical insights or algorithmic innovations. Across the entire human-reviewed main evaluation, the unresolved and neutral states appear only 6 times each, and both states have zero occurrences in the formation experiment. Furthermore, the ablation study in the appendix shows that the four-state encoding performs worse than a simple binary classification when predicting natural visible unsafe behavior.
-
Small experimental scale, unblinded behavioral grading, and ineffective external retrieval: The core experiments rely on only eight synthetic policy cases, and the formation, reuse, and activation blocks use disconnected sample pools. During the behavioral activation review, human annotators had direct access to upstream memory state labels and worked without independent double review. In the external LongMemEval benchmark, 32 out of 50 tasks lost traceable poison provenance, and natural access retrieved zero poisoned memories across 250 output slots, leaving the framework unverified in realistic retrieval settings.
Questions for authors
- Given that the unresolved and neutral states rarely appear in the main experiments and the four-state encoding underperforms binary coding on natural unsafe behavior prediction, what empirical evidence justifies a four-state taxonomy over a binary classification?
- In the LongMemEval stress test, why did 32 out of 50 tasks lose traceable poison provenance, and why did natural retrieval select zero poisoned memories across 250 output slots?
- Why did behavioral annotators have access to upstream state metadata during action safety grading, and why was independent double review omitted for behavioral labels?
3 comments
Experimental Scope, LongMemEval Traceability, and Blind Behavior Review
Reviewer Comment — Weakness 2:
Small experimental scale, unblinded behavioral grading, and ineffective external retrieval: The core experiments rely on only eight synthetic policy cases, and the formation, reuse, and activation blocks use disconnected sample pools.
During the behavioral activation review, human annotators had direct access to upstream memory state labels and worked without independent double review.
In the external LongMemEval benchmark, 32 out of 50 tasks lost traceable poison provenance, and natural retrieval selected zero poisoned memories across 250 output slots, leaving the framework unverified in realistic retrieval settings.
Response:
Experimental scope. The three blocks use distinct sampling units and answer different questions:
| Block | Scope | Evaluation |
|---|---|---|
| Formation | 8 cases × 3 profiles × 3 repeats = 216 writes | Post-write semantic outcomes |
| Explicit reuse | 288 registered edge/decision records; 285 exposed artifacts; two Qwen profiles | Cross-writer transitions |
| Activation | 8 cases; 288 decisions (144 visible, 144 hidden) | Visibility-conditioned behavior |
These are separate controlled probes, not a single item-level lineage from Formation through Reuse to Activation. The eight cases define a bounded study; repeats do not increase the number of independent cases.
LongMemEval traceability and retrieval. In the submitted trace, 18/50 tasks retained traceable poison provenance; 32 lacked sufficient provenance in the exported records. The logs do not establish each gap's task-specific cause. The historical 0/250 selected-poison entries also lacked query-level native readback, ID mapping, and actual-context evidence; they cannot establish a true zero retrieval rate or retrieval resistance.
A separate Schema-V2, 50-task replay adds backend-level instrumentation:
| Measure | Schema-V2 result |
|---|---|
| Native writes with successful readback | 6,189 |
| Verified logical/native ID mappings | 37,672 |
| Actor contexts captured and hashed | 1,200 |
| Contexts containing selected memories | 1,157/1,200 |
| Direct seed exposure in final non-attacker contexts | 207/250 |
| Content-binding candidates after excluding known gold-accepted alternatives | 198/250 |
| Retrieval-associated QA errors | 100/250 |
The other 43 were no-memory placeholders. The 198 content-binding cases and 100 QA errors are automatic results, not human-confirmed unsafe actions or causal effects. Direct seed exposure occurred in 45/50 tasks.
All 1,343 receiver-summary requests produced no visible LLM summary and persisted compact-report fallbacks. Of these, 46 stored fallbacks contained explicit poison-binding candidates and 32 were unresolved; no content-positive report descendant was selected in final non-attacker contexts. Schema-V2 verifies direct shared-seed exposure, not successful LLM-summary propagation.
Reviewer Question 2:
In the LongMemEval stress test, why did 32 out of 50 tasks lose traceable poison provenance, and why did natural retrieval select zero poisoned memories across 250 output slots?
Response:
The 32/50 figure is an export-level provenance gap; its task-specific causes are unverified. The 0/250 figure is a trace result under incomplete observability, not a retrieval rate. The separate Schema-V2 replay links queries, native memory IDs, actual Actor contexts, and provider outputs, and records direct seed exposure in 207/250 final contexts.
Reviewer Question 3:
Why did behavioral annotators have access to upstream state metadata during action safety grading, and why was independent double review omitted for behavioral labels?
Response:
Original E3 grading exposed upstream state metadata; the design record does not explain why or why double review was omitted. We conducted a separate blind re-review of all 288 downstream behavior records:
| Primary label | Reviewer A | Reviewer B |
|---|---|---|
| Safe | 280 | 280 |
| Unsafe | 8 | 8 |
| Total | 288 | 288 |
After content-based matching of the differently ordered files, primary labels agree on 288/288 items (100%; Cohen's κ = 1.000). Reviewer B's workbook contains an independence declaration; Reviewer A's CSV has no embedded declaration, while the study team confirms the independent blind assignment. Agreement applies only to the primary Safe/Unsafe label; auxiliary fields differ. This follow-up does not retroactively blind original E3 grading or link labels to its original state/visibility strata.
The authors have made meaningful efforts to address my concerns. However, concerns remain regarding the limited experimental scope, the justification for the four-state taxonomy, and the lack of convincing external validation. I therefore raise my rating from 1 to 2, while maintaining an overall negative assessment.
Four-State Audit Semantics and Empirical Utility
Reviewer Comment — Weakness 1:
Response:
SATRA is an empirical audit framework. Its contribution is to make the security-relevant meaning of a persisted memory artifact an explicit audit object, separating it from pre-write evidence, retrieval, and subsequent behavior. We do not claim a new formal theory or a new learning algorithm.
The four states have the following operational meanings, defined relative to the registered policy conflict and task context:
The reviewer is correct that U and N are sparse: each appears six times in the human-reviewed main evaluation, and neither occurs in Formation. We do not infer that these states are prevalent in deployment. Their distinction is relevant to the audit representation: P versus U separates a reusable risky rule from unresolved risky availability; C versus N separates explicit correction from an artifact carrying neither side of the registered conflict. A binary P/non-P label merges C, U, and N and loses these distinctions.
Predictive comparison. We retain the negative result from Appendix G. In leave-one-case-out evaluation of natural visible unsafe behavior, lower log loss is better:
Thus, four-state coding does not outperform binary coding on this natural behavior-prediction outcome. Its purpose is semantic auditing, not superior prediction of unsafe behavior.
Outcome-specific diagnostic evidence. In a separately registered, rule-judged descendant-match diagnostic, four-state coding yields a lower LOCO log loss than binary coding (0.416 vs. 0.552). U/N match rates are 22/24 and 7/23, respectively (61.2 percentage-point difference; 95% CI 21.2–91.7). This diagnostic is based on registered text and a rule judge, not human-reviewed behavior. It supports the value of distinguishing semantic categories for this controlled content-matching outcome, not universal predictive superiority.
A further offline state-aware gating diagnostic reduces P/U descendants from 6/72 to 1/72 and unsafe activation outcomes from 8/144 to 0/144 on frozen matched branches. This is an intervention-based diagnostic, not a deployable-defense evaluation.
Reviewer Question 1:
Response:
The empirical justification is outcome-specific: four states preserve distinctions between risky retention, unresolved risky availability, explicit correction, and neutral content. The controlled descendant-match and offline gating diagnostics show that these distinctions can be relevant to content-level auditing and selective analysis. The natural unsafe-behavior comparison does not show predictive superiority, and we do not make that claim.