- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces endogenous authorization laundering (EAL), a failure mode in which persistent agent memory incorrectly creates or retains authority and a downstream executor subsequently acts on that state. EAL-BENCH evaluates this writer–memory–executor chain across procurement, cybersecurity, and finance, using seven memory-writer models, two calibrated primary executors, typed and free-text memory, and one-shot versus incremental updating. The benchmark uses a hidden deterministic authorization ledger, matched authorized/unauthorized requests, frozen-memory replay, exact-state repair, and several mitigation and robustness studies.
The central within-benchmark result is convincing. Typed incremental memory forms false authority for up to 45.4% of unauthorized probes, and the tested executors act on erroneous memories in 386/390 trials, compared with 1/390 after oracle-exact memory replacement. The paper provides a valuable benchmark and unusually detailed mechanism analysis. My main reservations concern the effective experimental unit, dependence among pooled observations, coupling to a particular patch-based memory implementation, conditional selection of perfectly calibrated executors, and several claims that generalize beyond the evaluated architecture.
Critical strengths
-
Clear and useful problem formulation. The separation between false-authority formation (F) and downstream execution (G) in Section 3.1 (pp. 3–4) is conceptually useful. The deterministic ledger and exact-action scoring avoid relying on an LLM judge for the primary authorization outcomes.
-
Strong memory-level causal intervention. The exact-state repair experiment is a major strength. The request, executor, and other inputs are held fixed while the erroneous memory is replaced by oracle-exact state (Section 4.2; Table 11, p. 19). The reduction from 386/390 to 1/390 unauthorized actions convincingly establishes the stored memory artifact as a causal mediator in the evaluated system.
-
Broad and transparent empirical investigation. The paper evaluates seven writers, multiple memory representations and update strategies, two primary executors, a third-executor check, three domains, and three primary seeds. The appendices provide prompts, schemas, raw counts, software versions, writer–executor matrices, capacity ablations, and lineage information.
Critical weaknesses
- The inferential target and effective experimental unit are insufficiently clear
The benchmark contains only 12 procurement, 16 cybersecurity, and 8 finance base cases. Headline rates pool multiple probes derived from the same memory, stochastic runs on the same cases, seven fixed writers, and two executors replaying shared artifacts. Section 3.3 recognizes this dependence, but Table 2 reports no uncertainty, and the clustering units vary across analyses.
Consequently, the large request denominators should not be interpreted as independent evidence about a broader population of authorization histories. The paper should define the estimand for each main claim and report base-case-clustered intervals for Tables 2 and 10, keeping all probes, seeds, conditions, and executor replays derived from a case or memory together. Leave-one-case-out results would show whether any headline effect depends heavily on a small number of authored cases.
- The typed incremental result is partly coupled to the chosen update harness
Typed memory uses a relatively brittle PatchDoc protocol: nested arrays require dependent patches, writers receive only one validation-informed retry, and two failed attempts cause the previous profile to remain active (pp. 14–18). This recovery rule directly converts write failures into stale authorization.
The issue is especially important in cybersecurity: 503/529 attributed open-loop failures are rejected updates rather than semantic reinterpretations (Table 26, p. 31). In the single-seed capacity ablation, doubling capacity reduces baseline unauthorized actions from 12.5% to 1.6% (Table 34, p. 35).
This remains a real system failure, but it supports a narrower claim about the evaluated LangMem/PatchDoc/fail-stale configuration, not typed or incremental memory in general. The paper should report rejected-update rates beside behavioral results and either:
add a simpler incremental baseline that regenerates the complete typed state;
evaluate fail-closed or stale-state signaling after a rejected write; or
consistently narrow the claim to the tested patch-based implementation.
- The propagation and repair conclusions require more conditional wording
The executors are selected only if they make no errors under faithful benchmark memories, receive memory as their only case-specific authorization evidence, and are instructed to execute when that memory covers the request. Under this protocol, high (P(G\mid F)) is informative for causal localization but also partly induced by executor selection and interface design.
Moreover, oracle repair replaces the entire memory. It establishes a memory-level cause, but it does not prove that the particular false-authority record measured by (F) is the sole mediator; unrelated omissions, ordering, or wording may also change.
The paper should report how many executors were screened, provide results without conditioning on perfect calibration if available, and describe the 99% result as applying to “selected calibrated executors in the EAL-BENCH memory-only protocol.” A request-local minimal-repair arm would more specifically connect (F) to execution. Without an independent policy gateway or source lookup, the statement that executor-side safeguards “cannot” help is too broad.
Questions for authors
-
What are the confidence intervals for Tables 2 and 10 when resampling at the base-case level? Do the conclusions survive leave-one-case-out analysis?
-
What are the formation and unauthorized-action rates after separating semantic/lifecycle errors from rejected PatchDoc writes? Do the main results hold with full-state regeneration or a fail-closed recovery policy?
-
How many executor candidates were evaluated before selecting the two calibrated executors? What propagation rate is obtained without conditioning on perfect faithful-memory behavior?
-
Can the authors add a request-local repair condition that modifies only the fields determining (A_M(M_T,a_q)), rather than replacing the complete memory?
Thank you again for your positive assessment of our work and for your constructive suggestions! We wanted to follow up briefly to let you know that we have now updated the paper accordingly:
The updated artifact archive includes the new experiment under the
w2/directory, and the additional analyses underreview_checks/, with the saved records and code available for inspection in case you are interested :)We appreciate your support and careful engagement, which helped us make the experimental interpretation and manuscript clearer. Thank you again!