- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper asks whether the way remembered facts are delivered to an agent, and the way the agent’s reply is admitted to execution, change task outcomes when the facts themselves are held fixed. A deterministic selector extracts one or two scalar argument values from each task’s first reference tool call, redacts them from the user request, and restores the identical canonical JSON either inline in the user message or as the result of a fabricated historical memory.read exchange. On 146 selected tasks from AgentDojo, τ-bench (airline, retail) and ToolSandbox, with three sub-30B open models at one seed, historical delivery reaches 29.68% success against 23.52% for inline delivery (+6.16 pp). A follow-up on the 59 retail tasks with a quantized Qwen3.6-27B shows that the two interfaces propose identical tool calls in 58/59 cases but differ in whether the native communication check accepts the first reply (1/59 vs 24/59); admitting schema-valid mixed text-plus-tool replies raises both interfaces to 56/59. Context-state and content controls on AgentDojo find that placing the fact in a trailing assistant message is the worst construction and that correct content is worth roughly 40 pp in either interface.
Critical strengths
- This paper explores a practical question: whether the way remembered facts are delivered to an agent, and the way the agent’s reply is admitted to execution, change task outcomes when the facts themselves are held fixed.
- The paper isolates a stage of memory use that end-to-end benchmarks cannot see; the chain fact → message → proposed call → accepted reply → execution → score is a useful vocabulary.
- The response and protocol studies are the most informative part: 58/59 identical proposals with divergent acceptance (1/59 vs 24/59), and parity once mixed replies are admitted (56/59 each; Table 4, Fig. 3), cleanly show that the acceptance rule, not the decision, carries the difference.
Critical weaknesses
W1. The experiment does not measure what the paper says it measures. The stated object is the interface through which remembered facts enter an agent’s context. The design, however, takes the facts from the conversation itself: a value that already appears in the user request is cut out of the request and put back by one of two constructions (Section 3.1). What is being measured is therefore how well a model recovers a literal string that was removed from, and then reattached to, a damaged request. This is parameter recovery, as the authors call it, not memory use. Nothing is retrieved, nothing competes with the fact, and the request no longer resembles anything a user would write. The authors acknowledge in §8 that facts fixed from prior interactions would be the appropriate source; that design was not pursued.
W2. Both delivery arms are artificial, so the contrast between them is hard to interpret. The inline arm appends to the user message a data-handling sentence, a machine-readable version marker and a compact JSON object, a construction with no obvious counterpart in training data or deployment. The history arm fabricates an assistant memory.read call and its tool result and places the model after a trajectory it did not produce. The comparison is thus between two off-policy contexts.
W3. The method is training-free, yet it is evaluated only on small models, and the paper’s own mechanism predicts that the result depends on that choice. All experiments use sub-30B open models; the protocol studies use a single quantized checkpoint that also serves as the user simulator. What the paper finds is that under inline delivery these models more often produce replies that mix text with a tool call, and that τ-bench’s half-duplex rule rejects such replies; the two interfaces propose the same tool calls and differ only in this respect, and once schema-valid mixed replies are admitted they are indistinguishable (§6, Table 4). Format compliance of exactly this kind is what larger models do most reliably. By the paper’s own account, then, the delivery gap under the native rule should shrink or vanish on a capable model, as it already does under the relaxed rule. Because the manipulation requires only inference, a frontier-class model, or a harness that tolerates mixed replies, could have been included at little cost; without either, the direction of the headline result on the models that matter in practice is unverified.
Questions for authors
Q1. Does the History–Inline contrast survive on a frontier-class model, and does it survive on a harness that accepts replies mixing text with a tool call?
Q2. Would the authors consider the design described in §8, with facts fixed from prior interactions rather than taken from the current request, as the primary experiment, keeping the present redaction design as a secondary control?
Q3. If the consistency gate used for the context-state study (a minimum effect with the same direction across models and seeds) is applied to the main History–Inline contrast using the three-seed data, is it met?
Thank you for proposing prior-interaction records and an explicit consistency test. The new continuation experiment gives a direct memory-utility result: supplying earlier interaction records eliminates 29 repeated read queries and reduces model requests by 34.44% on 12 tasks. The original-weight GLM repeat separately tests the admission mechanism.
Q1 — Larger model and mixed-response-compatible execution. With the original GLM-4.7-Flash agent and Qwen3.6-27B simulator in BF16, we completed the 59-task delivery-by-admission experiment at seed 5201. Native admission gives 0/59 Inline and 10/59 History DB successes. Admitting schema-valid mixed text-and-tool responses raises these to 42/59 and 45/59; first responses are accepted on all 59 tasks in each arm. All ten native History-only successes propose the same first tool action as Inline, whose response is rejected.
The paired History–Inline difference is +16.95 percentage points under native admission (95% task-bootstrap interval [8.47, 27.12]) and +5.08 under permissive admission ([−6.78, 18.64]). Their interaction is −11.86 points ([−28.81, 5.08]). A positive residual delivery effect under permissive execution is therefore unestablished.
For the larger API agent, DeepSeek-V4-Pro-0813, we tested prior-interaction records under permissive execution: retrieved context succeeds on 12/12 tasks and History on 11/12, a paired difference of −8.33 points ([−25.00, 0.00]). This experiment uses temperature 0 and non-thinking mode. It tests natural continuation with records; the original 59-task redaction-based History–Inline contrast on this larger model remains unmeasured.
Q2 / W1–W2 — Prior interactions as the basis for the memory-use argument. Yes. We propose to lead the memory-use argument with prior-interaction records and use redaction as a controlled analysis of delivery and admission. The new 12-task panel is exploratory, alongside the existing 146-task controlled panel.
Each new current request is an intact user reply from a public GPT-4.1-generated benchmark conversation. The record contains preceding dialogue and read-only tool results, selected by a fixed task-ID hash order and prefix rule. No value is removed from the current request. A persistent store reads the session record and supplies identical serialized bytes through retrieved context or History. The assistant–tool exchange in History is constructed by the application; this tests a context-reset continuation with stored records, including that explicit delivery choice.
Both record deliveries use fewer requests on all 12 tasks. Mean paired reductions are 5.17 requests per task, with 95% task-bootstrap intervals [4.00, 6.17] for retrieved context and [4.25, 6.08] for History. Users can supply information again in every condition, accounting for the high no-record completion rate. A post-hoc trace check finds 29 exact repeats of earlier read queries without records and zero with either delivery, before the first non-read business call. On the ten tasks completed by both History and no-record conditions, History saves 5.50 requests on average. The measured benefit is reduced reacquisition effort, with no established History advantage over retrieved context. Retrieval competition and updating are untested.
Q3 — Does the full consistency gate pass? No. On the matched 40-task, three-model, three-seed panel, History succeeds in 75/360 cells versus 65/360 for Inline: +2.78 points, with a task-stratified 95% interval [−1.39, 7.22]. The exact two-sided whole-task sign-flip test gives p = 0.291, a single post-hoc contrast outside the original four-contrast Holm family. The minimum-effect (at least 2 points) and two-of-three pooled-model-direction components pass. The within-every-seed direction and statistical components fail: at seed 5202, Qwen, GLM, and Nemotron differ by 0.0, +10.0, and −5.0 points. Consequently, the three-seed data support no claim of a stable History–Inline advantage.