- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
LLM architecture-related & context management paper. This paper studies post-compaction memory access for long-horizon language agents. Its central observation is that trajectory compaction conflates storage with visibility: old raw interactions may still exist in an archive even though they are no longer available to the model's attention.
The paper introduces Triggered Sparse Attention (TSA), which separates these two decisions:
- A trigger determines whether archived tokens are eligible to participate in attention.
- A sparse-attention indexer then performs fine-grained token selection, separately for each layer and query. In the model-native setting, retrieved evidence does not need to be reinserted into the prompt.
The paper evaluates the idea along several complementary axes:
- native sparse-attention probes on synthetic traces, where query-conditioned opening is compared with closed and random archive access, including a rate-matched random control;
- two pre-registered WebShop waves, where selective access is compared with random and indiscriminate archive exposure at similar or lower archive volume;
- a frozen DeepSeek-V3.2 sparse indexer, showing that archived evidence can be recovered without retraining token-identity selection.
Overall, the main contribution is a useful reframing of compacted agent memory as an attention-side eligibility and control problem, rather than only a storage, summarization, or prompt-retrieval problem. The empirical validation includes matched controls, pre-registration, paired statistical testing, and separate mechanism-level and agent-level studies.
Critical strengths
- A clean and potentially important reframing of compacted agent memory.
- The paper separates memory storage from memory visibility and formulates post-compaction recall as an attention-eligibility problem (§1, §3).
- This leads to a conceptually clean two-stage design. A trigger decides whether archived memory may participate at all, and the model's own sparse indexer decides which individual tokens to read (§4, Eqs. 2–3).
- This matters because it provides a memory interface that differs from both summary-only systems and prompt-based retrieval, and it may generalize beyond the particular implementation studied here (§2; §7).
- Strong mechanism-oriented controls and unusually careful empirical methodology.
- The paper does more than compare against a closed-memory baseline. It also includes random access, rate-matched random access, always-open controls, fixed-total attention budgets, matched-content controls, paired task evaluation, and pre-registered WebShop replications (§5; App. C).
- The native sparse probe shows a large gap between Triggered-SA and random opening, even at matched opening rates. This supports the claim that where archive access occurs matters, not merely how much memory is exposed (§6, Table 1).
- Matched-content controls show that the effect is carried by the archived content rather than by extra attention mass (§6, l.372–375).
- The two WebShop waves reproduce the ordering of selective over random access under matched tasks and controlled prompts (§6, Fig. 3; App. Table 7).
Critical weaknesses
- The full proposed system is not demonstrated end-to-end in a live agent.
- The model-native experiments use Qwen3-4B with native sparse reading, but on synthetic traces and with a training-free score-margin trigger.
- The large WebShop agent study instead evaluates a learned access decision by restoring opened raw observations into the model context. It does not use the proposed native attention-side reading path (l.320; App. Table 6).
- The paper therefore provides strong evidence for each component separately. It does not directly validate the complete combination of a learned access gate, native sparse archive reading, and long-horizon agent execution at scale.
- This is consequential because the paper's main architectural contribution is precisely the attention-side interface, and the current agent-scale results do not instantiate it.
- The comparison to strong learned retrieval or prompt-memory systems is not yet sufficient to establish a broader systems advantage.
- The random, closed, and always-open controls are excellent for isolating the causal role of selective archive access. However, they are weaker competitors than a well-tuned learned retriever that selectively reinserts archived evidence under a comparable token, latency, and storage budget.
- The paper does report a matched-evidence comparison, in which attention-side admission yields a larger action-likelihood margin than prompt insertion (§6, l.374–375). This is narrower than an end-to-end comparison against a strong retrieval-based memory policy.
- This limits how strongly the results can be read as showing superiority over alternative long-term memory interfaces. At present they show that the proposed interface is viable and promising.
Questions for authors
Main questions:
Q1. What is the expected performance of the full proposed configuration, a learned NULL/OPEN gate combined with native attention-side sparse archive reading in a live agent? App. Table 6 indicates that the WebShop study restores opened observations to the context, whereas the native Qwen probe uses sparse top-k archive reading. Is there any existing experiment, even at smaller scale, that combines the learned gate from the agent setting with the native sparse reading path? Evidence that this full composition preserves the reported agent-level gains would materially strengthen my assessment.
Q2. How should the WebShop results be interpreted relative to the Visible-Only baseline? In the pooled WebShop evaluation, Triggered-SA improves mean reward substantially over random and always-open access, but only slightly over Visible-Only (0.171 vs. 0.168). The nonzero-reward count increases from 300 to 320 (App. Table 7). Which of these is the intended main claim?
- that Triggered-SA improves absolute agent capability over well-compacted memory; or
- that it mainly avoids the degradation caused by indiscriminate access while selectively recovering useful archived information? Clarifying this would affect how I assess the empirical significance.
Q3. Do you have a comparison against a learned prompt-retrieval baseline under approximately matched archive-selection and token budgets? The current random and always-open controls isolate selective access well. I would like to know whether the advantage remains against a learned retriever that selects one or more archived records and reinserts them into the prompt. This would materially affect whether I see the contribution mainly as a new, viable memory interface or as a practically superior alternative to prompt-based retrieval.
Minor questions:
- Title layout: The title breaks at an odd point on p.1. "Control" sits alone on the second line, followed by a blank line before "for Compacted Agent Memory" (l.001–004). Please reflow it, e.g., with a manual line break after "Hidden, Not Erased:".
- Figure size in the experiments: Figs. 2–4 (pp.7–8, l.337–417) are taller than their content needs and contain a lot of white space. Figs. 3 and 4 alone take most of p.8, and the legend in Fig. 4(e) overlaps the curves. Making these figures shorter and more compact, e.g., by reducing panel height or merging related panels into a single row, would free space. That space could bring key appendix material into the main text, such as App. Table 6, which summarizes what each experiment executes, or a fuller discussion of the WebShop and DeepSeek-V3.2 results.
6 comments
Response to Reviewer Ze82 (Part 2/5)
Summary of Revisions (continued)
- Presentation updates (Page 1 title break, Figs. 2–4 whitespace & Fig. 4e legend, App. Table 6 [now Main Table 1 / App. Table 7] in main text; PDF: Title p. 1, §5 Table 1 p. 5, Figs. 2–4 pp. 7–8, App. D.1 Table 7 p. 17): Reflowed p. 1 title across two balanced lines after Hidden, Not Erased:; compacted Figures 2–4 and separated the Figure 4(e) legend from the curves; promoted the experimental interface summary into Main §5 Table 1 (cross-referenced to App. D.1 Table 7) within 9 main pages.
W1 & Q1: Full Configuration (Learned NULL/OPEN Gate + Native Sparse Archive Reading) in a Live Agent
Q1: "What is the expected performance of the full proposed configuration, a learned NULL/OPEN gate combined with native attention-side sparse archive reading in a live agent? App. Table 6 [now Main Table 1 / App. Table 7] indicates that the WebShop study restores opened observations to the context, whereas the native Qwen probe uses sparse top-k archive reading. Is there any existing experiment, even at smaller scale, that combines the learned gate from the agent setting with the native sparse reading path?"
Main §5 Table 1 (p. 5) maps each experimental interface across backbones. To evaluate the fixed 8-feature learned gate composed directly with model-native DSA sparse reading, we ran four H200 evaluations on converted Qwen3-4B + native DSA + LoRA (top-k = 1024):
- Live 256-task WebShop with attention-side reading (§6.2 p. 8 & Appendix D.6 Table 14, p. 21):
We ran the first 256 unselected official WebShop tasks across two working-context token budgets (WCT = 2800 and tighter WCT = 1600), comparing Visible-Only, Learned-router DSA, Native score-margin DSA, Always-open DSA, and Matched-random DSA (all homogeneously with
expanded_l2= 0.0 across all 256 tasks). As a non-compacting reference, flat prompt snippet retrieval re-injects raw L2 text into the active prompt (expanded_l2= 0.87, 0 compactions; 0.4199 mean reward, 233/256 = 91.0% R>0 overall):- At tighter WCT = 1600 (49/256 tasks compacted, 8.47 compactions/task in Visible-Only): Visible-Only fails on 48/49 compacted tasks (1/49 = 2.0% R>0, 0.0122 reward, 15.35 steps). Learned-router DSA (triggering on only 13.7% of compacted steps, 5.8× sparser than Always-open 79.6%) lifts compacted R>0 11× to 11/49 (22.4%, +20.4 pp, 11W/1L, exact two-sided McNemar ) and compacted reward 5.6× to 0.0687 (paired Wilcoxon , permutation , 13.12 steps), matching Native score-margin DSA (11/49, 0.0687 at 53.1% trigger) and lifting 256-task overall reward to 0.3628 (200/256 = 78.1%) vs. 0.3608 (198/256) for Visible-Only and 0.3585 (199/256) for Matched-random DSA (+0.0042, 95% CI [+0.0004, +0.0099], 4W/0L).
- At WCT = 2800 (46/256 tasks compacted, 5.30 compactions/task): Learned-router DSA (15.8% compacted trigger rate) lifts compacted R > 0 from 4/46 (8.7%) to 9/46 (19.6%, 7W/2L) and compacted reward from 0.0289 to 0.0697 (Wilcoxon ; 0.0749 for Score-margin DSA, steps 15.43 → 14.09) while keeping 203/256 (79.3%, 0.3626) overall performance.
- 128-cell live synthetic multi-step agent loop (32 tasks × 4 policies, 964 generations; §6.2 pp. 7–8 & Appendix D.6 Table 13, p. 20):
Coupling the learned 8-feature gate () with native DSA across four multi-step tool families (
config_merge,relational_inventory,runbook_dependency,weighted_route; 8 tasks each, ITT scoring) solves 8/32 (25.0%, 8/8 onconfig_merge, 0 format errors) for both Learned gate + native DSA and Always-open native DSA vs. 0/32 for Visible-Only (, Holm ) and 5/32 (15.6%) for semantic prompt retrieval ().
Response to Reviewer Ze82 (Part 3/5)
- 192-state balanced 3-suite Gate Pareto (64 archive-detail + 64 stale-override + 64 visible-sufficient; §6.3 pp. 8–9 & Appendix D.8 Table 17, Fig. 5a, p. 23):
Testing false-open cost on Suite C (64 visible-sufficient states paired with multi-episode archived distractors), the fixed 8-feature gate at reaches 74.48% (143/192: 23.44% A [15/16 FP32, 16/16 BF16 on
config_merge], 100.0% B, 100.0% C) at 82.29% open rate. It opens on 46.88% of Suite C states, cutting unnecessary openings by 53.1% with zero accuracy loss and beating matched-rate Random (67.56%, +6.92 pp), 0-parameter DSA tail-margin (69.27% at 79.69% open, +5.21 pp), and Qwen3-Embedding-4B R2 (36.46%). At , it cuts Suite C false openings to 0.0% (50.0% overall open rate, +4.95 pp over Random). - BF16 Eq. 2 invariance,
DynamicCachefork parity, and hook-free FlashAttention-2 + PCIe Gen5 latency (Tables 17, 23–24, pp. 23, 26–27): In BF16 on H200, closed-gate execution matches a no-archive run on 32/32 (100.0%) greedy top-1 tokens (max |Δ| = 0.0 bitwise equality across 39 FP32DynamicCachefork forwards in Table 23), running 133.2× faster than R4 re-prefill in the 12K eager diagnostic harness (695.8 ms Open / 308.8 ms Closed vs. 92.7 s). In a hook-free merged-BF16 FlashAttention-2 benchmark (WCT = 2,048, 64-tok obs. prefill + 32-tok decode, Table 24), turn-latched Open takes 816.9 ms at 12K (1.39×) and 812.9 ms at 32K (2.90×) vs. 1,135.6 ms and 2,358.3 ms for R4 FA2 re-prefill, while PCIe Gen5 selectivetop-k = 1024host staging (9 KiB/tok HBM index) takes 803.6 ms (1.41×) and 839.2 ms (2.81×) (12.8 s vs. 37.9 s over a 16-turn 32K episode, 2.97×).
Scope (§6.2 and Limitations, p. 10): The composed learned gate + native DSA path operates without prompt restoration, cuts compacted-task WebShop openings 5.8× below Always-open and Suite C false openings by 53.1%, and significantly recovers compacted details (11/49 vs. 1/49 on 256-task WebShop at WCT = 1600, ; 8/8 live config_merge; 64/64 Suite B). Scaling live multi-hop chains to larger native backbones is a natural next step.
Q2: Interpreting WebShop Relative to Visible-Only (0.171 vs. 0.168, 320 vs. 300)
Q2: "How should the WebShop results be interpreted relative to the Visible-Only baseline? ... Which of these is the intended main claim: (1) that Triggered-SA improves absolute agent capability over well-compacted memory; or (2) that it mainly avoids the degradation caused by indiscriminate access while selectively recovering useful archived information?"
Our main claim on the 1,024-task Qwen3-30B-A3B WebShop study is (2): where summaries keep coarse cues, selective access avoids unselective-recall degradation while preserving closed-memory reliability. Claim (1) holds with clear statistical significance when compaction tightens or drops decisive identifiers, as shown in our live 256-task WebShop budget sweep (WCT = 2800 vs. WCT = 1600; Abstract p. 1, §1 p. 2, §6.2 pp. 7–8, Appendix D Tables 8–11 & 14):
- Claim (2) when summaries retain coarse cues (Qwen3-30B-A3B, n = 1,024, Tables 8–11): Visible-Only reaches 0.168 mean reward (300/1,024 R>0). Triggered-SA (0.171, 320 R>0) preserves closed-memory reliability (+0.003, 95% CI [−0.010, +0.016], 91W/71L, exact McNemar ) while eliminating unselective-recall loss: Random drops to 266 R>0 vs. 300 for Visible-Only (), whereas Triggered-SA gains +0.027 over Random () and +0.017 over Always-Open (291 R>0, ) at 2.7× fewer expansions/step.
- Claim (1) when compaction drops decisive details or tightens working memory (Tables 2, 13–14, 17, 19): Under tighter WebShop working memory on 256 tasks (WCT = 1600, Table 14), Visible-Only completes only 1/49 (2.0%) compacted tasks whereas native Triggered-SA completes 11/49 (22.4%, , reward 0.0122 → 0.0687, Wilcoxon ; and 4/46 → 9/46 at WCT = 2800). Consistent gains hold on DeepSeek-V3.2 (0/26 → 21/26, ), synthetic probes (Table 2), live 4B
config_merge(0/8 → 8/8, ), and 192-state decisions (68/192 → 143/192).
Response to Reviewer Ze82 (Part 1/5)
We thank Reviewer Ze82 for the supportive assessment, for highlighting our separation of memory storage from memory visibility, and for the layout suggestions. We ran seven experimental and kernel-level systems suites on H200 and incorporated all suggested presentation updates into the revised manuscript (9 main pages, Limitations starting on p. 10, no layout overflows). Notation used below: WCT denotes the working-context token budget that triggers compaction;
expanded_l2denotes raw archived (L2) text re-injected into the active prompt (0.0 on all native arms, vs. 0.87 for non-compacting flat snippet retrieval); R1–R4 denote the four prompt-injection formats of the text-retrieval baselines (R1 trailing user message, R2 inside the tool observation, R3 synthetic tool turn, R4 in-place history splice); and Suite A / B / C denote our 64-state archive-detail, stale-override, and visible-sufficient post-compaction suites.Summary of Revisions
expanded_l2= 0.0): at WCT = 1600 (49 compacted), Visible-Only completes 1/49 = 2.0% (0.0122 reward) while Learned-router DSA (13.7% compacted trigger rate, 5.8× < Always-open 79.6%) completes 11/49 = 22.4% (+20.4 pp, 11W/1L, exact McNemar p=0.00635; reward 0.0687, Wilcoxon p=0.0269, −2.22 steps), raising 256-task reward to 0.3628 vs. 0.3585 Matched-random (+0.0042, 4W/0L); at WCT = 2800 (46 compacted), compacted R>0 rises 4/46 (8.7%) → 9/46 (19.6%, reward 0.0289 → 0.0697). (2) Live 128-cell loop: 8/32 vs. 0/32 Visible-Only (p=0.0078; 8/8config_merge). (3) 192-state Pareto & latency: 74.48% (143/192) (−53.1% Suite C false opens), running 133.2× faster in the eager harness and 1.39×–2.90× (GPU) / 1.41×–2.81× (PCIe5) faster than hook-free FA2 R4 re-prefill at 12K–32K (Table 24).config_merge) for native Triggered-SA (p=0.0078 vs. R4). (2) Matched-gate dense & FLARE (N = 128): 79/128 (61.7%) vs. 1–2/128 (p≤1.22×10−4); Qwen3-Embedding-4B R2 degrades Suite C (100% → 95.31%, 36.46% on 192 states). (3) 128-trace copy & 0.65K → 15.8K scaling: turn-latching (88–90/128 = 68.8%–70.3%) beats Qwen3-Embedding-4B top-1 (72/128 = 56.3%); Global top-k and Equal-slice TSA reach 81.3% (13/16) at 10.6K vs. 12.5% R2.