- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes VISTA, a training-free context-management framework for long-horizon LLM agents. It exposes explicit runtime state through a dashboard and allows agents to archive and recover context blocks. The method is evaluated on LOCA-Bench, BrowseComp-Plus, and GAIA, with additional ablations and a small RL adaptation study.
Critical strengths
-
The paper studies an important and practical problem for long-horizon LLM agents: how to manage growing context under limited context budgets. The motivation is clear and easy to understand.
-
VISTA is simple, training-free, and model-agnostic. The explicit context dashboard together with recoverable archiving provides a practical interface that can be integrated into existing agent systems without modifying the underlying model.
Critical weaknesses
-
Both BrowseComp-Plus and GAIA are evaluated with DeepSeek-V4-Pro under a restrictive 12K active context window. For BrowseComp-Plus, the paper explicitly states that this setting is deliberately chosen to induce evidence eviction. While this can serve as a useful stress test, it may also disproportionately favor a method specifically designed for context management. The ReAct baselines on both BrowseComp-Plus and GAIA also appear relatively weak. Evaluation under more standard long-context settings, such as 128K or 256K, would make the results more convincing.
-
The cross-backbone evaluation is only conducted on LOCA-Bench. BrowseComp-Plus and GAIA are evaluated only with DeepSeek-V4-Pro, which makes the claim of model-agnostic generalization less convincing. It would be useful to verify whether the gains on these benchmarks also hold across different model families.
-
The RL objective mainly uses a trajectory-level outcome reward together with local heuristic penalties. However, context-management actions can have delayed effects across multiple workspace states: whether an archive or recovery action is beneficial may only become clear several segments later. The current formulation does not explicitly model such cross-segment reward propagation, making the credit assignment relatively coarse. Related work such as IterResearch [1] provides a more explicit treatment of multi-round state transitions and reward assignment, which may be relevant here.
-
The RL experiments are relatively limited, with only a small Qwen3-8B adaptation study. Important training details, including the number of training examples, rollout budget, group size, total training tokens, and update steps, are not clearly reported. Several of these are even listed as pending implementation checks in the appendix, making the RL results difficult to assess and reproduce.
-
The theoretical section formalizes several intuitive properties of the proposed design under highly stylized assumptions, such as the benefit of lossless recovery and explicit block-size information. However, these results appear only loosely connected to the actual empirical setting and do not seem necessary for explaining the main experimental findings. In several places, the assumptions already encode much of the behavior that the theory is intended to justify, while the corresponding theoretical quantities are not directly evaluated in the experiments. As a result, the extensive theoretical development feels somewhat disproportionate to the practical contribution of the paper. A more concise analysis, or a tighter connection between the theoretical results and measurable behavior in the actual benchmarks, would make this part more convincing.
Reference
[1] Guoxin Chen et al. IterResearch: Rethinking Long-Horizon Agents via Markovian State Reconstruction. ICLR 2026.
Questions for authors
As far as I know, the GAIA validation split contains 165 questions in total. Why does the paper refer to a “random 165-question subset”? Is this actually the full validation set, or was some other sampling procedure used?
Response to Reviewer x8Hi
Thank you. Two of your concerns are answered by the setting of our main benchmark, and we have corrected the paper on the others.
Standard long-context settings. The headline LOCA-Bench results already use a standard 128K active window (max-context-size = 128,000) with Gemini-3-Flash. There, VISTA solves 38/75 against ReAct's 17/75, paired b/c = 1/22, exact McNemar p = 5.7×10⁻⁶. The LOCA sweep keeps that 128K window and raises how much context each task accumulates. Where accumulation is heavy, the gain is large and stable: VISTA 57.3 vs ReAct 36.0 at 96K, 50.7 vs 22.7 at 128K, and 32.0 vs 12.0 at 256K (Table 8). So the effect does not depend on an artificially small window; it appears when the task outgrows the window, which is the regime the method is for.
The 12K window is used only for BrowseComp-Plus and GAIA, whose trajectories are around 100K and 10K tokens. With a 128K window those runs would rarely need to evict anything, and the context managers in Table 1 would behave much like ReAct. We agree that 12K is a stress setting chosen to create retention pressure. The revised §4.1 says so and no longer cites the gold-retrieval-depth statistic, because we cannot show it was computed on development data.
The ReAct baselines appear weak. ReAct is the harness-native append-until-truncation loop, with the same tools, prompt, budget and scoring as every other row; it is not tuned down. On the same 75 LOCA tasks at the 8K growth setting, where little management is needed, it solves 84.0%. It falls to 22.7% at 128K because the context fills, and that is the failure the paper studies.
Cross-backbone evaluation. We agree that the backbone evidence is LOCA-only, and Appendix J now says so. On LOCA at 128K, the same interface runs unchanged on four backbones (Figure 6). "Model-agnostic" in the paper means that the layer wraps a backbone without per-backbone tuning. We also report a fifth backbone we had run but not plotted, GPT-5.2, where both methods stay near the floor (VISTA 9.3%, ReAct 8.0%). We report it because a transfer claim should show its misses.
GAIA "random 165-question subset". It is the full 165-question validation split (Levels 1/2/3: 53/86/26). "Random subset" was a wording error, corrected in §4.1 and Appendix I.
RL credit assignment and details. We agree the objective is coarse: a trajectory outcome reward plus local penalties, with no cross-segment propagation. IterResearch's Markovian state reconstruction is the relevant treatment, and we will discuss it in related work. The details you asked for are now in Appendix M.1: group size G = 8, one training seed, PPO ratio clip 0.2/0.2 with dual-clip 3.0, gradient-norm clip 1.0, and maximum prompt and response lengths of 8,192 and 32,768 tokens. Per-variant update counts, training-set sizes and total training tokens are listed as open in M.1. With one seed, §4.4 is now labelled a single-seed pilot, and none of the frozen-model results depend on it.
Theory. We partly agree. Theorem 1's operative quantity is the rate of the size channel available to the agent: a low-rate interface forces make-room errors. The one place we measure that rate is the size-estimation probe. Without the ledger, four backbones misestimate total context size by a median relative error of 0.43–0.84 (Table 12), which is the low-rate regime the theorem describes. We will tighten §3.3 around this link and move the stylized constructions to the appendix.
Informativeness. The result we would point to as new is a dissociation: keeping every archive and recovery tool but removing only the per-block display costs about ten tasks on the matched launches, and every launch with the display is above every launch without it (34–39 solved versus 26–29; paired p = 0.013 on the submitted launches). Prior context managers ship tools and display together, so as far as we know this separation has not been measured before.