acceptodds
Under review as a conference paper at ICLR 2027

Traverse: Learning When to Remember, Reset, and Redirect for Long-Horizon Web Search

Abstract

Long-horizon information-seeking agents often accumulate noisy or misleading context, causing early mistakes to persist and making recovery increasingly difficult. We introduce an autonomous search harness in which the agent manages its own search process through three states: Rubric, Answer, and Verify. The agent first defines criteria for a valid answer, searches under these criteria, and then independently verifies the result before deciding whether to terminate or continue searching. It is further equipped with a Seal Memory tool that enables active context management. Training this behavior with reinforcement learning, however, can induce Seal Collapse, resulting in unstable training and preventing the agent from reliably learning when and how to use its memory tools. We solve this with a simple strategy that trains only the final segment after context management. Our 35B model achieves 72.83 on BrowseComp, outperforming comparable open-source systems, and consistently improves over the base model across BrowseComp-ZH, xbench, DeepSearchQA, WideSearch, financial investigation, and product search. Ablations show that autonomous compression outperforms automatic compaction and validate our RL design.

Primary area:
reinforcement learning
Keywords:
Search Agent, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2BbnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Traverse combines a Rubric–Answer–Verify workflow with agent-triggered context compression through Seal Memory. Its main training proposal is to optimize only the final context segment of each rollout. The resulting 35B model achieves 72.8 on BrowseComp and improves over its backbone across several search benchmarks.

Critical strengths

1)The paper studies a useful problem in long-horizon agent RL: how to optimize trajectories spanning multiple context resets.

2)Figures 2–3 show an interesting training result. Including the (1/K)-weighted all-segment baseline helps rule out trajectory reweighting as the sole cause of degradation.

3)The gains cover several search benchmarks. The active-compaction experiment also controls the summarizer model and memory format, which makes the comparison more informative.

Critical weaknesses

1)The benefit of selecting the final segment is not sufficiently isolated. Figures 2–3 establish an advantage over the two tested all-segment objectives. However, I would like to see a baseline that samples one segment per trajectory uniformly or in a round-robin manner. This would distinguish the benefit of terminal-segment selection from that of selective segment training more generally. The authors keep training hyperparameters consistent, but the optimized-token counts and update budgets would also help interpret this comparison.

2)Table 3 does not establish that adaptive timing is better than earlier fixed compression. Automatic compression triggers at roughly 230K tokens, whereas the median first active Seal occurs at 67.5K. A simpler policy that compresses earlier might recover part of the gain. Active Seal also increases average turns from 112.2 to 217.7, which the authors explicitly acknowledge. Longer productive search is a legitimate benefit, but earlier fixed-threshold baselines and accuracy–cost curves are needed to understand what adaptive triggering adds.

3)The evidence for improved memory control remains indirect. Section 3.2 provides an implicit-learning argument, and Appendix C introduces resource penalties, constraints on unnecessary Seal calls, and randomized context budgets. These can plausibly affect triggering behavior. However, successful Seal calls and their summaries in preceding segments are excluded from the RL loss. I am therefore not sure whether RL improves memory decisions themselves, or mainly improves final-segment search while preserving the SFT-initialized memory behavior. The current results support stable training more directly than improved memory construction.

Questions for authors

The following targeted experiments would help resolve my main concerns and could change my rating. Evaluation on the existing ablation benchmarks would be sufficient; I am not asking for a full benchmark rerun.

1)Is selecting the final segment specifically beneficial? Could you compare final-segment RL with uniformly sampling one segment per trajectory, using the same rollout collection budget and clearly reporting loss normalization and optimized-token counts? This would help distinguish the benefit of final-segment selection from that of training on fewer segments.

2)Does adaptive triggering outperform earlier fixed compression? Could you add fixed thresholds closer to the observed active Seal timing, such as 64K and 128K tokens, while keeping the model and memory format unchanged? Please report accuracy alongside actual token and tool costs under shared inference limits. The key question is whether state-dependent triggering adds value beyond simply compressing earlier.

3)What memory behavior actually improves during RL? Could you compare the SFT and RL policies on a fixed set of intermediate search states, examining Seal decisions and the utility of the generated summaries? For example, summary utility could be tested by holding the continuation policy fixed and measuring downstream success. This would provide more direct evidence of improved memory control beyond the implicit-learning argument in Section 3.2 and the training dynamics in Figure 3.

Reviewer_rLPKOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Traverse combines a Rubric–Answer–Verify search workflow with agent-triggered memory sealing and retrieval. Following SFT, it applies RL only to the final context segment, avoiding the training instability observed with two all-segment alternatives. The 35B system reports a score of 72.83 on BrowseComp and improvements across several search benchmarks. The paper also compares active sealing with automatic compaction triggered at 90% of the context window.

Critical strengths
  1. The training ablations address a meaningful alternative explanation. Figures 2–3 compare final-segment RL with standard all-segment RL and a $1/K$-weighted variant that equalizes total trajectory weights. Final-segment RL performs better and avoids the reported instability. These comparisons support the training approach and show that correcting trajectory weights alone does not resolve the observed degradation.

  2. The system demonstrates gains across diverse search tasks. The evaluation covers deep, multilingual, and breadth-oriented search, alongside additional transfer benchmarks. The memory interfaces and recovery examples also provide concrete illustrations of the proposed behavior (§4; Appendices B/F).

Critical weaknesses
  1. The compaction experiment does not distinguish adaptive timing from simply intervening earlier. In Table 3, the median first compaction occurs at 231.4K tokens for Auto Compaction, compared with 67.5K for Active Seal. This demonstrates an advantage over a late fixed trigger, but leaves open whether Active Seal would outperform an earlier fixed trigger. Its higher actual turn count does not, by itself, imply an unfair comparison; it may reflect longer productive search. The unresolved question is whether content-dependent timing provides benefits beyond earlier compression.

  2. The net benefit of self-verification remains unquantified. Figure 2 disables verification, appropriately isolating the effect of Answer-state training. However, it does not establish the separate contribution of the Verify stage to final-answer quality. Neither aggregate system scores nor selected recovery examples quantify how often verification corrects a wrong candidate or replaces a correct answer with an incorrect one. This could be assessed using existing full-system logs without additional training.

Questions for authors
  1. An optional inference-only comparison: Using the existing checkpoint, could the authors compare Active Seal with one preselected earlier automatic threshold, such as 64K, on a small, fixed subset? The maximum turn budget, search backend, and memory format should remain the same, but the actual numbers of turns and resets need not be matched. A single paired comparison would be informative; a threshold sweep is unnecessary. If this is impractical, please narrow the claim to an advantage over the tested 90% trigger rather than automatic compaction in general.

  2. An analysis of existing logs: Could the authors compare the first candidate answer entering Verify with the final answer in saved full-system runs? Please report wrong-to-correct and correct-to-wrong transitions, together with the additional verification and search cost. This would quantify the observed net benefit of the verification-and-revision cycle, although it would not establish superiority over all compute-matched alternatives.

Reviewer_mbcZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TRAVERSE, a long-horizon web search agent built on a Rubric–Answer–Verify state machine with agent-invoked context management (Seal/Read Memory tools). The agent first decomposes the query into verifiable criteria (Rubric), searches under these criteria while autonomously compacting its context into structured checkpoints (Answer), and self-verifies the candidate answer before deciding to terminate or resume (Verify). For training, the authors combine masked SFT on teacher trajectories with a final-segment-only RL strategy (GSPO/GRPO-style), motivated by their identification of "Seal Collapse"—unstable training when trajectory-level advantages are broadcast across all context segments. Their 35B model achieves 72.83 on BrowseComp and consistent gains across five search benchmarks plus OOD transfer to finance and e-commerce tasks.

Critical strengths
  1. Well-motivated problem and failure-mode analysis. The observation that early mistakes persist and bias long-horizon search, and that existing summarization methods only maintain state without self-verification, is convincing and well articulated.
  2. Thoughtful tool design. The Seal Memory schema (verified/conflicting/partial facts, dead-ends, frontier queue, next-step plan) is a genuinely useful structured-checkpoint design, clearly more expressive than unstructured compaction. The qualitative case studies (Appendix B/E) show the model learns to treat sealing as strategy reset, not just compression.
  3. Insightful RL analysis. The identification of Seal Collapse, together with the formalization of trajectory reweighting (Eq. 4) and the training-dynamics study (Fig. 3), is the most interesting technical contribution. The controlled comparison against both equal-weight and 1/K-weighted all-segment RL rules out the reweighting confound.
  4. Careful evaluation methodology. The BC185 subset is calibrated against historical full-set runs with explicit deviation tolerances; the Auto Compaction baseline uses the same model and memory format, making the 10.28-point Active Seal gain credible.
  5. Broad and honest evaluation. Coverage of multilingual (BC-ZH), breadth (WideSearch), synthesis (DSQA, xbench) plus OOD transfer (GAIA +18.45, FinSearchComp +19.70) is comprehensive.
Critical weaknesses
  1. Limited methodological novelty; components are largely assembled from prior work. Memory-as-action/context curation (Zhang et al., 2026; Sun et al., 2025), trajectory summarization (ReSum, Wu et al., 2026), rubric-guided verification (Wan et al., 2026), and state-machine decomposition inspired by DeepSeekMath-V2 are all cited by the authors themselves. The paper's contribution is a well-engineered integration, but each ingredient is close to existing work; the harness-heavy contribution may fit better at a systems venue.
  2. The final-segment-only RL strategy is a heuristic whose justification is incomplete. The two motivations ("implicit learning" and "less noisy reward signal") are plausible but informal. Critically, earlier segments—including the sealing decisions and memory content themselves—receive no gradient. How the model learns to write good memory beyond SFT imitation is not demonstrated; the claimed implicit encouragement of Seal invocation is asserted rather than verified (e.g., no analysis of gradient attribution or counterfactual probes). More principled alternatives (per-segment value estimates, segment-level credit assignment) are not compared—only two naive broadcast baselines are.
  3. The self-verification loop is under-analyzed. The Verify state uses the same model, raising the classic self-verification limitation: a model that cannot detect its own errors cannot recover. No statistics are given on verification precision/recall (does PASS actually correlate with correctness?), number of verify–answer loops, or failure cases where verification passes incorrectly. Notably, Fig. 2's ablations are all run with verification disabled, so the Verify state's actual contribution to the main numbers is unclear.
  4. Conflated contributions. The reported gains bundle (i) the WebShaper-style synthetic data pipeline, (ii) masked SFT on a strong teacher, (iii) the harness, and (iv) RL. The RL-over-SFT ablation (+4.8 on BC185) is helpful, but there is no ablation isolating the Rubric state or Verify state against a plain agent with Seal Memory alone. The headline BrowseComp number also includes a "Discard-All" fallback reset "following common baseline practice"—this hybrid mechanism makes it hard to attribute the 72.83 purely to the proposed method.
  5. Headline claims are stronger than the numbers. The abstract says 72.83 "outperforming comparable open-source systems," yet on BrowseComp the margin over FORT-Searcher (72.2) is 0.6 points, and Traverse trails FORT-Searcher on BC-ZH (70.7 vs 75.0) and matches (not beats) AREX-Turbo on xbench (57.0). "State-of-the-art among search-specialized models" on DSQA/WideSearch holds only within that slice. Suggest tempering the claims.
  6. Efficiency trade-off not discussed. Active Seal raises average trajectory length from 112.2 to 217.7 turns—nearly 2×. For a paper whose pitch includes context-budget efficiency, a cost/latency/token analysis of the accuracy-vs-length trade-off is missing.
  7. Read Memory is restricted to the most recent checkpoint only, so information sealed more than one segment back is permanently lost unless re-carried. This significant design limitation is not discussed.
  8. Reproducibility. Training uses 96 GPUs and distillation from a strong proprietary teacher; the synthetic data, teacher identity, and many reward hyperparameters (λ's, caps) are not fully released, making independent reproduction difficult.
Questions for authors
  1. What is the empirical precision/recall of the Verify state (P(PASS | correct) and P(PASS | incorrect))? How many verify–answer loops occur on average, and how often is the loop bounded?
  2. Since only the final segment receives gradients, how does the model improve its sealing content (what to preserve) beyond SFT? Have you measured memory quality (e.g., downstream success conditioned on sealed memory) over RL steps?
  3. In the BrowseComp main result, how frequently is the Discard-All fallback triggered relative to Seal? Would results change with Seal only?
  4. Can you ablate the Rubric state and Verify state independently (e.g., no-rubric, no-verify variants) to quantify each component's contribution to the final numbers?
  5. Given the ~2× increase in average turns with Active Seal, how do total tokens/latency compare against Auto Compaction at matched accuracy?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.