acceptodds
Under review as a conference paper at ICLR 2027

Micro-Turns for Efficient Context and State Management in Multi-Hop Retrieval Agents

Abstract

Long-running retrieval agents must continually decide what context and state to preserve, compress, rank, or discard. In many harnesses, these decisions are handled by fixed programmatic rules that truncate old messages, apply hand-written compression, or evict content when a capacity limit is reached. We introduce _micro-turns_, short harness-injected model calls dedicated to context and state management between main agent turns. A micro-turn branches from the current trajectory, has the model perform a scoped management operation, and returns a compact state update. After applying the update, the harness discards the micro-turn prompt and response rather than recording them in the persistent trajectory. This makes context and state management model-mediated without lengthening the main session or exposing later main turns to auxiliary management exchanges. We instantiate micro-turns in a multi-hop retrieval agent built on the Harness-1 retrieval framework, where they summarize retrieved chunks, resolve curated-set overflow, and order curated evidence. We empirically show that across a diverse suite of retrieval benchmarks with Qwen3.5-4B, micro-turns improve recall while reducing the persistent-context token and attention cost of every episode, and that ordered curation further strengthens ranking at a bounded, dataset-dependent comparison cost, suggesting a practical harness primitive for efficient, context-heavy retrieval agents.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language agents, AI agents, multi-hop retrieval, deep search, information retrieval, context management, state management, agent memory, efficient inference, micro-turns, agent harnesses, online summarization, active decuration, ordered curation, Harness-1, BrowseCompPlus

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_cEzQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper falls in the field of retrieval agent state management. Motivated by the limitations of fixed state management (including excessive trajectory growth, suboptimal document ordering and curation, repeated prefill costs for verbatim retrieved chunks), the paper proposes micro-turns, short harness-injected model calls that can manage the state and context between main agent turns.

The method contains 3 key components: online summarization (OS), active decuration (AD), and ordered curation (OC). OS can summarize the retrieved chunks to mitigate context truncation and trajectory growth issues due to rendering chunks verbatim. AD and OC are two curation-management methods, where AD explicitly directs the model to remove documents and restore capacity limit when needed, and OC uses binary comparison (via LLM pair-wise comparison) to insert each new document at the right position and maintain ordering.

With the Harness-1 method as a baseline, the paper compares several settings: Micro-turns + OS, Micro-turns + OS + AD, Micro-turns + OS + OC. Results on 5 different datasets are overall very positive, showing that the proposed method generally outperforms baseline, and that Micro-turns + OS + OC is generally the best-performing method.

The paper also conducts cost analysis from different dimensions. While Micro-turns + OS + OC can be more expensive than Harness-1, other methods reduce both processed tokens and estimated attention operations. All micro-turn variants have lower peak per-call context lengths, suggesting potential memory savings.

Critical strengths
  1. The method is simple, well motivated and effective.
  2. The experiments are well done, covering multiple datasets with very good overall performance. Tables 1 and 2 report five-seed means ± standard deviations, so the results seem relatively reliable. It's also great to see several in-depth analyses from different perspectives that help explain the method’s effectiveness in specific scenarios (top-n curated-set quality, and downstream answering quality).
  3. The tables also show improvements in cost, which relieves my concerns about potential cost overhead of more sophisticated state management. In most cases, the costs are even lower than the baseline.
Critical weaknesses

The experiments use only Qwen3.5-4B as the retrieval model. It is unclear whether the gains would hold with other models, especially stronger models that may already handle curation better, resulting in a more competitive Harness-1 baseline.

Questions for authors
  1. How reliable is binary insertion when the LLM’s comparisons are inconsistent (such as A > B, B > C, but C > A)? Could this cause OC to remove useful documents? Did you try changing the insertion order to see if the results change?
  2. Have you tried base models other than Qwen3.5-4B? Would the conclusions change with a different base model? For example, a stronger base model may improve Harness-1’s curation ability, thus improving the baseline’s performance.
4 comments
AuthorsOpenReview ↗

Response (1/2)

W1. Only Qwen3.5-4B; a stronger model may make the baseline more competitive

A stronger model may curate better under Harness-1, but the harness limits remain: truncated chunks, no total ordering, and coarse tag-based eviction. Micro-turns address these limits, and also benefit from a stronger model's summarization and comparison. To check, we re-ran Harness-1 (H1) and Online Summarization + Ordered Curation (OS+OC) on BC+ with a stronger model (Qwen3.5-9B, same BM25 setup) and, separately, with a stronger retriever (Qwen3.5-4B with hybrid BM25 + dense retrieval and reranking, both using Qwen3-Embedding-8B):

MethodRecallFARTrajectory RecallTraj. FAR
Paper: 4B, BM25
H10.428 ±0.012\scriptstyle\pm 0.0120.461 ±0.018\scriptstyle\pm 0.0180.509 ±0.004\scriptstyle\pm 0.0040.544 ±0.004\scriptstyle\pm 0.004
OS + OC0.503 ±0.007\scriptstyle\pm 0.0070.554 ±0.009\scriptstyle\pm 0.0090.526 ±0.008\scriptstyle\pm 0.0080.573 ±0.008\scriptstyle\pm 0.008
Δ\Delta over H1 (pp)+7.5+9.3+1.7+2.9
Stronger model: 9B, BM25
H10.474 ±0.013\scriptstyle\pm 0.0130.515 ±0.012\scriptstyle\pm 0.0120.555 ±0.013\scriptstyle\pm 0.0130.582 ±0.014\scriptstyle\pm 0.014
OS + OC0.560 ±0.012\scriptstyle\pm 0.0120.601 ±0.012\scriptstyle\pm 0.0120.597 ±0.014\scriptstyle\pm 0.0140.633 ±0.014\scriptstyle\pm 0.014
Δ\Delta over H1 (pp)+8.6+8.6+4.2+5.1
Stronger retriever: 4B, hybrid + reranker
H10.546 ±0.008\scriptstyle\pm 0.0080.595 ±0.010\scriptstyle\pm 0.0100.634 ±0.009\scriptstyle\pm 0.0090.680 ±0.014\scriptstyle\pm 0.014
OS + OC0.606 ±0.015\scriptstyle\pm 0.0150.662 ±0.021\scriptstyle\pm 0.0210.637 ±0.013\scriptstyle\pm 0.0130.689 ±0.016\scriptstyle\pm 0.016
Δ\Delta over H1 (pp)+6.0+6.7+0.3+0.9

Bold marks the best mean within each block. Changing the model: gains in all four metrics, with gains of 8.6pp and 8.6pp in Recall and FAR. Changing the retriever: average gains of 6.0pp in Recall and 6.7pp in FAR; Trajectory Recall saturates with modest gains, consistent with OC's gains coming from better selection rather than more discovery. A stronger model or retriever also lifts H1 itself (Recall +4.6pp and +11.8pp), and OS + OC still improves upon it.

AuthorsOpenReview ↗

Response (2/2)

Q1. Inconsistent comparisons (A > B, B > C, C > A)

Binary insertion compares an incoming document with only about log n retained ones, so a cycle is never queried or stored; the set is always a total order by construction. An inconsistent verdict can still misplace a document, and a useful one placed near the bottom could be evicted at overflow. We do not measure how often verdicts are inconsistent. In each comparison we show the incoming document first to benefit from prefix cache reuse: an incoming document can be cached and compared to multiple retained documents. In an offline replay of 25,182 comparisons, putting the existing document first (instead of the incoming document, as in the paper) processed 18.5% more fresh (uncached) comparison input tokens, even when allowing reuse of any previously cached matching prefix. We also changed which document is shown first and report the effect below.

First, we audit sensitivity to presentation order using the paper's original Qwen3.5-4B OS + OC trajectories from all 631 queries and five seeds. For each episode, we select up to ten comparisons between distinct documents, sampling across the early, middle and late portions of its comparison sequence. For each pair, we replay the exact logged model-visible conversation prefix and compare the same two summaries in the original and swapped presentation orders. Swapping the order changes 25.4% of the outcomes.

Audit measureResult
Agreement74.57%
Disagreement25.43%

Next, we measure the effect on downstream recall. We compare the order in the paper (incoming document first) with existing document first and with random order.

Qwen3.5-4B methodRecallFARTrajectory RecallTraj. FAR
H1 (paper)0.428 ±0.012\scriptstyle\pm 0.0120.461 ±0.018\scriptstyle\pm 0.0180.509 ±0.004\scriptstyle\pm 0.0040.544 ±0.004\scriptstyle\pm 0.004
OS + OC, incoming document first (paper)0.503 ±0.007\scriptstyle\pm 0.0070.554 ±0.009\scriptstyle\pm 0.0090.526 ±0.008\scriptstyle\pm 0.0080.573 ±0.008\scriptstyle\pm 0.008
Δ\Delta over H1 (pp)+7.5+9.3+1.7+2.9
OS + OC, existing document first0.488 ±0.009\scriptstyle\pm 0.0090.530 ±0.010\scriptstyle\pm 0.0100.510 ±0.007\scriptstyle\pm 0.0070.549 ±0.010\scriptstyle\pm 0.010
Δ\Delta over H1 (pp)+6.0+6.9+0.1+0.5
OS + OC, random order0.488 ±0.008\scriptstyle\pm 0.0080.529 ±0.005\scriptstyle\pm 0.0050.511 ±0.009\scriptstyle\pm 0.0090.548 ±0.005\scriptstyle\pm 0.005
Δ\Delta over H1 (pp)+6.0+6.8+0.2+0.4

OS + OC beats H1 under every order, with a gain of at least +6.0pp Recall (against +7.5pp in the paper), even though swapping the order changes 25.4% of verdicts.

Incoming documents within a round are inserted in the order the agent curates them. With a consistent comparator the final order does not depend on insertion order, so it can only matter through inconsistent verdicts. We will add an experiment that randomizes the insertion order within a round to the final version.

Q2. Other base models

Please see our response to W1 above: we re-ran Harness-1 and OS+OC with Qwen3.5-9B, including the stronger baseline.

Thanks for the new experiments. They largely resolved my concerns and added to the soundness of the paper. I adjusted my "soundness" score accordingly.

Out of curiosity: interestingly, random order performs the same as existing-first rather than in between incoming-first and existing-first. Is it possible that the comparator tends to favor the first- or second-presented document?

I also look forward to seeing the insertion-order randomization experiment in the final version.

Thank you for this observation and your encouraging assessment of the additional experiments.

To test whether the comparator favors either position, we presented the same document pairs in both orders, namely, incoming document first (paper order) and existing document first. Across 28,212 pairs with valid responses in both orders, the model chose the second-presented document in 52.49% of calls and the first in 47.51%. This modest second-position preference appeared across all five seeds.

Random presentation need not produce performance midway between the two fixed-order conditions because we randomize each comparison within an episode. A changed verdict can affect subsequent insertion decisions, retained documents, and the later retrieval trajectory. These effects can accumulate nonlinearly, so final recall does not necessarily interpolate between the fixed-order results. However, these findings do not establish that positional preference explains the similar average recall between existing-first and random order.

The similar aggregate results also conceal different episode outcomes. Comparing the two ordering conditions on the same 631 queries across five seeds (3,155 comparisons), random presentation achieves higher Recall in 844, existing-first presentation achieves higher Recall in 814, and 1,497 tie. These differences nearly cancel in the aggregate, yielding mean Recall of 0.48765 and 0.48788, respectively.

We will include the insertion-order randomization experiment and this discussion in the final version. We appreciate your constructive feedback. Given that the additional experiments have largely resolved your concerns, we would be grateful if you would consider revisiting your overall recommendation.

Reviewer_9EbpOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a harness-level mechanism that allows agents to make context and state-management decisions between main agent turns, without adding these intermediate steps to the main trajectory. The authors propose three variants: Online Summarization (OS), Active Curation (AD), and Ordered Curation (OC). They conducted experiments on BrowseCompPlus and other benchmarks show that OC improves retrieval recall and downstream answer accuracy, while OS and AD help reduce context-related computation.

Critical strengths
  1. The paper proposes a clear and useful framework. The distinction between the proposed micro-turns and existing approaches is well explained. The framework is simple and potentially applicable to other agent systems.

  2. The authors identify several problems in the current retrieval pipeline, including unrendered retrieved chunks, rule-based eviction, and inefficient summarization. The analysis provides a foundation for the proposed methods.

  3. The analysis of where the improvements come from is valuable. They show that the gains mainly come from better evidence selection rather than retrieving more relevant evidence. This helps explain why OC works and makes the results more convincing.

  4. The paper provides a reasonable cost analysis. The authors report both the computational savings and the additional costs introduced by the methods, including increased output tokens and pairwise comparisons.

Critical weaknesses
  1. The evaluation is limited to a single model and retrieval backend. Most experiments use Qwen3.5-4B with BM25 retrieval. Since the proposed methods rely heavily on model judgments, it is unclear whether the same conclusions hold for larger models or stronger retrieval systems. Experiments with different model sizes would strengthen the paper.

  2. The additional cost of OC deserves more attention. Although OC achieves the best recall, it requires many pairwise comparisons (around 484 per BrowseCompPlus episode). On some datasets, its computational cost even exceeds the baseline. It would be useful to better understand when the recall improvements justify this additional cost.

  3. The ablation study is incomplete. OC and AD are always combined with OS, making it difficult to isolate the contribution of each component. In particular, evaluating OC without OS would help clarify whether ordered curation provides benefits independently of summarization.

  4. Overall, this is a good empirical study, my main concern is its technical novelty, this is a good work for information retrieval venues, and maybe not a big loss for ICLR.

Questions for authors

NA

3 comments
AuthorsOpenReview ↗

Response (1/1)

W1. Single model and retrieval backend

The gains have two sources: harness limitations that a stronger model cannot remove (truncated chunks, no total ordering, coarse tag-based eviction), and the model's ability to summarize and compare, which a stronger model should only improve. We therefore expect the gains to persist with stronger models. We re-ran Harness-1 (H1) and Online Summarization + Ordered Curation (OS+OC) on BC+ with a stronger model (Qwen3.5-9B, same BM25 setup) and, separately, with a stronger retriever (Qwen3.5-4B with hybrid BM25 + dense retrieval and reranking, both using Qwen3-Embedding-8B):

MethodRecallFARTrajectory RecallTraj. FAR
Paper: 4B, BM25
H10.428 ±0.012\scriptstyle\pm 0.0120.461 ±0.018\scriptstyle\pm 0.0180.509 ±0.004\scriptstyle\pm 0.0040.544 ±0.004\scriptstyle\pm 0.004
OS + OC0.503 ±0.007\scriptstyle\pm 0.0070.554 ±0.009\scriptstyle\pm 0.0090.526 ±0.008\scriptstyle\pm 0.0080.573 ±0.008\scriptstyle\pm 0.008
Δ\Delta over H1 (pp)+7.5+9.3+1.7+2.9
Stronger model: 9B, BM25
H10.474 ±0.013\scriptstyle\pm 0.0130.515 ±0.012\scriptstyle\pm 0.0120.555 ±0.013\scriptstyle\pm 0.0130.582 ±0.014\scriptstyle\pm 0.014
OS + OC0.560 ±0.012\scriptstyle\pm 0.0120.601 ±0.012\scriptstyle\pm 0.0120.597 ±0.014\scriptstyle\pm 0.0140.633 ±0.014\scriptstyle\pm 0.014
Δ\Delta over H1 (pp)+8.6+8.6+4.2+5.1
Stronger retriever: 4B, hybrid + reranker
H10.546 ±0.008\scriptstyle\pm 0.0080.595 ±0.010\scriptstyle\pm 0.0100.634 ±0.009\scriptstyle\pm 0.0090.680 ±0.014\scriptstyle\pm 0.014
OS + OC0.606 ±0.015\scriptstyle\pm 0.0150.662 ±0.021\scriptstyle\pm 0.0210.637 ±0.013\scriptstyle\pm 0.0130.689 ±0.016\scriptstyle\pm 0.016
Δ\Delta over H1 (pp)+6.0+6.7+0.3+0.9

Bold marks the best mean within each block. Changing the model: gains in all four metrics, with gains of 8.6pp and 8.6pp in Recall and FAR. Changing the retriever: average gains of 6.0pp in Recall and 6.7pp in FAR; Trajectory Recall saturates with modest gains, consistent with OC's gains coming from better selection rather than more discovery. A stronger model or retriever also lifts H1 itself (Recall +4.6pp and +11.8pp), and OS + OC still improves upon it.

W2. Cost of OC

OC's value tracks the gap between trajectory recall (what the agent saw) and final curated recall (what it kept), which OC is designed to close. Where Harness-1 leaves a large gap, OC pays off: on Patents the gap is 0.187 and OC raises final recall by 0.180 (0.544 to 0.724), at 1.21× total attention ops. Where the gap is near zero (WebSimple 0.003, SECSimple 0.008), there is nothing to recover and OC adds cost without gain. There, OS or OS+AD is the cheaper choice, and both lower cost below Harness-1.

W3. OC and AD are always combined with OS

OS gives the compact summaries that Active Decuration (AD) and OC read, so we built them on top of OS. To isolate OC's own contribution, we ran OC without OS (no summaries), comparing either the raw chunks or snippets from the curated set instead of the raw chunks:

Qwen3.5-4B methodRecallFARTrajectory RecallTraj. FAR
H1 (paper)0.428 ±0.012\scriptstyle\pm 0.0120.461 ±0.018\scriptstyle\pm 0.0180.509 ±0.004\scriptstyle\pm 0.0040.544 ±0.004\scriptstyle\pm 0.004
OS, trajectory-conditioned (paper)0.466 ±0.007\scriptstyle\pm 0.0070.516 ±0.009\scriptstyle\pm 0.0090.514 ±0.009\scriptstyle\pm 0.0090.560 ±0.009\scriptstyle\pm 0.009
OS + OC, summaries (paper)0.503 ±0.007\scriptstyle\pm 0.0070.554 ±0.009\scriptstyle\pm 0.0090.526 ±0.008\scriptstyle\pm 0.0080.573 ±0.008\scriptstyle\pm 0.008
H1 + OC, full chunks0.450 ±0.009\scriptstyle\pm 0.0090.482 ±0.011\scriptstyle\pm 0.0110.512 ±0.012\scriptstyle\pm 0.0120.547 ±0.013\scriptstyle\pm 0.013
Change in OS + OC's gain over H1-71%-77%n/an/a
H1 + OC, snippets0.437 ±0.009\scriptstyle\pm 0.0090.471 ±0.017\scriptstyle\pm 0.0170.504 ±0.008\scriptstyle\pm 0.0080.537 ±0.015\scriptstyle\pm 0.015
Change in OS + OC's gain over H1-88%-89%n/an/a

Without OS, OC loses about 71% (full chunks) and 88% (snippets) of OS + OC's Recall gain over H1 (+2.2pp and +0.9pp, against +7.5pp). A one-line snippet cannot separate documents, and full chunks are too dense for reliable comparison and make every comparison longer and costlier. Summaries give OC the right level of information, which is precisely why OC and AD are paired with OS in the paper.

Thanks for your additional experiments and results, and I feel it partially resolved my questions. I changed my score to reflect it.

Thank you for considering the additional experiments and updating your assessment. We appreciate your constructive feedback. If any questions remain, we would be happy to address them.

Reviewer_JsGsOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces micro-turns, brief model calls inserted by the harness between main agent turns to manage context and state. Each exchange is discarded, retaining only its state update. Implemented in Harness-1 using Qwen3.5-4B and BM25, the approach has three variants: online summarization of retrieved chunks (OS), model-guided removal when the curated set exceeds capacity (AD), and pairwise binary-insertion ordering (OC). Across BrowseComp-Plus and four Harness-1 benchmarks, the methods improve final curated recall (BC+: 0.428 to 0.503; Patents: 0.544 to 0.724), Recall@n, and downstream answer accuracy. OS and AD reduce modeled token and attention costs, whereas OC increases total cost on some datasets.

Critical strengths
  • The primitive is simple and general, and it is clearly framed: harness-scheduled, scoped, and discarded after use.
  • Recall gains are consistent and sizable on the benchmarks with headroom (BC+, Patents, Web for OC), with 5 seeds.
  • The efficiency accounting is unusually thorough: cache-aware attention ops, peak context, and a fresh/cached/output split. Reports the 1.7–3.0× higher decode cost.
  • OC’s Recall@n and accuracy@n gains are practically relevant. The accuracy@n curve is nearly flat from n=30 to n=5.
  • The diagnostic appendices are useful (exposure anatomy, overflow accounting, and tag precision).
Critical weaknesses
  • Limited novelty (particularly, OS resembles Search-o1/CompAct; OC comes from PRP).
  • The value of trajectory-conditioned micro-turns is unclear (provide a query-only summarization baseline).
  • Latency or throughput of models is missing.
  • Ablation experiments are missing (e.g., micro-turn exchanges in the main trajectory).
  • Comparison of OC with other reranking baseline alternatives is missing.
  • Fixing the new document’s summary as Summary A across every comparison and placing it before Summary B can cause position bias (no experiments to justify).
Questions for authors

Does OS pass summaries or raw chunks? What are the latency and throughput?

4 comments
AuthorsOpenReview ↗

Response (1/2)

W1. Contribution and Novelty

Our main contribution is the micro-turn mechanism for one problem: maintaining a high-quality curated set across turns of a retrieval agent. Summarization and pairwise comparison are the building blocks of our algorithm. The design follows from the Harness-1's failure here: its importance tags are rarely revisited (1.9% evidence-free revisions on BC+) and coarse: half of the "high" tier (0.476 precision) is not annotated evidence, with no way to rank within it (Appendix C). So we keep a fully ordered set by binary insertion, with an LLM as comparator. Online Summarization (OS) gives the compact content that Active Decuration (AD) and Ordered Curation (OC) read.

As we discussed in the paper (Appendix A lines 627-631), Search-o1 and CompAct are the closest to document-level compression like our on-line summarization and PRP reranks a fixed list for a single query. However, they do not address maintaining a bounded, ordered, curated set across turns.

The table below provides a compact representation of our contribution

MethodC1C2C3C4C5
CompAct✓\checkmark×\times×\times×\times×\times
Search-o1✓\checkmark✓\checkmark✓\checkmark×\times×\times
PRP×\times×\times×\times✓\checkmark×\times
Micro-turns✓\checkmark✓\checkmark✓\checkmark✓\checkmark✓\checkmark
  • C1: Query-only summarization of retrieved content.
  • C2: Trajectory-conditioned summarization.
  • C3: Multi-turn retrieval.
  • C4: Pairwise ranking.
  • C5: Maintains a bounded, ordered document set across turns, with retained document identities and insertion/eviction as new search results arrive. An ordered set requires pairwise comparisons (C4) inside the multi-turn loop (C3), which we run on trajectory-conditioned summaries (C2). This is our main contribution.

Each component helps, and the full combination helps most: Recall over H1 is +2.1pp with query-only summarization (C1) (W2 below), +3.8pp with trajectory-conditioned summarization (C2) (W2 below), +2.2pp with ranking alone (C4) (W3 of our response to Reviewer 9Ebp), and +7.5pp with all of them together (W3 of our response to Reviewer 9Ebp).

W2. Value of trajectory-conditioned micro-turns; query-only baseline

Each OS call sees the main trajectory prefix plus one chunk, so a summary can reflect the agent's current query, findings and curated state, and the prefix is reused from the KV cache. As suggested, we ran a query-only OS baseline: the same summarization micro-turn with the trajectory prefix removed, seeing only the task query and the chunk.

Qwen3.5-4B methodRecallFARTrajectory RecallTraj. FAR
H1 (paper)0.428 ±0.012\scriptstyle\pm 0.0120.461 ±0.018\scriptstyle\pm 0.0180.509 ±0.004\scriptstyle\pm 0.0040.544 ±0.004\scriptstyle\pm 0.004
OS, trajectory-conditioned (paper)0.466 ±0.007\scriptstyle\pm 0.0070.516 ±0.009\scriptstyle\pm 0.0090.514 ±0.009\scriptstyle\pm 0.0090.560 ±0.009\scriptstyle\pm 0.009
OS, query-only0.449 ±0.003\scriptstyle\pm 0.0030.490 ±0.008\scriptstyle\pm 0.0080.501 ±0.004\scriptstyle\pm 0.0040.536 ±0.012\scriptstyle\pm 0.012
Change in OS's gain over H1, query-only-45%-47%n/an/a

Query-only OS loses about 45% of its Recall gain over H1 (from +3.8pp to +2.1pp) and about 47% of its FAR gain (from +5.5pp to +2.9pp), i.e., drops of 1.7pp and 2.6pp. As in the paper, Traj. Recall and Traj. FAR are secondary diagnostic metrics that only bound the curated set from above, so we report the drop for Recall and FAR. These results support the value of trajectory-conditioned summarization.

We also compared the cost of the two, using the paper's attention-ops accounting on the same 631 queries and 5 seeds:

MetricQuery-only OSTrajectory-conditioned OSChange
Attention operations (B) ↓\downarrow1.6252.216+36%×\times
Fresh input tokens (k) ↓\downarrow385.8356.3-7.6%✓\checkmark
Output tokens (k) ↓\downarrow24.820.9-16.0%✓\checkmark
Peak main context (k) ↓\downarrow12.812.0-5.8%✓\checkmark
Recall ↑\uparrow0.4490.466+1.7pp✓\checkmark
FAR ↑\uparrow0.4900.516+2.6pp✓\checkmark

Conditioning on the trajectory costs about 36% more attention operations than query-only OS, but gives higher Recall and FAR with 16% fewer output tokens, 8% fewer fresh input tokens, and a 6% smaller peak main context (since the query-only baseline generated longer summaries and the agent used more turns before terminating).

AuthorsOpenReview ↗

Response (2/2)

W3. Latency and throughput

Wall-clock latency and throughput depend heavily on hardware, serving stack, batching, parallelism, and cache configuration, so they do not cleanly isolate what the algorithm itself costs. We therefore report a hardware-independent measure: cache-aware attention operations, with prefill and decode charged separately from the fresh, cached and output token counts of every call (accounting in the Appendix, "How the Efficiency Numbers Are Computed"). This is the standard causal-attention operation count with cache-hit prefixes charged once, so it reflects and isolates how each variant algorithmically changes compute.

W4. Missing ablations

The appendix has many analyses of where the gains come from (exposure anatomy, overflow accounting, tag revisions and precision). For the example given, keeping micro-turn exchanges in the main trajectory would grow the context quickly, which is exactly what micro-turns avoid: OS can issue up to 22 summarization calls per retrieval step, and OC about 484 comparisons per BC+ episode. We have also added the following ablations: query-only OS (W2 above), OC ordering (W6 below), and OC without OS (W3 of our response to Reviewer 9Ebp). Please let us know which other ablations would be useful.

W5. Comparison with other reranking baselines

As discussed in our response to W1, the pairwise comparator is one building block within our framework, not the contribution. The contribution is the micro-turn mechanism for maintaining a curated set across turns. OC is one instantiation, and we expect other rankers could be substituted in the same slot. Optimizing the internal design of any single micro-turn, including benchmarking alternative rerankers, is beyond the scope of this paper, and it does not undermine our key contribution which is the functionality that each micro-turn provides (here, maintaining an ordered curated set).

W6. Position bias from fixing the incoming document as Summary A

LLM comparators are known to be order-sensitive (e.g., PRP queries both orderings of each pair). However, we present the incoming document first to the LLM to benefit from prefix cache reuse. An incoming document can be cached and compared to multiple documents. In an offline replay of 25,182 comparisons, putting the existing document first (instead of the incoming document, as in the paper) processed 18.5% more fresh (uncached) comparison input tokens, even when allowing reuse of any previously cached matching prefix. Next we analyze the consequences of this design choice.

First, we audit sensitivity to presentation order using the paper's original Qwen3.5-4B OS + OC trajectories from all 631 queries and five seeds. For each episode, we select up to ten comparisons between distinct documents, sampling across the early, middle and late portions of its comparison sequence. For each pair, we replay the exact logged conversation prefix and compare the same two summaries in the original and swapped presentation orders. Swapping the order changes 25.4% of the outcomes.

Audit measureResult
Agreement74.57%
Disagreement25.43%

Next, we measure the effect on the downstream recall. We compare the order in the paper (incoming document first) with existing document first and with random order.

Qwen3.5-4B methodRecallFARTrajectory RecallTraj. FAR
H1 (paper)0.428 ±0.012\scriptstyle\pm 0.0120.461 ±0.018\scriptstyle\pm 0.0180.509 ±0.004\scriptstyle\pm 0.0040.544 ±0.004\scriptstyle\pm 0.004
OS + OC, incoming document first (paper)0.503 ±0.007\scriptstyle\pm 0.0070.554 ±0.009\scriptstyle\pm 0.0090.526 ±0.008\scriptstyle\pm 0.0080.573 ±0.008\scriptstyle\pm 0.008
Δ\Delta over H1 (pp)+7.5+9.3+1.7+2.9
OS + OC, existing document first0.488 ±0.009\scriptstyle\pm 0.0090.530 ±0.010\scriptstyle\pm 0.0100.510 ±0.007\scriptstyle\pm 0.0070.549 ±0.010\scriptstyle\pm 0.010
Δ\Delta over H1 (pp)+6.0+6.9+0.1+0.5
OS + OC, random order0.488 ±0.008\scriptstyle\pm 0.0080.529 ±0.005\scriptstyle\pm 0.0050.511 ±0.009\scriptstyle\pm 0.0090.548 ±0.005\scriptstyle\pm 0.005
Δ\Delta over H1 (pp)+6.0+6.8+0.2+0.4

OS + OC beats H1 under every order, with a gain of at least +6.0pp Recall (against +7.5pp in the paper), even though swapping the order changes 25.4% of verdicts.

Q1. Does OS pass summaries or raw chunks? Latency and throughput?

OS passes summaries. After an OS micro-turn, only the summary of each retrieved chunk remains in the main agent's context, and the raw chunk is seen only inside the discarded micro-turn. This is what lowers the prefill cost in later main turns. For latency and throughput, please see our response to W3.

Thank you for providing detailed clarifications. My concerns are well addressed. I am raising my rating to weak acceptance.

Thank you for your constructive feedback and for updating your assessment. We are pleased that our clarifications have addressed your concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Common Response

We thank all reviewers. Based on the suggestions of multiple reviewers, we have conducted the following additional experiments.

  • Stronger base models (Reviewers 9Ebp and cEzQ): Harness-1 (H1) and Online Summarization + Ordered Curation (OS+OC) with Qwen3.5-9B on BrowseCompPlus (BC+). OS + OC improves over 9B H1 (same model) by 8.6pp Recall and 8.6pp Final Answer Recall (FAR). For reference, 9B H1 is itself 4.6pp above 4B H1.
  • Stronger retrieval backend (Reviewers 9Ebp and cEzQ): Harness-1 and OS+OC with Qwen3.5-4B and hybrid retrieval + reranker on BC+. OS + OC improves over H1 with the same retriever by 6.0pp Recall and 6.7pp FAR. For reference, H1 with the stronger retriever is itself 11.8pp above BM25 H1.
  • Position robustness of OC (Reviewers JsGs and cEzQ): incoming document first (paper), existing document first, and random order in each comparison. OS + OC beats H1 by at least 6pp Recall under every order (7.5pp with the paper's order). The paper's order (incoming document first) is also cheaper: putting the existing document first needs 18.5% more fresh (uncached) comparison input tokens, since the incoming document's prefix can be cached and reused across its comparisons.
  • Query-only OS baseline (Reviewer JsGs): OS without the trajectory prefix on BC+ with Qwen3.5-4B and BM25. Across 5 seeds, it loses about 45% of OS's Recall gain over H1 and about 47% of its FAR gain, demonstrating the value of trajectory conditioning.

Details are in the individual responses below. We hope these results address the main concerns, and we welcome further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.