- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper falls in the field of retrieval agent state management. Motivated by the limitations of fixed state management (including excessive trajectory growth, suboptimal document ordering and curation, repeated prefill costs for verbatim retrieved chunks), the paper proposes micro-turns, short harness-injected model calls that can manage the state and context between main agent turns.
The method contains 3 key components: online summarization (OS), active decuration (AD), and ordered curation (OC). OS can summarize the retrieved chunks to mitigate context truncation and trajectory growth issues due to rendering chunks verbatim. AD and OC are two curation-management methods, where AD explicitly directs the model to remove documents and restore capacity limit when needed, and OC uses binary comparison (via LLM pair-wise comparison) to insert each new document at the right position and maintain ordering.
With the Harness-1 method as a baseline, the paper compares several settings: Micro-turns + OS, Micro-turns + OS + AD, Micro-turns + OS + OC. Results on 5 different datasets are overall very positive, showing that the proposed method generally outperforms baseline, and that Micro-turns + OS + OC is generally the best-performing method.
The paper also conducts cost analysis from different dimensions. While Micro-turns + OS + OC can be more expensive than Harness-1, other methods reduce both processed tokens and estimated attention operations. All micro-turn variants have lower peak per-call context lengths, suggesting potential memory savings.
Critical strengths
- The method is simple, well motivated and effective.
- The experiments are well done, covering multiple datasets with very good overall performance. Tables 1 and 2 report five-seed means ± standard deviations, so the results seem relatively reliable. It's also great to see several in-depth analyses from different perspectives that help explain the method’s effectiveness in specific scenarios (top-n curated-set quality, and downstream answering quality).
- The tables also show improvements in cost, which relieves my concerns about potential cost overhead of more sophisticated state management. In most cases, the costs are even lower than the baseline.
Critical weaknesses
The experiments use only Qwen3.5-4B as the retrieval model. It is unclear whether the gains would hold with other models, especially stronger models that may already handle curation better, resulting in a more competitive Harness-1 baseline.
Questions for authors
- How reliable is binary insertion when the LLM’s comparisons are inconsistent (such as A > B, B > C, but C > A)? Could this cause OC to remove useful documents? Did you try changing the insertion order to see if the results change?
- Have you tried base models other than Qwen3.5-4B? Would the conclusions change with a different base model? For example, a stronger base model may improve Harness-1’s curation ability, thus improving the baseline’s performance.
4 comments
Response (2/2)
Q1. Inconsistent comparisons (A > B, B > C, C > A)
Binary insertion compares an incoming document with only about log n retained ones, so a cycle is never queried or stored; the set is always a total order by construction. An inconsistent verdict can still misplace a document, and a useful one placed near the bottom could be evicted at overflow. We do not measure how often verdicts are inconsistent. In each comparison we show the incoming document first to benefit from prefix cache reuse: an incoming document can be cached and compared to multiple retained documents. In an offline replay of 25,182 comparisons, putting the existing document first (instead of the incoming document, as in the paper) processed 18.5% more fresh (uncached) comparison input tokens, even when allowing reuse of any previously cached matching prefix. We also changed which document is shown first and report the effect below.
First, we audit sensitivity to presentation order using the paper's original Qwen3.5-4B OS + OC trajectories from all 631 queries and five seeds. For each episode, we select up to ten comparisons between distinct documents, sampling across the early, middle and late portions of its comparison sequence. For each pair, we replay the exact logged model-visible conversation prefix and compare the same two summaries in the original and swapped presentation orders. Swapping the order changes 25.4% of the outcomes.
| Audit measure | Result |
|---|---|
| Agreement | 74.57% |
| Disagreement | 25.43% |
Next, we measure the effect on downstream recall. We compare the order in the paper (incoming document first) with existing document first and with random order.
| Qwen3.5-4B method | Recall | FAR | Trajectory Recall | Traj. FAR |
|---|---|---|---|---|
| H1 (paper) | 0.428 | 0.461 | 0.509 | 0.544 |
| OS + OC, incoming document first (paper) | 0.503 | 0.554 | 0.526 | 0.573 |
| over H1 (pp) | +7.5 | +9.3 | +1.7 | +2.9 |
| OS + OC, existing document first | 0.488 | 0.530 | 0.510 | 0.549 |
| over H1 (pp) | +6.0 | +6.9 | +0.1 | +0.5 |
| OS + OC, random order | 0.488 | 0.529 | 0.511 | 0.548 |
| over H1 (pp) | +6.0 | +6.8 | +0.2 | +0.4 |
OS + OC beats H1 under every order, with a gain of at least +6.0pp Recall (against +7.5pp in the paper), even though swapping the order changes 25.4% of verdicts.
Incoming documents within a round are inserted in the order the agent curates them. With a consistent comparator the final order does not depend on insertion order, so it can only matter through inconsistent verdicts. We will add an experiment that randomizes the insertion order within a round to the final version.
Q2. Other base models
Please see our response to W1 above: we re-ran Harness-1 and OS+OC with Qwen3.5-9B, including the stronger baseline.
Thanks for the new experiments. They largely resolved my concerns and added to the soundness of the paper. I adjusted my "soundness" score accordingly.
Out of curiosity: interestingly, random order performs the same as existing-first rather than in between incoming-first and existing-first. Is it possible that the comparator tends to favor the first- or second-presented document?
I also look forward to seeing the insertion-order randomization experiment in the final version.
Thank you for this observation and your encouraging assessment of the additional experiments.
To test whether the comparator favors either position, we presented the same document pairs in both orders, namely, incoming document first (paper order) and existing document first. Across 28,212 pairs with valid responses in both orders, the model chose the second-presented document in 52.49% of calls and the first in 47.51%. This modest second-position preference appeared across all five seeds.
Random presentation need not produce performance midway between the two fixed-order conditions because we randomize each comparison within an episode. A changed verdict can affect subsequent insertion decisions, retained documents, and the later retrieval trajectory. These effects can accumulate nonlinearly, so final recall does not necessarily interpolate between the fixed-order results. However, these findings do not establish that positional preference explains the similar average recall between existing-first and random order.
The similar aggregate results also conceal different episode outcomes. Comparing the two ordering conditions on the same 631 queries across five seeds (3,155 comparisons), random presentation achieves higher Recall in 844, existing-first presentation achieves higher Recall in 814, and 1,497 tie. These differences nearly cancel in the aggregate, yielding mean Recall of 0.48765 and 0.48788, respectively.
We will include the insertion-order randomization experiment and this discussion in the final version. We appreciate your constructive feedback. Given that the additional experiments have largely resolved your concerns, we would be grateful if you would consider revisiting your overall recommendation.
Response (1/2)
W1. Only Qwen3.5-4B; a stronger model may make the baseline more competitive
A stronger model may curate better under Harness-1, but the harness limits remain: truncated chunks, no total ordering, and coarse tag-based eviction. Micro-turns address these limits, and also benefit from a stronger model's summarization and comparison. To check, we re-ran Harness-1 (H1) and Online Summarization + Ordered Curation (OS+OC) on BC+ with a stronger model (Qwen3.5-9B, same BM25 setup) and, separately, with a stronger retriever (Qwen3.5-4B with hybrid BM25 + dense retrieval and reranking, both using Qwen3-Embedding-8B):
Bold marks the best mean within each block. Changing the model: gains in all four metrics, with gains of 8.6pp and 8.6pp in Recall and FAR. Changing the retriever: average gains of 6.0pp in Recall and 6.7pp in FAR; Trajectory Recall saturates with modest gains, consistent with OC's gains coming from better selection rather than more discovery. A stronger model or retriever also lifts H1 itself (Recall +4.6pp and +11.8pp), and OS + OC still improves upon it.