acceptodds
Under review as a conference paper at ICLR 2027

Reinforcement Learning for Search Agents with State-Externalizing Harnesses

Abstract

Search agents are often trained as policies over growing transcripts: the model must decide how to search while also remembering what it has seen, which evidence is useful, which constraints remain open, and which claims have actually been checked. We argue that this formulation puts too much routine state management inside the policy: reinforcement learning is forced to optimize both semantic search decisions and recoverable bookkeeping that the environment can maintain more reliably. We introduce Harness-1, a 20B search agent (retrieval subagent) trained with reinforcement learning inside a stateful search harness. The harness maintains environment-side working memory, including a candidate pool, an importance-tagged curated set, compact evidence links, verification records, compressed and deduplicated observations, and budget-aware context rendering. The policy retains the semantic decisions: what to search, which documents to keep or discard, what to verify, and when to stop. Across eight retrieval benchmarks spanning web, finance, patents, and multi-hop QA, Harness-1 achieves 0.730 average curated recall, outperforming the next strongest open search subagent by +11.4 points and remaining competitive with much larger frontier-model searchers. Its gains over the Context-1 search agent are especially strong on held-out transfer benchmarks, suggesting that reinforcement learning over explicit search state can produce retrieval behaviors that generalize beyond the training domains.

Primary area:
foundation or frontier models, including LLMs
Keywords:
long-horizon agent, information retrieval, search agent

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_aWYVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that RL-trained search agents spend policy capacity on routine state bookkeeping, and proposes “stateful cognitive offloading.” Harness-1 is a 20B retrieval subagent (gpt-oss-20b) run inside a harness that maintains environment-side working memory. This memory has an importance-tagged curated set (cap 30), a candidate pool, an evidence graph built by regex entity extraction, an LLM-based verify tool, BM25 sentence compression, MinHash deduplication, budget-aware rendering, and per-turn programmatic nudges. The first search auto-seeds the curated set with 8 documents. Training is SFT on 899 filtered GPT-5.4 trajectories collected in the same harness, then CISPO RL on SEC queries only, with a recall-weighted terminal reward plus a tool-diversity bonus. On eight retrieval benchmarks, Harness-1 reaches 0.730 average curated recall. That is +11.4 points over the next open agent and +12.7 over Context-1, and above four of six frontier models run in Context-1’s harness. The paper reports larger gains over Context-1 on four held-out benchmarks, inference-time ablations of each harness mechanism, a harness-swap experiment, a tool-diversity RL ablation, and modular-RAG answer accuracy on BC+ and SEC.

Critical strengths

Large, consistent gains over open baselines, with an inexpensive system. Harness-1 beats Context-1 on all eight benchmarks and Tongyi DeepResearch by 11.4 points on average. It costs about $0.03 per episode and comes with a promised release of weights, harness, and recipe. For retrieval subagents, this is a practically significant result. Unusually candid, self-testing evaluation. The authors run a harness-swap experiment on their own confound (Appendix P). They report an ablation that goes against them (content dedup raises recall) and check that the answer-miss penalty does not discourage discovery. They also include an SFT-vs-RL breakdown, a trajectory with a failed hypothesis (Case E), and an explicit note that Opus-4.6’s smaller sets give higher answer accuracy.

Critical weaknesses
  • Inference-time ablations: Table 3 removes mechanisms from a checkpoint that was trained with all of them. This measures how much this policy depends on its training interface, which is an out-of-distribution test. It does not measure the value of the design for learning. No run trains the same SFT+RL recipe in a thin harness (e.g., Context-1’s), or with auto-seeding, the evidence graph, or compression removed at training time. The only training-time ablation is the diversity reward (Fig. 5), which concerns reward shaping and looks like one run per arm.
  • The “three trainability requirements” are asserted, not shown: Auto-seeding is motivated by identical empty-set rewards in early RL, but there is no RL run with and without it. Its inference-time effect is −0.3% Recall. Compression is +0.2% Recall. Selective reading of Table 3: On the primary metric (Recall), three of seven ablations raise it (compression +0.2%, review_docs +2.4%, dedup +4.6%). “Almost every mechanism matters” rests on final-answer recall changes over 166 queries (9–15 “hard-fail” queries per ablation), with no confidence intervals.
  • Recipe confound: Harness-1 differs from Context-1 in the teacher (GPT-5.4 live in the harness with turn-level guidance), SFT data, reward, and RL data, as well as the harness. The swap in Appendix P shows policy–harness coupling but does not show the harness is the cause.
  • Set size: Recall rises with the number of returned documents. Harness-1 usually fills the 30-document cap, while frontier searchers return about 4–10 (Appendix N). Precision and F-beta are not reported. In Fig. 8 the small Opus-4.6 sets give higher answer accuracy than Harness-1’s sets. So “beats four of six frontier models” partly measures set size. Harness mismatch: Frontier models run in Context-1’s harness while Harness-1 runs in its own. Appendix P measures that our harness adds +4.0 (GPT-5.4) and +3.3 (Opus-4.6) points on the four source-family benchmarks. From Table 2, I get GPT-5.4’s eight-benchmark average as about 0.709, only about 2 points below Harness-1’s 0.730. That is smaller than the measured harness effect, so the win over
  • GPT-5.4 may not survive harness matching. Matched-harness runs exist only on the four source-family benchmarks, which are in-distribution (SFT on all four, RL on SEC). There, Harness-1 beats GPT-5.4 by 2.2 points and loses to Opus (0.713 vs 0.769). “97% of Opus at 4% of its cost” uses Opus in the weaker harness; in its matched harness the figure is about 93%.
Questions for authors
  • Can you train the same SFT+RL recipe (same teacher and data) in a thin, Context-1-style harness, and with auto-seeding, the evidence graph, and the nudges removed at training time? (Determines whether the gain comes from the harness design or the recipe.) Can you ablate the programmatic nudges (Appendix G) individually, and say which nudge categories fire most often in successful trajectories?
  • How do results change at matched set size? Please report recall@k or precision/F2 curves, or truncate Harness-1’s output to the frontier models’ set sizes. Can all frontier models be run in Harness-1’s harness on all eight benchmarks, including the held-out ones?
  • Do the “three runs” in Table 2 cover evaluation repeats of one trained model or independent training runs? Can you add bootstrap confidence intervals and, ideally, ≥2 additional RL seeds? How was the live-web evaluation controlled (same time window, cached results)?
  • What are the base-model and SFT-only results in Harness-1’s harness on the four held-out benchmarks? How does the transfer gain look relative to the base or SFT checkpoint rather than Context-1?
  • Does the $0.031 per-episode cost include the gpt-4o-mini verifier and Qwen3-Reranker-8B calls? How is the self-hosted fine-tuned model priced versus API models?
  • The verify tool uses gpt-4o-mini, which is now a legacy model. Is the exact snapshot pinned, and would an open-weight or current small model (including Harness-1 itself as verifier) give similar verify accuracy and downstream results? This matters for reproducibility of the released recipe.
7 comments
AuthorsOpenReview ↗

Response to Reviewer aWYV (1/3)

Dear Reviewer aWYV,

Thank you for a careful and very concrete review. Your central question is whether the harness helps the policy learn, or only helps the trained policy at test time. We ran the training-time control you proposed to answer it, and we revised each claim you flagged. All changes are marked in blue in the revised PDF.

W1, W3 & Q1. Does the harness help learning, or does the gain come from the recipe?

No run trains the same SFT+RL recipe in a thin harness (e.g., Context-1's) ...

We trained our full recipe in Context-1's harness, the thin harness the paper already uses for the frontier baselines. Tool outputs stay in the transcript, and there is no working memory, curated set, evidence graph, verification or nudges. When the transcript nears its budget, the policy is asked to prune its context or stop, and it ends by listing up to 30 documents. We did not use a plain transcript with automatic truncation, because deep-search episodes fill the context within a few searches, and silently dropping old turns would handicap the transcript run for reasons unrelated to learning. Everything else is shared with Harness-1: the base model, the GPT-5.4 teacher (run inside each harness on the same SFT queries, with the same recall filter), the SFT settings, the SEC RL data, the reward (computed on each model's final document set), the CISPO settings and budget (80 steps × 1,024 rollouts), and the retrieval backend, reranker and 30-document cap. Each model is evaluated in its own harness, as in Table 2.

Together with Context-1's own result, this separates recipe from harness, as W3 asks.

RecipeHarness (train = eval)StageBC+WebPatentsSECHeld-out
OursContext-1SFT0.1860.3020.4420.1710.308
OursContext-1SFT + RL0.4430.5910.8100.4120.501
OursHarness-1SFT0.2260.3440.5050.2420.376
OursHarness-1SFT + RL0.5840.7870.8900.5890.748
Context-1'sContext-1SFT + RL0.5000.6890.8570.4670.579

With the recipe fixed (rows 2 and 4), the stateful model is only a few points ahead after SFT but far ahead after RL, and the gap is widest on the held-out benchmarks that no training stage uses. The difference therefore comes from RL, where data, reward and budget are identical.

With the harness fixed (rows 2 and 5), our recipe ends below Context-1, whose recipe is larger in scale (new Table 21 in Appendix Q.1). So the recipe is not what puts Harness-1 ahead of Context-1. The harness effect is larger than Harness-1's entire lead, and our smaller recipe gives part of it back.

(continued below)

AuthorsOpenReview ↗

Response to Reviewer aWYV (2/3)

The training curves on the SEC training queries (5-step averages) show where the RL difference comes from:

HarnessRecall, step 20 / 40 / 60Answer docs found but not promoted, start → end
Context-10.26 / 0.37 / 0.410.14 → 0.10
Harness-10.40 / 0.56 / 0.590.17 → 0.07

With explicit state, recall rises faster, and the share of answer documents that the policy finds but leaves out of its final set drops further. In the stateful run, the policy gets there by promoting more of what it finds, not by finding less (Appendix D). Selection is the skill that explicit state is meant to make learnable, and it is the one that improves.

W2. The trainability requirements and Table 3.

The "three trainability requirements" are asserted, not shown ... Selective reading of Table 3 ...

You are right that the paper presented the three requirements as if each had been shown on its own. A separate SFT+RL run per mechanism (each RL run samples 81,920 multi-turn episodes) does not fit in the discussion period, so we spent the budget on the controlled comparison above, which tests the design as a whole. The introduction now says so: Table 4 tests the requirements together, and Fig. 5 tests only the diversity incentive. On the nudges (Q1), the transcript harness has none at all, and switching all of them off at inference moves BC+ FA recall by less than one point (0.667 to 0.660). Appendix G now also reports how often each category fires. [ACTION_REQUIRED] and [NEXT] follow almost every search in both successful and failed episodes, [WARN] appears on fewer than 3% of turns, and [TIP] fires more often in failed episodes, which read full documents and use exact search less.

We also agree that our paragraph on Table 3 read the table selectively, and we rewrote it. It now reports both metrics and states the pattern plainly: removing six of the seven mechanisms lowers FA recall, while removing compression (+0.1 points), review_docs (+1.4) or dedup (+2.7) slightly raises curated recall. The two metrics count different documents. Recall covers every annotated relevant document, FA recall only those that contain the answer, so a change can add supporting documents and still lose answer documents, as happens with review_docs (§M.1.6). The dedup gain is mechanical: some BC+ gold sets contain near-duplicate pairs, and dedup keeps one document from each pair (§M.1.7). The heading "Almost every mechanism matters" is now "Most mechanisms protect answer evidence", and no conclusion rests on a single small difference. The final version will add paired bootstrap 95% intervals over the 166 queries for every row.

W4 & Q2. Set size.

... So "beats four of six frontier models" partly measures set size.

We agree that recall at the cap cannot, by itself, show that a set is better rather than bigger, and Appendix N now addresses this directly. Precision, F2 and recall@k each fix a price for extra documents (through β or k), but the right price depends on who reads the set. Here a generator reads the whole set, so we measure that price directly with answer accuracy: if Harness-1 padded its sets with weak documents, its accuracy would drop. It does not. In Fig. 8, Harness-1's sets give higher accuracy than the smaller sets of GPT-5.4, Sonnet-4.6 and Kimi-K2.5 in 21 of 24 generator–benchmark–searcher combinations, and beat GPT-5.4's sets in all eight, each time by at least 6 points. Opus-4.6 is the exception, in line with its higher FA recall (0.736 vs. 0.667 on BC+), and the text says so.

The budget is also part of the task. Every system gets the same 30 slots, and the frontier models' harness asks for up to 30 ranked documents, yet they return 4–10. How many slots to use is the searcher's own decision, like when to stop searching. The open-agent comparison does not depend on the cap at all: Harness-1's capped recall (0.730) is higher than Tongyi DeepResearch's uncapped trajectory recall (0.673), and pooling five Search-R1 samples per query reaches only 0.082 on BC+, against 0.584.

(continued below)

AuthorsOpenReview ↗

Response to Reviewer aWYV (3/3)

W5 & Q2. Matched harness and the cost claim.

"97% of Opus at 4% of its cost" uses Opus in the weaker harness; in its matched harness the figure is about 93%.

Your arithmetic is right, and we have dropped "97% of Opus-4.6 at 4% of its cost" from the paper. That figure compared Harness-1 in its own harness with Opus-4.6 in the weaker one. Sec. 3.2 and Appendix N now quote the same-harness figure instead: Harness-1 reaches 93% of Opus-4.6's source-family recall, and in that comparison it is still ahead of GPT-5.4 on average. Our cost estimates were measured with Opus-4.6 in the Context-1 harness, so putting them next to the 93% would mix two settings, and the paper now reports cost on its own.

The paper is also explicit about which harness each comparison uses. Sec. 3.2 and the Conclusion say that the frontier rows of Table 2 run in the shared Context-1 harness, and Sec. 3.2 refers readers to Appendix P for same-harness results. Table 2 compares systems as they are evaluated; Appendix P compares policies inside one harness. Our matched runs cover only the four source-family benchmarks, and we make no same-harness claim beyond them.

Q3. Runs, seeds, intervals and the live web. The three runs in Table 2 are independent evaluation runs (temperature 1.0) of a single checkpoint, and the camera-ready version will report their standard deviations and paired bootstrap intervals. For now, the new Appendix Q.2 tests the headline gains against evaluation noise using only the reported numbers: a sign test across benchmarks, and a distribution-free bound on evaluation noise that both gains exceed by more than a factor of two (details in our reply to Reviewer RYSk, W2). We could not fit extra RL seeds into the discussion period alongside the controlled training runs, and the Limitations now say that each configuration is trained once. On the web benchmarks, all systems query the same live Serper+Jina endpoint, and results are not cached (Appendix N).

Q4. Base model and SFT-only checkpoint on the held-out benchmarks.

CheckpointLongSealSeal0FRAMESHotpotQAHeld-out avg.Source avg.
Base gpt-oss-20b0.2530.2500.3910.3950.3220.202
SFT only (step 550, RL start)0.3260.3140.4140.4510.3760.329
SFT + RL (Harness-1)0.8910.5510.7100.8410.7480.713

The SFT and SFT+RL rows run in our harness. The base row repeats Table 2, where the untrained baselines run in the shared Context-1 harness. On the held-out benchmarks, SFT adds about five points over the base model, and RL, trained only on SEC, then nearly doubles recall and improves all four. Measured from the SFT checkpoint, RL gains about as much on these benchmarks as on the source families, even though neither training stage sees them, and the control above shows that this gain is far larger with explicit state. We use one harness configuration for all eight benchmarks and tuned nothing on the held-out ones (Appendix D). The table is Table 9 in Appendix D, and Sec. 3.2 cites it.

Q5. Cost. No. The per-episode cost counts only the policy's own input and output tokens, priced at gpt-oss-20b's open-weight rate, and every searcher in the cost table of Appendix N is counted the same way. The reranker and search backend serve all systems, so they are left out for all of them. The one extra LLM call our harness makes is verify, which costs a small fraction of a cent per episode (Appendix M.1.5). We use gpt-oss-20b's rate for the fine-tuned model because the rank-32 LoRA adapter adds almost no inference compute.

Q6. Verifier. We call gpt-4o-mini by model name at temperature 0; Appendix I.2 now lists the full settings. Reproducing the reported numbers does not require the API: each verify judgment is stored in the per-query logs of every evaluation run, so all runs can be re-scored offline. The only alternative we have tested is an NLI model, which finds about half as many supported claims (Table 14). We have not tested whether a small open-weight LLM can serve as the verifier, and Appendix M.1.5 now says so.


Thank you again for this review. It asked for the one experiment the paper was missing, and that experiment shows that explicit state makes RL much more effective and that our recipe does not explain the gain. We have also rewritten every claim you questioned so that it says no more than the evidence shows. If these changes address your concerns, we would be grateful if you would raise your score, and we are happy to discuss anything that remains.

Reviewer_cWxhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Harness-1, a 20B search agent trained with reinforcement learning inside a stateful harness. The harness maintains working memory containing candidate documents, curated evidence, evidence links, and verification records. The policy decides what to search for, which documents to retain, what claims to verify, and when to stop. Training consists of supervised fine-tuning on teacher-generated trajectories followed by reinforcement learning over complete search episodes. The system is evaluated on eight retrieval benchmarks and through downstream question answering, where its curated documents are passed to frozen answer generators.

Critical strengths
  • The harness offers a clear and practical design for maintaining search state, organizing evidence, and supporting verification.
  • The 20B agent achieves strong results across eight retrieval benchmarks, with substantial gains on held-out tasks.
Critical weaknesses

The benefit of stateful offloading during training is not isolated. The paper argues that maintaining search state in the harness reduces the policy’s bookkeeping burden and makes RL more effective. However, the component ablation (Table 3) removes harness mechanisms only at inference time, and the harness-swap experiment changes the harness of an already-trained policy. These experiments demonstrate the learned policy’s dependence on its environment, but do not establish whether externalizing state improves learning. The training-time analyses show gains from RL within the proposed harness and test the tool-diversity reward, but neither compares training with and without the external working state. A controlled comparison should train the same base model under stateful and transcript-style harnesses, with matched tasks, supervision, retrieval capabilities, rewards, and training budgets. Without this control, the contribution of stateful offloading to learning stability, sample efficiency, and final performance remains unclear.

Questions for authors

NA

5 comments
AuthorsOpenReview ↗

Response to Reviewer cWxh (1/2)

Dear Reviewer cWxh,

Thank you for the clear and focused review. We agree with your diagnosis. Table 3 and the harness swap show that the trained policy depends on its interface, but not that externalized state helps it learn. To answer that question, we ran the controlled comparison you describe. It is now Table 4 in Sec. 3.2, with full details in the new Appendix Q.1 (changes in blue).

Weakness: Does externalizing state improve learning?

A controlled comparison should train the same base model under stateful and transcript-style harnesses, with matched tasks, supervision, retrieval capabilities, rewards, and training budgets.

Setup. We trained our recipe twice from the same base model (gpt-oss-20b): once in our stateful harness, and once in Context-1's transcript-style harness, which the paper already uses for the frontier baselines. In the transcript-style harness, tool outputs pile up in the transcript, and there is no working memory, curated set, evidence graph, verification or nudges. When the transcript nears its token limit, the policy is asked to prune chunks or stop, and it ends the episode by listing up to 30 documents. We chose this over a plain transcript with automatic truncation, because deep-search episodes fill the context within a few searches, and silently dropping old turns would handicap the transcript run for reasons unrelated to learning. It is also a strong environment for training: Context-1 itself was trained with RL in it.

Everything else is the same in both runs, which covers each condition you list. The two runs use the same SFT queries and RL data (SEC, 3,453 queries) and the same GPT-5.4 teacher, run inside each harness with the same 0.10 recall filter. They also share the corpora, search backend, reranker and 30-document cap, the reward (computed on each model's final document set), the SFT settings, and the CISPO settings and RL budget (80 steps × 1,024 rollouts). Each model is evaluated in the harness it was trained in, as in Table 2.

RecipeHarness (train = eval)StageBC+WebPatentsSECHeld-out
OursContext-1SFT0.1860.3020.4420.1710.308
OursContext-1SFT + RL0.4430.5910.8100.4120.501
OursHarness-1SFT0.2260.3440.5050.2420.376
OursHarness-1SFT + RL0.5840.7870.8900.5890.748
Context-1'sContext-1SFT + RL0.5000.6890.8570.4670.579

Rows 2 and 4 share the recipe and differ only in the harness. Below, "source-family" is the average of the first four columns.

  • After SFT, the stateful model is only a few points ahead: 5.4 on the source-family benchmarks and 6.8 on the held-out ones.
  • After RL, it is ahead by 14.8 and 24.7 points. RL adds 38.3 and 37.2 points with explicit state, but only 28.9 and 19.3 without it.

So the gap opens during RL, where the data, reward and budget are identical, and it is widest on the held-out benchmarks, which neither SFT nor RL ever sees. The SFT results matter too. The teacher has to act through each harness's own tools, so the two SFT datasets cannot be identical, yet they produce models that start only a few points apart. The difference comes from what RL can learn in each harness.

(continued below)

AuthorsOpenReview ↗

Response to Reviewer cWxh (2/2)

Sample efficiency, stability and final performance. Your review names these three as the open questions. Final performance is in the table above. For sample efficiency, here are the training curves on the SEC training queries:

HarnessRecall, step 20 / 40 / 60Answer docs found but not promoted, start → end
Context-10.26 / 0.37 / 0.410.14 → 0.10
Harness-10.40 / 0.56 / 0.590.17 → 0.07

With explicit state, RL raises recall faster and closes more of the gap between finding answer documents and keeping them in the final set. In the stateful run, this happens because the policy promotes more of the answer documents it finds, not because it finds fewer (Appendix D). Selection is the skill that explicit state was designed to make learnable, and it is the skill that improves.

Stability across seeds is the one part we cannot measure, because each configuration is trained once, and the Limitations now say so. The difference does not look like seed noise, though: the stateful model is ahead on every benchmark in the table, after SFT and after RL, and at every logged training step.

The recipe does not explain Harness-1's lead over Context-1. Row 5 is Context-1 trained in its own harness. Our recipe in the same harness (row 2) ends 6.4 source-family and 7.8 held-out points below it. This is what one would expect, since Context-1's recipe is larger on every axis of scale:

Context-1Harness-1
SFT data>8K synthetic tasks899 queries
RL data9,159 queries, 6 splits3,453 queries, SEC only
RL rollouts≈230 steps × 1,02480 steps × 1,024

Harness-1's lead over Context-1 (8.4 source-family and about 17 held-out points, Fig. 3) is therefore the harness effect (14.8 and 24.7) minus a recipe deficit. The harness accounts for more than the whole gain.

Scope. The experiment tests the stateful design as a whole rather than each mechanism separately. The revised introduction says this directly: the three design requirements are tested together at training time (Sec. 3.2), and the diversity incentive alone in Fig. 5.


Thank you for asking for this experiment. It fills the main gap in the paper and shows directly that explicit state helps the policy learn. If this resolves your concern, we would be grateful if you would raise your score.

AuthorsOpenReview ↗

Follow-up to Reviewer cWxh

Hi Reviewer cWxh,

Thank you again for your kind and helpful review. You asked whether keeping state outside the transcript helps the model learn. The experiment you suggested shows that it does. After SFT, our model is only a little better than the same model trained in a transcript-style harness. After RL, it is far better, especially on benchmarks it never saw in training (Table 4). We would be glad to hear whether this answers your question. Thank you for helping us improve the paper.

Reviewer_RYSkOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes Harness-1, agentic harness that separates memory management from decision making. Authors LoRA finetuned gpt-oss-20b using SFT and RL.

The harness maintains the entire knowledge in a highly structured way through candidate pool, Importance tagged curated set, full text document store, regex based evidence graph, cached tool calls and results. The harness applies many techniques to keep the retrieved knowledge and LLM context as concise as possible through document compression, deduplication, working memory creation. The agent interacts with environment using specific tools to retrieve more information, curate and verify the existing information.

SFT uses 899 GPT-5.4 trajectories to improve agent in using the harness specific tool calls. RL uses SEC train split on the harness-1 to improve overall performance.

The paper shows improvement of around 11.4 points over tongyi deepresearch on 8 benchmarks. Closed source models also showed improvement when using harness-1 instead of Context-1 harness.

Critical strengths
  • Table 2 shows harness-1 achieves highest recall over 8 different benchmarks over any other open source model.
  • Table 12 shows that harness-1 is extremely cost efficient compared to any frontier model even with context-1 harness
  • Table 3 shows clear importance of each of the harness components. Content deduplication though reduces the recall, shown no loss in information is explained but no final answer correctness evaluation is done
  • Authors well documented several practical design choices (autoseeding empty set on first search, Diversity bonus against collapse of repeated search calls, Answer miss penalty weight not to discourage discovery)
  • Extremely detailed appendix for better understanding and reproducibility
  • Figure 9a shows the harness improves the frontier models performance as well
Critical weaknesses
  • Authors say "harness makes no retrieval decisions" but per-turn programmatic nudges still guide through [ACTION_REQUIRED], [WARN], [TIP]
  • Table 2 says averaged over three runs but reported no variance
  • Final answer accuracy is shown only on SEC and BC+, what about other benchmarks?
Questions for authors
  • How much effect does per-turn nudges have?
  • Does $0.031 cost include harness made llm calls (verify, reranker)?
  • How are annotated sets built for open web search benchmarks like FRAMES, SealQA, Hotpot QA?
5 comments
AuthorsOpenReview ↗

Response to Reviewer RYSk (1/2)

Dear Reviewer RYSk,

Thank you for the careful review and the precise questions. We answer each one below, most of them with new results, and the corresponding changes are in the revised PDF (in blue).


W1. Does the harness make retrieval decisions?

Authors say "harness makes no retrieval decisions" but per-turn programmatic nudges still guide through [ACTION_REQUIRED], [WARN], [TIP]

You are right that "makes no retrieval decisions" was too broad. What we meant is that the harness never makes a semantic choice: it never writes a query, chooses a document to keep or drop, writes a claim to verify, or decides when to stop. It does act without being asked in two places, and Sec. 2 now says so explicitly. Auto-seeding places the top-8 results of the first search in the curated set as tentative fair candidates, which the policy must then promote or remove. The result summaries (Appendix G) add procedural reminders such as "curate before searching again". These reminders are about the order of actions (search, then curate). They never say which query to issue or which document to keep, and Q1 shows that the trained policy finds the answer evidence without them.

Q1. How much do the per-turn nudges matter?

We switched all prescriptive lines off at inference on BC+, keeping the factual [STATUS] and curated-set lines, the tools, the working memory and the checkpoint. Table 3's all-off row removes the nudges together with every other mechanism, so this is the first run that isolates them:

BC+ conditionRecallFA Recall
Full Harness-10.5840.667
Prescriptive lines off0.5610.660

FA recall drops by only 0.7 points, less than when any of the six state mechanisms that lower it in Table 3 is removed. Recall drops by 2.3, about as much as without importance tags. The nudges help the policy collect supporting documents, but it finds the answer evidence on its own.

The firing counts explain why. We counted every firing in the BC+ evaluation episodes of Table 3, separately for successful episodes (FA recall ≥ 0.5, as in Appendix M) and failed ones:

NudgeFires per episode (success / failure)Fires per turn (success / failure)
[ACTION_REQUIRED]17.5 / 20.70.50 / 0.52
[WARN]0.7 / 1.20.02 / 0.03
[TIP]12.5 / 17.10.36 / 0.43
[NEXT]17.5 / 20.70.50 / 0.52

[ACTION_REQUIRED] and [NEXT] fire after almost every search, equally often in both groups. [WARN] fires when the policy goes two turns without curating, and it appears on fewer than 3% of turns, so the policy keeps the search-then-curate rhythm by itself. The two groups differ most in [TIP]: failed episodes read full documents and use exact search less often, so they get more reminders to do so, which matches the failure pattern in Appendix M. Both tables are in Appendix G. As for training, the transcript-style harness in our new training experiment (see our replies to Reviewers cWxh and aWYV) has no nudges at all.

W2. Variance across runs.

Table 2 says averaged over three runs but reported no variance

The three runs in Table 2 are independent evaluation runs (temperature 1.0) of one trained checkpoint. We will add their standard deviations and paired bootstrap intervals in the camera-ready version. Two checks that use only the reported numbers already rule out evaluation noise as an explanation for the headline gains (new Appendix Q.2):

  • Harness-1 is ahead of Context-1 on all eight benchmarks (one-sided sign test, p = 0.004) and ahead of Tongyi DeepResearch on seven of eight (p = 0.035).
  • Each query's recall lies in [0, 1], so the difference between two systems on a query lies in [−1, 1]. With the benchmark sizes in Appendix E (111 to 824 queries), Hoeffding's inequality bounds the 95% interval of any eight-benchmark average difference by ±5.6 points, whatever the per-query variance. The gains over Tongyi DeepResearch (+11.4) and Context-1 (+12.7) are both more than twice this.

Each configuration is trained with RL once, which the Limitations now state.

(continued below)

AuthorsOpenReview ↗

Response to Reviewer RYSk (2/2)

W3. Answer accuracy beyond BC+ and SEC.

Final answer accuracy is shown only on SEC and BC+, what about other benchmarks?

We ran the same protocol (GPT-5.4 as generator, same judge; all queries, averaged over three runs) on the four held-out benchmarks, the ones furthest from the training data:

Evidence given to the generatorLongSealSeal0FRAMESHotpotQAAvg.
None (closed-book)0.2000.0670.5560.5760.350
Naive RAG (one search, top-10)0.2550.0570.5410.5890.361
Context-1's curated set0.5170.1810.6600.7060.516
Harness-1's curated set0.5660.1950.7280.7110.550

Harness-1's sets give the highest accuracy on all four benchmarks, 3.4 points above Context-1's on average. Over closed-book answering, they add 20.0 points of accuracy, against 16.6 for Context-1's sets. The margins are smaller than the recall margins largely because GPT-5.4 already answers many FRAMES and HotpotQA questions from memory (closed-book 0.556 and 0.576), which leaves less room for retrieval to help. This is consistent with Fig. 8: on SEC, Harness-1's sets beat Context-1's for all four generators, by 4 to 11 points, and across BC+ and SEC they beat the sets of GPT-5.4, Sonnet-4.6 and Kimi-K2.5 in 21 of 24 generator–benchmark–searcher combinations. The new table is in Appendix O.

Q2. Does the $0.031 include the verifier and reranker?

Does $0.031 cost include harness made llm calls (verify, reranker)?

No. The cost table in Appendix N compares searchers under one accounting rule: each row counts only the searcher's own input and output tokens, and for Harness-1 this gives the $0.031. The reranker and search backend are shared services that every system calls, including the helper the open-web read tool uses for long pages, so they are left out for all systems alike.

verify is the one LLM call that only our harness makes, so we account for it explicitly. Each check sends the claim and one document to gpt-4o-mini and costs about $0.0002 (Appendix M.1.5), a small fraction of a cent per episode. Including it therefore does not change any comparison in the cost table.

Q3. How are gold evidence sets built for the open-web benchmarks?

How are annotated sets built for open web search benchmarks like FRAMES, SealQA, Hotpot QA?

We do not build gold sets ourselves. Each benchmark comes with human-annotated evidence, and we use it unchanged:

  • FRAMES (Krishna et al., 2025): expert annotators wrote each of the 824 questions from several Wikipedia articles and recorded the articles needed to answer it (2 to 15 per question). The authors re-annotated the questions three months later and removed those whose answers were no longer true. Our gold set is this article list.
  • HotpotQA (Yang et al., 2018): crowd workers were shown two Wikipedia paragraphs (connected by a hyperlink for bridge questions, or two entities from the same list for comparison questions), wrote a question that needs both, and provided the sentences they used to answer it. Our gold set is the two articles containing these supporting facts.
  • Seal0QA (Pham et al., 2025): NLP researchers wrote each question together with webpages that justify the answer. Each question was reviewed by at least two graduate-level annotators and approved by an expert, and the supporting URLs were verified during data cleaning. Seal-0 keeps the 111 questions that GPT-4o, GPT-4.1, their mini variants and Llama-4-Scout, with and without browsing, all failed to answer across 10–15 attempts. Our gold set is the annotated URLs.
  • LongSealQA (Pham et al., 2025): each question comes with one helpful document taken from the annotators' supporting webpages and up to 50 hard negatives retrieved with Google search. The helpful document is the gold.

We match at the document level. FRAMES and HotpotQA search only Wikipedia (site:wikipedia.org, read through the Wikipedia API), so every retrieved document is a Wikipedia URL; URLs are normalized by dropping the fragment (/wiki/X#Section matches /wiki/X), and Seal0QA URLs are matched the same way. LongSealQA documents are indexed in one Chroma collection, so IDs match exactly. All systems share the same backend and the same gold sets, so the comparison is fair, and answer accuracy (W3), which does not use the gold annotations at all, gives the same result. Appendix E now describes each dataset's annotation and our matching rule.


We believe these answers address each of your questions, and the new results make the paper's case stronger than before. If you agree that your concerns are resolved, we would be grateful if you would raise your score. We are happy to discuss anything further.

AuthorsOpenReview ↗

Follow-up to Reviewer RYSk

Hi Reviewer RYSk,

Thank you again for your review and your positive assessment. Two new results answer your questions most directly: switching all nudges off at inference barely changes the answer evidence the policy finds, and Harness-1 gives the highest answer accuracy on all four held-out benchmarks. Together with the new training control, they support the paper’s main point, that keeping search state in the harness is what lets a 20B open model outperform frontier models such as GPT-5.4 under a shared protocol. We would be glad to hear whether these results address your concerns, and we are happy to clarify anything in the remaining time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.