- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that RL-trained search agents spend policy capacity on routine state bookkeeping, and proposes “stateful cognitive offloading.” Harness-1 is a 20B retrieval subagent (gpt-oss-20b) run inside a harness that maintains environment-side working memory. This memory has an importance-tagged curated set (cap 30), a candidate pool, an evidence graph built by regex entity extraction, an LLM-based verify tool, BM25 sentence compression, MinHash deduplication, budget-aware rendering, and per-turn programmatic nudges. The first search auto-seeds the curated set with 8 documents. Training is SFT on 899 filtered GPT-5.4 trajectories collected in the same harness, then CISPO RL on SEC queries only, with a recall-weighted terminal reward plus a tool-diversity bonus. On eight retrieval benchmarks, Harness-1 reaches 0.730 average curated recall. That is +11.4 points over the next open agent and +12.7 over Context-1, and above four of six frontier models run in Context-1’s harness. The paper reports larger gains over Context-1 on four held-out benchmarks, inference-time ablations of each harness mechanism, a harness-swap experiment, a tool-diversity RL ablation, and modular-RAG answer accuracy on BC+ and SEC.
Critical strengths
Large, consistent gains over open baselines, with an inexpensive system. Harness-1 beats Context-1 on all eight benchmarks and Tongyi DeepResearch by 11.4 points on average. It costs about $0.03 per episode and comes with a promised release of weights, harness, and recipe. For retrieval subagents, this is a practically significant result. Unusually candid, self-testing evaluation. The authors run a harness-swap experiment on their own confound (Appendix P). They report an ablation that goes against them (content dedup raises recall) and check that the answer-miss penalty does not discourage discovery. They also include an SFT-vs-RL breakdown, a trajectory with a failed hypothesis (Case E), and an explicit note that Opus-4.6’s smaller sets give higher answer accuracy.
Critical weaknesses
- Inference-time ablations: Table 3 removes mechanisms from a checkpoint that was trained with all of them. This measures how much this policy depends on its training interface, which is an out-of-distribution test. It does not measure the value of the design for learning. No run trains the same SFT+RL recipe in a thin harness (e.g., Context-1’s), or with auto-seeding, the evidence graph, or compression removed at training time. The only training-time ablation is the diversity reward (Fig. 5), which concerns reward shaping and looks like one run per arm.
- The “three trainability requirements” are asserted, not shown: Auto-seeding is motivated by identical empty-set rewards in early RL, but there is no RL run with and without it. Its inference-time effect is −0.3% Recall. Compression is +0.2% Recall. Selective reading of Table 3: On the primary metric (Recall), three of seven ablations raise it (compression +0.2%, review_docs +2.4%, dedup +4.6%). “Almost every mechanism matters” rests on final-answer recall changes over 166 queries (9–15 “hard-fail” queries per ablation), with no confidence intervals.
- Recipe confound: Harness-1 differs from Context-1 in the teacher (GPT-5.4 live in the harness with turn-level guidance), SFT data, reward, and RL data, as well as the harness. The swap in Appendix P shows policy–harness coupling but does not show the harness is the cause.
- Set size: Recall rises with the number of returned documents. Harness-1 usually fills the 30-document cap, while frontier searchers return about 4–10 (Appendix N). Precision and F-beta are not reported. In Fig. 8 the small Opus-4.6 sets give higher answer accuracy than Harness-1’s sets. So “beats four of six frontier models” partly measures set size. Harness mismatch: Frontier models run in Context-1’s harness while Harness-1 runs in its own. Appendix P measures that our harness adds +4.0 (GPT-5.4) and +3.3 (Opus-4.6) points on the four source-family benchmarks. From Table 2, I get GPT-5.4’s eight-benchmark average as about 0.709, only about 2 points below Harness-1’s 0.730. That is smaller than the measured harness effect, so the win over
- GPT-5.4 may not survive harness matching. Matched-harness runs exist only on the four source-family benchmarks, which are in-distribution (SFT on all four, RL on SEC). There, Harness-1 beats GPT-5.4 by 2.2 points and loses to Opus (0.713 vs 0.769). “97% of Opus at 4% of its cost” uses Opus in the weaker harness; in its matched harness the figure is about 93%.
Questions for authors
- Can you train the same SFT+RL recipe (same teacher and data) in a thin, Context-1-style harness, and with auto-seeding, the evidence graph, and the nudges removed at training time? (Determines whether the gain comes from the harness design or the recipe.) Can you ablate the programmatic nudges (Appendix G) individually, and say which nudge categories fire most often in successful trajectories?
- How do results change at matched set size? Please report recall@k or precision/F2 curves, or truncate Harness-1’s output to the frontier models’ set sizes. Can all frontier models be run in Harness-1’s harness on all eight benchmarks, including the held-out ones?
- Do the “three runs” in Table 2 cover evaluation repeats of one trained model or independent training runs? Can you add bootstrap confidence intervals and, ideally, ≥2 additional RL seeds? How was the live-web evaluation controlled (same time window, cached results)?
- What are the base-model and SFT-only results in Harness-1’s harness on the four held-out benchmarks? How does the transfer gain look relative to the base or SFT checkpoint rather than Context-1?
- Does the $0.031 per-episode cost include the gpt-4o-mini verifier and Qwen3-Reranker-8B calls? How is the self-hosted fine-tuned model priced versus API models?
- The verify tool uses gpt-4o-mini, which is now a legacy model. Is the exact snapshot pinned, and would an open-weight or current small model (including Harness-1 itself as verifier) give similar verify accuracy and downstream results? This matters for reproducibility of the released recipe.
7 comments
Response to Reviewer aWYV (2/3)
The training curves on the SEC training queries (5-step averages) show where the RL difference comes from:
| Harness | Recall, step 20 / 40 / 60 | Answer docs found but not promoted, start → end |
|---|---|---|
| Context-1 | 0.26 / 0.37 / 0.41 | 0.14 → 0.10 |
| Harness-1 | 0.40 / 0.56 / 0.59 | 0.17 → 0.07 |
With explicit state, recall rises faster, and the share of answer documents that the policy finds but leaves out of its final set drops further. In the stateful run, the policy gets there by promoting more of what it finds, not by finding less (Appendix D). Selection is the skill that explicit state is meant to make learnable, and it is the one that improves.
W2. The trainability requirements and Table 3.
The "three trainability requirements" are asserted, not shown ... Selective reading of Table 3 ...
You are right that the paper presented the three requirements as if each had been shown on its own. A separate SFT+RL run per mechanism (each RL run samples 81,920 multi-turn episodes) does not fit in the discussion period, so we spent the budget on the controlled comparison above, which tests the design as a whole. The introduction now says so: Table 4 tests the requirements together, and Fig. 5 tests only the diversity incentive. On the nudges (Q1), the transcript harness has none at all, and switching all of them off at inference moves BC+ FA recall by less than one point (0.667 to 0.660). Appendix G now also reports how often each category fires. [ACTION_REQUIRED] and [NEXT] follow almost every search in both successful and failed episodes, [WARN] appears on fewer than 3% of turns, and [TIP] fires more often in failed episodes, which read full documents and use exact search less.
We also agree that our paragraph on Table 3 read the table selectively, and we rewrote it. It now reports both metrics and states the pattern plainly: removing six of the seven mechanisms lowers FA recall, while removing compression (+0.1 points), review_docs (+1.4) or dedup (+2.7) slightly raises curated recall. The two metrics count different documents. Recall covers every annotated relevant document, FA recall only those that contain the answer, so a change can add supporting documents and still lose answer documents, as happens with review_docs (§M.1.6). The dedup gain is mechanical: some BC+ gold sets contain near-duplicate pairs, and dedup keeps one document from each pair (§M.1.7). The heading "Almost every mechanism matters" is now "Most mechanisms protect answer evidence", and no conclusion rests on a single small difference. The final version will add paired bootstrap 95% intervals over the 166 queries for every row.
W4 & Q2. Set size.
... So "beats four of six frontier models" partly measures set size.
We agree that recall at the cap cannot, by itself, show that a set is better rather than bigger, and Appendix N now addresses this directly. Precision, F2 and recall@k each fix a price for extra documents (through β or k), but the right price depends on who reads the set. Here a generator reads the whole set, so we measure that price directly with answer accuracy: if Harness-1 padded its sets with weak documents, its accuracy would drop. It does not. In Fig. 8, Harness-1's sets give higher accuracy than the smaller sets of GPT-5.4, Sonnet-4.6 and Kimi-K2.5 in 21 of 24 generator–benchmark–searcher combinations, and beat GPT-5.4's sets in all eight, each time by at least 6 points. Opus-4.6 is the exception, in line with its higher FA recall (0.736 vs. 0.667 on BC+), and the text says so.
The budget is also part of the task. Every system gets the same 30 slots, and the frontier models' harness asks for up to 30 ranked documents, yet they return 4–10. How many slots to use is the searcher's own decision, like when to stop searching. The open-agent comparison does not depend on the cap at all: Harness-1's capped recall (0.730) is higher than Tongyi DeepResearch's uncapped trajectory recall (0.673), and pooling five Search-R1 samples per query reaches only 0.082 on BC+, against 0.584.
(continued below)
Response to Reviewer aWYV (3/3)
W5 & Q2. Matched harness and the cost claim.
"97% of Opus at 4% of its cost" uses Opus in the weaker harness; in its matched harness the figure is about 93%.
Your arithmetic is right, and we have dropped "97% of Opus-4.6 at 4% of its cost" from the paper. That figure compared Harness-1 in its own harness with Opus-4.6 in the weaker one. Sec. 3.2 and Appendix N now quote the same-harness figure instead: Harness-1 reaches 93% of Opus-4.6's source-family recall, and in that comparison it is still ahead of GPT-5.4 on average. Our cost estimates were measured with Opus-4.6 in the Context-1 harness, so putting them next to the 93% would mix two settings, and the paper now reports cost on its own.
The paper is also explicit about which harness each comparison uses. Sec. 3.2 and the Conclusion say that the frontier rows of Table 2 run in the shared Context-1 harness, and Sec. 3.2 refers readers to Appendix P for same-harness results. Table 2 compares systems as they are evaluated; Appendix P compares policies inside one harness. Our matched runs cover only the four source-family benchmarks, and we make no same-harness claim beyond them.
Q3. Runs, seeds, intervals and the live web. The three runs in Table 2 are independent evaluation runs (temperature 1.0) of a single checkpoint, and the camera-ready version will report their standard deviations and paired bootstrap intervals. For now, the new Appendix Q.2 tests the headline gains against evaluation noise using only the reported numbers: a sign test across benchmarks, and a distribution-free bound on evaluation noise that both gains exceed by more than a factor of two (details in our reply to Reviewer RYSk, W2). We could not fit extra RL seeds into the discussion period alongside the controlled training runs, and the Limitations now say that each configuration is trained once. On the web benchmarks, all systems query the same live Serper+Jina endpoint, and results are not cached (Appendix N).
Q4. Base model and SFT-only checkpoint on the held-out benchmarks.
| Checkpoint | LongSeal | Seal0 | FRAMES | HotpotQA | Held-out avg. | Source avg. |
|---|---|---|---|---|---|---|
| Base gpt-oss-20b | 0.253 | 0.250 | 0.391 | 0.395 | 0.322 | 0.202 |
| SFT only (step 550, RL start) | 0.326 | 0.314 | 0.414 | 0.451 | 0.376 | 0.329 |
| SFT + RL (Harness-1) | 0.891 | 0.551 | 0.710 | 0.841 | 0.748 | 0.713 |
The SFT and SFT+RL rows run in our harness. The base row repeats Table 2, where the untrained baselines run in the shared Context-1 harness. On the held-out benchmarks, SFT adds about five points over the base model, and RL, trained only on SEC, then nearly doubles recall and improves all four. Measured from the SFT checkpoint, RL gains about as much on these benchmarks as on the source families, even though neither training stage sees them, and the control above shows that this gain is far larger with explicit state. We use one harness configuration for all eight benchmarks and tuned nothing on the held-out ones (Appendix D). The table is Table 9 in Appendix D, and Sec. 3.2 cites it.
Q5. Cost. No. The per-episode cost counts only the policy's own input and output tokens, priced at gpt-oss-20b's open-weight rate, and every searcher in the cost table of Appendix N is counted the same way. The reranker and search backend serve all systems, so they are left out for all of them. The one extra LLM call our harness makes is verify, which costs a small fraction of a cent per episode (Appendix M.1.5). We use gpt-oss-20b's rate for the fine-tuned model because the rank-32 LoRA adapter adds almost no inference compute.
Q6. Verifier. We call gpt-4o-mini by model name at temperature 0; Appendix I.2 now lists the full settings. Reproducing the reported numbers does not require the API: each verify judgment is stored in the per-query logs of every evaluation run, so all runs can be re-scored offline. The only alternative we have tested is an NLI model, which finds about half as many supported claims (Table 14). We have not tested whether a small open-weight LLM can serve as the verifier, and Appendix M.1.5 now says so.
Thank you again for this review. It asked for the one experiment the paper was missing, and that experiment shows that explicit state makes RL much more effective and that our recipe does not explain the gain. We have also rewritten every claim you questioned so that it says no more than the evidence shows. If these changes address your concerns, we would be grateful if you would raise your score, and we are happy to discuss anything that remains.
Response to Reviewer aWYV (1/3)
Dear Reviewer aWYV,
Thank you for a careful and very concrete review. Your central question is whether the harness helps the policy learn, or only helps the trained policy at test time. We ran the training-time control you proposed to answer it, and we revised each claim you flagged. All changes are marked in blue in the revised PDF.
W1, W3 & Q1. Does the harness help learning, or does the gain come from the recipe?
We trained our full recipe in Context-1's harness, the thin harness the paper already uses for the frontier baselines. Tool outputs stay in the transcript, and there is no working memory, curated set, evidence graph, verification or nudges. When the transcript nears its budget, the policy is asked to prune its context or stop, and it ends by listing up to 30 documents. We did not use a plain transcript with automatic truncation, because deep-search episodes fill the context within a few searches, and silently dropping old turns would handicap the transcript run for reasons unrelated to learning. Everything else is shared with Harness-1: the base model, the GPT-5.4 teacher (run inside each harness on the same SFT queries, with the same recall filter), the SFT settings, the SEC RL data, the reward (computed on each model's final document set), the CISPO settings and budget (80 steps × 1,024 rollouts), and the retrieval backend, reranker and 30-document cap. Each model is evaluated in its own harness, as in Table 2.
Together with Context-1's own result, this separates recipe from harness, as W3 asks.
With the recipe fixed (rows 2 and 4), the stateful model is only a few points ahead after SFT but far ahead after RL, and the gap is widest on the held-out benchmarks that no training stage uses. The difference therefore comes from RL, where data, reward and budget are identical.
With the harness fixed (rows 2 and 5), our recipe ends below Context-1, whose recipe is larger in scale (new Table 21 in Appendix Q.1). So the recipe is not what puts Harness-1 ahead of Context-1. The harness effect is larger than Harness-1's entire lead, and our smaller recipe gives part of it back.
(continued below)