acceptodds
Under review as a conference paper at ICLR 2027

GrepSeek: Training Search Agents for Direct Corpus Interaction

Abstract

Large Language Model (LLM) search agents have shown strong promise on knowledge-intensive tasks through iterative reasoning and retrieval. Most existing systems rely on retrievers that return ranked documents from a pre-built index. We explore a complementary paradigm in which the agent treats the corpus as the search environment and finds evidence through executable shell commands. We introduce GrepSeek, an optimized direct corpus interaction (DCI) agent that learns to find, filter, and compose evidence over large text corpora. To stabilize reinforcement learning (RL) over large corpora, we train in two stages: first, we initialize the policy using verified, causally grounded search trajectories generated by an answer-aware Tutor and an answer-blind Planner; then, we refine the policy using Group Relative Policy Optimization (GRPO). To make DCI practical at scale, we introduce two semantics-preserving execution optimizations: Pruned Adaptive Command Execution, which reduces shell-based search latency by up to 77× on a 14 GB corpus with 21 million documents using a compact auxiliary structure, and Sharded-Parallel Corpus Search, which achieves up to 7.6× speedup without additional preprocessing; both preserve equivalence with sequential execution. Across eight open-domain QA benchmarks, GrepSeek achieves the strongest overall performance, with a statistically significant relative improvement of 5.7% over the best baseline. Our analysis shows how DCI-optimized agents conduct flexible and effective compositional search through direct corpus interaction.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Direct Corpus Interaction, Search Agents, Agentic Search, Information Retrieval

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_FgnpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GrepSeek, a search agent that interacts directly with text corpora using shell commands instead of conventional retrieval systems. The authors train the agent in two stages. They first construct verified search trajectories using an answer-aware Tutor and an answer-blind Planner, then apply SFT and GRPO to improve the agent's search behavior. To reduce the cost of searching large corpora, they introduce two execution optimizations, PACE and SPCS. Experiments on eight QA benchmarks show that GrepSeek outperforms several retrieval-based baselines, with particularly strong results on multi-hop questions.

Critical strengths
  1. The motivation is clear, and the proposed method directly addresses the problem. Allowing agents to search corpora through executable commands provides more flexibility than conventional document retrieval, particularly for tasks involving exact matching and multiple filtering steps.

  2. The training pipeline is well designed. Using verified trajectories for cold-start SFT before RL seems reasonable, and the ablation studies show that both training stages contribute to performance. The experiments also cover different model sizes and a good range of QA benchmarks.

  3. I appreciate that the paper considers the efficiency of direct corpus search. Both PACE and SPCS achieve substantial improvements in command execution speed, which is important if this approach is to be used on large corpora.

Critical weaknesses
  1. Although the proposed optimizations make individual search commands faster, the end-to-end inference latency is still higher than that of retrieval-based baselines. For example, GrepSeek with PACE takes 7.93 seconds per query, compared with 4.77 seconds for E5 and 6.07 seconds for Qwen3 embeddings. The additional time spent on LLM reasoning remains a bottleneck.

  2. The technical novelty of the execution optimizations seems limited. PACE builds on existing indexing techniques, while SPCS uses parallel execution to accelerate corpus search. Their integration into the agent framework is useful, but I would like to see a clearer discussion of how these methods differ from existing retrieval and database optimizations.

  3. Direct corpus search relies heavily on lexical matching, which may limit its applicability. The results on PopQA already show some of these limitations. More experiments on datasets with less lexical overlap or greater linguistic variation would help clarify when this approach is preferable to semantic retrieval.

Questions for authors

See Weaknesses

3 comments

We thank the reviewer for the positive assessment of our motivation, training pipeline and efficiency work.

W1: End-to-end latency is still higher than retrieval-based baselines

We agree that GrepSeek's end-to-end latency (7.93 s per query with PACE) is higher than that of the retrieval baselines (4.77 to 6.07 s), and, as the reviewer notes, the bottleneck is LLM generation rather than search. With PACE, tool time is 0.07 s per query, on par with BM25 (0.06 s), which we have now added to Figure 4, and faster than E5 (0.18 s) and Qwen3-Embedding (1.84 s). Generation takes 7.86 s against 4.23 to 4.84 s for the baselines, because GrepSeek reasons longer and takes more turns.

MethodTool time (s / query)End-to-end (s / query)Memory (GB)Offline preparationMacro-avg (Table 1)
Search-R1 + BM250.064.9016.117 min, 1 CPU thread0.4674
Search-R1 + E5-110M0.184.77703.2 A100-hours0.4704
Search-R1 + Qwen3-Emb-4B1.846.0722162.4 A100-hours0.4945
GrepSeek + SPCS0.818.6714~1 min, CPU0.5226
GrepSeek + PACE0.077.9316.821 min, CPU0.5226

Tool time and end-to-end latency are per query, as in Figure 4(a); memory includes the 14 GB corpus for every method.

  • The goal of PACE and SPCS is to make corpus search itself cheap, and they do. Without them, search alone takes 5.39 s per query (Figure 4(d)), which would put the total at about 13 s; PACE brings it to 7.93 s.
  • The remaining overhead is generation length. It is orthogonal to the retrieval interface and could be reduced, for example with a length penalty during RL. We leave this to future work.
  • The extra time buys accuracy and lower cost. GrepSeek is the most accurate method (0.5226 against at most 0.4945 macro-average). Compared with dense retrievers, it needs 4 to 16 times less memory and no GPU indexing.

W2: Technical novelty of the execution optimizations

We would like to clarify the role of PACE and SPCS. The main contribution of this work is the optimization of DCI models: training a compact agent to search the corpus directly, through verified cold-start data, SFT and GRPO. PACE and SPCS are not proposed as new indexing methods. We present them to show that DCI-style models can be used efficiently at scale, and we adopt them to make GrepSeek usable in real-world settings. They build on established techniques (posting lists, suffix arrays, Elias–Fano compression, sharding and k-way merging), adapted to the requirements of direct corpus interaction, which standard retrieval indexes and database optimizers do not meet:

  • Substring search, not word lookup. The inverted indexes behind retrievers such as BM25 map whole words to documents, so they can only answer word-level queries, and they return a ranked list rather than every exact match. Shell search is different: rg -F "16" must also match "2016" and "b16", and rg -F "(ON)CHOH" must match a chemical formula with punctuation inside it (Example 1). PACE supports this with a suffix array over the vocabulary. Each query term is matched against every vocabulary term that contains it as a substring, and the union of their posting lists gives the candidate passages.
  • Exactly the same output as the original command. PACE uses its index only to decide where to run a command, never what it returns. It runs the agent's original pipeline on the candidate passages, in corpus order, so the output is byte-identical to a full scan, including ordering, head -n truncation, counts and exit status. Inverted filters (e.g., -v) do not contribute to pruning, and commands that cannot be pruned safely fall back to a full scan. SPCS likewise parallelizes only pipelines whose output can be reassembled exactly from shards (concatenation, head -n, wc -l, and sort/uniq through a k-way merge). It runs anything that depends on global state (line numbers -n, counts -c, context flags) sequentially.
  • Free-form pipelines, no schema. Database optimizations such as predicate pushdown, or ordering conjunctive filters by selectivity (which PACE's cost-aware intersection resembles), require a schema and a declarative query language. PACE and SPCS instead accept the multi-stage shell pipelines the agent writes, so the agent's interface stays unchanged.

Together, PACE and SPCS make shell search 77 times faster (PACE) and 7.6 times faster (SPCS with 32 shards) on 21M passages without changing what the agent observes, so the same trained policy runs unchanged with or without them. We added this discussion to Appendix B.

W3: Reliance on lexical matching; datasets with less lexical overlap

We agree that DCI relies on lexical matching, and PopQA already exposes this. Following the reviewer's suggestion, we evaluated all methods on the full test sets of three additional datasets with less lexical overlap or more linguistic variation:

  • WebQuestions: 2,032 questions issued to Google, worded independently of Wikipedia.
  • EntityQuestions: 22,075 questions about long-tail entities.
  • PopQA-TP: all 14,267 PopQA test questions, with each question's relation reworded by a template paraphrase.

The results are in the new Figure 6 and discussed in Section 3.3.

MethodRetrieverWebQuestionsEntityQuestionsPopQA-TP
Direct—0.34720.26930.2151
RAGBM250.29730.47320.3076
E5-110M0.37990.42490.4088
Qwen3-Emb-4B0.39520.51230.4785
Search-R1BM250.40930.53110.4052
E5-110M0.42190.47620.4498
Qwen3-Emb-4B0.43640.52460.4853
GrepSeek—0.45600.58040.4668

F1, full test sets, Qwen3.5-9B backbone.

  • GrepSeek is best on WebQuestions and EntityQuestions. It leads the best Search-R1 agent by +0.020 F1 on WebQuestions (95% paired bootstrap interval [+0.007, +0.032]) and by +0.049 on EntityQuestions ([+0.045, +0.054]). This holds on WebQuestions even though a single dense retrieval there is clearly better than a single BM25 retrieval (+0.098 F1 for RAG with Qwen3-Embedding over BM25).
  • PopQA-TP is where semantic retrieval wins, as on the original PopQA. Search-R1 with Qwen3-Embedding leads GrepSeek by 0.019 F1 ([0.012, 0.026]), similar to its lead on the original PopQA in Table 1. GrepSeek still beats the E5 and BM25 agents.
  • Paraphrasing does not widen the gap. On the 13,745 PopQA-TP questions whose wording differs from the original, rewording costs GrepSeek 0.020 F1, comparable to the baselines (0.014 to 0.039).
Reviewer_YGJFOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies search agents that interact directly with a raw text corpus through executable shell commands rather than a conventional retrieval interface. The authors introduce GrepSeek, a compact agent trained in two stages: supervised fine-tuning on cold-start trajectories constructed by an answer-aware Tutor and an answer-blind Planner, followed by reinforcement learning with GRPO. To make shell-based search practical at scale, they propose two semantics-preserving execution optimizations: Pruned Adaptive Command Execution, which restricts compatible commands to candidate documents, and Sharded-Parallel Corpus Search, which parallelizes execution across corpus shards. Across eight open-domain QA benchmarks, GrepSeek achieves the highest macro-average score among the evaluated methods.

Critical strengths
  • Practical execution engine. The semantics-preserving optimizations make shell-based search over large corpora fast enough for training and inference, with modest memory and preprocessing costs.
  • Careful cold-start data construction. The Tutor–Planner pipeline produces verified, causally grounded trajectories with explicit safeguards against answer leakage, which makes RL training of a compact DCI agent stable.
Critical weaknesses
  1. The claimed benefits of shell-based corpus interaction are not sufficiently validated. The paper motivates shell operations through finer-grained text access and explicit control over matching, filtering, and evidence composition. However, the reported behavior mainly involves matching and filtering presegmented passages, stored one per line, using rg and head, without demonstrating benefits from accessing text beyond predefined passage boundaries. The qualitative examples primarily illustrate the value of exact lexical matching relative to dense retrieval, leaving unclear what additional advantage comes from the shell-based interface itself. Harness-1 [1], for example, already combines indexed hybrid retrieval, reranking, corpus-wide regex search, and full-text access, providing a relevant alternative for obtaining precise evidence. The paper needs to establish which benefits its shell-based corpus operations actually deliver, and whether these translate into better accuracy or lower cost than capable lexical or hybrid retrieval interfaces.
  2. The reported gains are not separated from the benefits of teacher-generated cold-start supervision. GrepSeek receives 10K trajectories constructed by an answer-aware 27B Tutor and an answer-blind Planner before GRPO, but no retriever-based baseline is documented as receiving comparable teacher supervision. A critical missing comparison is a Search-R1-style agent first fine-tuned on high-quality teacher-generated retrieval trajectories and then optimized with GRPO, using the same backbone and comparable supervision and training budgets. This would test whether stronger initialization and more stable RL training can close the reported performance gap without adopting DCI.

[1] Harness-1: Reinforcement Learning for Search Agents with State-Externalizing Harnesses.

Questions for authors

NA

6 comments

We thank the reviewer for the thoughtful and detailed review.

W1: Benefits of shell-based corpus operations beyond lexical matching; comparison with Harness-1

We agree that GrepSeek operates on passage-level lines and does not exploit access beyond passage boundaries. What the shell interface delivers is control over what must match and how much to read. GrepSeek uses literal (fixed-string) matching in nearly all searches, chains filters in 70% of its commands (e.g., rg -F "The Joggers" corpus.jsonl | rg -i -F "singer"), and sets the amount of retrieved text with head -n (Section 3.3). A ranked lexical retriever cannot express these constraints: BM25 returns the top-k passages for a bag-of-words query.

Accuracy: better than a lexical retrieval interface at equal training.

MethodTable 1 avgWebQuestionsEntityQuestionsPopQA-TP
Search-R1 + BM250.46740.40930.53110.4052
GrepSeek0.52260.45600.58040.4668

F1; the Table 1 average also includes GPT-4.1-judged accuracy on BrowseComp-Plus. Giving Search-R1 + BM25 GrepSeek's two-stage training (SFT on 10K teacher trajectories, then GRPO) does not close the gap (0.4630).

The interface helps even without training. On BrowseComp-Plus, the untrained Qwen3.5-9B with our shell tool reaches 0.4157 judged accuracy. That is higher than every trained Search-R1 agent (at most 0.2711), under the same 16-turn budget.

Cost. GrepSeek costs about the same as a lexical index: PACE needs 16.8 GB and 21 minutes of CPU preprocessing, against 16.1 GB and 17 minutes for BM25. Dense retrieval costs far more (70 to 221 GB and 3.2 to 62.4 A100-hours, Figure 4), and a hybrid interface would need that dense index too.

Harness-1. Harness-1 and GrepSeek answer different questions and are complementary:

  • Search interface. Harness-1's main search tools run over indexed corpora: hybrid BM25 + dense retrieval, fused with reciprocal rank fusion and re-ranked by an 8B reranker, alongside a regex tool and full-document reads. GrepSeek has no index, embedding model or reranker: every search is a shell pipeline that the agent writes over the raw corpus. This is why GrepSeek avoids the dense indexing cost shown in Figure 4 (3.2 to 62.4 A100-hours and 70 to 221 GB).
  • Research question. Harness-1 moves search bookkeeping (candidate pools, evidence links, verification records) out of the policy and into the harness. We ask whether a compact model can learn direct corpus interaction itself, and how to make such search fast at scale. PACE and SPCS, which accelerate shell search over 21M passages while keeping its outputs identical, have no counterpart in Harness-1.
  • Training. Harness-1 fine-tunes a 20B mixture-of-experts model on trajectories from a proprietary model (GPT-5.4). GrepSeek trains a 9B (and a 4B) model on trajectories from an open 27B teacher, built backward from verified evidence and screened for answer leakage.
  • Task and evaluation. Harness-1 is a retrieval subagent: it returns a curated evidence set, evaluated by recall, and a separate frontier model writes the answer. GrepSeek searches and answers end to end, evaluated by answer F1 and judged accuracy. Their numbers are therefore not directly comparable, even on shared benchmarks such as BrowseComp-Plus and HotpotQA.

The two approaches can be combined. GrepSeek's shell tool could serve as a search tool inside a harness like Harness-1's, and moving search state into a harness could help GrepSeek on long-horizon tasks such as BrowseComp-Plus, which we identify as a limitation. Finally, Harness-1 appeared on arXiv after the first public version of this work and has not been published at a peer-reviewed venue, so under the ICLR 2027 reviewer guidelines it is not a required comparison.

W2: Gains not separated from teacher-generated cold-start supervision

We agree that this comparison was missing, and we ran it as the reviewer described. Search-R1 is first fine-tuned on 10K teacher-generated retrieval trajectories from the same Qwen3.5-27B teacher used for GrepSeek (the Rejection Sampling checkpoint in Table 1). It is then optimized with GRPO under the same budget as GrepSeek (200 steps, group size 5, batch size 256), on the same Qwen3.5-9B backbone. The results are now in Table 1 and Table 6 as "Search-R1 (SFT+RL)".

MethodRetrieverNQ*TriviaQAPopQAHotpotQA*2WikiMuSiQueBamboogleBrowseComp-PlusAvg
SFT only (Rejection Sampling)BM250.38250.72280.38270.55300.39720.23960.58190.26510.4406
E5-110M0.42380.72560.42980.53530.36780.24380.64410.14100.4389
Qwen3-Emb-4B0.42940.72580.46300.54420.42550.26970.65690.17590.4613
RL only (Search-R1)BM250.45920.77330.41940.56590.40280.25100.60130.26630.4674
E5-110M0.50690.77340.47470.54240.37270.27200.67630.14460.4704
Qwen3-Emb-4B0.50670.76930.51010.55910.42990.28780.69890.19400.4945
SFT + RL (new)BM250.43080.74780.40170.56780.41340.25280.61890.27110.4630
E5-110M0.50380.75390.46260.55800.39890.27870.65170.12770.4669
Qwen3-Emb-4B0.48980.75070.50100.57100.45020.29000.65310.19040.4870
GrepSeek—0.52230.76730.48610.62310.51780.30060.62120.34220.5226

F1 on the seven QA datasets; GPT-4.1-judged accuracy on BrowseComp-Plus. * = used in training. Exact match shows the same pattern: macro EM is 0.3882 to 0.4042 for Search-R1 (SFT+RL) against 0.4347 for GrepSeek (Table 6).

  • Stronger initialization does not close the gap without DCI. Search-R1 (SFT+RL) stays within 0.008 of RL-only Search-R1 for every retriever and 0.036 to 0.060 below GrepSeek in macro-average.
  • The SFT stage matters for DCI specifically. It is essential for GrepSeek (0.3235 without it, 0.5226 with it), likely because writing effective shell commands is much harder to learn from scratch than writing natural-language queries.

Thank you for the response. My remaining question is whether GrepSeek and the retrieval baselines operate under comparable reading budgets.

Your response lists control over “how much to read” as one of the benefits of the shell interface. In the paper, however, the retrieval baselines receive a fixed top-3 per query, whereas GrepSeek's head -n grows from 4.9 to 10.4 lines during RL (Table 9), with up to 2,048 tokens per tool output. Table 9 also shows that the filter-chain rate and pipes per command stay flat, so the main structural change RL makes is reading more.

I do not think this control is specific to the shell. The original Search-R1 implementation (Jin et al., 2025) issues a single query per search call with top-3 by default. However, later Search-R1-style recipes in verl and other RL frameworks accept a list of queries per call, and in my experience, policies trained in such setups learn to issue multiple queries per call, i.e., to adjust how much they retrieve. A single-query, top-3 interface is therefore a restrictive default rather than an inherent limitation of retrieval agents. I believe this emergent multi-query behavior serves a similar purpose to GrepSeek’s adaptive n in head -n, and it could be extended to a fully dynamic budget by letting the agent set top-k through a modified search API.

Could you clarify whether your Search-R1 baselines issue a single query per call? Retraining Search-R1 with an adaptive budget may not be feasible within the discussion period, but I would like to see the comparison under the same reading budget, for example by evaluating the existing Search-R1 checkpoints with top-k matched to the average number of passages GrepSeek actually reads per command.

Reviewer_ZSXdOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents GrepSeek, a search agent that finds evidence by executing shell commands directly over a text corpus. It trains a compact language model using verified search trajectories generated by an answer-aware Tutor and an answer-blind Planner, followed by supervised fine-tuning and GRPO. Two execution strategies accelerate corpus search while preserving command outputs. Across eight question-answering benchmarks, GrepSeek achieves the highest overall performance among the evaluated baselines, with a 5.7% relative improvement over the strongest one.

Critical strengths
  1. GrepSeek introduces a method for generating verified, causally grounded search trajectories, which train the model to find evidence and produce answers.
  2. GrepSeek uses less memory than the evaluated dense-retrieval baselines. Its Pruned Adaptive Command Execution and Sharded-Parallel Corpus Search strategies also reduce search latency.
Critical weaknesses
  1. Limited differentiation from prior DCI work. GrepSeek adopts the same core direct corpus interaction paradigm studied in [1]. Its distinct contributions appear to lie in training a small model and accelerating command execution.
  2. Training and retrieval interface are confounded in the main comparison. The ablations show that both cold-start SFT and GRPO improve GrepSeek. However, the strongest retrieval-based baseline is not trained with the same two-stage pipeline.

[1] Li, Zhuofeng, et al. "Beyond semantic similarity: Rethinking retrieval for agentic search via direct corpus interaction." Neurips 2026

Questions for authors
  1. Could the paper evaluate Qwen3.5-9B and Qwen3.5-27B in thinking mode within the same DCI environment, before any task-specific training? This would establish how much direct corpus interaction capability the models already possess and clarify the contribution beyond [1].

  2. Could Search-R1 be trained with a comparable cold-start SFT stage followed by RL? This would help determine whether GrepSeek’s gains stem primarily from its training pipeline or from the DCI interface.

  3. The paper uses a Tutor-based judge to reject SFT trajectories that reveal information before it becomes observable. How reliable is this judgment? A manual audit of accepted and rejected trajectories, with representative failure cases, would strengthen confidence in the cold-start data.

[1] Li, Zhuofeng, et al. "Beyond semantic similarity: Rethinking retrieval for agentic search via direct corpus interaction." Neurips 2026

4 comments

We thank the reviewer for the constructive review and for the concrete suggestions, which we have followed.

W1: Differentiation from prior DCI work [1]

We agree that GrepSeek shares the direct corpus interaction (DCI) interface with Li et al. [1], which we discuss as concurrent work. Our contributions are complementary.

  • Learning DCI with a compact open model. [1] prompts large proprietary models such as Claude at inference time, which can take an hour or more per query. We show that DCI can be learned by a compact open 9B model, through verified cold-start trajectories followed by GRPO. The experiment the reviewer suggested (Q1 below) quantifies this. Without our training, the same 9B model reaches a 0.3180 macro-average, and even an untrained model three times larger (27B) reaches only 0.3840, against 0.5226 for GrepSeek.
  • Making DCI practical at scale. PACE and SPCS are two execution optimizations that preserve byte-exact command outputs. On a 14 GB corpus of 21M passages, they reduce search latency from 5.39 s to 0.07 s (PACE) and 0.71 s (SPCS with 32 shards), as shown in Figure 4(d).

W2 / Q2: Training and retrieval interface are confounded; Search-R1 with cold-start SFT followed by RL

Thank you for this suggestion. We trained Search-R1 with GrepSeek's two-stage recipe. It starts from supervised fine-tuning on 10K trajectories generated by the same Qwen3.5-27B teacher (the Rejection Sampling checkpoint in Table 1). It then runs 200 steps of GRPO with group size 5 and batch size 256, on the same Qwen3.5-9B backbone. The results are now in Table 1 and Table 6 as "Search-R1 (SFT+RL)".

MethodRetrieverNQ*TriviaQAPopQAHotpotQA*2WikiMuSiQueBamboogleBrowseComp-PlusAvg
SFT only (Rejection Sampling)BM250.38250.72280.38270.55300.39720.23960.58190.26510.4406
E5-110M0.42380.72560.42980.53530.36780.24380.64410.14100.4389
Qwen3-Emb-4B0.42940.72580.46300.54420.42550.26970.65690.17590.4613
RL only (Search-R1)BM250.45920.77330.41940.56590.40280.25100.60130.26630.4674
E5-110M0.50690.77340.47470.54240.37270.27200.67630.14460.4704
Qwen3-Emb-4B0.50670.76930.51010.55910.42990.28780.69890.19400.4945
SFT + RL (new)BM250.43080.74780.40170.56780.41340.25280.61890.27110.4630
E5-110M0.50380.75390.46260.55800.39890.27870.65170.12770.4669
Qwen3-Emb-4B0.48980.75070.50100.57100.45020.29000.65310.19040.4870
GrepSeek—0.52230.76730.48610.62310.51780.30060.62120.34220.5226

F1 on the seven QA datasets; GPT-4.1-judged accuracy on BrowseComp-Plus. * = used in training. Exact match shows the same pattern: macro EM is 0.3882 to 0.4042 for Search-R1 (SFT+RL) against 0.4347 for GrepSeek (Table 6).

  • The same recipe does not close the gap. Search-R1 (SFT+RL) stays within 0.008 of RL-only Search-R1 for every retriever and 0.036 to 0.060 below GrepSeek in macro-average.
  • SFT matters for DCI, not for retrieval agents. The SFT stage is essential for GrepSeek (0.3235 without it, 0.5226 with it), likely because writing effective shell commands is much harder to learn from scratch than writing natural-language queries.
  • Both ingredients are needed. Together with the untrained baselines (Q1 below), this shows that GrepSeek's gains require both direct corpus interaction and our training. The shell interface without training falls short, and the training recipe without the shell interface falls short.

Q1: Untrained Qwen3.5-9B and Qwen3.5-27B in the same DCI environment

Thank you for this suggestion. We ran both models in thinking mode, without any task-specific training, with exactly the same shell tool, system prompt, and turn, context and tool-output limits as GrepSeek. The results have been added to Figure 5(b) and Table 7 and are discussed in Appendix D.

MethodNQ*TriviaQAPopQAHotpotQA*2WikiMuSiQueBamboogleBrowseComp-PlusAvg
GrepSeek (Qwen3.5-9B)0.52230.76730.48610.62310.51780.30060.62120.34220.5226
− w/o GRPO0.38800.63900.39030.47380.29560.20690.42310.22530.3803
− w/o SFT0.28960.54510.31630.37050.18380.12910.35440.39880.3235
Untrained Qwen3.5-9B0.27510.49940.33470.34090.21410.12810.33600.41570.3180
Untrained Qwen3.5-27B0.33180.59780.36530.40540.25070.16490.39600.56020.3840

F1 on the seven QA datasets; GPT-4.1-judged accuracy on BrowseComp-Plus. * = used in training.

Exact match (EM) shows the same pattern: macro-average EM is 0.4347 for GrepSeek, 0.2737 for the untrained 9B and 0.3240 for the untrained 27B (Table 8).

  • The base models have the mechanics of DCI but not an effective search strategy. Their tool calls are almost always well formed (94% to 99.7%), but within the six-turn budget they often run out of turns before answering. They trail GrepSeek on every QA dataset, by 0.15–0.30 F1 (9B) and 0.12–0.27 F1 (27B). The trained 9B model outperforms the untrained 27B, a model three times its size, by 0.14 macro-average. Counting malformed tool calls, rejected commands and empty answers as wrong changes the untrained scores by at most 0.06 and does not change these conclusions.
  • BrowseComp-Plus is an exception, and we report it transparently. Both untrained models outperform GrepSeek there (0.4157 and 0.5602 against 0.3422), as does the RL-only variant. We attribute this to a mismatch in horizon. GrepSeek is trained on short-answer QA with a six-turn budget, which favors precise, early-stopping search, whereas BrowseComp-Plus requires long-horizon evidence gathering over up to 16 turns. We now discuss this in Appendix D and leave training on long-horizon trajectories to future work.

Q3: Reliability of the leakage judge (manual audit)

Following the reviewer's suggestion, we manually audited 100 trajectories: 50 that the judge accepted and 50 that it rejected (Appendix D).

Judge decisionAuditedHumans agree with the judge
Accepted5039 (78%)
Rejected5036 (72%)
Overall10075 (75%)

The judge's 75% agreement with humans is comparable to the 73% agreement between the annotators themselves.

Representative failure cases:

  • False rejection: the judge rejected "Sacramento" for "where is the capital city located in California" because the answer is a name rather than a location. Errors of this kind only cost training data.
  • Missed leak: a trajectory searched for "Gadsden flag" when asked who created the "Don't Tread on Me" flag, before any output mentioned Gadsden. Almost all missed leaks (10 of 11) rely on widely known facts like this one.
Reviewer_ZSXdOpenReview ↗

Official Comment by Reviewer ZSXd

Thank you for your comprehensive experiments. I will maintain my original score and recommend acceptance.

Thank you for your careful consideration of our rebuttal and your recommendation for acceptance. If the additional experiments have resolved your main concerns, we would appreciate your reconsidering whether the updated evidence warrants a stronger score assessment. If any concerns still limit your confidence, we would welcome the opportunity to clarify them.

Reviewer_22L8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GrepSeek, a search agent based on Direct Corpus Interaction (DCI), which interacts with raw text corpora through executable shell commands rather than conventional document retrieval systems. The training framework combines an answer-aware Tutor and an answer-blind Planner to construct search trajectories, followed by SFT and GRPO optimization.

The main contributions lie in two execution optimization methods: PACE (Pruned Adaptive Command Execution), which uses auxiliary inverted structures to prune candidate documents, and SPCS (Sharded-Parallel Corpus Search), which parallelizes compatible shell pipelines while preserving execution semantics.

Experiments across eight open-domain QA benchmarks show that GrepSeek outperforms the baselines such as Search-R1. Also, PACE and SPCS provides up to 77x and 7.6× speedup compared to directly interacting with the corpora without further processing.

Critical strengths
  • Practical execution optimizations for direct corpus interaction. The paper proposes PACE and SPCS to substantially improve the execution efficiency of shell-based corpus search. Both methods are technically interesting, and the preservation of search semantics makes the optimizations particularly relevant to practical DCI systems.
  • A relatively comprehensive training and evaluation framework. The combination of Tutor–Planner-based trajectory construction, SFT, and GRPO provides a systematic approach to training DCI agents. The experiments across multiple QA benchmarks and the ablation studies offer useful evidence of the effectiveness of the overall framework.
Critical weaknesses
  • Insufficient comparison with BM25-based retrieval. The paper compares GrepSeek primarily with dense retrieval baselines, but lacks a direct comparison with BM25, a fundamental and highly relevant baseline for lexical search. Since GrepSeek relies heavily on lexical matching and shell-based filtering, it is important to understand how it compares with BM25 in both retrieval effectiveness and latency. In particular, the substantial performance gain over the reported baselines is not sufficiently explained.
  • Unclear attribution of the performance improvements. It is difficult to determine whether the observed gains mainly come from the DCI search paradigm or from the proposed data construction and training strategy. The paper does not sufficiently isolate these factors, making it unclear whether similar training data and trajectory construction methods could also improve conventional retrieval-based agents.
  • Limited evidence of end-to-end efficiency gains. Although PACE and SPCS significantly accelerate corpus search execution, GrepSeek's reported end-to-end latency remains higher than those of the E5 and Qwen3-4B retrieval baselines. This raises questions about whether the execution-level optimizations translate into practical end-to-end efficiency improvements.
Questions for authors
  • Comparison with BM25. Could the authors provide a direct comparison with BM25 in terms of both retrieval effectiveness and latency under the same corpus and evaluation settings? Given the lexical nature of DCI, it would be particularly interesting to understand why GrepSeek achieves substantially better QA performance than conventional retrieval baselines, and what types of queries or evidence retrieval scenarios account for these gains.
  • Attribution of performance improvements. How much of the observed improvement comes from the DCI search mechanism itself, as opposed to the proposed data construction and training strategy? For example, if the same training data and trajectory construction pipeline were used to train agents based on BM25 or E5 retrieval, how would their performance compare with GrepSeek? Such experiments would help isolate the contribution of the DCI paradigm from that of the training methodology.
3 comments

We thank the reviewer for the detailed review and for recognizing the practicality of PACE and SPCS and the comprehensiveness of our training and evaluation framework.

W1 / Q1: Comparison with BM25 (effectiveness and latency)

Effectiveness. BM25 is already a baseline throughout the paper. Table 1 (F1) and Table 6 (EM) evaluate every method (RAG, IRCoT, Search-O1, Rejection Sampling and Search-R1) with BM25, E5 and Qwen3-Embedding, all on the same 21M-passage corpus and with the same backbone. The strongest BM25 agent, Search-R1 with BM25, scores below GrepSeek on 7 of the 8 datasets:

MethodNQ*TriviaQAPopQAHotpotQA*2WikiMuSiQueBamboogleBrowseComp-PlusAvg
Search-R1 + BM250.45920.77330.41940.56590.40280.25100.60130.26630.4674
GrepSeek0.52230.76730.48610.62310.51780.30060.62120.34220.5226

F1 on the seven QA datasets; GPT-4.1-judged accuracy on BrowseComp-Plus. * = used in training.

Latency and cost. Following the reviewer's suggestion, we added BM25 to the efficiency comparison in Figure 4, measured with the same protocol as the other methods (50 questions from each FlashRAG dataset). Tool time and end-to-end latency are per query, as in Figure 4(a); memory includes the 14 GB corpus for every method.

MethodTool time (s / query)End-to-end (s / query)Memory (GB)Offline preparationMacro-avg (Table 1)
Search-R1 + BM250.064.9016.117 min, 1 CPU thread0.4674
Search-R1 + E5-110M0.184.77703.2 A100-hours0.4704
Search-R1 + Qwen3-Emb-4B1.846.0722162.4 A100-hours0.4945
GrepSeek + SPCS0.818.6714~1 min, CPU0.5226
GrepSeek + PACE0.077.9316.821 min, CPU0.5226

BM25 is the cheapest retriever. Its tool time is on par with PACE, and its memory and preparation cost are similar. GrepSeek's advantage over BM25 is therefore accuracy rather than cost. Against the dense retrievers, GrepSeek also needs 4 to 16 times less memory and no GPU indexing.

Why GrepSeek outperforms BM25 although both rely on lexical matching. BM25 returns the top-3 passages for a bag-of-words query, so the agent cannot require an exact phrase, require several terms to appear in the same passage, or decide how much text to read. Shell commands give the agent exactly these controls. GrepSeek uses literal (fixed-string) matching in nearly all searches, chains filters in 70% of its commands (e.g., rg -F "The Joggers" corpus.jsonl | rg -i -F "singer"), and sets the amount of retrieved text with head -n (Section 3.3).

The same holds on three additional datasets with less lexical overlap that we added in the revision (Figure 6, Section 3.3). GrepSeek outperforms Search-R1 with BM25 by +0.047 F1 on WebQuestions, +0.049 on EntityQuestions and +0.062 on PopQA-TP:

MethodWebQuestionsEntityQuestionsPopQA-TP
Search-R1 + BM250.40930.53110.4052
GrepSeek0.45600.58040.4668

F1, full test sets.

W2 / Q2: How much of the gain comes from DCI, and how much from the data construction and training pipeline?

We address this question with two complementary experiments.

(1) DCI without training. We ran the untrained Qwen3.5-9B and Qwen3.5-27B with exactly the same shell tool, system prompt, and turn, context and tool-output limits as GrepSeek. They reach macro-averages of 0.3180 and 0.3840, against 0.5226 for GrepSeek, and trail it on every QA dataset (Figure 5(b), Table 7, Appendix D). The shell interface alone therefore does not produce GrepSeek's results: training is necessary for the agent to use direct corpus interaction effectively.

MethodNQ*TriviaQAPopQAHotpotQA*2WikiMuSiQueBamboogleBrowseComp-PlusAvg
GrepSeek (Qwen3.5-9B)0.52230.76730.48610.62310.51780.30060.62120.34220.5226
− w/o GRPO0.38800.63900.39030.47380.29560.20690.42310.22530.3803
− w/o SFT0.28960.54510.31630.37050.18380.12910.35440.39880.3235
Untrained Qwen3.5-9B0.27510.49940.33470.34090.21410.12810.33600.41570.3180
Untrained Qwen3.5-27B0.33180.59780.36530.40540.25070.16490.39600.56020.3840

F1 on the seven QA datasets; GPT-4.1-judged accuracy on BrowseComp-Plus. * = used in training.

Exact match (EM) shows the same pattern: macro-average EM is 0.4347 for GrepSeek, 0.2737 for the untrained 9B and 0.3240 for the untrained 27B (Table 8).

On BrowseComp-Plus, the untrained models score higher than GrepSeek. GrepSeek is trained on short-answer QA with a six-turn budget, which favors precise, early-stopping search, whereas BrowseComp-Plus requires long-horizon evidence gathering over up to 16 turns. We discuss this in Appendix D.

(2) Retrieval agents trained with the same recipe. Our cold-start trajectories consist of shell commands and their outputs, so they cannot be used directly to train an agent whose actions are retrieval queries. The closest equivalent is to give the retrieval agent the same teacher, the same amount of supervision and the same training recipe, so we trained Search-R1 with GrepSeek's two-stage recipe. It starts from supervised fine-tuning on 10K trajectories that the same Qwen3.5-27B teacher generated with the retriever (the Rejection Sampling checkpoint in Table 1). It then runs 200 steps of GRPO with group size 5 and batch size 256, on the same Qwen3.5-9B backbone. The results are now in Table 1 and Table 6 as "Search-R1 (SFT+RL)".

MethodRetrieverNQ*TriviaQAPopQAHotpotQA*2WikiMuSiQueBamboogleBrowseComp-PlusAvg
SFT only (Rejection Sampling)BM250.38250.72280.38270.55300.39720.23960.58190.26510.4406
E5-110M0.42380.72560.42980.53530.36780.24380.64410.14100.4389
Qwen3-Emb-4B0.42940.72580.46300.54420.42550.26970.65690.17590.4613
RL only (Search-R1)BM250.45920.77330.41940.56590.40280.25100.60130.26630.4674
E5-110M0.50690.77340.47470.54240.37270.27200.67630.14460.4704
Qwen3-Emb-4B0.50670.76930.51010.55910.42990.28780.69890.19400.4945
SFT + RL (new)BM250.43080.74780.40170.56780.41340.25280.61890.27110.4630
E5-110M0.50380.75390.46260.55800.39890.27870.65170.12770.4669
Qwen3-Emb-4B0.48980.75070.50100.57100.45020.29000.65310.19040.4870
GrepSeek—0.52230.76730.48610.62310.51780.30060.62120.34220.5226

F1 on the seven QA datasets; GPT-4.1-judged accuracy on BrowseComp-Plus. * = used in training. Exact match shows the same pattern: macro EM is 0.3882 to 0.4042 for Search-R1 (SFT+RL) against 0.4347 for GrepSeek (Table 6).

The same recipe does not close the gap. Search-R1 (SFT+RL) stays within 0.008 of RL-only Search-R1 for every retriever and 0.036 to 0.060 below GrepSeek in macro-average. In contrast, the SFT stage is essential for GrepSeek (0.3235 without it, 0.5226 with it), likely because writing effective shell commands is much harder to learn from scratch than writing natural-language queries.

Together, the two experiments show that GrepSeek's gains require both direct corpus interaction and our training. The shell interface without training falls short, and the training recipe without the shell interface falls short.

W3: End-to-end efficiency

We agree that GrepSeek's end-to-end latency (7.93 s per query with PACE) is higher than that of the retrieval baselines (4.77 to 6.07 s). The gap does not come from search. With PACE, tool time is 0.07 s per query, on par with BM25 (0.06 s) and faster than E5 (0.18 s) and Qwen3-Embedding (1.84 s). It comes from LLM generation (7.86 s against 4.23 to 4.84 s), because GrepSeek reasons longer and takes more turns.

MethodTool time (s / query)End-to-end (s / query)Memory (GB)Offline preparationMacro-avg (Table 1)
Search-R1 + BM250.064.9016.117 min, 1 CPU thread0.4674
Search-R1 + E5-110M0.184.77703.2 A100-hours0.4704
Search-R1 + Qwen3-Emb-4B1.846.0722162.4 A100-hours0.4945
GrepSeek + SPCS0.818.6714~1 min, CPU0.5226
GrepSeek + PACE0.077.9316.821 min, CPU0.5226

Tool time and end-to-end latency are per query, as in Figure 4(a); memory includes the 14 GB corpus for every method.

  • The execution optimizations do translate into end-to-end gains for DCI. Without them, search alone takes 5.39 s per query (Figure 4(d)), which would put the total at about 13 s; PACE brings it to 7.93 s.
  • The remaining overhead is generation length. It is orthogonal to the retrieval interface and could be reduced, for example with a length penalty during RL. We leave this to future work.
  • The extra time buys accuracy and lower cost. GrepSeek is the most accurate method (0.5226 against at most 0.4945 macro-average). Compared with dense retrievers, it needs 4 to 16 times less memory and no GPU indexing. For reference, prompting-based DCI with proprietary models (Li et al., 2026) can take an hour or more per query.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response (to all reviewers and AC)

We thank all reviewers for their careful and constructive feedback. We have uploaded a revised paper. The main changes:

  • Untrained models in the same DCI environment. Figure 5(b), Table 7 and Appendix D now report the untrained Qwen3.5-9B and Qwen3.5-27B, run with the same shell tool, prompt and limits as GrepSeek. This addresses ZSXd Q1 and part of 22L8 W2.
  • BM25 in the efficiency comparison. Figure 4 and Section 3.2 now include BM25. This addresses 22L8 W1/W3 and Fgnp W1.
  • Search-R1 with GrepSeek's two-stage training. Tables 1 and 6 now include Search-R1 fine-tuned on 10K trajectories from the same 27B teacher and then trained with GRPO. It does not close the gap to GrepSeek. This addresses ZSXd W2/Q2, YGJF W2 and 22L8 W2/Q2.
  • Three datasets with less lexical overlap. A new Figure 6 and paragraph in Section 3.3 add WebQuestions, EntityQuestions and PopQA-TP. GrepSeek is best on the first two and trails the strongest dense agent on PopQA-TP. This addresses Fgnp W3 and 22L8 W1.
  • Human audit of the leakage judge. Appendix D reports a manual audit of 100 trajectories: the judge agrees with the human labels on 75%, comparable to the 73% agreement between annotators. This addresses ZSXd Q3.
  • PACE and SPCS compared with existing optimizations. Appendix B now explains how they differ from word-level retrieval indexes and database optimizers. This addresses Fgnp W2.
AuthorsOpenReview ↗

Summary of the discussion for the AC and reviewers

We thank the AC and the reviewers for their time and constructive feedback. The reviewers agreed on the paper's strengths: a practical execution engine that preserves shell semantics (all four reviewers), carefully verified cold-start data (ZSXd, YGJF, Fgnp), and a comprehensive training and evaluation framework (22L8, Fgnp). We ran every experiment the reviewers asked for. All new results are in the revised PDF, except the matched-budget comparison, which we posted in the discussion on 17 Nov.

Main concerns and what we added

  1. Interface vs. training (22L8, ZSXd, YGJF). Search-R1 trained with GrepSeek's full recipe (SFT on 10K trajectories from the same 27B teacher, then the same GRPO) reaches a 0.4630 to 0.4870 macro-average, against 0.5226 for GrepSeek. The untrained Qwen3.5-9B and 27B with the same shell tool reach 0.3180 and 0.3840. The gains therefore need both the shell interface and our training.
  2. BM25 (22L8, YGJF). GrepSeek outperforms the strongest BM25 agent (0.5226 vs. 0.4674) at about the same cost: 0.07 vs. 0.06 s of tool time per query and 16.8 vs. 16.1 GB (Figure 4). The dense retrievers need 70 to 221 GB and 3.2 to 62.4 A100-hours of indexing.
  3. Reliance on lexical matching (Fgnp). On three new datasets with less lexical overlap (Figure 6), GrepSeek is best on WebQuestions and EntityQuestions and trails only the Qwen3-Embedding methods on PopQA-TP.
  4. End-to-end latency (22L8, Fgnp). PACE reduces search from 5.39 s to 0.07 s per query with byte-identical outputs. The remaining end-to-end gap comes from longer LLM generation.
  5. Reliability of the leakage judge (ZSXd). In a human audit of 100 trajectories, the judge agrees with the human labels on 75%, comparable to the 73% agreement between annotators (Appendix D).
  6. Positioning (ZSXd, YGJF, Fgnp). We clarified our contributions relative to prompting-based DCI and Harness-1, and how PACE and SPCS differ from retrieval indexes and database optimizers (Appendix B).

Follow-up from Reviewer YGJF: GrepSeek still wins under a matched reading budget

Reviewer YGJF asked whether GrepSeek and the retrieval baselines read comparable amounts of text, since GrepSeek's head -n grows to 10.4 lines during RL while Search-R1 receives a fixed top-3 per query. Our Search-R1 baselines follow the original implementation: one query per call, top-3 passages. The head -n value is only an upper bound. Many commands match fewer passages than requested, so GrepSeek actually sees 5.1 passages per command on the QA sets and 2.9 on BrowseComp-Plus.

As the reviewer suggested, we re-evaluated our strongest Search-R1 checkpoint (Qwen3-Embedding-4B) with top-6, at least as many passages per call as GrepSeek reads. We used five datasets that cover in-distribution (NQ, HotpotQA), held-out multi-hop (2Wiki), long-tail entity (PopQA) and long-horizon (BrowseComp-Plus) settings. Even with this larger budget, Search-R1 trails GrepSeek on every dataset except PopQA, where dense retrieval is known to have an advantage (as in Table 1). The gaps are largest on multi-hop and long-horizon search: +0.045 on HotpotQA, +0.071 on 2Wiki and +0.134 on BrowseComp-Plus. The reading budget therefore does not explain GrepSeek's advantage. We agree with the reviewer that adapting how much to read is not unique to the shell. Multi-query calls is orthogonal to the search interface and could benefit both approaches. We will report all datasets and retrievers in the camera-ready version.

Discussion status

Reviewer ZSXd thanked us for the comprehensive experiments and recommends acceptance. We answered Reviewer YGJF's follow-up with the matched-budget experiment above. We have not yet heard from Reviewers 22L8 and Fgnp. As the discussion closes today, we would be grateful if Reviewers 22L8, Fgnp and YGJF could let us know whether our responses address their concerns and, if so, consider updating their assessments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.