- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes GrepSeek, a search agent that interacts directly with text corpora using shell commands instead of conventional retrieval systems. The authors train the agent in two stages. They first construct verified search trajectories using an answer-aware Tutor and an answer-blind Planner, then apply SFT and GRPO to improve the agent's search behavior. To reduce the cost of searching large corpora, they introduce two execution optimizations, PACE and SPCS. Experiments on eight QA benchmarks show that GrepSeek outperforms several retrieval-based baselines, with particularly strong results on multi-hop questions.
Critical strengths
-
The motivation is clear, and the proposed method directly addresses the problem. Allowing agents to search corpora through executable commands provides more flexibility than conventional document retrieval, particularly for tasks involving exact matching and multiple filtering steps.
-
The training pipeline is well designed. Using verified trajectories for cold-start SFT before RL seems reasonable, and the ablation studies show that both training stages contribute to performance. The experiments also cover different model sizes and a good range of QA benchmarks.
-
I appreciate that the paper considers the efficiency of direct corpus search. Both PACE and SPCS achieve substantial improvements in command execution speed, which is important if this approach is to be used on large corpora.
Critical weaknesses
-
Although the proposed optimizations make individual search commands faster, the end-to-end inference latency is still higher than that of retrieval-based baselines. For example, GrepSeek with PACE takes 7.93 seconds per query, compared with 4.77 seconds for E5 and 6.07 seconds for Qwen3 embeddings. The additional time spent on LLM reasoning remains a bottleneck.
-
The technical novelty of the execution optimizations seems limited. PACE builds on existing indexing techniques, while SPCS uses parallel execution to accelerate corpus search. Their integration into the agent framework is useful, but I would like to see a clearer discussion of how these methods differ from existing retrieval and database optimizations.
-
Direct corpus search relies heavily on lexical matching, which may limit its applicability. The results on PopQA already show some of these limitations. More experiments on datasets with less lexical overlap or greater linguistic variation would help clarify when this approach is preferable to semantic retrieval.
Questions for authors
See Weaknesses
3 comments
W2: Technical novelty of the execution optimizations
We would like to clarify the role of PACE and SPCS. The main contribution of this work is the optimization of DCI models: training a compact agent to search the corpus directly, through verified cold-start data, SFT and GRPO. PACE and SPCS are not proposed as new indexing methods. We present them to show that DCI-style models can be used efficiently at scale, and we adopt them to make GrepSeek usable in real-world settings. They build on established techniques (posting lists, suffix arrays, Elias–Fano compression, sharding and k-way merging), adapted to the requirements of direct corpus interaction, which standard retrieval indexes and database optimizers do not meet:
- Substring search, not word lookup. The inverted indexes behind retrievers such as BM25 map whole words to documents, so they can only answer word-level queries, and they return a ranked list rather than every exact match. Shell search is different:
rg -F "16"must also match "2016" and "b16", andrg -F "(ON)CHOH"must match a chemical formula with punctuation inside it (Example 1). PACE supports this with a suffix array over the vocabulary. Each query term is matched against every vocabulary term that contains it as a substring, and the union of their posting lists gives the candidate passages. - Exactly the same output as the original command. PACE uses its index only to decide where to run a command, never what it returns. It runs the agent's original pipeline on the candidate passages, in corpus order, so the output is byte-identical to a full scan, including ordering,
head -ntruncation, counts and exit status. Inverted filters (e.g.,-v) do not contribute to pruning, and commands that cannot be pruned safely fall back to a full scan. SPCS likewise parallelizes only pipelines whose output can be reassembled exactly from shards (concatenation,head -n,wc -l, andsort/uniqthrough a k-way merge). It runs anything that depends on global state (line numbers-n, counts-c, context flags) sequentially. - Free-form pipelines, no schema. Database optimizations such as predicate pushdown, or ordering conjunctive filters by selectivity (which PACE's cost-aware intersection resembles), require a schema and a declarative query language. PACE and SPCS instead accept the multi-stage shell pipelines the agent writes, so the agent's interface stays unchanged.
Together, PACE and SPCS make shell search 77 times faster (PACE) and 7.6 times faster (SPCS with 32 shards) on 21M passages without changing what the agent observes, so the same trained policy runs unchanged with or without them. We added this discussion to Appendix B.
W3: Reliance on lexical matching; datasets with less lexical overlap
We agree that DCI relies on lexical matching, and PopQA already exposes this. Following the reviewer's suggestion, we evaluated all methods on the full test sets of three additional datasets with less lexical overlap or more linguistic variation:
- WebQuestions: 2,032 questions issued to Google, worded independently of Wikipedia.
- EntityQuestions: 22,075 questions about long-tail entities.
- PopQA-TP: all 14,267 PopQA test questions, with each question's relation reworded by a template paraphrase.
The results are in the new Figure 6 and discussed in Section 3.3.
| Method | Retriever | WebQuestions | EntityQuestions | PopQA-TP |
|---|---|---|---|---|
| Direct | — | 0.3472 | 0.2693 | 0.2151 |
| RAG | BM25 | 0.2973 | 0.4732 | 0.3076 |
| E5-110M | 0.3799 | 0.4249 | 0.4088 | |
| Qwen3-Emb-4B | 0.3952 | 0.5123 | 0.4785 | |
| Search-R1 | BM25 | 0.4093 | 0.5311 | 0.4052 |
| E5-110M | 0.4219 | 0.4762 | 0.4498 | |
| Qwen3-Emb-4B | 0.4364 | 0.5246 | 0.4853 | |
| GrepSeek | — | 0.4560 | 0.5804 | 0.4668 |
F1, full test sets, Qwen3.5-9B backbone.
- GrepSeek is best on WebQuestions and EntityQuestions. It leads the best Search-R1 agent by +0.020 F1 on WebQuestions (95% paired bootstrap interval [+0.007, +0.032]) and by +0.049 on EntityQuestions ([+0.045, +0.054]). This holds on WebQuestions even though a single dense retrieval there is clearly better than a single BM25 retrieval (+0.098 F1 for RAG with Qwen3-Embedding over BM25).
- PopQA-TP is where semantic retrieval wins, as on the original PopQA. Search-R1 with Qwen3-Embedding leads GrepSeek by 0.019 F1 ([0.012, 0.026]), similar to its lead on the original PopQA in Table 1. GrepSeek still beats the E5 and BM25 agents.
- Paraphrasing does not widen the gap. On the 13,745 PopQA-TP questions whose wording differs from the original, rewording costs GrepSeek 0.020 F1, comparable to the baselines (0.014 to 0.039).
We thank the reviewer for the positive assessment of our motivation, training pipeline and efficiency work.
W1: End-to-end latency is still higher than retrieval-based baselines
We agree that GrepSeek's end-to-end latency (7.93 s per query with PACE) is higher than that of the retrieval baselines (4.77 to 6.07 s), and, as the reviewer notes, the bottleneck is LLM generation rather than search. With PACE, tool time is 0.07 s per query, on par with BM25 (0.06 s), which we have now added to Figure 4, and faster than E5 (0.18 s) and Qwen3-Embedding (1.84 s). Generation takes 7.86 s against 4.23 to 4.84 s for the baselines, because GrepSeek reasons longer and takes more turns.
Tool time and end-to-end latency are per query, as in Figure 4(a); memory includes the 14 GB corpus for every method.