- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SEARCHSKILL tackles a real annoyance: search agents tend to just dump the whole multi-hop question into the retriever. The fix is a select–read–act loop, before each step the model picks a skill from a SkillBank, reads its card, then issues a query or answers. The bank isn't hand-written; it grows in four curriculum stages (B0→B4, six → ~20 cards), driven by failure analysis, with skills from successful rollouts "protected" from later revision.
Critical strengths
-
The core idea is clean and actually testable. Making query planning explicit — pick a skill, read the card, then act — turns a vague "search better" goal into a token-level, trainable interface. The factorization in Eq. 1 is simple, and the two-stage SFT genuinely aligns training with the inference protocol (Stage II removes the skill hint, so the model must really route). That's a nicer story than bolting on another RL reward.
-
The SkillBank evolution is a real mechanism, not window dressing. Staged growth from direct lookup → bridge → comparison → long-hop, with protection of validated skills and revision driven by failure traces, is principled. Table 4 shows staging beats one-shot by ~5 points. And the bank is portable: frozen B4 helps three closed-source models (Fig. 5) and transfers to WebWalker with zero retraining.
-
Diagnostics go well beyond EM. Copy rate, atomic-hop rate, searches per question, Correct@≤3 — these actually test the motivating claim instead of just reporting accuracy. The paired SFT-vs-RL analysis (Fig. 6) is honest, and they openly admit GRPO doesn't help accuracy. Case studies are readable and convincing. Also: 1.2h of LoRA SFT vs ~10h of GRPO is a genuinely cheap recipe.
Critical weaknesses
-
The headline number rides on a proprietary teacher. The 42.34 comes from GPT-5.4-generated skills and GPT-5.4 trajectories. The self-generated variant gets 38.60 — i.e., basically ties Search-R1's 38.68. So "self-directed skill discovery elicits latent capability" is oversold. Table 3's distillation control doesn't rescue it either: the SkillBank itself is GPT-5.4-authored, so it's distillation in a different costume, and the control differs in more than "skill conditioning."
-
Baseline fairness and statistical rigor are shaky. Baselines look copied from original papers rather than rerun under this E5/top-3/budget-5 setup, and some are implausible (s3 at 15.00 avg). Evaluation is 16 validation records, greedy decoding, single run, no seeds or significance tests, EM only. Worse, the bank is built from HotpotQA/2Wiki/MuSiQue — the same datasets where gains concentrate — while MuSiQue stays stuck at ~15 EM.
Questions for authors
see weakness
5 comments
Response to Reviewer WSbx (Part 2/3)
W2: Baseline fairness and statistical rigor are shaky. Baselines look copied from original papers rather than rerun under this E5/top-3/budget-5 setup, and some are implausible (s3 at 15.00 avg). Evaluation is 16 validation records, greedy decoding, single run, no seeds or significance tests, EM only. Worse, the bank is built from HotpotQA/2Wiki/MuSiQue — the same datasets where gains concentrate — while MuSiQue stays stuck at ~15 EM.
1) Baselines are re-run, and s3 is scored by strict EM
We thank the reviewer for the careful reading. As stated in the Evaluation paragraph of Sec. 4.1, the main table re-ran every baseline with the shared E5 retriever, top-3 passages, five-search budget and greedy decoding. To address the concern further, we enlarged the datasets to the 1,000-question scale; Bamboogle officially has only 125 test questions. The EM results are as follows:
| Method | HotpotQA | 2Wiki | MuSiQue | Bamboogle | NQ | TriviaQA | PopQA | Avg. |
|---|---|---|---|---|---|---|---|---|
| ZeroSearch | 30.4 | 27.8 | 7.2 | 22.4 | 36.0 | 58.9 | 47.3 | 32.9 |
| Search-o1 | 33.0 | 31.8 | 14.3 | 36.8 | 31.4 | 58.9 | 44.6 | 35.8 |
| Search-R1 | 39.7 | 33.2 | 16.7 | 38.4 | 34.5 | 53.6 | 46.1 | 37.5 |
| StepSearch | 36.9 | 40.2 | 21.8 | 44.0 | 33.7 | 52.0 | 45.5 | 39.2 |
| Smart-Searcher | 38.9 | 40.6 | 18.7 | 43.2 | 35.9 | 58.9 | 47.3 | 40.5 |
| SearchSkill-Base, seed 42 | 41.9 | 42.2 | 16.9 | 40.0 | 38.4 | 61.0 | 51.7 | 41.7 |
| SearchSkill-Base, seed 73531 | 40.6 | 42.7 | 19.7 | 38.4 | 38.8 | 61.3 | 51.3 | 41.8 |
| SearchSkill-Base, seed 75531 | 41.9 | 43.1 | 17.7 | 36.8 | 39.6 | 61.6 | 52.9 | 42.0 |
| SearchSkill-Base, mean | 41.5 | 42.7 | 18.1 | 38.4 | 38.9 | 61.3 | 52.0 | 41.8 ± 0.1 |
| SearchSkill-Instruct, seed 42 | 41.2 | 42.2 | 18.9 | 43.2 | 39.4 | 61.9 | 51.1 | 42.6 |
| SearchSkill-Instruct, seed 75531 | 42.4 | 41.5 | 18.2 | 43.2 | 38.3 | 62.5 | 51.8 | 42.6 |
| SearchSkill-Instruct, seed 2903 | 42.2 | 40.1 | 18.9 | 40.8 | 39.3 | 62.3 | 49.7 | 41.9 |
| SearchSkill-Instruct, mean | 41.9 | 41.3 | 18.7 | 42.4 | 39.0 | 62.2 | 50.9 | 42.3 ± 0.4 |
Paired significance tests against every baseline:
| ZeroSearch | Search-o1 | Search-R1 | StepSearch | Smart-Searcher | |
|---|---|---|---|---|---|
| McNemar p, largest over all six seeds | < 2e-35 | < 3e-26 | < 1e-15 | < 3e-10 | < 5e-4 |
| Instruct mean gain, paired bootstrap 95% CI | +9.5 [+7.9, +10.9] | +6.5 [+5.1, +8.0] | +4.9 [+3.6, +6.3] | +3.2 [+1.7, +4.6] | +1.8 [+0.4, +3.2] |
Our method leads every baseline, and the advantage passes significance tests.
s3 itself has strong search ability. Its 15.00 comes from strict EM, whereas its authors use span containment, so we scored our re-run of s3 in our harness under both criteria:
| s3 | HotpotQA | 2Wiki | MuSiQue | Bamboogle | NQ | TriviaQA | PopQA |
|---|---|---|---|---|---|---|---|
| Strict EM, as in the main table | 20.0 | 14.4 | 3.5 | 10.4 | 6.5 | 36.3 | 17.3 |
| Span containment, its own criterion | 48.9 | 51.7 | 20.5 | 35.2 | 58.4 | 72.6 | 63.6 |
Most mainstream methods report EM, so for fairness we keep EM as the common standard, while we acknowledge that s3 also has strong search ability.
Response to Reviewer WSbx (Part 3/3)
2) Evaluation set, seeds, metrics and decoding
Actually, the 16 records are the SFT validation split used to monitor training, not the evaluation set. The paper evaluates 3,725 questions, and the enlarged sets above contain 5,799.
The two tables above already give multi-run greedy results. Greedy decoding has no run-to-run variance: we re-ran every baseline three times and show one row because the runs are identical. To address the concern further, we also report sampling decoding against Search-R1 and F1 for every baseline. Sampling uses temperature 0.7 with three runs on the paper's benchmarks, in EM:
| HotpotQA | 2Wiki | MuSiQue | Bamboogle | NQ | TriviaQA | PopQA | Avg. | |
|---|---|---|---|---|---|---|---|---|
| Search-R1, run 1 | 33.0 | 27.0 | 11.5 | 29.6 | 36.0 | 52.8 | 44.9 | 33.5 |
| Search-R1, run 2 | 34.0 | 26.5 | 10.5 | 32.0 | 33.3 | 53.0 | 44.3 | 33.4 |
| Search-R1, run 3 | 33.5 | 25.5 | 11.5 | 32.8 | 33.7 | 53.7 | 44.9 | 33.7 |
| Search-R1, mean | 33.5 | 26.3 | 11.2 | 31.5 | 34.3 | 53.2 | 44.7 | 33.5 |
| SearchSkill, seed 42, run 1 | 39.5 | 43.0 | 13.5 | 30.4 | 36.4 | 59.5 | 51.2 | 39.1 |
| SearchSkill, seed 42, run 2 | 40.0 | 42.5 | 13.5 | 35.2 | 37.1 | 60.3 | 50.6 | 39.9 |
| SearchSkill, seed 42, run 3 | 38.5 | 43.5 | 16.0 | 40.8 | 37.3 | 59.9 | 51.1 | 41.0 |
| SearchSkill, seed 42, mean | 39.3 | 43.0 | 14.3 | 35.5 | 36.9 | 59.9 | 51.0 | 40.0 |
F1 on the enlarged benchmarks for every baseline:
| Method | HotpotQA | 2Wiki | MuSiQue | Bamboogle | NQ | TriviaQA | PopQA | Avg. |
|---|---|---|---|---|---|---|---|---|
| ZeroSearch | 40.9 | 34.9 | 15.0 | 34.5 | 46.2 | 67.5 | 56.1 | 42.2 |
| Search-o1 | 44.4 | 40.7 | 22.6 | 45.2 | 42.4 | 68.4 | 53.2 | 45.3 |
| Search-R1 | 49.1 | 36.6 | 22.6 | 46.2 | 41.2 | 59.3 | 51.0 | 43.7 |
| StepSearch | 47.8 | 47.4 | 32.6 | 54.1 | 42.3 | 60.3 | 52.8 | 48.2 |
| Smart-Searcher | 49.9 | 48.1 | 28.2 | 57.2 | 46.5 | 67.4 | 55.7 | 50.4 |
| SearchSkill-Instruct, seed 42 | 52.5 | 48.7 | 27.3 | 53.3 | 48.3 | 69.8 | 57.8 | 51.1 |
| SearchSkill-Instruct, seed 75531 | 53.4 | 48.8 | 26.9 | 53.5 | 48.3 | 70.2 | 58.2 | 51.3 |
| SearchSkill-Instruct, seed 2903 | 52.9 | 47.3 | 27.8 | 51.7 | 48.8 | 69.8 | 56.6 | 50.7 |
| SearchSkill-Instruct, mean | 52.9 | 48.3 | 27.3 | 52.8 | 48.5 | 69.9 | 57.5 | 51.0 ± 0.3 |
Under sampling SearchSkill leads Search-R1 by 6.5 EM, and under F1 it leads every baseline, so our advantage holds under these changes. The gains are not concentrated on the three evolution datasets. On the enlarged MuSiQue every seed reaches 18 EM or more, so it is not stuck at 15.
We are deeply grateful for your thorough and constructive review, which helped us separate the role of skills from the teacher and strengthen the statistical evidence. We hope these results resolve your concerns and would be glad to discuss any questions.
Response to Reviewer WSbx (Part 1/3)
We sincerely thank you for the thoughtful, candid and encouraging review. We greatly appreciate your recognition of the clean select–read–act interface, the SkillBank evolution as a real mechanism and the diagnostics beyond EM. Your questions on teacher dependence and evaluation rigor guided us to separate skills from distillation and to re-test all baselines more rigorously, which strengthens both our claims and their empirical support.
1) Separating skills from teacher distillation
The paper's Table 3 shows that our skill workflow elicits extra capability. This comes from two sources: the model learns to select and apply skills in its reasoning, and the cards themselves carry content.
First, we exclude GPT-5.4's knowledge, that is, whether the gain is plain distillation from GPT-5.4 without our workflow. The paper's Table 3 gives this control on 7B-Instruct, in EM:
Distilling GPT-5.4's knowledge on these questions is not enough without our workflow. With the same teacher and budget, skill-structured training still outperforms direct trajectory distillation.
Second, we restrict the test to skill conditioning: we reuse our whole pipeline and only shuffle the content of the skill cards. EM of 7B-Instruct, seed 42, on the enlarged benchmarks:
Without correct card content, EM drops on every dataset, by 1.8 on average (McNemar p = 0.003, 95% CI [+0.4, +2.9]). The card content is thus essential for reasoning, which further shows the role of skills. We will add this part to the main text with an explanation.
2) Self-generated skills against large-scale RL
Our workflow is not designed to rely on a particular model for its gains. We use GPT-5.4 mainly for practicality and efficiency, since an API makes the bank easy to build. The training cost of the self-generated variant against Search-R1, 7B-Instruct on the paper's benchmarks:
Our original aim is to elicit the model's capability in a more efficient way, and we do achieve it. With 4 H100 GPUs and under one hour of SFT, the 7B model writing its own skills matches the capability of large-scale RL, which shows that self-directed skill discovery does elicit the latent capability of the 7B model.