- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies whether rule-generated fictional environments can train search agents that transfer to real-world multi-hop question answering. Building on PhantomWiki, it creates templated document collections and questions with Prolog-verifiable answers, and fine-tunes four instruction models using GRPO on approximately 55,000 questions. The models improve across six evaluation benchmarks and generalize to unseen fictional universes. Comparisons with equally sized NQ+HotpotQA training sets show that real-data training performs better on the older benchmark group, while synthetic training is competitive on the newer group and performs better for the Qwen models. Further experiments investigate search-call scaling, retrieval configurations, and the effects of adding comparison and constraint questions.
Critical strengths
-
Consistent transfer from a simple, verifiable training environment. The gains occur across four models from three families, six benchmarks, and two training seeds. Generalization to new fictional universes, including a tenfold larger universe, makes memorization of the training world's facts an insufficient explanation. The real-data comparison also matches question counts and the training retriever, and Appendix A.3 checks the evaluation configuration. Together, these experiments make a persuasive case that rule-generated environments can be a useful source of search-agent training data.
-
Informative analysis of environment design. The comparison of hop, comparison, and constraint questions goes beyond reporting an aggregate improvement. Comparison training preferentially benefits the corresponding evaluation category, while constraint-rich training can encourage unhelpful query shortcuts. The category-level analyses and example trajectories provide concrete guidance for designing synthetic environments and show why additional complexity need not improve transfer.
Critical weaknesses
-
The explanation based on factual overlap and memorization is stronger than the controls establish. Section 4.2 treats Wikipedia snapshot dates as evidence that the older and newer benchmark groups differ in factual overlap with training. However, a later snapshot can contain older facts, and it does not establish which facts are absent from a model's pretraining or RL data. The real-versus-synthetic training comparison also changes question distributions, linguistic form, and corpus properties. The SynthWorlds mirror comparison is useful evidence, but entity replacement can affect retrieval difficulty as well as access to parametric knowledge; no corresponding evidence-recall control is reported here. The results support transfer and a reduced real/synthetic performance gap, but do not isolate memorization as the explanation for the relative performance of the two training sources. This matters because robustness beyond knowledge cutoffs is a prominent part of the paper's interpretation.
-
The deployment claims extend beyond the evaluated retrieval setting. The evaluation indices are assembled from the selected benchmarks' gold and distractor passages, with FRAMES using its reference articles. Pooling these collections is a useful stress test, but still evaluates search over a benchmark-derived collection with known supporting evidence. It does not establish the same robustness in a substantially larger, less curated corpus. Moreover, Table 7 shows the synthetic-trained Llama model dropping from 35.1 to 29.9 average F1 under pooling, so the degradation is not uniformly slight. The paper acknowledges open-web evaluation as a limitation, but the stronger claims about readiness for large-scale deployment should be narrowed or supported with a corpus-size and retrieval-recall analysis.
Questions for authors
-
Clarification and targeted control for the memorization interpretation. Can you measure actual supporting-fact overlap with the RL corpus, or separate questions by closed-book answerability, rather than using snapshot dates alone? On the SynthWorlds mirror pair, does the knowledge-advantage gap persist after controlling for retrieval of the required evidence?
-
Targeted validation of retrieval robustness. What are the absolute passage counts and supporting-evidence recall rates for the individual and pooled indices? On a fixed subset of questions, how do the base, synthetic-trained, and real-data-trained agents compare as additional unrelated and hard-negative passages are introduced under the same retrieval and interaction budgets?
-
Clarification of checkpoint selection and uncertainty. Appendix A.2 replaces some unstable runs with earlier checkpoints selected by held-out performance. Which held-out sets and selection rule were used for each condition, and do per-seed synthetic-versus-real comparisons remain consistent when the same checkpoint-selection rule is applied throughout?
5 comments
Rebuttal (2/3)
W2: The deployment claims ... stronger claims about readiness for large-scale deployment should be narrowed or supported with a corpus-size and retrieval-recall analysis.
Q2: Targeted validation of retrieval robustness. What are the absolute passage counts and supporting-evidence recall rates for the individual and pooled indices? On a fixed subset of questions, how do the base, synthetic-trained, and real-data-trained agents compare as additional unrelated and hard-negative passages are introduced under the same retrieval and interaction budgets?
Passage counts. HotpotQA 4,956; 2Wiki 3,347; MuSiQue 6,841; SynthWorlds-RM/SM 1,212 each; FRAMES 35,251; pooled index 52,819.
Evidence recall on the own and pooled indices (new). We measure recall as above: the fraction of a question’s gold evidence that the agent retrieves in any of its searches. For FRAMES that labels whole articles rather than passages, an article counts as found if any of its passages is retrieved. Averaged over the six benchmarks, for Qwen2.5-7B:
| Model | Own index | Pooled index |
|---|---|---|
| Qwen2.5-7B | 64.2 ± 0.5 | 60.0 ± 0.5 |
| + Synthetic | 85.2 ± 0.5 | 82.5 ± 0.5 |
The pooled index is 1.5–44× larger than each benchmark’s own index, yet both agents lose only 3–4 points of recall, and the synthetic-trained agent still finds far more evidence than the base model.
More unrelated passages. Together with the new Search-R1 evaluation below, we now answer this question at three corpus sizes: each benchmark’s own index (1K–35K passages), the pooled index (53K), and full Wikipedia (21M).
Finally, your concern about “robustness in a substantially larger, less curated corpus” is well-taken. We now add the Search-R1 evaluation benchmarks against the full Wikipedia index. This retrieval corpus contains 21M uncurated docs, which is ≈400x our pooled index.
We evaluate base and trained counterparts of Qwen2.5-7B and Llama-3.2-3B models, and find that both + Synthetic and + NQ+HotpotQA training environments improve agent performance: 2x improvement for Qwen2.5-7B and 3x for Llama-3.2-3B. For instance, Qwen2.5-7B goes from 23.3% accuracy to 43.7% with + Synthetic, and to 45.5% with + NQ+HotpotQA.
These results show that agents trained in our synthetic environments keep their gains when searching all of Wikipedia, not just the benchmark corpora. This adds to our discussion in Section 4.1 on robustness to large-scale deployment.
| Model | Average | NQ | TriviaQA | PopQA | HotpotQA | 2Wiki | MuSiQue | Bamboogle |
|---|---|---|---|---|---|---|---|---|
| Qwen2.5-7B | 23.3 ± 0.5 | 31.7 | 42.9 | 28.0 | 21.5 | 16.6 | 6.5 | 16.0 |
| + Synthetic | 43.7 ± 0.7 | 43.9 | 61.4 | 48.7 | 41.7 | 40.3 | 23.8 | 46.4 |
| + NQ+HotpotQA | 45.5 ± 0.6 | 50.4 | 66.3 | 47.5 | 46.4 | 42.6 | 21.7 | 43.6 |
| Llama-3.2-3B | 11.4 ± 0.4 | 15.7 | 20.0 | 10.7 | 9.1 | 9.1 | 2.7 | 12.8 |
| + Synthetic | 36.3 ± 0.7 | 47.8 | 59.4 | 44.8 | 31.1 | 20.8 | 13.1 | 36.8 |
| + NQ+HotpotQA | 39.4 ± 0.8 | 47.1 | 63.7 | 45.6 | 38.6 | 36.9 | 13.0 | 30.8 |
Real vs synthetic training. While real-world training is slightly better than synthetic, we find the jump from base -> trained agent as the main takeaway here. In fact, in Section 4.2 we had already observed that real-world training is better than synthetic, when training-test distributions overlap. The Search-R1 Wikipedia-2018 evaluation corpus is the same as that of NQ+HotpotQA, our real-world training environment. Search-R1 evaluation questions also share factual knowledge overlap to the real-world training set.
Search-R1 evaluation setup. We follow the Search-R1 paper setup: (1) official validation split of ≈51K questions, (2) 21M docs from 2018 Wikipedia encoded with the e5-base-v2 model, (3) report the Exact Match accuracy % (EM = 100% if any predicted answer, after splitting model’s answer by commas, matches any gold answer). We use our paper’s evaluation prompts and infrastructure—up to 20 turns, 32K context, top-3 docs—that performs better than the official Search-R1 checkpoints and code. For every model x training combination, we had trained with two training seeds, so the EM scores here are averaged across two training seeds and all evaluation questions.
Rebuttal (3/3)
Q3: Clarification of checkpoint selection and uncertainty. Appendix A.2 replaces some unstable runs with earlier checkpoints selected by held-out performance. Which held-out sets and selection rule were used for each condition, and do per-seed synthetic-versus-real comparisons remain consistent when the same checkpoint-selection rule is applied throughout?
Details on checkpoint selection. We agree that the wording was a bit loose, so we have added this clarification in the appendix. Only three of our runs degenerated mid-training due to entropy spikes. For the two degenerated NQ+HotpotQA runs, we picked the saved checkpoints right before collapse in training reward (we had been saving checkpoints at every 10% of total training steps). Llama model run had seen 30K of 55K questions, and Phi had seen 25K. For the one Hops+Constraints run, we used the training-distribution validation questions. The model had seen 48K questions. We never use the final evaluation benchmarks to pick checkpoints.
Instability appeared only for Phi and Llama, in both real-data and synthetic training, and never for Qwen models. As discussed in the appendix A.2, RL instability is common in practice. For our runs, we believe it originates from the open-source RL infrastructure, due to a mismatch between the sampler's (vLLM) and the trainer's log-probabilities: entropy spikes right after spikes in this log-prob mismatch.
Rebuttal (1/3)
Thank you for your comments and clarifying questions.
Our evidence comes from the paired SynthWorlds evaluation, which preserves the underlying question structure while replacing real-world entities with synthetic counterparts. For all four models, synthetic-trained agents have a smaller RM–SM gap than real-data-trained agents (Figure 4; other models in the appendix). The smaller gap after synthetic training also persists under both evaluation configurations in Appendix A.3. This supports our interpretation that synthetic training improves agents’ ability to operate without familiar factual content.
Evidence recall on the SynthWorlds pair (new). We agree that replacing entities can change retrieval difficulty, so we report evidence-recall rates here from saved trajectories. For each episode, we collect every document returned across all of the agent’s searches and compute the fraction of the question’s gold documents in that set.
The synthetic-trained 7B agent queries correct evidence at the same rates in both RM and SM evaluations, whereas the base and real-trained agent show a gap of 8–9 points. For Qwen2.5-3B and Llama-3.2-3B models, the recall gap of synthetic-trained is non-zero but still less than real-trained.
So as you point out, the knowledge-advantage gap in end-to-end evaluation also comes from retrieval, in line with SynthWorlds authors’ finding (their Sec. 6). However, the worse retrieval quality in SM could be due to the embedding model handling unfamiliar entities poorly or the agent issuing worse queries for unfamiliar entities—this is hard to disentangle. Since all agents share the same retriever, the synthetic-trained agent’s equal recall suggests that the agent’s queries play a role. We have added these results to the appendix, and revised Sec. 4.2 to say that the RM–SM gap reflects reliance on both retriever and memorized factual knowledge. We present memorization as a hypothesis, not an established mechanism.
Closed-book evaluation (new). To isolate memorized knowledge of models, we prompt each Qwen2.5-7B model to answer without the search tool, using the closed-book prompt from SynthWorlds paper. Closed-book F1 (%), averaged over two training seeds:
To your question, the synthetic RL corpus has zero overlap with evaluation, and NQ+HotpotQA retrieval corpus of Wikipedia 2018 contains the HotpotQA, 2Wiki, MuSiQue gold passages. The NQ+HotpotQA training corpus could share facts with SynthWorlds and FRAMES corpora, but we don’t see a clear way to measure this: both LLM-as-a-judge and n-gram overlaps will be dubious estimates. Finally, we did ablate the choice of retriever in Appendix A.3 (e5 vs Qwen3) and our claimed trends on SynthWorlds RM vs SM hold.