acceptodds
Under review as a conference paper at ICLR 2027

On Training LLM Agents in Fictional Worlds

Abstract

Training LLM agents with reinforcement learning (RL) is bottlenecked by environments, which must provide verifiable rewards, support long-horizon interaction, and scale cheaply. Existing approaches rely on costly human-curated data or on LLM-generated environments that risk hallucinations and benchmark contamination. We show that LLMs can instead be trained into capable search agents using synthetic environments generated entirely by rules, whose generation requires no LLM and has zero marginal cost. We build multi-turn RL environments from fictional worlds, where agents must search a corpus of templated articles to answer multi-hop questions. Despite sharing no facts with the real world, these strikingly simple environments yield agents that transfer to real-world multi-hop search benchmarks, often outperforming real-world training data on newer benchmarks. Trained agents generalize to unseen fictional universes, and Qwen models learn to scale their search budget roughly linearly with question difficulty, suggesting emergent search scaling from environment interaction alone. Ablating environment complexity reveals that hop count drives transfer more than constraints or comparisons: even the simplest rule-generated environments are a surprisingly effective, free resource for training generalizable LLM agents.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language models, agent training, search agents, synthetic RL environments, RL fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_9MHMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies whether rule-generated fictional environments can train search agents that transfer to real-world multi-hop question answering. Building on PhantomWiki, it creates templated document collections and questions with Prolog-verifiable answers, and fine-tunes four instruction models using GRPO on approximately 55,000 questions. The models improve across six evaluation benchmarks and generalize to unseen fictional universes. Comparisons with equally sized NQ+HotpotQA training sets show that real-data training performs better on the older benchmark group, while synthetic training is competitive on the newer group and performs better for the Qwen models. Further experiments investigate search-call scaling, retrieval configurations, and the effects of adding comparison and constraint questions.

Critical strengths
  1. Consistent transfer from a simple, verifiable training environment. The gains occur across four models from three families, six benchmarks, and two training seeds. Generalization to new fictional universes, including a tenfold larger universe, makes memorization of the training world's facts an insufficient explanation. The real-data comparison also matches question counts and the training retriever, and Appendix A.3 checks the evaluation configuration. Together, these experiments make a persuasive case that rule-generated environments can be a useful source of search-agent training data.

  2. Informative analysis of environment design. The comparison of hop, comparison, and constraint questions goes beyond reporting an aggregate improvement. Comparison training preferentially benefits the corresponding evaluation category, while constraint-rich training can encourage unhelpful query shortcuts. The category-level analyses and example trajectories provide concrete guidance for designing synthetic environments and show why additional complexity need not improve transfer.

Critical weaknesses
  1. The explanation based on factual overlap and memorization is stronger than the controls establish. Section 4.2 treats Wikipedia snapshot dates as evidence that the older and newer benchmark groups differ in factual overlap with training. However, a later snapshot can contain older facts, and it does not establish which facts are absent from a model's pretraining or RL data. The real-versus-synthetic training comparison also changes question distributions, linguistic form, and corpus properties. The SynthWorlds mirror comparison is useful evidence, but entity replacement can affect retrieval difficulty as well as access to parametric knowledge; no corresponding evidence-recall control is reported here. The results support transfer and a reduced real/synthetic performance gap, but do not isolate memorization as the explanation for the relative performance of the two training sources. This matters because robustness beyond knowledge cutoffs is a prominent part of the paper's interpretation.

  2. The deployment claims extend beyond the evaluated retrieval setting. The evaluation indices are assembled from the selected benchmarks' gold and distractor passages, with FRAMES using its reference articles. Pooling these collections is a useful stress test, but still evaluates search over a benchmark-derived collection with known supporting evidence. It does not establish the same robustness in a substantially larger, less curated corpus. Moreover, Table 7 shows the synthetic-trained Llama model dropping from 35.1 to 29.9 average F1 under pooling, so the degradation is not uniformly slight. The paper acknowledges open-web evaluation as a limitation, but the stronger claims about readiness for large-scale deployment should be narrowed or supported with a corpus-size and retrieval-recall analysis.

Questions for authors
  1. Clarification and targeted control for the memorization interpretation. Can you measure actual supporting-fact overlap with the RL corpus, or separate questions by closed-book answerability, rather than using snapshot dates alone? On the SynthWorlds mirror pair, does the knowledge-advantage gap persist after controlling for retrieval of the required evidence?

  2. Targeted validation of retrieval robustness. What are the absolute passage counts and supporting-evidence recall rates for the individual and pooled indices? On a fixed subset of questions, how do the base, synthetic-trained, and real-data-trained agents compare as additional unrelated and hard-negative passages are introduced under the same retrieval and interaction budgets?

  3. Clarification of checkpoint selection and uncertainty. Appendix A.2 replaces some unstable runs with earlier checkpoints selected by held-out performance. Which held-out sets and selection rule were used for each condition, and do per-seed synthetic-versus-real comparisons remain consistent when the same checkpoint-selection rule is applied throughout?

5 comments
AuthorsOpenReview ↗

Rebuttal (1/3)

Thank you for your comments and clarifying questions.

W1: The explanation based on factual overlap ... no corresponding evidence-recall control is reported here. The results support transfer and a reduced real/synthetic performance gap, but do not isolate memorization as the explanation for the relative performance of the two training sources. This matters because robustness beyond knowledge cutoffs is a prominent part of the paper's interpretation.

Q1: Clarification and targeted control for the memorization interpretation. Can you measure actual supporting-fact overlap with the RL corpus, or separate questions by closed-book answerability, rather than using snapshot dates alone? On the SynthWorlds mirror pair, does the knowledge-advantage gap persist after controlling for retrieval of the required evidence?

Our evidence comes from the paired SynthWorlds evaluation, which preserves the underlying question structure while replacing real-world entities with synthetic counterparts. For all four models, synthetic-trained agents have a smaller RM–SM gap than real-data-trained agents (Figure 4; other models in the appendix). The smaller gap after synthetic training also persists under both evaluation configurations in Appendix A.3. This supports our interpretation that synthetic training improves agents’ ability to operate without familiar factual content.

Evidence recall on the SynthWorlds pair (new). We agree that replacing entities can change retrieval difficulty, so we report evidence-recall rates here from saved trajectories. For each episode, we collect every document returned across all of the agent’s searches and compute the fraction of the question’s gold documents in that set.

ModelSynthWorlds-RM recallSynthWorlds-SM recall
Qwen2.5-7B55.6 ± 1.146.4 ± 0.9
+ Synthetic78.0 ± 2.378.2 ± 0.8
+ NQ+HotpotQA72.4 ± 1.064.1 ± 1.1

The synthetic-trained 7B agent queries correct evidence at the same rates in both RM and SM evaluations, whereas the base and real-trained agent show a gap of 8–9 points. For Qwen2.5-3B and Llama-3.2-3B models, the recall gap of synthetic-trained is non-zero but still less than real-trained.

So as you point out, the knowledge-advantage gap in end-to-end evaluation also comes from retrieval, in line with SynthWorlds authors’ finding (their Sec. 6). However, the worse retrieval quality in SM could be due to the embedding model handling unfamiliar entities poorly or the agent issuing worse queries for unfamiliar entities—this is hard to disentangle. Since all agents share the same retriever, the synthetic-trained agent’s equal recall suggests that the agent’s queries play a role. We have added these results to the appendix, and revised Sec. 4.2 to say that the RM–SM gap reflects reliance on both retriever and memorized factual knowledge. We present memorization as a hypothesis, not an established mechanism.

Closed-book evaluation (new). To isolate memorized knowledge of models, we prompt each Qwen2.5-7B model to answer without the search tool, using the closed-book prompt from SynthWorlds paper. Closed-book F1 (%), averaged over two training seeds:

ModelHotpotQA2WikiMuSiQueSynthWorlds-RMSynthWorlds-SMFRAMES
Qwen2.5-7B26.1 ± 1.825.0 ± 1.810.5 ± 1.114.2 ± 1.40.7 ± 0.214.3 ± 1.1
+ Synthetic27.2 ± 1.325.4 ± 2.310.5 ± 1.215.6 ± 1.11.4 ± 0.214.9 ± 1.1
+ NQ+HotpotQA28.9 ± 1.530.8 ± 1.512.6 ± 0.915.6 ± 1.00.9 ± 0.215.0 ± 1.2
  • HotpotQA and 2Wiki are easier to answer from memory than newer benchmarks SynthWorlds and FRAMES. Closed-book answerability is thus more direct evidence than snapshot dates; we report it in Sec. 4.2 and the appendix.
  • Models can answer some RM questions from memory but not SM. Closed-book F1 scores for SynthWorlds-RM is 14–16% F1 but SM scores are about 1%. RL training does not change what the base model knows about the real entities.
  • Synthetic-trained shows a Knowledge Advantage gap in closed-book eval (RM - SM ≈ 14%), but not in search agent evaluation (Fig 8 in paper, RM - SM <0). So RL training in synthetic environments did reduce bias from memorized knowledge.

To your question, the synthetic RL corpus has zero overlap with evaluation, and NQ+HotpotQA retrieval corpus of Wikipedia 2018 contains the HotpotQA, 2Wiki, MuSiQue gold passages. The NQ+HotpotQA training corpus could share facts with SynthWorlds and FRAMES corpora, but we don’t see a clear way to measure this: both LLM-as-a-judge and n-gram overlaps will be dubious estimates. Finally, we did ablate the choice of retriever in Appendix A.3 (e5 vs Qwen3) and our claimed trends on SynthWorlds RM vs SM hold.

AuthorsOpenReview ↗

Rebuttal (2/3)

W2: The deployment claims ... stronger claims about readiness for large-scale deployment should be narrowed or supported with a corpus-size and retrieval-recall analysis.

Q2: Targeted validation of retrieval robustness. What are the absolute passage counts and supporting-evidence recall rates for the individual and pooled indices? On a fixed subset of questions, how do the base, synthetic-trained, and real-data-trained agents compare as additional unrelated and hard-negative passages are introduced under the same retrieval and interaction budgets?

Passage counts. HotpotQA 4,956; 2Wiki 3,347; MuSiQue 6,841; SynthWorlds-RM/SM 1,212 each; FRAMES 35,251; pooled index 52,819.

Evidence recall on the own and pooled indices (new). We measure recall as above: the fraction of a question’s gold evidence that the agent retrieves in any of its searches. For FRAMES that labels whole articles rather than passages, an article counts as found if any of its passages is retrieved. Averaged over the six benchmarks, for Qwen2.5-7B:

ModelOwn indexPooled index
Qwen2.5-7B64.2 ± 0.560.0 ± 0.5
+ Synthetic85.2 ± 0.582.5 ± 0.5

The pooled index is 1.5–44× larger than each benchmark’s own index, yet both agents lose only 3–4 points of recall, and the synthetic-trained agent still finds far more evidence than the base model.

More unrelated passages. Together with the new Search-R1 evaluation below, we now answer this question at three corpus sizes: each benchmark’s own index (1K–35K passages), the pooled index (53K), and full Wikipedia (21M).

Finally, your concern about “robustness in a substantially larger, less curated corpus” is well-taken. We now add the Search-R1 evaluation benchmarks against the full Wikipedia index. This retrieval corpus contains 21M uncurated docs, which is ≈400x our pooled index.

We evaluate base and trained counterparts of Qwen2.5-7B and Llama-3.2-3B models, and find that both + Synthetic and + NQ+HotpotQA training environments improve agent performance: 2x improvement for Qwen2.5-7B and 3x for Llama-3.2-3B. For instance, Qwen2.5-7B goes from 23.3% accuracy to 43.7% with + Synthetic, and to 45.5% with + NQ+HotpotQA.

These results show that agents trained in our synthetic environments keep their gains when searching all of Wikipedia, not just the benchmark corpora. This adds to our discussion in Section 4.1 on robustness to large-scale deployment.

ModelAverageNQTriviaQAPopQAHotpotQA2WikiMuSiQueBamboogle
Qwen2.5-7B23.3 ± 0.531.742.928.021.516.66.516.0
+ Synthetic43.7 ± 0.743.961.448.741.740.323.846.4
+ NQ+HotpotQA45.5 ± 0.650.466.347.546.442.621.743.6
Llama-3.2-3B11.4 ± 0.415.720.010.79.19.12.712.8
+ Synthetic36.3 ± 0.747.859.444.831.120.813.136.8
+ NQ+HotpotQA39.4 ± 0.847.163.745.638.636.913.030.8

Real vs synthetic training. While real-world training is slightly better than synthetic, we find the jump from base -> trained agent as the main takeaway here. In fact, in Section 4.2 we had already observed that real-world training is better than synthetic, when training-test distributions overlap. The Search-R1 Wikipedia-2018 evaluation corpus is the same as that of NQ+HotpotQA, our real-world training environment. Search-R1 evaluation questions also share factual knowledge overlap to the real-world training set.

Search-R1 evaluation setup. We follow the Search-R1 paper setup: (1) official validation split of ≈51K questions, (2) 21M docs from 2018 Wikipedia encoded with the e5-base-v2 model, (3) report the Exact Match accuracy % (EM = 100% if any predicted answer, after splitting model’s answer by commas, matches any gold answer). We use our paper’s evaluation prompts and infrastructure—up to 20 turns, 32K context, top-3 docs—that performs better than the official Search-R1 checkpoints and code. For every model x training combination, we had trained with two training seeds, so the EM scores here are averaged across two training seeds and all evaluation questions.

AuthorsOpenReview ↗

Rebuttal (3/3)

Q3: Clarification of checkpoint selection and uncertainty. Appendix A.2 replaces some unstable runs with earlier checkpoints selected by held-out performance. Which held-out sets and selection rule were used for each condition, and do per-seed synthetic-versus-real comparisons remain consistent when the same checkpoint-selection rule is applied throughout?

Details on checkpoint selection. We agree that the wording was a bit loose, so we have added this clarification in the appendix. Only three of our runs degenerated mid-training due to entropy spikes. For the two degenerated NQ+HotpotQA runs, we picked the saved checkpoints right before collapse in training reward (we had been saving checkpoints at every 10% of total training steps). Llama model run had seen 30K of 55K questions, and Phi had seen 25K. For the one Hops+Constraints run, we used the training-distribution validation questions. The model had seen 48K questions. We never use the final evaluation benchmarks to pick checkpoints.

Instability appeared only for Phi and Llama, in both real-data and synthetic training, and never for Qwen models. As discussed in the appendix A.2, RL instability is common in practice. For our runs, we believe it originates from the open-source RL infrastructure, due to a mismatch between the sampler's (vLLM) and the trainer's log-probabilities: entropy spikes right after spikes in this log-prob mismatch.

Reviewer_nVaoOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether LLM search agents can be trained through reinforcement learning in entirely rule-generated fictional worlds. It builds an interactive retrieval environment from PhantomWiki, using templated articles and multi-hop questions with verifiable answers. Four models are trained with GRPO and evaluated on six question-answering benchmarks. Synthetic training consistently improves over the base models and is competitive with real-data training on some benchmarks. Additional experiments examine generalization to unseen fictional worlds and the effects of adding comparison and constraint questions.

Critical strengths

The empirical transfer result is convincing and potentially useful. Improvements appear across four models and multiple benchmarks, despite the substantial difference between the synthetic training documents and the evaluation corpora. For example, Qwen2.5-7B improves from 16.2 to 35.4 F1 on FRAMES. The unseen-universe experiments also provide evidence that the agents learn behavior that transfers beyond the particular fictional facts encountered during training. These results establish procedural environments as a viable source of training data for search agents.

Critical weaknesses
  1. The advance over prior work appears incremental. The paper builds on PhantomWiki for environment generation and follows an established Search-R1-style training recipe. Section 2 also acknowledges prior work showing that rule-generated synthetic data can improve real-world reasoning. Extending this finding to interactive retrieval is meaningful, since agents must formulate queries and gather evidence themselves. However, I am not convinced that the paper establishes a sufficiently substantial new insight beyond this extension. An empirical contribution does not require a new algorithm, but it should reveal something consequential that was previously unclear. Here, the results mainly demonstrate that the combination works, while leaving the source and limits of its effectiveness unresolved.

  2. The analyses do not sufficiently strengthen the paper’s claim to new understanding. The conclusion that linear hops drive transfer is based on comparisons among environments that all include multi-hop questions. These experiments show that adding comparisons or constraints does not consistently improve average performance, but do not isolate the benefit of multi-hop training over simpler retrieval training.

Questions for authors
  1. Clarification of the contribution: Given the prior synthetic-to-real reasoning results discussed in Section 2, what do the authors regard as the main new scientific finding enabled by the interactive search setting? Which experiment most directly establishes it?

  2. Evidence for the current transfer claim: How much of the improvement remains when training is restricted to single-hop retrieval under the same budget? This comparison would help determine whether the contribution depends on learning multi-step search behavior or can largely be explained by simpler tool-use training.

3 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

We thank the reviewer for engaging with the contribution question. Please also see the general response.

W1: The advance over prior work appears incremental. The paper builds on PhantomWiki for environment generation and follows an established Search-R1-style training recipe. Section 2 also acknowledges prior work showing that rule-generated synthetic data can improve real-world reasoning. Extending this finding to interactive retrieval is meaningful, since agents must formulate queries and gather evidence themselves. However, I am not convinced that the paper establishes a sufficiently substantial new insight beyond this extension. An empirical contribution does not require a new algorithm, but it should reveal something consequential that was previously unclear. Here, the results mainly demonstrate that the combination works, while leaving the source and limits of its effectiveness unresolved.

Q1: Clarification of the contribution: Given the prior synthetic-to-real reasoning results discussed in Section 2, what do the authors regard as the main new scientific finding enabled by the interactive search setting? Which experiment most directly establishes it?

Our main scientific finding is that rule-generated fictional environments can train a search agent that transfers to real world benchmarks. Table 2 establishes this agent transfer result across four models—the agents must retrieve evidence through its own queries, which is a significantly harder task than prior transfer work on in-context reasoning. Section 4.1 analyzes what properties our synthetic environments teach LLM agents (generalizability, robustness to bigger corpus, etc.). Moreover, we conduct extensive ablations on how to design RL environments:

  • Synthetic is competitive with real-world training environments (Section 4.2), and we have discussion using SynthWorlds benchmark on when one is better than the other.
  • In Section 5, adding constraint questions taught models to shortcut multi-hop search and adding comparison questions improved performance on CofCA and 2Wiki benchmark’s comparison questions.

The new ablation on hop count (below) adds evidence about the source of transfer: single-hop training yields a small gain, while training with longer chains produces substantially larger gains. We had significantly extended PhantomWiki’s generator and sampling to implement constraints, comparisons, and hop controls (Appendix A.4). Finally in Table 8 in the appendix, we observe that RL fine-tuning leaves pre-trained knowledge and fluency intact.

All prior work on rule-generated synthetic-to-real transfer is on improving in-context reasoning in 1 turn, where models do not need to interact with external environments. As you also remark and we discuss in Section 2, multi-turn interaction is substantially harder because the model by itself should figure out how to search, what to search, and in what sequence. In-context reasoning is easier because models do not have to search documents at all. While our algorithm is not novel (simple RL like Search-R1), the fact that you only need rule-generated synthetic RL environments—no costly real-world or LLM-generated environments—to train agentic reasoning is novel.

AuthorsOpenReview ↗

Rebuttal (2/2)

W2: The analyses do not sufficiently strengthen the paper's claim to new understanding. The conclusion that linear hops drive transfer is based on comparisons among environments that all include multi-hop questions. These experiments show that adding comparisons or constraints does not consistently improve average performance, but do not isolate the benefit of multi-hop training over simpler retrieval training.

Q2: Evidence for the current transfer claim: How much of the improvement remains when training is restricted to single-hop retrieval under the same budget? This comparison would help determine whether the contribution depends on learning multi-step search behavior or can largely be explained by simpler tool-use training.

Good ablation idea, thanks. We RL fine-tune Qwen2.5-3B model on our synthetic environments, controlling the number of training hops in the environments at <=1, <=2 and <=4 hops, and evaluate on the real-world benchmarks to report the average F1 score. E.g. hops <=1 contains only single-retrieval questions (e.g. "Who is the parent of X?", "Who is the person whose occupation is Y?").

In Section 4, our paper already reports <=0 hops (i.e. base model) and <=7 hops (i.e. + Synthetic). So this ablation study further gives us a plot of Avg. F1 vs training hops <= t. We have added this plot to Figure 5 (a) in Section 5; here we present the plot as a table. We find that increasing the number of training hops from 0 to 4 monotonically increases real-world performance from 22.1% to 43.0%, thus hop count does drive transfer. F1 scores for training hops <=4 and hops <=7 are similar, so further increasing environment complexity by increasing hop count does not lead to better performance on these benchmarks (hops in eval questions are generally <= 4).

ModelAvg. F1HotpotQA2WikiMuSiQueSynthWorlds-RMSynthWorlds-SMFRAMES
Qwen2.5-3B (paper)22.1 ± 0.640.130.321.917.79.912.6
+ Synthetic, training hops <=1 (new)24.4 ± 0.746.628.824.321.911.613.6
+ Synthetic, training hops <=2 (new)32.4 ± 0.748.636.936.231.324.117.6
+ Synthetic, training hops <=4 (new)43.0 ± 0.862.261.639.935.231.327.6
+ Synthetic, training hops <=7 (paper, seed=1)42.8 ± 0.761.556.041.035.934.827.6
+ NQ+HotpotQA (paper, seed=1)42.3 ± 0.865.962.541.437.625.121.3
  • Models are learning multi-turn search, not just search-tool-use format. At hops <= 1, the agent learns to call the search tool format yet task performance improves only marginally over the base model. Moreover, performance increases when we increase hops to 2, and then again with training hops up to 4.
  • Synthetic hops up to 4 ≈ real-world NQ+HotpotQA training. Evaluation benchmarks like SynthWorlds and FRAMES have >= 2-hop questions, so synthetic training hops <= 2 is not enough to match real-world training but <= 4 is sufficient in our synthetic environments.

Training environment details. We used the same universe of 1K individuals for the synthetic environments, and generated more questions with less number of hops, so that every row in the table has the same training budget of ≈55K questions. We trained with one seed due to time constraints, will add another in the final version.

Dear reviewer nVao, we hope to have addressed all your concerns with new results, especially on hop ablation. We have updated the PDF and highlighted changes.

Please let us know if you have any more questions. We’d appreciate it if you can raise your score. Thank you for your constructive suggestions.

Reviewer_BchWOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether LLM search agents can be trained using reinforcement learning in entirely rule-generated fictional environments, without relying on human-annotated or LLM-generated training data. The authors build on PhantomWiki to generate synthetic documents and multi-hop questions, and use GRPO to train agents with final-answer rewards. Experiments show improvements on real-world multi-hop QA benchmarks, suggesting that search skills learned in fictional environments can transfer to real-world tasks.

Critical strengths
  1. The idea of using fully rule-generated environments for agent training is interesting. It provides a scalable way to generate training tasks with verifiable answers without relying on expensive LLM-generated data.
  2. The analysis of different environment complexities is insightful. In particular, the finding that additional constraints can encourage shortcutting behavior highlights an important challenge in designing RL environments.
Critical weaknesses
  1. The real-world baseline is trained on NQ+HotpotQA, which mostly contains single-hop and 2-hop questions. Synthetic training includes up to 7 hops. Since Sec. 5 shows that hop count is the main factor affecting transfer, the gains might simply come from training on harder multi-hop questions rather than using fictional environments. Also, synthetic training only outperforms real-world training on OOD benchmarks for two out of four models.

  2. It is unclear whether the models actually learn generalizable search strategies or just learn the fixed patterns in the synthetic environment. Both documents and questions are generated using predefined templates and grammars, and even the unseen-universe evaluation in Fig. 3 follows the same generation rules. This makes it hard to tell how well the learned strategies would generalize to more diverse settings.

  3. The evaluation setting is simpler than real-world agentic search. Each benchmark uses a search corpus containing only gold and distractor documents, rather than a full Wikipedia index. It is unclear whether the improvements would hold with a much larger number of irrelevant documents. Evaluating on a full Wikipedia index and comparing against existing search agents like Search-R1 or ZeroSearch under the same retrieval setup would make the results more convincing.

Questions for authors
  1. How much of the performance gain comes from learning the search-tool protocol rather than multi-hop reasoning? Could the authors include a 1-hop synthetic RL baseline to isolate these effects?
3 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

Thank you, we appreciate your feedback. Indeed, fully rule-generated environments provide a scalable data generation pipeline for agent training without relying on costly LLMs. We are glad that you also find our analysis on environment complexities insightful. Addressing your comments below.

W1: The real-world baseline is trained on NQ+HotpotQA, which mostly contains single-hop and 2-hop questions. Synthetic training includes up to 7 hops …

Q1: How much of the performance gain comes from learning the search-tool protocol rather than multi-hop reasoning? Could the authors include a 1-hop synthetic RL baseline to isolate these effects?

Thanks, this is a good ablation suggestion. We RL fine-tune Qwen2.5-3B model on our synthetic environments, controlling the number of training hops in the environments at <=1, <=2 and <=4 hops, and evaluate on the real-world benchmarks to report the average F1 score. E.g. hops <=1 contains only single-retrieval questions (e.g. "Who is the parent of X?", "Who is the person whose occupation is Y?").

In Section 4, our paper already reports <=0 hops (i.e. base model) and <=7 hops (i.e. + Synthetic). So this ablation study further gives us a plot of Avg. F1 vs training hops <= t. We have added this plot to Figure 5 (a) in Section 5; here we present the plot as a table. We find that increasing the number of training hops from 0 to 4 monotonically increases real-world performance from 22.1% to 43.0%, thus hop count does drive transfer. F1 scores for training hops <=4 and hops <=7 are similar, so further increasing environment complexity by increasing hop count does not lead to better performance on these benchmarks (hops in eval questions are generally <= 4).

ModelAvg. F1HotpotQA2WikiMuSiQueSynthWorlds-RMSynthWorlds-SMFRAMES
Qwen2.5-3B (paper)22.1 ± 0.640.130.321.917.79.912.6
+ Synthetic, training hops <=1 (new)24.4 ± 0.746.628.824.321.911.613.6
+ Synthetic, training hops <=2 (new)32.4 ± 0.748.636.936.231.324.117.6
+ Synthetic, training hops <=4 (new)43.0 ± 0.862.261.639.935.231.327.6
+ Synthetic, training hops <=7 (paper, seed=1)42.8 ± 0.761.556.041.035.934.827.6
+ NQ+HotpotQA (paper, seed=1)42.3 ± 0.865.962.541.437.625.121.3
  • Models are learning multi-turn search, not just search-tool-use format. At hops <= 1, the agent learns to call the search tool format yet task performance improves only marginally over the base model. Moreover, performance increases when we increase hops to 2, and then again with training hops up to 4.
  • Synthetic hops up to ≈ real-world NQ+HotpotQA training. Evaluation benchmarks like SynthWorlds and FRAMES have >= 2-hop questions, so synthetic training hops <= 2 is not enough to match real-world training but <= 4 is sufficient in our synthetic environments.

Training environment details. We used the same universe of 1K individuals for the synthetic environments, and generated more questions with less number of hops, so that every row in the table has the same training budget of ≈55K questions. We trained with one seed due to time constraints, will add another in the final version.

W2: It is unclear whether the models actually learn generalizable search strategies or just learn the fixed patterns in the synthetic environment. Both documents and questions are generated using predefined templates and grammars, and even the unseen-universe evaluation in Fig. 3 follows the same generation rules. This makes it hard to tell how well the learned strategies would generalize to more diverse settings.

Table 2 directly tests transfer beyond the training generator: synthetic training improves all four models on independently constructed evaluation benchmarks. Figure 3 addresses a complementary question, showing generalization to new fictional universes generated under the same rules. We therefore base the claim of transfer beyond the generator on Table 2, while using Figure 3 to test dependence on the particular training universe. The new full-Wikipedia evaluation (below) provides additional evidence that the gains extend to searching in uncurated benchmark corpora.

AuthorsOpenReview ↗

Rebuttal (2/2)

W3: The evaluation setting is simpler than real-world agentic search. Each benchmark uses a search corpus containing only gold and distractor documents, rather than a full Wikipedia index. It is unclear whether the improvements would hold with a much larger number of irrelevant documents. Evaluating on a full Wikipedia index and comparing against existing search agents like Search-R1 or ZeroSearch under the same retrieval setup would make the results more convincing.

Good suggestion. We now add the official Search-R1 evaluation benchmark against the full Wikipedia index. This retrieval corpus contains 21M uncurated docs, which is ≈400x our pooled index.

We evaluate base and trained counterparts of Qwen2.5-7B and Llama-3.2-3B models, and find that both + Synthetic and + NQ+HotpotQA training environments improve agent performance: 2x improvement for Qwen2.5-7B and 3x for Llama-3.2-3B. For instance, Qwen2.5-7B goes from 23.3% accuracy to 43.7% with + Synthetic, and to 45.5% with + NQ+HotpotQA.

These results show that agents trained in our synthetic environments keep their gains when searching all of Wikipedia, not just the benchmark corpora. This adds to our discussion in Section 4.1 on robustness to large-scale deployment.

ModelAverageNQTriviaQAPopQAHotpotQA2WikiMuSiQueBamboogle
Qwen2.5-7B23.3 ± 0.531.742.928.021.516.66.516.0
+ Synthetic43.7 ± 0.743.961.448.741.740.323.846.4
+ NQ+HotpotQA45.5 ± 0.650.466.347.546.442.621.743.6
Llama-3.2-3B11.4 ± 0.415.720.010.79.19.12.712.8
+ Synthetic36.3 ± 0.747.859.444.831.120.813.136.8
+ NQ+HotpotQA39.4 ± 0.847.163.745.638.636.913.030.8

Real vs synthetic training. While real-world training is slightly better than synthetic, we find the jump from base -> trained agent as the main takeaway here. In fact, in Section 4.2 we had already observed that real-world training is better than synthetic, when training-test distributions overlap. The Search-R1 Wikipedia-2018 evaluation corpus is the same as that of NQ+HotpotQA, our real-world training environment. Search-R1 evaluation questions also share factual knowledge overlap to the real-world training set.

Search-R1 evaluation setup. We follow the Search-R1 paper setup: (1) official validation split of ≈51K questions, (2) 21M docs from 2018 Wikipedia encoded with the e5-base-v2 model, (3) report the Exact Match accuracy % (EM = 100% if any predicted answer, after splitting model’s answer by commas, matches any gold answer). We use our paper’s evaluation prompts and infrastructure—up to 20 turns, 32K context, top-3 docs—that performs better than the official Search-R1 checkpoints and code. For every model x training combination, we had trained with two training seeds, so the EM scores here are averaged across two training seeds and all evaluation questions.

Dear reviewer BchW, we hope we addressed all your concerns with new results, on hop ablation and full Wikipedia index evaluation. We have highlighted the PDF with changes.

If your concerns are met, we’d appreciate it if you can raise your score. Thank you for your concrete experiment suggestions.

Reviewer_CYs1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper turns PhantomWiki fictional universes into an LLM-free RL search environment. A universe has 1K people with templated articles; CFG questions have up to 7 hops and Prolog-verified answers. Training uses the Search-R1 recipe on Qwen2.5-3B/7B, Llama-3.2-3B, and Phi-4-mini. Evaluation covers HotpotQA, 2Wiki, MuSiQue, SynthWorlds-RM/SM, and FRAMES. Each benchmark has its own closed corpus and is searched with a stronger setup.

Critical strengths
  1. A well-controlled existence proof that LLM-free, template-only worlds teach search behavior that transfers. Table 2 improves every cell across four model families.

  2. A matched-budget comparison with real data that reports where synthetic training loses, plus a useful negative result. Real NQ+HotpotQA training wins on the in-domain benchmarks for every model, by 4.5 to 14.1 F1.

Critical weaknesses
  1. The paper focuses on a vertical domain of fiction, which is too narrow to demonstrate the effectiveness and generalization of the methodology.

  2. The advantage of "newer" benchmarks over real data is narrower than the abstract and Sec. 4.2 suggest, and the memorization explanation is not tested.

  3. Sec. 5 does not test the abstract's claim that "hop count drives transfer", and the gain over the base model is not decomposed.

Questions for authors
  1. Please add hop-matched controls. For example, train on a PhantomWiki environment limited to 1–2 hops, and/or a real multi-hop training set with deeper chains at the same budget. If synthetic training still matches or beats real data on SynthWorlds-RM and FRAMES under these controls, W1 and W2 are largely resolved.

  2. App. A.2 says that several NQ+HotpotQA runs (Llama, Phi) and one Hops+Constraints seed degenerated, and that you report "the prior saved policy checkpoint with the best held-out set performance". Which held-out set is this: training-distribution validation questions or the evaluation benchmarks? How much of the 55K budget had the selected real-data checkpoints seen?

  3. Sec. 3.1 and App. A.4 say a question can have several gold answers. App. A.4.1 step 4, however, keeps a sample only if the Prolog query "returns a unique answer". What fraction of the training questions have more than one answer?

3 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

Thank you for the careful review and concrete ablation suggestions.

W1: The paper focuses on a vertical domain of fiction, which is too narrow to demonstrate the effectiveness and generalization of the methodology.

We read your concern as the following. By fictional worlds, we mean that the synthetic environments do not contain any real-world facts; all individuals and relationships and hobbies are made-up. Even so, the models learn transferable search behavior to the complex real world—this transfer is not obvious.

Moreover, training in our synthetic search environments guarantees that no factual knowledge leaks into evaluation. Yet the models generalize to human-written questions over real-world corpora, which is our key contribution. To further test generalization, we extend our evaluation to 7 additional QA benchmarks, retrieving from the full 2018 Wikipedia index (21M docs, 400x size of our pooled search index). We adopt the benchmark suite and retrieval setup of Search-R1 to enable direct comparison of synthetic and real-world training on their QA benchmarks.

We evaluate base and trained counterparts of Qwen2.5-7B and Llama-3.2-3B models, and find that both + Synthetic and + NQ+HotpotQA training environments improve agent performance: 2x improvement for Qwen2.5-7B and 3x for Llama-3.2-3B. For instance, Qwen2.5-7B goes from 23.3% accuracy to 43.7% with + Synthetic, and to 45.5% with + NQ+HotpotQA.

These results show that agents trained in our synthetic environments keep their gains when searching all of Wikipedia, not just the benchmark corpora. This adds to our discussion in Section 4.1 on robustness to large-scale deployment.

ModelAverageNQTriviaQAPopQAHotpotQA2WikiMuSiQueBamboogle
Qwen2.5-7B23.3 ± 0.531.742.928.021.516.66.516.0
+ Synthetic43.7 ± 0.743.961.448.741.740.323.846.4
+ NQ+HotpotQA45.5 ± 0.650.466.347.546.442.621.743.6
Llama-3.2-3B11.4 ± 0.415.720.010.79.19.12.712.8
+ Synthetic36.3 ± 0.747.859.444.831.120.813.136.8
+ NQ+HotpotQA39.4 ± 0.847.163.745.638.636.913.030.8

Real vs synthetic training. While real-world training is slightly better than synthetic, we find the jump from base -> trained agent as the main takeaway here. In fact, in Section 4.2 we had already observed that real-world training is better than synthetic, when training-test distributions overlap. The Search-R1 Wikipedia-2018 evaluation corpus is the same as that of NQ+HotpotQA, our real-world training environment. Search-R1 evaluation questions also share factual knowledge overlap to the real-world training set.

Search-R1 evaluation setup. We follow the Search-R1 paper setup: (1) official validation split of ≈51K questions, (2) 21M docs from 2018 Wikipedia encoded with the e5-base-v2 model, (3) report the Exact Match accuracy % (EM = 100% if any predicted answer, after splitting model’s answer by commas, matches any gold answer). We use our paper’s evaluation prompts and infrastructure—up to 20 turns, 32K context, top-3 docs—that performs better than the official Search-R1 checkpoints and code. For every model x training combination, we had trained with two training seeds, so the EM scores here are averaged across two training seeds and all evaluation questions.

W2: The advantage of "newer" benchmarks over real data is narrower than the abstract and Sec. 4.2 suggest, and the memorization explanation is not tested.

We fairly characterize the advantage in the text: in introduction lines 85-87 and Section 4.2 we remark that this advantage of synthetic over real data is narrow or absent for some models, and present for the Qwen models. In the abstract and Section 4.2 we have adjusted our writing to not make a blanket statement for all models: Synthetic training beats real data on the newer benchmarks only for the Qwen models. In the introduction and Section 4.2, we now present memorization as a hypothesis rather than an established explanation.

SynthWorlds RM vs SM pair is a clear test for knowledge memorization, according to the paper https://arxiv.org/abs/2510.24427. They propose the Knowledge Advantage Gap (RM accuracy - SM accuracy) test to find that LLMs with retrieval-augmented generation (RAG) indeed answer from memorized knowledge; we find the same phenomena with agentic search.

AuthorsOpenReview ↗

Rebuttal (2/2)

W3: Sec. 5 does not test the abstract's claim that "hop count drives transfer", and the gain over the base model is not decomposed.

Q1: Please add hop-matched controls. ... If synthetic training still matches or beats real data on SynthWorlds-RM and FRAMES under these controls, W1 and W2 are largely resolved.

Good idea! We present results of this new ablation study. We RL fine-tune Qwen2.5-3B model on our synthetic environments, controlling the number of training hops in the environments at <=1, <=2 and <=4 hops, and evaluate on the real-world benchmarks to report the average F1 score. E.g. hops <=1 contains only single-retrieval questions (e.g. "Who is the parent of X?", "Who is the person whose occupation is Y?").

In Section 4, our paper already reports <=0 hops (i.e. base model) and <=7 hops (i.e. + Synthetic). So this ablation study further gives us a plot of Avg. F1 vs training hops <= t. We have added this plot to Figure 5 (a) in Section 5; here we present the plot as a table. We find that increasing the number of training hops from 0 to 4 monotonically increases real-world performance from 22.1% to 43.0%, thus hop count does drive transfer. F1 scores for training hops <=4 and hops <=7 are similar, so further increasing environment complexity by increasing hop count does not lead to better performance on these benchmarks (hops in eval questions are generally <= 4).

ModelAvg. F1HotpotQA2WikiMuSiQueSynthWorlds-RMSynthWorlds-SMFRAMES
Qwen2.5-3B (paper)22.1 ± 0.640.130.321.917.79.912.6
+ Synthetic, training hops <=1 (new)24.4 ± 0.746.628.824.321.911.613.6
+ Synthetic, training hops <=2 (new)32.4 ± 0.748.636.936.231.324.117.6
+ Synthetic, training hops <=4 (new)43.0 ± 0.862.261.639.935.231.327.6
+ Synthetic, training hops <=7 (paper, seed=1)42.8 ± 0.761.556.041.035.934.827.6
+ NQ+HotpotQA (paper, seed=1)42.3 ± 0.865.962.541.437.625.121.3
  • Models are learning multi-turn search, not just search-tool-use format. At hops <= 1, the agent learns to call the search tool format yet task performance improves only marginally over the base model. Moreover, performance increases when we increase hops to 2, and then again with training hops up to 4.
  • Synthetic hops up to 4 ≈ real-world NQ+HotpotQA training. Evaluation benchmarks like SynthWorlds and FRAMES have >= 2-hop questions, so synthetic training hops <= 2 is not enough to match real-world training but <= 4 is sufficient in our synthetic environments.

Training environment details. We used the same universe of 1K individuals for the synthetic environments, and generated more questions with less number of hops, so that every row in the table has the same training budget of ≈55K questions. We trained with one seed due to time constraints, will add another in the final version.

Q2: App. A.2 says that several NQ+HotpotQA runs (Llama, Phi) and one Hops+Constraints seed degenerated, and that you report "the prior saved policy checkpoint with the best held-out set performance". Which held-out set is this: training-distribution validation questions or the evaluation benchmarks? How much of the 55K budget had the selected real-data checkpoints seen?

We agree that the wording was a bit loose, so we have added this clarification in the appendix. For the two degenerated NQ+HotpotQA runs, we picked the saved checkpoints right before collapse in training reward (we had been saving checkpoints at every 10% of total training steps). Llama model run had seen 30K of 55K questions, and Phi had seen 25K. For the one Hops+Constraints run, we used the training-distribution validation questions. The model had seen 48K questions. We never use the final evaluation benchmarks to pick checkpoints.

Instability appeared only for Phi and Llama, in both real-data and synthetic training, and never for Qwen models. As discussed in the appendix A.2, RL instability is common in practice. For our runs, we believe it originates from the open-source RL infrastructure, due to a mismatch between the sampler's (vLLM) and the trainer's log-probabilities: entropy spikes right after spikes in this log-prob mismatch.

Q3: Sec. 3.1 and App. A.4 say a question can have several gold answers. App. A.4.1 step 4, however, keeps a sample only if the Prolog query "returns a unique answer". What fraction of the training questions have more than one answer?

Sorry that’s a typo deep within the appendix; thanks for catching the inconsistency between the main text and appendix. Our generator does not discard questions with multiple answers, and we have removed the App. A.4.1 step 4. About 75% of the Hops training questions have more than one gold answer, 65% of Hops+Constraints and 36% of Hops+Comparisons.

Dear reviewer CYs1, we hope our clarifications and new hop ablation results address your points. For your convenience we also highlighted the PDF with changes.

We’d appreciate it if you can raise your score if all questions are resolved. Thank you for the good feedback.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response (1/2)

We thank all reviewers for their feedback. We ran new experiments to address the general concerns and summarize those results here. We have updated the manuscript incorporating feedback and new results.

1. Search-R1 evaluation over full Wikipedia index of 21M docs

Our six evaluation benchmarks’ retrieval corpora contain respective gold/distractor docs, which is limited in size. We reported an ablation in Table 3 on a “pooled” search index of 53K docs, up to 44x the size of some benchmarks’ corpora.

Several reviewers BchW, 9MHM, CYs1 suggested that this ablation is still on a curated corpora, and they would like evaluation on an uncurated Wikipedia index, like Search-R1 paper’s setup. We now add this evaluation set from the Search-R1 paper against the full Wikipedia index containing 21M uncurated docs, which is 400x size of our pooled index.

We evaluate base and trained counterparts of Qwen2.5-7B and Llama-3.2-3B models, and find that both + Synthetic and + NQ+HotpotQA training environments improve agent performance: 2x improvement for Qwen2.5-7B and 3x for Llama-3.2-3B. For instance, Qwen2.5-7B goes from 23.3% accuracy to 43.7% with + Synthetic, and to 45.5% with + NQ+HotpotQA.

These results show that agents trained in our synthetic environments keep their gains when searching all of Wikipedia, not just the benchmark corpora. This adds to our discussion in Section 4.1 on robustness to large-scale deployment.

ModelAverageNQTriviaQAPopQAHotpotQA2WikiMuSiQueBamboogle
Qwen2.5-7B23.3 ± 0.531.742.928.021.516.66.516.0
+ Synthetic43.7 ± 0.743.961.448.741.740.323.846.4
+ NQ+HotpotQA45.5 ± 0.650.466.347.546.442.621.743.6
Llama-3.2-3B11.4 ± 0.415.720.010.79.19.12.712.8
+ Synthetic36.3 ± 0.747.859.444.831.120.813.136.8
+ NQ+HotpotQA39.4 ± 0.847.163.745.638.636.913.030.8

Real vs synthetic training. While real-world training is slightly better than synthetic, we find the jump from base -> trained agent as the main takeaway here. In fact, in Section 4.2 we had already observed that real-world training is better than synthetic, when training-test distributions overlap. The Search-R1 Wikipedia-2018 evaluation corpus is the same as that of NQ+HotpotQA, our real-world training environment. Search-R1 evaluation questions also share factual knowledge overlap to the real-world training set.

Search-R1 evaluation setup. We follow the Search-R1 paper setup: (1) official validation split of ≈51K questions, (2) 21M docs from 2018 Wikipedia encoded with the e5-base-v2 model, (3) report the Exact Match accuracy % (EM = 100% if any predicted answer, after splitting model’s answer by commas, matches any gold answer). We use our paper’s evaluation prompts and infrastructure—up to 20 turns, 32K context, top-3 docs—that performs better than the official Search-R1 checkpoints and code. For every model x training combination, we had trained with two training seeds, so the EM scores here are averaged across two training seeds and all evaluation questions.

AuthorsOpenReview ↗

General response (2/2)

2. Ablation on training hops

Reviewers CYs1, BchW, nVao suggested that we add an ablation on # hops in training environments, to substantiate our claim that linear hops drive transfer. We RL fine-tune Qwen2.5-3B model on our synthetic environments, controlling the number of training hops in the environments at <=1, <=2 and <=4 hops, and evaluate on the real-world benchmarks to report the average F1 score.

In Section 4, our paper already reports <=0 hops (i.e. base model) and <=7 hops (i.e. + Synthetic). So this ablation study further gives us a plot of Avg. F1 vs training hops <= t. We have added this plot to Figure 5 (a) in Section 5; here we present the plot as a table. We find that increasing the number of training hops from 0 to 4 monotonically increases real-world performance from 22.1% to 43.0%, thus hop count does drive transfer. F1 scores for training hops <=4 and hops <=7 are similar, so further increasing environment complexity by increasing hop count does not lead to better performance on these benchmarks (hops in eval questions are generally <= 4).

ModelAvg. F1HotpotQA2WikiMuSiQueSynthWorlds-RMSynthWorlds-SMFRAMES
Qwen2.5-3B (paper)22.1 ± 0.640.130.321.917.79.912.6
+ Synthetic, training hops <=1 (new)24.4 ± 0.746.628.824.321.911.613.6
+ Synthetic, training hops <=2 (new)32.4 ± 0.748.636.936.231.324.117.6
+ Synthetic, training hops <=4 (new)43.0 ± 0.862.261.639.935.231.327.6
+ Synthetic, training hops <=7 (paper, seed=1)42.8 ± 0.761.556.041.035.934.827.6
+ NQ+HotpotQA (paper, seed=1)42.3 ± 0.865.962.541.437.625.121.3
  • Models are learning multi-turn search, not just search-tool-use format. At hops <= 1, the agent learns to call the search tool format yet task performance improves only marginally over the base model. Moreover, performance increases when we increase hops to 2, and then again with training hops up to 4.
  • Synthetic hops up to 4 ≈ real-world NQ+HotpotQA training. Evaluation benchmarks like SynthWorlds and FRAMES have >= 2-hop questions, so synthetic training hops <= 2 is not enough to match real-world training but <= 4 is sufficient in our synthetic environments.

Training environment details. We used the same universe of 1K individuals for the synthetic environments, and generated more questions with less number of hops, so that every row in the table has the same training budget of ≈55K questions. We trained with one seed due to time constraints, will add another in the final version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.