acceptodds
Under review as a conference paper at ICLR 2027

Retrieval is Cheap, Show Me the Code: Executable Multi-Hop Reasoning for Retrieval-Augmented Generation

Abstract

Retrieval-Augmented Generation (RAG) has become a standard approach for knowledge-intensive question answering, but existing systems remain brittle on multi-hop questions, where solving the task requires chaining multiple retrieval and reasoning steps. Key challenges are that current methods represent reasoning through free-form natural language, where intermediate states are implicit, retrieval queries can drift from intended entities, and errors are detected by the same model that produces them making self-reflection an unreliable, ungrounded signal. We observe that multi-hop question answering is a typical form of step-by-step computation, and that this structured process aligns closely with how code-specialized language models are trained to operate. Motivated by this, we introduce \pyrag, a framework that reformulates multi-hop RAG as program synthesis and execution. Instead of free-form reasoning trajectories, \pyrag represents the reasoning process as an executable Python program over retrieval and QA tools, exposing intermediate states as variables, producing deterministic feedback through execution, and yielding an inspectable trace of the entire reasoning process. This formulation further enables compiler-grounded self-repair and execution-driven adaptive retrieval without any additional training. Experiments on five QA benchmarks (PopQA, HotpotQA, 2WikiMultihopQA, MuSiQue, and Bamboogle) show that \pyrag consistently outperforms strong baselines under both training-free and RL-trained settings, with especially large gains on compositional multi-hop datasets.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large language models, Retrieve-augmented generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_s4NJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PyRAG, a framework that represents multi-hop RAG as an executable intermediate representation. A query-specific Python program invokes retrieval and QA tools, binds intermediate answers to variables, and connects subsequent steps through explicit data dependencies. The experimental results that covers five question-answering benchmarks show that PyRAG achieves well performance in both training-free and RL settings. Ablation studies and efficiency analysis further demonstrate the effectiveness of PyRAG.

Critical strengths
  1. It achieves competitive results in both training-free and RL-trained settings, improving average EM by 11.8 points over Vanilla RAG and outperforming Search-R1 and StepSearch. The improvements across different datasets and model families also show good generalization.
  2. Using explicit variables and executable programs makes intermediate reasoning steps easier to inspect. The ablation and efficiency experiments show the benefits of different modules and reductions in both token usage as well as latency compared with Search-R1.
Critical weaknesses
  1. The main ideas such as explicit variables and executable tool calls, have already been explored in ReWOO, LLMCompiler, and CodeAct. The paper does not clearly explain what is fundamentally new, and lacks direct comparisons with these works.
  2. Table 4 shows that Basic Execution only improves EM from 25.8% to 28.5%, while larger gains come from adaptive retrieval and self-repair. It is therefore unclear how much improvement is due to executable representation rather than these extra mechanisms.
Questions for authors
  1. For the 3B backbone, PyRAG-RL only outperforms StepSearch by 0.3 EM points (29.9 vs. 29.6), while performing worse on HotpotQA and MuSiQue. Could the authors report standard deviations or significance tests to show whether this improvement is reliable?
  2. Bamboogle contains only 125 examples, yet some of the largest improvements are reported on this dataset. Could the authors provide per-run scores or confidence intervals to show how stable these gains are?
  3. Figure 6 shows that missing retrieval evidence accounts for roughly half of the 100 sampled failures on HotpotQA. How often does the sentinel-based adaptive retrieval detect these cases, and what fraction of the triggered re-retrievals successfully recover the correct answer?
4 comments
AuthorsOpenReview ↗

Rebuttal Part 1

Thank you for your valuable feedback. We address each point below:

W1: We thank the reviewer for highlighting these closely related studies. Our existing related work (Appendix C and C.1) discusses several relevant research directions: iterative multi-hop retrieval methods such as Self-Ask, IRCoT, and ITER-RETGEN; search-policy learning methods including Search-R1, R1-Searcher, and StepSearch; program-guided reasoning approaches such as PAL, Program-of-Thoughts, Binder, Faithful-CoT, Logic-LM, and ProgramFC; and LM pipeline optimization through DSPy. These discussions position PyRAG in relation to retrieval-based reasoning, executable and symbolic computation, and pipeline optimization.

Nevertheless, our coverage of this rapidly developing field is incomplete, and we apologize for omitting an explicit discussion of ReWOO, LLMCompiler, and CodeAct. We appreciate the reviewer bringing these relevant connections to our attention and will incorporate them into the revised manuscript. Below, we clarify how PyRAG differs from these approaches in its reasoning formulation, reinforcement-learning-based policy optimization, and systematic empirical analysis.

ReWOO. ReWOO separates reasoning from observations: a Planner generates a plan with evidence placeholders, Workers execute the corresponding tool calls, and a Solver integrates the resulting evidence into a final answer. PyRAG expresses both evidence acquisition and answer composition within a query-specific executable program. Intermediate answers are bound to runtime variables and used in subsequent retrieval queries and native Python operations. When composition has well-defined semantics, such as arithmetic, comparison, or Boolean operations, it is performed directly by the program; an answer model remains available for semantic synthesis. Our focus therefore extends beyond separating planning from tool execution to explicitly representing and executing computations over retrieved information.

LLMCompiler. LLMCompiler generates a dependency graph of function calls and schedules their execution in parallel, with support for replanning based on intermediate results. PyRAG focuses on the executable representation of the multi-hop reasoning procedure: retrieval, evidence-conditioned answering, and computations over intermediate answers are expressed within the same program. Whereas dependency-aware scheduling is central to LLMCompiler, our study examines how this program representation supports intermediate-state management, answer composition, and targeted recovery. These directions are complementary, and parallel scheduling could also be incorporated into PyRAG.

CodeAct. CodeAct represents agent actions as executable Python code and trains CodeActAgent through instruction tuning on multi-turn interaction trajectories. Beyond adopting executable code, PyRAG-RL explicitly optimizes the policies responsible for decomposition, program generation, and evidence-conditioned answering through reinforcement learning. Specifically, we use GRPO with role-specific LoRA adapters, training the Answer, Plan, and Decompose roles sequentially while freezing the other roles at each stage. Rewards are derived from the final QA outcome of the executed pipeline, using a combination of answer F1 and EM. This allows policy optimization to be guided by downstream task success rather than solely by imitation of demonstrated interaction trajectories. We distinguish this RL extension from the training-free PyRAG framework, which can be used without additional optimization.

We reproduced ReWOO, LLMCompiler, and CodeAct using their official implementations adapted to our retrieval setup. ProgramFC is designed and evaluated for complex-claim fact-checking on HOVER and FEVEROUS-S. Its analysis experiments also focus on fact-checking accuracy, evidence retrieval, and reasoning-program errors. Although its execution pipeline includes a question-answering submodule, the paper does not report a standalone evaluation on open-domain QA benchmarks. We therefore discuss ProgramFC as a methodological predecessor and prioritize ReWOO, LLMCompiler, and CodeAct for our additional empirical comparisons.

Shared settings. All LLM components use Qwen2.5-7B-Instruct, served through vLLM with tensor parallel size 2. The three baselines and training-free PyRAG use this checkpoint without additional fine-tuning. We use the same E5-base dense retriever over the Wikipedia 2018 corpus and evaluate on identical HotpotQA, 2WikiMQA, MuSiQue, and Bamboogle splits, containing 7,405, 12,576, 2,417, and 125 questions, respectively. All LLM calls use temperature 0.7, top-p 1.0, and a maximum of 16,384 generated tokens per call. Each baseline retrieval call returns the top-5 passages without additional passage truncation, and no cumulative passage cap is imposed per question. We retain the original prompt templates for the three baselines.

AuthorsOpenReview ↗

Rebuttal Part 2

ReWOO. We retain the planner–worker–solver workflow and evidence placeholders. The planner and solver both use Qwen2.5-7B-Instruct, and the toolset contains only the retrieval tool connected to the shared E5-base service.

LLMCompiler. We retain dependency-aware planning, task execution, and final answer aggregation. Both the planner and joiner use Qwen2.5-7B-Instruct, and the available tool set contains only search, backed by the shared retrieval service.

CodeAct. We retain the iterative Python-action execution loop, with Qwen2.5-7B-Instruct generating code actions and receiving execution feedback. The only external tool available to the Python agent is search, backed by the same retrieval service.

PyRAG. We use the configuration described in the paper, including top-5 retrieval, adaptive expansion to top-10 for insufficient-evidence responses, and up to three runtime self-repair rounds. We additionally report PyRAG-RL with the role-specific adapters described in Appendix E.1.

Backbone: Qwen2.5-7B-Instruct. EM (%) ± SD (percentage points)

MethodHotpotQA2WikiMQAMuSiQueBamboogleAvg.
ReWOO31.6 ± 0.623.8 ± 0.97.3 ± 0.523.2 ± 2.421.5 ± 1.1
LLMCompiler32.7 ± 1.127.5 ± 0.810.6 ± 0.428.8 ± 2.924.9 ± 1.3
CodeAct33.7 ± 1.331.2 ± 0.79.3 ± 1.232.0 ± 3.526.6 ± 1.6
PyRAG w/o Training34.0 ± 2.333.4 ± 2.011.8 ± 0.741.6 ± 4.030.2 ± 1.4
PyRAG-RL40.5 ± 2.949.4 ± 2.620.7 ± 1.346.4 ± 4.239.3 ± 1.9

Training-free PyRAG achieves the highest mean EM on all four benchmarks among the training-free methods, with a macro-average of 30.2, compared with 26.6 for CodeAct, the strongest baseline by macro-average EM. The margin over the best baseline varies across datasets, from 0.3 points on HotpotQA (34.0 vs. 33.7) to 9.6 points on Bamboogle (41.6 vs. 32.0). These comparisons evaluate the full systems under a shared backbone and retrieval environment, with each method retaining its own execution and recovery procedures. PyRAG-RL reaches 39.3 macro-average EM with role-specific RL training.

Moreover, our contribution also includes an empirical investigation of when and how the framework is effective. In Section 3.3 and Figure 5, we compare Qwen2.5-7B-Instruct with Qwen2.5-Coder-7B-Instruct. Code specialization has little effect on Vanilla RAG in this comparison, whereas it improves PyRAG by 1.8, 6.9, and 2.0 EM points on HotpotQA, 2WikiMQA, and Bamboogle, respectively. Meanwhile, PyRAG already improves substantially over Vanilla RAG with the instruction-tuned backbone. These findings indicate that the framework’s benefits do not require a code-specialized model, while code specialization can provide additional gains.

We further analyze structured versus executable representations, the contributions of adaptive retrieval and self-repair, fixed versus adaptive retrieval budgets, token consumption and latency, and residual failure modes. Together, these analyses characterize the framework’s effectiveness, computational trade-offs, and remaining limitations. We will distinguish the evidence for the complete system from the narrower evidence for individual mechanisms; in particular, the full-system improvement should not be interpreted as an execution-only effect.

Accordingly, our contribution combines an executable formulation of multi-hop RAG, outcome-driven policy optimization, and systematic empirical analysis. We will revise the related work and contribution statements to make these distinctions explicit, while acknowledging the foundational ideas shared with prior executable-agent frameworks.

W2:

Thank you for this comment. We would like to clarify the distinction between the two reference conditions. In Table 4, Basic Execution improves average EM from 21.3 (Vanilla RAG) to 28.5, yielding a substantial gain of 7.2 percentage points without either adaptive retrieval or self-repair. The value of 25.8 refers instead to the Structured IR condition in Figure 3, which already incorporates typed operations, named outputs, and explicit dependencies.

These comparisons address different questions. The improvement from 21.3 to 28.5 demonstrates the effectiveness of the basic program-based reasoning pipeline before either refinement mechanism is introduced. The improvement from 25.8 to 28.5 measures the additional benefit of execution relative to an already structured representation. Consequently, the latter comparison should not be interpreted as the entire contribution of the basic PyRAG framework.

Table 4 further shows that adding adaptive retrieval or self-repair individually raises performance from 28.5 to 31.5 or 32.4, respectively, while combining them achieves 36.3. These results support a substantial contribution from the basic framework, with further improvements from the two recovery mechanisms.

AuthorsOpenReview ↗

Rebuttal Part 3

We will clarify these reference conditions and distinguish the benefit of the basic executable reasoning pipeline from the narrower incremental benefit of execution over Structured IR. We do not attribute the entire 7.2-point improvement to the interpreter alone, but the ablation clearly shows that the framework provides substantial gains even when both refinement mechanisms are disabled.

Q1:

Supplement to Table 2 — Training-free PyRAG

EM (%) ± SD (percentage points).

BackbonePopQAHotpotQA2WikiMQAMuSiQueBamboogleAvg.
Qwen2.5-7B-Instruct33.5 ± 1.534.0 ± 2.333.4 ± 2.011.8 ± 0.741.6 ± 4.030.9 ± 0.9
Qwen2.5-72B-Instruct45.5 ± 2.552.0 ± 1.844.4 ± 2.816.9 ± 1.145.6 ± 3.240.9 ± 1.2

Supplement to Table 3 — PyRAG-RL

EM (%) ± SD (percentage points).

BackboneHotpotQA2WikiMQAMuSiQueBamboogleAvg.
Qwen2.5-7B-Instruct40.5 ± 2.949.4 ± 2.620.7 ± 1.346.4 ± 4.239.3 ± 1.9
Qwen2.5-3B-Instruct34.2 ± 3.232.5 ± 3.516.8 ± 0.836.0 ± 4.529.9 ± 1.7
Qwen3-4B-Instruct38.4 ± 2.245.1 ± 3.018.6 ± 0.643.2 ± 3.736.3 ± 1.4
LLaMA-3.1-8B-Instruct43.2 ± 2.750.1 ± 2.122.1 ± 2.448.0 ± 4.940.9 ± 1.6

Thank you for raising the question of statistical reliability. Across three runs, PyRAG-RL-3B achieves 29.9 ± 1.7 EM (mean ± sample standard deviation) when macro-averaged over the four benchmarks. The supplement to Table 3 reports the corresponding statistics for each benchmark. StepSearch reports a macro-average EM of 29.6. PyRAG-RL scores higher on 2WikiMQA and Bamboogle, while StepSearch scores higher on HotpotQA and MuSiQue.

The two methods differ in both training data and supervision. StepSearch trains on MuSiQue, while PyRAG-RL follows the Search-R1 training setup and uses single-hop Natural Questions and multi-hop HotpotQA. MuSiQue is therefore an in-domain benchmark for StepSearch and an out-of-domain benchmark for PyRAG-RL.

StepSearch also uses GPT-4o to enrich decomposed question–answer pairs and generate candidate search queries for individual reasoning steps. Its training objective combines final-answer feedback with a reward for matching reference queries and per-round retrieval rewards based on information gain and redundancy. PyRAG-RL optimizes its role-specific policies using only the final-answer reward after executing the generated program. The similar average EM shows that PyRAG-RL achieves comparable aggregate performance at this scale with outcome-level supervision alone.

Given that we have only published aggregate scores for StepSearch, we report the 0.3-point difference as a descriptive comparison.

Q2:

MethodBackboneRun 1 EM (%)Run 2 EM (%)Run 3 EM (%)Mean EM (%) ± sample SD (pp)
Training-free PyRAGQwen2.5-7B-Instruct37.645.641.641.6 ± 4.0
PyRAG-RLQwen2.5-7B-Instruct44.051.244.046.4 ± 4.2

Thank you for highlighting the small size of Bamboogle. With 125 questions, a single additional correct answer changes EM by 0.8 percentage points. Across three runs, PyRAG-RL with Qwen2.5-7B-Instruct achieves 46.4 ± 4.2 EM (mean ± sample standard deviation). This standard deviation describes run-to-run variability for PyRAG-RL.

In the training-free setting, 7B PyRAG achieves 41.6 EM, compared with 29.6 EM for ITER-RETGEN, an observed difference of 12.0 percentage points on Bamboogle. Estimating a paired confidence interval for this difference would require question-level predictions from both methods.

Q3: Thank you for asking how often the sentinel detects missing retrieval evidence and how often re-retrieval recovers the answer. Figure 6 characterizes errors among 100 incorrect final predictions on HotpotQA. To measure detection and recovery, we examined 100 HotpotQA sub-steps for which the initial top-5 passages were judged insufficient to answer the sub-question.

The Answer Agent returned a sentinel in 64 of these 100 sub-steps, yielding a detection recall of 64%. Each sentinel response automatically triggered an expansion from top-5 to top-10. Of the 64 triggered re-retrievals, 9 produced a correct sub-answer, corresponding to a 14.1% recovery rate per triggered step.

These measurements complement the question-level ablation in Table 4, where adding adaptive retrieval to Basic Execution improves HotpotQA EM from 31.5 to 32.6. We will add the detection and recovery counts to the revised analysis and specify the denominator for each rate.

Once again, thank you for your thorough review. We hope these points can address your concerns.

Thank you for the detailed response and the supplementary experiments.

The additional comparisons with ReWOO, LLMCompiler, and CodeAct are helpful and provide further support the effectiveness of the PyRAG framework. However, these comparisons are mainly conducted at the system level, and the methods differ in their adaptive retrieval and error recovery mechanisms. Consequently, the observed performance gains cannot be fully attributed to the executable inference representation itself.

The authors have further clarified the 2.7 EM improvement from Structured IR to Basic Execution. I acknowledge this distinction, but there is still a lack of direct comparison of executable and non-executable representations under the same retrieval extension and retry conditions. As a result, the specific contribution of the proposed representation remains insufficiently isolated.

The newly reported standard deviations and multiple-run results on Bamboogle provide additional evidence of experimental stability. However, among the 100 sub-steps with insufficient evidence, 64 trigger re-retrieval, but only 9 are successfully recovered.

Overall, I still have reservations about the independent contribution of the executable representation and its technical novelty compared with existing methods. Therefore, I maintain my original score.

Reviewer_svuxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PyRAG, which reformulates multi-hop RAG as program synthesis followed by execution. The Decompose Agent breaks the question down into its atomic sub-queries while the Plan Agent composes a Python program from two tool primitives, retrieval and answering. The interpreter then executes the program, binding intermediate answers to variables that later steps combine through explicit data flow or native Python operations. Two training-free refinements are added. One is compiler-based self-repair, which re-prompts the planner with the traceback for as many as three rounds should the program throw an exception. The other is execution-driven adaptive retrieval, which re-runs a step with top-k raised from 5 to 10 if answer() yields a sentinel like “unknown”. As for PyRAG-RL, it fine-tunes the three roles through GRPO and role-specific LoRA adapters, with the Answer, Plan and Decompose adapters trained in that order. The method is validated with Exact Match on PopQA, HotpotQA, 2WikiMQA, MuSiQue and Bamboogle under the Search-R1 retrieval setup. In the training-free setting of Table 2, PyRAG with Qwen2.5-7B-Instruct averages 30.9 EM, compared with 26.2 for ITER-RETGEN and 19.0 for Vanilla RAG, and the same trend holds at 72B. In the RL setting of Table 3, PyRAG-RL reaches an average of 39.3 at 7B, compared with 34.5 for StepSearch and 32.4 for Search-R1, and similar results are reported on Qwen3-4B and LLaMA-3.1-8B. The ablations in Figure 3 and Table 4 separate the structured plan, basic execution, adaptive retrieval and self-repair, and Figure 4 reports roughly 61 to 67 percent fewer tokens and 25 to 32 percent lower latency than Search-R1. The appendix adds a retrieval-budget-matched control, an LLM-as-a-judge check, a failure analysis and the full prompts.

Critical strengths
  1. The formulation is clean and inspectable, and it delivers tangible gains exactly where the method is meant to help. Converting the multi-hop chain into a program with persistent variables is a sensible way to counter entity drift, and the largest gains indeed appear on the compositional datasets. In the training-free setting at 7B, PyRAG improves on Vanilla RAG by 14.5 points on 2WikiMQA and by 25.6 on Bamboogle. PyRAG-RL at 7B reaches 49.4 and 46.4 EM on the same two datasets, well above Search-R1 and StepSearch. The improvement holds in both the training-free and RL settings. The approach is also practically appealing, since it uses far fewer tokens and less latency than Search-R1 despite making slightly more LLM calls (Figure 4).

  2. The appendix is thorough on both reproducibility and diagnostics. All prompts and repair templates are given verbatim. The retrieval-budget-matched control in Table 6, where full PyRAG reaches 41.6 EM with 6.2 documents per query against 39.2 EM with a fixed k=10, pre-empts the obvious objection that the method simply reads more documents. The failure analysis and the worked failure cases F1 to F5 are honest about what execution does not fix. This makes the contribution easy to verify and to build on.

Critical weaknesses
  1. The main comparisons omit several relevant baselines, and where a comparison is made, the margin over prior methods is often small. Table 3 reports ZeroSearch, Search-R1 and StepSearch only for Qwen2.5-3B and 7B. For Qwen3-4B and LLaMA-3.1-8B, the only baselines are the authors’ own RAG-SFT and RAG-RL. The claim that PyRAG-RL transfers across model families is therefore never tested against an external RL baseline, even though Search-R1 is open source and could have been run. PopQA appears in Table 2 but is missing from Table 3 without explanation, even though Search-R1 reports it. The gains over the external baselines are modest. PyRAG-RL at 3B is 0.3 points ahead of StepSearch on average but behind on HotpotQA (34.2 vs. 34.5) and MuSiQue (16.8 vs. 17.4). The 7B model is also behind on MuSiQue (20.7 vs. 22.6). In the training-free setting, the 7B model gains roughly one point over IRCoT on PopQA and HotpotQA (33.5 vs. 32.6 and 34.0 vs. 32.7), and the same pattern holds at 72B (45.5 vs. 44.6 and 52.0 vs. 50.9). The PyRAG figures are means over three runs, but no standard deviation or confidence interval is reported, and the external RL baselines in Table 3 are single numbers copied from their original papers. Taken together, the evidence that the executable representation beats the strongest existing methods, rather than Vanilla RAG, is weaker than the abstract suggests.

  2. The ablation does not isolate the stated contribution, and the related work omits the most pertinent prior formulations. In Figure 3, the Free-form Plan numbers are identical to Vanilla RAG (28.9, 18.9 and 16.0, averaging 21.3). Table 4 confirms that the 7.2-point gain is measured against Vanilla RAG. There is therefore no like-for-like comparison between the executable representation and a free-form multi-step plan. ITER-RETGEN scores 32.5, 28.9 and 29.6 on the three datasets (average 30.3). PyRAG with basic execution is lower on every dataset at 31.5, 28.3 and 25.6 (average 28.5), and the adaptive retrieval variant merely draws level at an average of 31.5. The authors themselves note in Section 3.3 that the non-executable baseline has no matched retrieval-expansion or retry mechanism, and the same applies to ITER-RETGEN and IRCoT, so the margin comes from adaptive retrieval and self-repair rather than from the representation. Self-repair raises a further question. It yields an average gain of 3.9 points and 8.0 on Bamboogle, which is hard to reconcile with Appendix E.3, where program errors account for only 5 percent of failures and genuine Python exceptions for under 20 percent of those program errors. Finally, binding tool outputs to variables in a planner-generated program is the core idea of ReWOO (Xu et al., 2023), LLMCompiler (Kim et al., 2024) and CodeAct (Wang et al., 2024), none of which is discussed or compared against, and this limits the novelty of the representational claim.

References:

  • Xu, B., Peng, Z., Lei, B., Mukherjee, S., Liu, Y., and Xu, D. (2023). ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Models. arXiv:2305.18323.

  • Kim, S., Moon, S., Tabrizi, R., Lee, N., Mahoney, M. W., Keutzer, K., and Gholami, A. (2024). An LLM Compiler for Parallel Function Calling. In Proceedings of the 41st International Conference on Machine Learning (ICML 2024).

  • Wang, X., Chen, Y., Yuan, L., Zhang, Y., Li, Y., Peng, H., and Ji, H. (2024). Executable Code Actions Elicit Better LLM Agents. In Proceedings of the 41st International Conference on Machine Learning (ICML 2024).

Questions for authors

The following are clarification requests, and the answers would affect my assessment.

  1. Could you complete Table 3 with Search-R1, and ideally StepSearch and ZeroSearch, on Qwen3-4B and LLaMA-3.1-8B, and add PopQA to Table 3? If these runs were attempted and then omitted, please explain why.

  2. Is the Free-form Plan condition in Figure 3 literally the Vanilla RAG system? If so, please re-run Figure 3 and Table 4 with ITER-RETGEN or IRCoT as the free-form multi-step baseline, and explain why basic execution should be credited with a gain from execution when its average of 28.5 is below the 30.3 that ITER-RETGEN reaches on the same three datasets (Tables 2 and 4).

  3. Please report standard deviations or confidence intervals over the three runs for all PyRAG rows in Tables 2 and 3. Please also run the strongest training-free baseline, ITER-RETGEN, with matched refinements, namely a k=10 fallback on unknown or cannot-answer responses and up to three repair retries, so that the gain attributable to the executable representation alone can be read off. Table 6 provides such a control only for Vanilla RAG at a fixed budget.

  4. How does the training cost of the three-stage curriculum, with three LoRA adapters trained in separate GRPO runs, compare with Search-R1 and StepSearch in rollouts or GPU-hours?

  5. Could you discuss ReWOO, LLMCompiler and CodeAct, which also plan tool calls with variable dependencies, and, if feasible, compare PyRAG against them?

3 comments
AuthorsOpenReview ↗

Rebuttal Part 1

Thank you for your valuable feedback. We address each point below:

W1: Thank you for raising these concerns. We will clarify the scope of the comparisons in Tables 2 and 3 and report run-to-run variability for PyRAG.

Backbone selection and baseline coverage. Most external RL baselines in Table 3 report results for Qwen2.5-3B and Qwen2.5-7B-Instruct, which explains the broader baseline coverage for these backbones. To evaluate PyRAG-RL beyond Qwen2.5, we also tested Qwen3-4B-Instruct and LLaMA-3.1-8B-Instruct, training RAG-SFT and RAG-RL controls on each corresponding backbone. PyRAG-RL improves over both controls on these backbones. We will clarify the scope of this comparison in the revised manuscript. Matched retrieval budget. We evaluated Search-R1-3B and PyRAG-RL-3B on the same HotpotQA questions using the same retriever, corpus, and EM scoring protocol. Both systems operated under a shared cap of 20 retrieved passages per question, counted cumulatively across retrieval calls, including expansions and retries. Under this cap, PyRAG-RL-3B achieves 32.1 EM, compared with 28.6 EM for Search-R1-3B, a difference of 3.5 percentage points.

PopQA. We include PopQA in Table 2 as an out-of-domain, single-hop benchmark for the training-free setting. Table 3 focuses on multi-hop QA to assess the retrieval and reasoning capabilities targeted by PyRAG-RL. We will make this evaluation scope explicit in the Table 3 caption.

Training-free performance. Across the five benchmarks in Table 2, PyRAG achieves a macro-average EM of 30.9 at 7B and 40.9 at 72B. ITER-RETGEN, the strongest baseline by macro-average EM at both scales, achieves 26.2 and 36.3, respectively. The average gains are 4.7 and 4.6 percentage points, with smaller margins over IRCoT on PopQA and HotpotQA. We will revise the abstract to identify these gains as macro-averages and clarify that they vary across benchmarks.

RL margins and uncertainty. Across three runs, PyRAG-RL achieves four-benchmark macro-average EM scores of 29.9 ± 1.7 at 3B and 39.3 ± 1.9 at 7B (mean ± sample standard deviation). The supplements to Tables 2 and 3 report the corresponding statistics for each benchmark and backbone. At 3B, PyRAG-RL is competitive with StepSearch, which reports 29.6 average EM and scores higher on HotpotQA and MuSiQue. We treat the 0.3-point gap as descriptive given the available aggregate results for StepSearch. At 7B, PyRAG-RL achieves 39.3 average EM versus 34.5 for StepSearch, a 4.8-point difference, while StepSearch scores higher on MuSiQue.

Supplement to Table 2 — Training-free PyRAG

EM (%) ± SD (percentage points).

BackbonePopQAHotpotQA2WikiMQAMuSiQueBamboogleAvg.
Qwen2.5-7B-Instruct33.5 ± 1.534.0 ± 2.333.4 ± 2.011.8 ± 0.741.6 ± 4.030.9 ± 0.9
Qwen2.5-72B-Instruct45.5 ± 2.552.0 ± 1.844.4 ± 2.816.9 ± 1.145.6 ± 3.240.9 ± 1.2

Supplement to Table 3 — PyRAG-RL

EM (%) ± SD (percentage points).

BackboneHotpotQA2WikiMQAMuSiQueBamboogleAvg.
Qwen2.5-7B-Instruct40.5 ± 2.949.4 ± 2.620.7 ± 1.346.4 ± 4.239.3 ± 1.9
Qwen2.5-3B-Instruct34.2 ± 3.232.5 ± 3.516.8 ± 0.836.0 ± 4.529.9 ± 1.7
Qwen3-4B-Instruct38.4 ± 2.245.1 ± 3.018.6 ± 0.643.2 ± 3.736.3 ± 1.4
LLaMA-3.1-8B-Instruct43.2 ± 2.750.1 ± 2.122.1 ± 2.448.0 ± 4.940.9 ± 1.6

W2: Free-form Plan in Figure 3. The condition labeled “Free-form Plan” is Vanilla RAG. It retrieves evidence once using the original question, then generates the answer through multi-step reasoning in natural language. This explains why its scores are identical to the Vanilla RAG row in Table 4. We will relabel it as “Vanilla RAG (single retrieval + textual reasoning)”. The 7.2-point gain over this baseline measures the combined effect of structured planning and basic execution. The comparison with Structured IR gives a narrower estimate of 2.7 points with both refinements disabled.

AuthorsOpenReview ↗

Rebuttal Part 2

Comparison with ITER-RETGEN. We agree that Basic Execution scores below ITER-RETGEN on the three ablation datasets. Basic Execution is a component ablation that measures how PyRAG performs when adaptive retrieval and self-repair are disabled. The end-to-end comparison evaluates the complete PyRAG system, whose design integrates all three components. Its runtime can identify the specific QA step that returned an insufficient-evidence response and expand retrieval for that step. It can also pass an execution exception to the planner for targeted repair. Adding analogous behavior to ITER-RETGEN would require defining which parts of its textual reasoning trace can trigger an intervention and how to retry them while preserving the surrounding reasoning. Matching resource limits alone would leave those design choices open. We will clarify that the current comparison establishes the performance of the integrated system, while the gap under matched recovery policies remains unmeasured.

Self-repair and the failure analysis. Appendix E.3 analyzes HotpotQA questions whose final answers remained incorrect after execution. Its error proportions describe residual failures after repair, so cases successfully recovered by repair fall outside that sample. Table 4 measures the change in EM when self-repair is enabled on Bamboogle. Longer reasoning chains, such as those encountered on Bamboogle, give intermediate execution errors more opportunities to affect the final answer. This may help explain why repair yields a larger gain on Bamboogle. We will clarify the difference in datasets and sampling conditions in the revision.

Relationship to prior work. ReWOO separates planning from tool observations and uses the collected evidence to synthesize an answer. LLMCompiler uses a dependency graph to schedule tool calls and supports replanning. CodeAct uses executable Python as an action space and instruction-tunes CodeActAgent on multi-turn interaction trajectories. PyRAG shares the use of explicit intermediate state with these works. Its specific contribution is to integrate evidence acquisition, operations over intermediate answers, and targeted recovery within a query-specific program for multi-hop QA. PyRAG-RL additionally uses the final QA outcome after program execution to optimize the decomposition, planning, and answering policies. We will add these comparisons to the related-work section and clarify the scope of our contribution. Please also refer to our responses for Question 5 that we compare PyRAG with these baselines.

Q1: Please see our response to Weakness 1, where we explain the baseline coverage for Qwen3-4B and LLaMA-3.1-8B and why the RL evaluation in Table 3 focuses on multi-hop benchmarks.

Q2: Yes. The “Free-form Plan” condition in Figure 3 is Vanilla RAG. It retrieves evidence once for the original question and then performs multi-step reasoning in natural language. We will relabel it accordingly and include the existing ITER-RETGEN results from Table 2 as a separate iterative retrieval reference in Figure 3 and Table 4.

Basic Execution achieves 28.5 average EM, below ITER-RETGEN’s 30.3. The evidence for an incremental benefit from execution comes from the comparison with Structured IR: EM rises from 25.8 to 28.5 with adaptive retrieval and self-repair disabled in both conditions. This gain is specific to that ablation. Please see our response to Weakness 2 for the interpretation of the full-system comparison.

Q3: Thank you for requesting run-to-run statistics and a comparison under matched refinement policies.

We provided the supplements to Tables 2 and 3 report mean EM and sample standard deviations across three runs for every PyRAG and PyRAG-RL row, including each benchmark and the overall average. For the overall average, we compute the macro-average EM within each run and then calculate the sample standard deviation across the three run-level averages.

ITER-RETGEN performs multi-step reasoning in free-form text, which makes it difficult to identify and retry a specific reasoning step or expand retrieval at that step. PyRAG executes each step explicitly. When the Answer Agent returns unknown, the runtime expands retrieval for the affected sub-question. When execution raises an exception, the runtime passes the diagnostic to the planner so it can revise and retry the program. The executable representation thus provides a direct way to locate failures and intervene during reasoning.

AuthorsOpenReview ↗

Rebuttal Part 3

Q4: PyRAG training cost. We train three role-specific LoRA adapters sequentially on a shared frozen backbone. The Answer stage uses 32 prompts per iteration and samples 8 rollouts per prompt. The Plan and Decompose stages each use 64 prompts per iteration and sample 4 rollouts per prompt. Each stage therefore samples 256 rollouts per iteration. With approximately 100 iterations per stage, the three stages sample approximately 76,800 rollouts in total. The full training run takes approximately 12 hours on eight A100 GPUs, corresponding to 96 A100 GPU-hours, including rollout generation and policy updates.

Comparison with Search-R1 and StepSearch. Appendix B of Search-R1 reports 500 PPO training steps with a global batch size of 512 on eight H100 GPUs. Appendix B of StepSearch reports 500 training steps with a global batch size of 256 on sixteen H800 GPUs. Assuming each step samples one fresh trajectory per prompt in the global batch, these configurations imply approximately 256,000 rollouts for Search-R1 and 128,000 rollouts for StepSearch. These estimates compare the number of sampled trajectories. The compute required per trajectory can vary with generation length, retrieval interactions, and the policy update procedure.

Q5:

We reproduced ReWOO, LLMCompiler, and CodeAct using their official implementations adapted to our retrieval setup. ProgramFC is designed and evaluated for complex-claim fact-checking on HOVER and FEVEROUS-S. Its analysis experiments also focus on fact-checking accuracy, evidence retrieval, and reasoning-program errors. Although its execution pipeline includes a question-answering submodule, the paper does not report a standalone evaluation on open-domain QA benchmarks. We therefore discuss ProgramFC as a methodological predecessor and prioritize ReWOO, LLMCompiler, and CodeAct for our additional empirical comparisons.

Shared settings. All LLM components use Qwen2.5-7B-Instruct, served through vLLM with tensor parallel size 2. The three baselines and training-free PyRAG use this checkpoint without additional fine-tuning. We use the same E5-base dense retriever over the Wikipedia 2018 corpus and evaluate on identical HotpotQA, 2WikiMQA, MuSiQue, and Bamboogle splits, containing 7,405, 12,576, 2,417, and 125 questions, respectively. All LLM calls use temperature 0.7, top-p 1.0, and a maximum of 16,384 generated tokens per call. Each baseline retrieval call returns the top-5 passages without additional passage truncation, and no cumulative passage cap is imposed per question. We retain the original prompt templates for the three baselines. Each baseline is evaluated over three runs with different inference seeds, using the same answer normalization and EM-scoring procedure.

ReWOO. We retain the planner–worker–solver workflow and evidence placeholders. The planner and solver both use Qwen2.5-7B-Instruct, and the toolset contains only the retrieval tool connected to the shared E5-base service.

LLMCompiler. We retain dependency-aware planning, task execution, and final answer aggregation. Both the planner and joiner use Qwen2.5-7B-Instruct, and the available tool set contains only search, backed by the shared retrieval service.

CodeAct. We retain the iterative Python-action execution loop, with Qwen2.5-7B-Instruct generating code actions and receiving execution feedback. The only external tool available to the Python agent is search, backed by the same retrieval service.

PyRAG. We use the configuration described in the paper, including top-5 retrieval, adaptive expansion to top-10 for insufficient-evidence responses, and up to three runtime self-repair rounds. We additionally report PyRAG-RL with the role-specific adapters described in Appendix E.1.

Backbone: Qwen2.5-7B-Instruct. EM (%) ± SD (percentage points)

MethodHotpotQA2WikiMQAMuSiQueBamboogleAvg.
ReWOO31.6 ± 0.623.8 ± 0.97.3 ± 0.523.2 ± 2.421.5 ± 1.1
LLMCompiler32.7 ± 1.127.5 ± 0.810.6 ± 0.428.8 ± 2.924.9 ± 1.3
CodeAct33.7 ± 1.331.2 ± 0.79.3 ± 1.232.0 ± 3.526.6 ± 1.6
PyRAG w/o Training34.0 ± 2.333.4 ± 2.011.8 ± 0.741.6 ± 4.030.2 ± 1.4
PyRAG-RL40.5 ± 2.949.4 ± 2.620.7 ± 1.346.4 ± 4.239.3 ± 1.9

Training-free PyRAG achieves the highest mean EM on all four benchmarks among the training-free methods, with a macro-average of 30.2, compared with 26.6 for CodeAct, the strongest baseline by macro-average EM. The margin over the best baseline varies across datasets, from 0.3 points on HotpotQA (34.0 vs. 33.7) to 9.6 points on Bamboogle (41.6 vs. 32.0). These comparisons evaluate the full systems under a shared backbone and retrieval environment, with each method retaining its own execution and recovery procedures. PyRAG-RL reaches 39.3 macro-average EM with role-specific RL training.

We hope these points can address your concerns.

Reviewer_x3WmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

PyRAG addresses multi-hop retrieval-augmented question answering by generating and executing query-specific Python programs. Decomposition, planning, and answering roles use persistent variables and explicit data dependencies, supported by runtime self-repair and adaptive retrieval. Experiments on five QA benchmarks report improvements in training-free and RL-trained settings, with additional ablations, backbone comparisons, and efficiency analyses.

Critical strengths
  1. The executable interface connects retrieval, answering, and deterministic computation through explicit intermediate variables, making reasoning states and failures easier to inspect.
  2. The evaluation covers multiple benchmarks, model sizes, and training settings. The appendices provide detailed prompts, implementation settings, and informative failure analyses.
Critical weaknesses
  1. Technical novelty is limited. The distinction from ProgramFC [1], CodeAct [2], and LLMCompiler [3] remains unclear, particularly regarding executable plans, intermediate variables, and dependency-aware tool use. Please identify the new mechanisms and include a closely related executable-agent baseline.

    [1] Fact-Checking Complex Claims with Program-Guided Reasoning

    [2] Executable Code Actions Elicit Better LLM Agents

    [3] An LLM Compiler for Parallel Function Calling

  2. The contribution of execution remains only partially isolated. Basic execution improves over Structured IR by 2.7 EM, while runtime refinements add another 7.8 points. However, the non-executable control lacks matched retrieval expansion and retry mechanisms. Please specify the Structured IR implementation and compare both representations under matched refinement policies and budget limits.

  3. RL baseline comparability needs clarification. Table 3 imports published results while claiming identical retrieval settings, but matching the corpus and retriever does not establish matched inference budgets. Please document configuration differences and evaluate a strong baseline under a common inference setup. Uncertainty estimates are also needed for small margins, particularly the 0.3-point average advantage over StepSearch at 3B.

  4. Efficiency accounting needs clarification. Please break down the reported 3.1 average LLM calls per HotpotQA question by role and specify whether decomposition, planning, answering, and retries are all included. Table 6 should also distinguish documents retrieved per call from cumulative documents retrieved per question.

Questions for authors
  1. Which mechanisms are technically new relative to ProgramFC, CodeAct, and LLMCompiler? How does PyRAG compare with a closely related executable-agent baseline using the same backbone and tools?
  2. How does Structured IR invoke tools and maintain intermediate state? What execution advantage remains with matched retrieval expansion, retry policies, and budget limits?
  3. Which configurations produced the imported Table 3 results? Do the main comparisons hold under a common inference setup, and how stable are the small reported margins?
  4. How are the 3.1 average calls distributed across roles and questions? Do Table 6 costs represent cumulative per-question usage, including all retrievals and retries?
4 comments
AuthorsOpenReview ↗

Rebuttal Part 1

Thank you for your valuable feedback. We address each point below:

W1: Thank you for highlighting these closely related works. Our related work discusses program-guided reasoning methods such as PAL, Program-of-Thoughts, Binder, Faithful-CoT, Logic-LM, and ProgramFC, as well as iterative retrieval and search-policy learning methods. ProgramFC is already discussed in Appendix C.1, but the current discussion does not explain the distinction sufficiently. We will expand that comparison and explicitly discuss CodeAct and LLMCompiler. We agree that executable plans, intermediate variables, and dependency-aware tool use are shared foundations. The distinctions concern what the program represents, how execution handles failures, and how the policies are trained.

ProgramFC. The submitted manuscript explicitly discusses ProgramFC in Appendix C.1 (Program-Guided Reasoning), where we describe how it converts natural-language claims into Python-style verification programs and executes them through specialized fact-checking modules. We appreciate the reviewer’s request for a more explicit comparison and will expand this discussion in the revision.

Both systems use generated programs to organize multi-step reasoning. ProgramFC uses these programs to verify complex claims by composing specialized operations for question answering, claim verification, and logical reasoning. PyRAG uses a generated program to coordinate open-domain multi-hop question answering. Intermediate answers can determine subsequent retrieval queries or feed directly into Python computations. Its runtime also uses two signals for targeted recovery. An insufficient-evidence response triggers broader retrieval for the affected sub-question, and a program exception provides diagnostic feedback to the planner for revision. We will clarify that our contribution lies in how these mechanisms are integrated into the retrieval-and-answering program.

CodeAct. CodeAct represents agent actions as executable Python and trains CodeActAgent through instruction tuning on multi-turn interaction trajectories. PyRAG-RL uses reinforcement learning to optimize the decomposition, planning, and evidence-conditioned answering policies against the final QA outcome. Specifically, it trains three role-specific LoRA adapters sequentially with GRPO, using a reward of 0.7 F1+0.3 EM0.7\,\mathrm{F1}+0.3\,\mathrm{EM} computed after executing the full pipeline. This training objective allows a decomposition or plan to receive feedback based on whether it ultimately leads to a correct answer.

LLMCompiler. LLMCompiler constructs a dependency graph of function calls and schedules independent tasks in parallel, with support for replanning based on intermediate results. PyRAG focuses on specifying the reasoning computation within the program itself. An intermediate answer can become the input to a subsequent search, a numerical comparison, or a conditional branch. We study how this representation supports evidence acquisition, answer composition, and recovery at specific points in a multi-hop execution. Dependency-aware parallel scheduling could complement this design.

We reproduced ReWOO, LLMCompiler, and CodeAct using their official implementations adapted to our retrieval setup. ProgramFC is designed and evaluated for complex-claim fact-checking on HOVER and FEVEROUS-S. Its analysis experiments also focus on fact-checking accuracy, evidence retrieval, and reasoning-program errors. Although its execution pipeline includes a question-answering submodule, the paper does not report a standalone evaluation on open-domain QA benchmarks. We therefore discuss ProgramFC as a methodological predecessor and prioritize ReWOO, LLMCompiler, and CodeAct for our additional empirical comparisons.

Shared settings. All LLM components use Qwen2.5-7B-Instruct, served through vLLM with tensor parallel size 2. The three baselines and training-free PyRAG use this checkpoint without additional fine-tuning. We use the same E5-base dense retriever over the Wikipedia 2018 corpus and evaluate on identical HotpotQA, 2WikiMQA, MuSiQue, and Bamboogle splits, containing 7,405, 12,576, 2,417, and 125 questions, respectively. All LLM calls use temperature 0.7, top-p 1.0, and a maximum of 16,384 generated tokens per call. Each baseline retrieval call returns the top-5 passages without additional passage truncation, and no cumulative passage cap is imposed per question. We retain the original prompt templates for the three baselines. Each baseline is evaluated over three runs with different inference seeds, using the same answer normalization and EM-scoring procedure. Timeouts, parsing failures, and missing final answers receive an EM score of zero.

AuthorsOpenReview ↗

Rebuttal Part 2

ReWOO. We retain the planner–worker–solver workflow and evidence placeholders. The planner and solver both use Qwen2.5-7B-Instruct, and the toolset contains only the retrieval tool connected to the shared E5-base service.

LLMCompiler. We retain dependency-aware planning, task execution, and final answer aggregation. Both the planner and joiner use Qwen2.5-7B-Instruct, and the available tool set contains only search, backed by the shared retrieval service.

CodeAct. We retain the iterative Python-action execution loop, with Qwen2.5-7B-Instruct generating code actions and receiving execution feedback. The only external tool available to the Python agent is search, backed by the same retrieval service.

PyRAG. We use the configuration described in the paper, including top-5 retrieval, adaptive expansion to top-10 for insufficient-evidence responses, and up to three runtime self-repair rounds. We additionally report PyRAG-RL with the role-specific adapters described in Appendix E.1.

Backbone: Qwen2.5-7B-Instruct. EM (%) ± SD (percentage points).

MethodHotpotQA2WikiMQAMuSiQueBamboogleAvg.
ReWOO31.6 ± 0.623.8 ± 0.97.3 ± 0.523.2 ± 2.421.5 ± 1.1
LLMCompiler32.7 ± 1.127.5 ± 0.810.6 ± 0.428.8 ± 2.924.9 ± 1.3
CodeAct33.7 ± 1.331.2 ± 0.79.3 ± 1.232.0 ± 3.526.6 ± 1.6
PyRAG w/o Training34.0 ± 2.333.4 ± 2.011.8 ± 0.741.6 ± 4.030.2 ± 1.4
PyRAG-RL40.5 ± 2.949.4 ± 2.620.7 ± 1.346.4 ± 4.239.3 ± 1.9

Training-free PyRAG achieves the highest mean EM on all four benchmarks among the training-free methods, with a macro-average of 30.2, compared with 26.6 for CodeAct, the strongest baseline by macro-average EM. The margin over the best baseline varies across datasets, from 0.3 points on HotpotQA (34.0 vs. 33.7) to 9.6 points on Bamboogle (41.6 vs. 32.0). These comparisons evaluate the full systems under a shared backbone and retrieval environment, with each method retaining its own execution and recovery procedures. PyRAG-RL reaches 39.3 macro-average EM with role-specific RL training.

In addition to comparing with other baselines, we also study whether PyRAG’s gains depend on a code-specialized backbone. In Figure 5, PyRAG already improves over Vanilla RAG with Qwen2.5-7B-Instruct. Switching to Qwen2.5-Coder-7B-Instruct has little effect on Vanilla RAG in this comparison but further improves PyRAG by 1.8, 6.9, and 2.0 EM points on HotpotQA, 2WikiMQA, and Bamboogle, respectively. We separately analyze basic execution, adaptive retrieval, and self-repair, alongside retrieval costs, latency, and failure modes. These analyses characterize when the framework helps and where it still fails.

W2: The Structured IR in Figure 3 is a non-executable, JSON-style plan that specifies operations, named outputs, and dependencies. Adding step-level retrieval expansion and retries would require a separate controller that tracks intermediate results, identifies the affected step, and decides when to intervene.

This is a practical advantage of PyRAG’s executable formulation. The runtime can observe an unknown response at a specific QA step and expand retrieval for that step. It can also capture a program exception and pass the diagnostic to the planner for repair. Both mechanisms use feedback from the same runtime that executes the reasoning plan.

The 2.7-point EM gain measures the difference between Structured IR and Basic Execution with both refinements disabled. The additional 7.8 points measure the benefit of enabling these mechanisms within PyRAG. A comparison with a separately engineered Structured IR controller would help determine how much of that gain can be reproduced through external orchestration.

W3:

Matched retrieval budget with Search-R1-3B. We evaluate Search-R1-3B and PyRAG-RL-3B on the same test questions using the same retriever, corpus, and EM scoring protocol. Both systems operate under a shared cap of 20 retrieved passages per question, counted cumulatively across all retrieval calls, including expansions and retries. Each system retains its adaptive search policy and may stop before reaching the cap.

Under this cap, Search-R1-3B achieves 28.6 EM on HotpotQA, and PyRAG-RL-3B achieves 32.1 EM, an improvement of 3.5 percentage points. We report per-benchmark EM, actual passages retrieved, retrieval calls, and input and output tokens to show how each system uses the shared retrieval budget.

AuthorsOpenReview ↗

Rebuttal Part 3

W4: Thank you for asking us to clarify the efficiency accounting. Decomposition and planning are performed jointly in one LLM call per question in our implemetation. The remaining calls are made to the Answer Agent, which averages 2.1 calls on HotpotQA and 3.2 on MuSiQue. This gives totals of 3.1 and 4.2 LLM calls per question, respectively.

Table 6 tests whether selectively expanding retrieval improves accuracy compared with using a larger fixed retrieval depth at every step. Within the PyRAG framework, adaptive retrieval reaches 41.6 EM on Bamboogle, compared with 39.2 EM for fixed (k=10). We agree that the table should distinguish retrieval depth per step from the cumulative number of passages returned per original question.

Vanilla RAG with (k=5) retrieves once for each original question, returning 5 passages per question and 625 passages across the 125 Bamboogle questions. Full PyRAG decomposes each original question into approximately 4 sub-questions and retrieves an average of 6.2 passages per sub-question. This corresponds to approximately 24.8 passages per original question, or 3,100 passages across Bamboogle under this accounting. We will revise Table 6 to report both retrieval depth and cumulative retrieval volume for each condition.

Q1: Please see our responses in weakness 1.

Q2:

Structured IR tool use and state. The Structured IR in Figure 3 is a JSON-style plan that specifies tool operations, named outputs, and dependencies. The model reads the plan to decide which retrieval or answering tool to invoke. Tool outputs are passed back to the model as context, and the model uses the named outputs and dependencies in the plan to interpret earlier results. PyRAG instead executes the generated program in a runtime that binds tool outputs to variables and passes those values directly to downstream operations.

Execution and refinement. With both refinements disabled, Basic Execution improves average EM from 25.8 to 28.5, a gain of 2.7 points over Structured IR. In full PyRAG, the runtime can identify the QA step that returned unknown and expand retrieval for that step. It can also capture a program exception and pass the diagnostic to the planner for repair. Supporting comparable interventions in our Structured IR baseline would require an additional controller to track intermediate results and decide when to expand retrieval or retry.

The 2.7-point difference is the measured gap with refinements disabled. The remaining gap when both systems have matched retrieval expansion, retry allowances, and inference budget limits has yet to be measured. We will clarify this distinction in the ablation discussion.

Q3: Table 3 configurations. The ZeroSearch, Search-R1, and StepSearch rows report scores from the original papers for the corresponding Qwen2.5-3B and 7B settings. Those scores reflect each paper’s inference configuration. In our PyRAG-RL runs, the Decompose, Plan, and Answer roles share the backbone listed in Table 3 and use separate LoRA adapters trained with GRPO. We use the E5-base dense retriever over Wikipedia 2018, retrieve five passages by default, expand to ten when the Answer Agent indicates insufficient evidence, and allow up to three runtime self-repair attempts. We will revise the Table 3 caption to distinguish the published results from those evaluated under our protocol.

Comparison under a common retrieval budget. We evaluated Search-R1-3B and PyRAG-RL-3B on the same HotpotQA questions with the same retriever, corpus, and EM scoring. Both methods operated under a cap of 20 retrieved passages per question, counted cumulatively across retrieval calls, including expansions and retries. Under this shared cap, PyRAG-RL-3B achieves 32.1 EM and Search-R1-3B achieves 28.6 EM, a difference of 3.5 percentage points on HotpotQA.

Run-to-run variability. We report mean EM and sample standard deviations across three runs in the supplement to Table 3. We compute the macro-average across the four benchmarks within each run and then calculate the sample SD across those three averages. PyRAG-RL achieves 29.9 ± 1.7 EM at 3B and 39.3 ± 1.9 EM at 7B. At 3B, StepSearch reports a macro-average of 29.6 and scores higher on HotpotQA and MuSiQue. The 0.3-point macro-average gap is descriptive given the available published scores for StepSearch. At 7B, PyRAG-RL’s macro-average is 4.8 points higher than StepSearch’s published 34.5.

AuthorsOpenReview ↗

Rebuttal Part 4

Supplement to Table 2 — Training-free PyRAG

EM (%) ± SD (percentage points).

BackbonePopQAHotpotQA2WikiMQAMuSiQueBamboogleAvg.
Qwen2.5-7B-Instruct33.5 ± 1.534.0 ± 2.333.4 ± 2.011.8 ± 0.741.6 ± 4.030.9 ± 0.9
Qwen2.5-72B-Instruct45.5 ± 2.552.0 ± 1.844.4 ± 2.816.9 ± 1.145.6 ± 3.240.9 ± 1.2

Supplement to Table 3 — PyRAG-RL

EM (%) ± SD (percentage points).

BackboneHotpotQA2WikiMQAMuSiQueBamboogleAvg.
Qwen2.5-7B-Instruct40.5 ± 2.949.4 ± 2.620.7 ± 1.346.4 ± 4.239.3 ± 1.9
Qwen2.5-3B-Instruct34.2 ± 3.232.5 ± 3.516.8 ± 0.836.0 ± 4.529.9 ± 1.7
Qwen3-4B-Instruct38.4 ± 2.245.1 ± 3.018.6 ± 0.643.2 ± 3.736.3 ± 1.4
LLaMA-3.1-8B-Instruct43.2 ± 2.750.1 ± 2.122.1 ± 2.448.0 ± 4.940.9 ± 1.6

Q4: Please see our responses in weakness 4.

Once again, thank you for your thorough review. We hope these points can address your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.