- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces PyRAG, a framework that represents multi-hop RAG as an executable intermediate representation. A query-specific Python program invokes retrieval and QA tools, binds intermediate answers to variables, and connects subsequent steps through explicit data dependencies. The experimental results that covers five question-answering benchmarks show that PyRAG achieves well performance in both training-free and RL settings. Ablation studies and efficiency analysis further demonstrate the effectiveness of PyRAG.
Critical strengths
- It achieves competitive results in both training-free and RL-trained settings, improving average EM by 11.8 points over Vanilla RAG and outperforming Search-R1 and StepSearch. The improvements across different datasets and model families also show good generalization.
- Using explicit variables and executable programs makes intermediate reasoning steps easier to inspect. The ablation and efficiency experiments show the benefits of different modules and reductions in both token usage as well as latency compared with Search-R1.
Critical weaknesses
- The main ideas such as explicit variables and executable tool calls, have already been explored in ReWOO, LLMCompiler, and CodeAct. The paper does not clearly explain what is fundamentally new, and lacks direct comparisons with these works.
- Table 4 shows that Basic Execution only improves EM from 25.8% to 28.5%, while larger gains come from adaptive retrieval and self-repair. It is therefore unclear how much improvement is due to executable representation rather than these extra mechanisms.
Questions for authors
- For the 3B backbone, PyRAG-RL only outperforms StepSearch by 0.3 EM points (29.9 vs. 29.6), while performing worse on HotpotQA and MuSiQue. Could the authors report standard deviations or significance tests to show whether this improvement is reliable?
- Bamboogle contains only 125 examples, yet some of the largest improvements are reported on this dataset. Could the authors provide per-run scores or confidence intervals to show how stable these gains are?
- Figure 6 shows that missing retrieval evidence accounts for roughly half of the 100 sampled failures on HotpotQA. How often does the sentinel-based adaptive retrieval detect these cases, and what fraction of the triggered re-retrievals successfully recover the correct answer?
4 comments
Rebuttal Part 2
ReWOO. We retain the planner–worker–solver workflow and evidence placeholders. The planner and solver both use Qwen2.5-7B-Instruct, and the toolset contains only the retrieval tool connected to the shared E5-base service.
LLMCompiler. We retain dependency-aware planning, task execution, and final answer aggregation. Both the planner and joiner use Qwen2.5-7B-Instruct, and the available tool set contains only search, backed by the shared retrieval service.
CodeAct. We retain the iterative Python-action execution loop, with Qwen2.5-7B-Instruct generating code actions and receiving execution feedback. The only external tool available to the Python agent is search, backed by the same retrieval service.
PyRAG. We use the configuration described in the paper, including top-5 retrieval, adaptive expansion to top-10 for insufficient-evidence responses, and up to three runtime self-repair rounds. We additionally report PyRAG-RL with the role-specific adapters described in Appendix E.1.
Backbone: Qwen2.5-7B-Instruct. EM (%) ± SD (percentage points)
| Method | HotpotQA | 2WikiMQA | MuSiQue | Bamboogle | Avg. |
|---|---|---|---|---|---|
| ReWOO | 31.6 ± 0.6 | 23.8 ± 0.9 | 7.3 ± 0.5 | 23.2 ± 2.4 | 21.5 ± 1.1 |
| LLMCompiler | 32.7 ± 1.1 | 27.5 ± 0.8 | 10.6 ± 0.4 | 28.8 ± 2.9 | 24.9 ± 1.3 |
| CodeAct | 33.7 ± 1.3 | 31.2 ± 0.7 | 9.3 ± 1.2 | 32.0 ± 3.5 | 26.6 ± 1.6 |
| PyRAG w/o Training | 34.0 ± 2.3 | 33.4 ± 2.0 | 11.8 ± 0.7 | 41.6 ± 4.0 | 30.2 ± 1.4 |
| PyRAG-RL | 40.5 ± 2.9 | 49.4 ± 2.6 | 20.7 ± 1.3 | 46.4 ± 4.2 | 39.3 ± 1.9 |
Training-free PyRAG achieves the highest mean EM on all four benchmarks among the training-free methods, with a macro-average of 30.2, compared with 26.6 for CodeAct, the strongest baseline by macro-average EM. The margin over the best baseline varies across datasets, from 0.3 points on HotpotQA (34.0 vs. 33.7) to 9.6 points on Bamboogle (41.6 vs. 32.0). These comparisons evaluate the full systems under a shared backbone and retrieval environment, with each method retaining its own execution and recovery procedures. PyRAG-RL reaches 39.3 macro-average EM with role-specific RL training.
Moreover, our contribution also includes an empirical investigation of when and how the framework is effective. In Section 3.3 and Figure 5, we compare Qwen2.5-7B-Instruct with Qwen2.5-Coder-7B-Instruct. Code specialization has little effect on Vanilla RAG in this comparison, whereas it improves PyRAG by 1.8, 6.9, and 2.0 EM points on HotpotQA, 2WikiMQA, and Bamboogle, respectively. Meanwhile, PyRAG already improves substantially over Vanilla RAG with the instruction-tuned backbone. These findings indicate that the framework’s benefits do not require a code-specialized model, while code specialization can provide additional gains.
We further analyze structured versus executable representations, the contributions of adaptive retrieval and self-repair, fixed versus adaptive retrieval budgets, token consumption and latency, and residual failure modes. Together, these analyses characterize the framework’s effectiveness, computational trade-offs, and remaining limitations. We will distinguish the evidence for the complete system from the narrower evidence for individual mechanisms; in particular, the full-system improvement should not be interpreted as an execution-only effect.
Accordingly, our contribution combines an executable formulation of multi-hop RAG, outcome-driven policy optimization, and systematic empirical analysis. We will revise the related work and contribution statements to make these distinctions explicit, while acknowledging the foundational ideas shared with prior executable-agent frameworks.
W2:
Thank you for this comment. We would like to clarify the distinction between the two reference conditions. In Table 4, Basic Execution improves average EM from 21.3 (Vanilla RAG) to 28.5, yielding a substantial gain of 7.2 percentage points without either adaptive retrieval or self-repair. The value of 25.8 refers instead to the Structured IR condition in Figure 3, which already incorporates typed operations, named outputs, and explicit dependencies.
These comparisons address different questions. The improvement from 21.3 to 28.5 demonstrates the effectiveness of the basic program-based reasoning pipeline before either refinement mechanism is introduced. The improvement from 25.8 to 28.5 measures the additional benefit of execution relative to an already structured representation. Consequently, the latter comparison should not be interpreted as the entire contribution of the basic PyRAG framework.
Table 4 further shows that adding adaptive retrieval or self-repair individually raises performance from 28.5 to 31.5 or 32.4, respectively, while combining them achieves 36.3. These results support a substantial contribution from the basic framework, with further improvements from the two recovery mechanisms.
Rebuttal Part 3
We will clarify these reference conditions and distinguish the benefit of the basic executable reasoning pipeline from the narrower incremental benefit of execution over Structured IR. We do not attribute the entire 7.2-point improvement to the interpreter alone, but the ablation clearly shows that the framework provides substantial gains even when both refinement mechanisms are disabled.
Q1:
Supplement to Table 2 — Training-free PyRAG
EM (%) ± SD (percentage points).
| Backbone | PopQA | HotpotQA | 2WikiMQA | MuSiQue | Bamboogle | Avg. |
|---|---|---|---|---|---|---|
| Qwen2.5-7B-Instruct | 33.5 ± 1.5 | 34.0 ± 2.3 | 33.4 ± 2.0 | 11.8 ± 0.7 | 41.6 ± 4.0 | 30.9 ± 0.9 |
| Qwen2.5-72B-Instruct | 45.5 ± 2.5 | 52.0 ± 1.8 | 44.4 ± 2.8 | 16.9 ± 1.1 | 45.6 ± 3.2 | 40.9 ± 1.2 |
Supplement to Table 3 — PyRAG-RL
EM (%) ± SD (percentage points).
| Backbone | HotpotQA | 2WikiMQA | MuSiQue | Bamboogle | Avg. |
|---|---|---|---|---|---|
| Qwen2.5-7B-Instruct | 40.5 ± 2.9 | 49.4 ± 2.6 | 20.7 ± 1.3 | 46.4 ± 4.2 | 39.3 ± 1.9 |
| Qwen2.5-3B-Instruct | 34.2 ± 3.2 | 32.5 ± 3.5 | 16.8 ± 0.8 | 36.0 ± 4.5 | 29.9 ± 1.7 |
| Qwen3-4B-Instruct | 38.4 ± 2.2 | 45.1 ± 3.0 | 18.6 ± 0.6 | 43.2 ± 3.7 | 36.3 ± 1.4 |
| LLaMA-3.1-8B-Instruct | 43.2 ± 2.7 | 50.1 ± 2.1 | 22.1 ± 2.4 | 48.0 ± 4.9 | 40.9 ± 1.6 |
Thank you for raising the question of statistical reliability. Across three runs, PyRAG-RL-3B achieves 29.9 ± 1.7 EM (mean ± sample standard deviation) when macro-averaged over the four benchmarks. The supplement to Table 3 reports the corresponding statistics for each benchmark. StepSearch reports a macro-average EM of 29.6. PyRAG-RL scores higher on 2WikiMQA and Bamboogle, while StepSearch scores higher on HotpotQA and MuSiQue.
The two methods differ in both training data and supervision. StepSearch trains on MuSiQue, while PyRAG-RL follows the Search-R1 training setup and uses single-hop Natural Questions and multi-hop HotpotQA. MuSiQue is therefore an in-domain benchmark for StepSearch and an out-of-domain benchmark for PyRAG-RL.
StepSearch also uses GPT-4o to enrich decomposed question–answer pairs and generate candidate search queries for individual reasoning steps. Its training objective combines final-answer feedback with a reward for matching reference queries and per-round retrieval rewards based on information gain and redundancy. PyRAG-RL optimizes its role-specific policies using only the final-answer reward after executing the generated program. The similar average EM shows that PyRAG-RL achieves comparable aggregate performance at this scale with outcome-level supervision alone.
Given that we have only published aggregate scores for StepSearch, we report the 0.3-point difference as a descriptive comparison.
Q2:
| Method | Backbone | Run 1 EM (%) | Run 2 EM (%) | Run 3 EM (%) | Mean EM (%) ± sample SD (pp) |
|---|---|---|---|---|---|
| Training-free PyRAG | Qwen2.5-7B-Instruct | 37.6 | 45.6 | 41.6 | 41.6 ± 4.0 |
| PyRAG-RL | Qwen2.5-7B-Instruct | 44.0 | 51.2 | 44.0 | 46.4 ± 4.2 |
Thank you for highlighting the small size of Bamboogle. With 125 questions, a single additional correct answer changes EM by 0.8 percentage points. Across three runs, PyRAG-RL with Qwen2.5-7B-Instruct achieves 46.4 ± 4.2 EM (mean ± sample standard deviation). This standard deviation describes run-to-run variability for PyRAG-RL.
In the training-free setting, 7B PyRAG achieves 41.6 EM, compared with 29.6 EM for ITER-RETGEN, an observed difference of 12.0 percentage points on Bamboogle. Estimating a paired confidence interval for this difference would require question-level predictions from both methods.
Q3: Thank you for asking how often the sentinel detects missing retrieval evidence and how often re-retrieval recovers the answer. Figure 6 characterizes errors among 100 incorrect final predictions on HotpotQA. To measure detection and recovery, we examined 100 HotpotQA sub-steps for which the initial top-5 passages were judged insufficient to answer the sub-question.
The Answer Agent returned a sentinel in 64 of these 100 sub-steps, yielding a detection recall of 64%. Each sentinel response automatically triggered an expansion from top-5 to top-10. Of the 64 triggered re-retrievals, 9 produced a correct sub-answer, corresponding to a 14.1% recovery rate per triggered step.
These measurements complement the question-level ablation in Table 4, where adding adaptive retrieval to Basic Execution improves HotpotQA EM from 31.5 to 32.6. We will add the detection and recovery counts to the revised analysis and specify the denominator for each rate.
Once again, thank you for your thorough review. We hope these points can address your concerns.
Thank you for the detailed response and the supplementary experiments.
The additional comparisons with ReWOO, LLMCompiler, and CodeAct are helpful and provide further support the effectiveness of the PyRAG framework. However, these comparisons are mainly conducted at the system level, and the methods differ in their adaptive retrieval and error recovery mechanisms. Consequently, the observed performance gains cannot be fully attributed to the executable inference representation itself.
The authors have further clarified the 2.7 EM improvement from Structured IR to Basic Execution. I acknowledge this distinction, but there is still a lack of direct comparison of executable and non-executable representations under the same retrieval extension and retry conditions. As a result, the specific contribution of the proposed representation remains insufficiently isolated.
The newly reported standard deviations and multiple-run results on Bamboogle provide additional evidence of experimental stability. However, among the 100 sub-steps with insufficient evidence, 64 trigger re-retrieval, but only 9 are successfully recovered.
Overall, I still have reservations about the independent contribution of the executable representation and its technical novelty compared with existing methods. Therefore, I maintain my original score.
Rebuttal Part 1
Thank you for your valuable feedback. We address each point below:
W1: We thank the reviewer for highlighting these closely related studies. Our existing related work (Appendix C and C.1) discusses several relevant research directions: iterative multi-hop retrieval methods such as Self-Ask, IRCoT, and ITER-RETGEN; search-policy learning methods including Search-R1, R1-Searcher, and StepSearch; program-guided reasoning approaches such as PAL, Program-of-Thoughts, Binder, Faithful-CoT, Logic-LM, and ProgramFC; and LM pipeline optimization through DSPy. These discussions position PyRAG in relation to retrieval-based reasoning, executable and symbolic computation, and pipeline optimization.
Nevertheless, our coverage of this rapidly developing field is incomplete, and we apologize for omitting an explicit discussion of ReWOO, LLMCompiler, and CodeAct. We appreciate the reviewer bringing these relevant connections to our attention and will incorporate them into the revised manuscript. Below, we clarify how PyRAG differs from these approaches in its reasoning formulation, reinforcement-learning-based policy optimization, and systematic empirical analysis.
ReWOO. ReWOO separates reasoning from observations: a Planner generates a plan with evidence placeholders, Workers execute the corresponding tool calls, and a Solver integrates the resulting evidence into a final answer. PyRAG expresses both evidence acquisition and answer composition within a query-specific executable program. Intermediate answers are bound to runtime variables and used in subsequent retrieval queries and native Python operations. When composition has well-defined semantics, such as arithmetic, comparison, or Boolean operations, it is performed directly by the program; an answer model remains available for semantic synthesis. Our focus therefore extends beyond separating planning from tool execution to explicitly representing and executing computations over retrieved information.
LLMCompiler. LLMCompiler generates a dependency graph of function calls and schedules their execution in parallel, with support for replanning based on intermediate results. PyRAG focuses on the executable representation of the multi-hop reasoning procedure: retrieval, evidence-conditioned answering, and computations over intermediate answers are expressed within the same program. Whereas dependency-aware scheduling is central to LLMCompiler, our study examines how this program representation supports intermediate-state management, answer composition, and targeted recovery. These directions are complementary, and parallel scheduling could also be incorporated into PyRAG.
CodeAct. CodeAct represents agent actions as executable Python code and trains CodeActAgent through instruction tuning on multi-turn interaction trajectories. Beyond adopting executable code, PyRAG-RL explicitly optimizes the policies responsible for decomposition, program generation, and evidence-conditioned answering through reinforcement learning. Specifically, we use GRPO with role-specific LoRA adapters, training the Answer, Plan, and Decompose roles sequentially while freezing the other roles at each stage. Rewards are derived from the final QA outcome of the executed pipeline, using a combination of answer F1 and EM. This allows policy optimization to be guided by downstream task success rather than solely by imitation of demonstrated interaction trajectories. We distinguish this RL extension from the training-free PyRAG framework, which can be used without additional optimization.
We reproduced ReWOO, LLMCompiler, and CodeAct using their official implementations adapted to our retrieval setup. ProgramFC is designed and evaluated for complex-claim fact-checking on HOVER and FEVEROUS-S. Its analysis experiments also focus on fact-checking accuracy, evidence retrieval, and reasoning-program errors. Although its execution pipeline includes a question-answering submodule, the paper does not report a standalone evaluation on open-domain QA benchmarks. We therefore discuss ProgramFC as a methodological predecessor and prioritize ReWOO, LLMCompiler, and CodeAct for our additional empirical comparisons.
Shared settings. All LLM components use Qwen2.5-7B-Instruct, served through vLLM with tensor parallel size 2. The three baselines and training-free PyRAG use this checkpoint without additional fine-tuning. We use the same E5-base dense retriever over the Wikipedia 2018 corpus and evaluate on identical HotpotQA, 2WikiMQA, MuSiQue, and Bamboogle splits, containing 7,405, 12,576, 2,417, and 125 questions, respectively. All LLM calls use temperature 0.7, top-p 1.0, and a maximum of 16,384 generated tokens per call. Each baseline retrieval call returns the top-5 passages without additional passage truncation, and no cumulative passage cap is imposed per question. We retain the original prompt templates for the three baselines.