- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This work presents BindRAG, a RAG framework for multi-hop question answering. The method first prompts an LLM for a decomposed retrieval plan, executes the plan by binding and storing retrieved entities from previous hops/steps and leverages them for subsequent hops dependent on said entity. Throughout, the method maintains alternative options, if produced, and completes a final full resolution step considering them before producing an answer. The authors formalize the problem at hand and present experimental results where BindRAG is evaluated on three 1k question subsets of common multi-hop QA datasets and compared against a set of graph-based RAG methods. The work reports that the presented method beats out every baseline in every single setting.
Critical strengths
The presentation of the problem with running examples and the formalization are good. However, it is evident that many parts of the submitted work are strongly LLM-edited.
Critical weaknesses
I’m mainly concerned with the novelty of the method. Analyzing the main contributions:
(1) Binding-aware execution has been proposed in ReWOO (Xu et al., 2023) and LLMCompiler (Kim et al., 2024) which are both correctly cited, but also other un-cited works like Break It Down (Wolfson et al., 2020) present the idea of re-inserting previous answers in their BREAK method, and more recently pi-CoT (Gong et al., 2026) leverages prolog for binding answers.
(2) Uncertainty-aware execution has been targeted by Multi-Chain Reasoning (Yoran et al., 2024) and Beam Retrieval (Zhang et al., 2024), which are both not cited.
(3) While the works’ experimental results may be a valid, they don’t compare to the most competitive and most highly related methods.
Additionally, it is hard to verify the correctness of the evaluation results when no code artifacts are submitted. Also, the perfect trend in Table 1 of Str-Acc < LLM-Acc for HotpotQA and MusiQue, whereas the inverse holds perfectly for 2Wiki, with no deviations whatsoever raises questions.
Questions for authors
- Why was temperature 0 chosen? This does not guarantee determinism and also does not reflect how a regular user would use the language model with default, model-provider suggested settings.
- Why is the LightRAG reference listed with author Tian Ao, instead of Tu Ao? I was unable to find an online reference where the author is listed as Tian instead of Tu.
- Can you explain the choice of models used? It is not clear to me why Qwen2.5 was chosen over 3.5 or 3.6 and why gpt-4o-mini was leveraged as the judge.
- How do you explain the perfect scores in Table 1 (please see weaknesses)?
4 comments
[Question 3: Choice of backbone and evaluation judge]
We selected Qwen2.5-32B-Instruct because it is a publicly available backbone supported by DIGIMON, enabling consistent comparisons with the graph-based baselines while balancing model capability and computational cost. We additionally evaluated GPT-4o-mini and Llama-3.1-8B-Instruct to examine backbone dependence (Table B.1).
We used gpt-4o-mini as the LLM judge for consistency with DIGIMON and LogicRAG. We acknowledge potential self-evaluation bias when GPT-4o-mini is also the generator; therefore, we report Str-Acc as a complementary judge-independent metric for these settings.
[Question 4: Consistent Str-Acc and LLM-Acc patterns across datasets]
Thank you for highlighting this unusual pattern. We verified that Str-Acc is consistently lower than LLM-Acc on HotpotQA and MuSiQue, while the reverse holds on 2WikiMultiHopQA across all methods in Table 1. We therefore examined question-level disagreements between the two metrics.
The metrics agree on 88–99% of questions, depending on the method and dataset. On HotpotQA and MuSiQue, disagreements commonly involve shorter but semantically correct predictions that fail string containment against longer gold answers. On 2WikiMultiHopQA, gold answers are often short entities or categories, so string containment may accept predictions with additional, semantically incorrect qualifiers that the LLM judge rejects.
For example, "Jean Cocteau" may fail string matching against a longer full-name gold answer, whereas "Chinese American" may pass substring matching against "Chinese" despite not being equivalent.
These findings suggest that the consistent direction across methods reflects dataset-specific answer characteristics interacting with the two evaluation metrics, rather than a method-specific effect. We have clarified this behavior and the limitations of both metrics in the revised manuscript.
Thank you for the extensive response to my review. Please see my response below.
W1-3:
- Regarding bullet point 1, it is still unclear to me what is novel about the formulation/formalization. The problem BindRAG tackles has been formalized in very similar ways in previous works, including in the ones you cite yourself (bullet point 1).
- I appreciate the revision of the contributions section and, based on my understanding, the key methodological contribution lies in providing the final reader (prompt) with more context than existing works have done (bullet point 2). Looking at Table H.1, the biggest contribution (+16.1) comes from slot substitution (see my concerns about related works in my initial review on this) and +4.8 from providing the program, bindings and grouping the intermediate results by hops.
- With Table 2 added, it seems now that your interpretation of the experimental results has shifted from pure performance over other methods, to include reduced token cost (bullet point 3)? I believe a separation of this would be beneficial.
I understand that comparisons to methods that require training or don't provide the source code, so your revisions to the related works section resolves my concern for these two.
Writing issues: While I appreciate the revisions made and the authors conceding on the contribution claims, I still believe there need to be key revisions done outside of the results section (this is where the current version shows most blue-colored differences). I am less concerned with the revised introduction and the related works, but starting in section 3, a lot of the writing could be drastically simplified.
Q1: Understood. Q2: I can confirm this has been corrected.
Q3: Digimon appears to support a wide variety of open and closed-weight models. This does not resolve my question for the choice of Qwen version. I understand the choice of wanting to use a consistent judge, but have you evaluated any discrepancies in judging when using other models as the judge (even on a small subset, not even all evaluation results)?
Q4: This addresses my question.
[Weaknesses 1–3: Novelty and relationship to prior work]
Thank you for highlighting these closely related methods. We agree that variable substitution, dependency-aware execution, and uncertainty handling are not individually novel. We have therefore clarified our contribution and expanded the related-work discussion to include Break It Down, π-CoT, Multi-Chain Reasoning (MCR), and Beam Retrieval.
Our key distinction lies in how intermediate dependencies are represented and used during final answer synthesis. ReWOO preserves execution plans and tool observations, while IRCoT accumulates reasoning steps and retrieved evidence. BindRAG instead maintains an explicit typed dependency program, resolved answer bindings, and hop-wise supporting evidence as structured inputs to the final reader. Thus, intermediate answers are not only substituted into downstream retrieval queries but also retained with their dependency relationships for final synthesis.
Our controlled experiments help identify the contribution of this design. Removing binding propagation reduces MuSiQue accuracy by 17.0 points, while removing query-aware memory reduces it by 4.4 points with the answer prompt fixed. Moreover, removing the program and resolved bindings from the final reader reduces accuracy by 4.8 points with retrieved evidence held constant. These results support the importance of explicitly preserving and using dependency information beyond retrieval.
We also added direct comparisons with ReWOO, IRCoT, SeqProp, and adapted MCR. BindRAG outperforms ReWOO by 25.0 points with each method's own reader and 17.7 points with a shared flat reader. However, BindRAG and SeqProp are statistically tied under the shared reader, indicating that the additional end-to-end benefit comes primarily from binding-aware final synthesis rather than a fundamentally different substitution mechanism. Against IRCoT, BindRAG achieves comparable flat-reader accuracy with substantially fewer tokens.
BindRAG uses 5.8k tokens per question. Str-Acc differences with 95% CIs (Table 2).
We do not include three of the methods you mention in our comparison. LLMCompiler's main advantage is parallel execution, whereas most MuSiQue (86.9% chain) and HotpotQA (81.1% bridge) questions are sequential; ReWOO represents this plan-and-execute family. Beam Retrieval requires supervised retriever training, unlike the training-free methods compared here. For π-CoT, we found no public code (its latest arXiv version gives the prompts in an appendix), so we discuss the differences in the related-work section without an experimental comparison.
Finally, our ablations show that intermediate candidate enumeration does not provide a statistically significant accuracy gain in isolation. We therefore position uncertainty handling as a supporting mechanism rather than an independent novelty claim.
In summary, our contribution is not the invention of variable binding or evidence accumulation, but a structured retrieval-and-synthesis framework that preserves explicit answer dependencies from execution through final answer generation. We have revised the manuscript to make this distinction and its empirical support clearer.
[Weakness: Code availability]
We have uploaded an anonymized supplementary package containing the implementation, prompts, configurations, question IDs, predictions, evaluation scripts, and documented baseline adaptations. This is intended to make the experimental setup and reported comparisons reproducible.
[Weakness: Writing issues and limitations]
Thank you for pointing out the writing and presentation issues. We have corrected the identified errors and clarified the scope of our conclusions. In particular, we now explicitly acknowledge settings where BindRAG does not outperform the baselines, including certain 2WikiMultiHopQA question types and shared-reader comparisons.
[Question 1: Why temperature 0?]
We followed DIGIMON's greedy-decoding configuration to keep generation settings consistent across methods and reduce sampling variance. We agree that temperature 0 does not guarantee determinism or represent typical provider defaults; our objective was a controlled comparison rather than a deployment-oriented evaluation.
[Question 2: LightRAG author name]
Thank you for catching this error. The correct author name is Tu Ao, and we have corrected the reference.