- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
Doctor-RAG (DR-RAG) is a framework that fixes errors in Agentic RAG systems during complex multi-step tasks by avoiding slow, expensive full reruns. Instead of starting from scratch, it uses a distilled model to diagnose the exact breakpoint and type of error (format, reasoning, retriever, or search), and then applies a targeted local repair right where things went wrong. By only fixing the mistake and reusing all the correct evidence gathered before that point, DR-RAG significantly boosts accuracy and repair rates while running 2 to 3 times faster than traditional methods.
Critical strengths
The standout advantage of Doctor-RAG is its fast, cost-effective, and highly accurate approach to fixing errors. Instead of wasting computational power by rerunning a whole process from scratch, it uses a smart diagnostic model to instantly pinpoint the exact error type and breakpoint in a single pass. By applying targeted local repairs and reusing previously correct steps, it runs 2 to 3 times faster than standard reruns and up to 10 times faster than RAG-Critic, all while drastically improving repair rates and final answer accuracy. Plus, it is highly versatile, working seamlessly across various base LLMs and Agentic RAG frameworks like ReAct and Search-o1.
Critical weaknesses
Despite its strengths, the Doctor RAG framework has a few limitations. It struggles to accurately diagnose search errors compared to other types because distinguishing them from retriever errors requires highly complex analysis. It also shows limited improvement on extremely challenging datasets like MuSiQue, where the tasks are simply too complex for after-the-fact repairs to help much. Finally, while the system is somewhat forgiving, occasional misdiagnoses still occur, and getting the initial diagnosis completely right remains crucial for achieving the highest possible repair rate.
Questions for authors
All my questions are as stated above.
3 comments
Response to the rebuttal
Thank you to the authors for the detailed response of my concerns. Although the authors have address some of my concern, my concern regarding the limited recovery rates on complex multi-hop benchmark as well as the clarity of presentation. Therefore, I will maintan my current rating.
Follow-up on Recovery Rate and Presentation
Thank you for taking the time to consider our rebuttal and provide further feedback. We would like to clarify the practical significance of recovery on difficult tasks by considering both repair effectiveness and recomputation cost. On MuSiQue, Tables 2 and 4 report the following results with Qwen3-8B.
| Trajectory source | Rerun: repair rate / time | DR-RAG: repair rate / time |
|---|---|---|
| ReAct | 3.8% / 927 s | 7.7% / 233 s |
| Search-o1 | 7.3% / 817 s | 18.1% / 250 s |
| Search-R1 | 7.2% / 773 s | 8.3% / 394 s |
Repair rates are measured on initially failed trajectories. Times are totals for processing the failed trajectories in each setting, and DR-RAG includes both diagnosis and repair. These results show that reusing failed trajectories can recover more answers while requiring substantially less recomputation, including on MuSiQue.
Regarding presentation clarity, could you point us to the specific sections, definitions, or explanations that remain unclear? We would appreciate concrete examples so that we can provide targeted clarification and improve the manuscript accordingly.
Response to Reviewer VTf4
Response to W1. Distinguishing Search and Retriever Errors
We agree that Search and Retriever errors are difficult to distinguish because they can occur sequentially in the same trajectory. Under our earliest-error criterion, the diagnoser must trace the trajectory to determine whether the initial failure occurred in query formulation or retrieval. Subsequent error propagation makes this attribution harder. The following two examples illustrate this distinction.
Seven Women (1944 Film).The repair framework has some tolerance to this confusion because both operators can acquire additional evidence and regenerate the answer. Appendix D.2.4, Table 9 evaluates this through controlled rerouting on 587 misclassified trajectories, holding the repair model, retriever, and evidence context fixed.
Of all misclassifications, 62% occur between Search and Retriever. Correcting the routing improves repair rates for these two reference classes by 1.1 and 2.2 percentage points, respectively. These modest aggregate gains support partial tolerance to imperfect classification.
Response to W2. Repair Performance on MuSiQue
MuSiQue is indeed challenging. Our repair evaluation uses fixed failed trajectories generated by Qwen3-4B. Recovering these trajectories still requires obtaining missing evidence and completing multi-hop reasoning, so repair performance also depends on the capabilities of the diagnosis and repair models. Table 2 and Appendix D.1, Table 6 show that increasing their backbone from Qwen3-4B to Qwen3-8B improves MuSiQue repair rates across all three trajectory sources.
With Qwen3-8B, DR-RAG repairs 18.1% of failed Search-o1 trajectories on MuSiQue, compared with 10.8% for the strongest baseline. These results show that useful post-hoc recovery remains possible on MuSiQue and that a larger diagnosis and repair backbone can recover additional failures.
Response to W3. Impact of Diagnostic Accuracy on Repair
Diagnosis accuracy does affect repair quality. All main results use error types and locations predicted by the distilled diagnoser, so the reported gains already include its diagnostic errors. Table 3 shows that these predicted signals improve repair over removing either taxonomy-based routing or localization. The values below are ΔEM on the failed subset, averaged over the three agentic RAG baselines.
Appendix D.2.4, Table 9 further quantifies the benefit of improving error-type predictions: on 587 misclassified trajectories, replacing the predicted types with teacher labels increases repair success from 17.2% to 19.8%. Thus, more accurate diagnosis offers additional gains, while the existing predicted diagnosis already provides useful guidance for repair.