acceptodds
Under review as a conference paper at ICLR 2027

Doctor-RAG: A Failure-Aware Repair Framework for Agentic Retrieval-Augmented Generation

Abstract

Agentic Retrieval-Augmented Generation interleaves retrieval and reasoning for multi-hop QA and complex knowledge tasks. As reasoning trajectories lengthen, failures become more frequent, while existing methods often either stop at diagnosis or rely on coarse replanning and rerun-style recovery, incurring high computational cost. We propose Doctor-RAG (DR-RAG), a diagnose-and-repair framework that corrects failures via explicit error localization and prefix reuse. DR-RAG operates in two stages: (i) trajectory-level failure diagnosis, where a distilled diagnosis model jointly assesses evidence sufficiency, classifies the failure type, and localizes the earliest failure point; and (ii) tool-conditioned local repair that intervenes only at the diagnosed point while reusing conditionally valid prefixes and retrieved evidence. By separating error attribution from correction, DR-RAG avoids blind reruns in a post-hoc repair setting and enables targeted, efficient correction of known failed trajectories. Experiments on three multi-hop QA benchmarks across multiple agentic RAG baselines and backbone models show substantial improvements in answer accuracy.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Retrieval-Augmented Generation, Agentic RAG, Failure Diagnosis, Localized Repair, Multi-hop Question Answering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VTf4OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

Doctor-RAG (DR-RAG) is a framework that fixes errors in Agentic RAG systems during complex multi-step tasks by avoiding slow, expensive full reruns. Instead of starting from scratch, it uses a distilled model to diagnose the exact breakpoint and type of error (format, reasoning, retriever, or search), and then applies a targeted local repair right where things went wrong. By only fixing the mistake and reusing all the correct evidence gathered before that point, DR-RAG significantly boosts accuracy and repair rates while running 2 to 3 times faster than traditional methods.

Critical strengths

The standout advantage of Doctor-RAG is its fast, cost-effective, and highly accurate approach to fixing errors. Instead of wasting computational power by rerunning a whole process from scratch, it uses a smart diagnostic model to instantly pinpoint the exact error type and breakpoint in a single pass. By applying targeted local repairs and reusing previously correct steps, it runs 2 to 3 times faster than standard reruns and up to 10 times faster than RAG-Critic, all while drastically improving repair rates and final answer accuracy. Plus, it is highly versatile, working seamlessly across various base LLMs and Agentic RAG frameworks like ReAct and Search-o1.

Critical weaknesses

Despite its strengths, the Doctor RAG framework has a few limitations. It struggles to accurately diagnose search errors compared to other types because distinguishing them from retriever errors requires highly complex analysis. It also shows limited improvement on extremely challenging datasets like MuSiQue, where the tasks are simply too complex for after-the-fact repairs to help much. Finally, while the system is somewhat forgiving, occasional misdiagnoses still occur, and getting the initial diagnosis completely right remains crucial for achieving the highest possible repair rate.

Questions for authors

All my questions are as stated above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer VTf4

Response to W1. Distinguishing Search and Retriever Errors

We agree that Search and Retriever errors are difficult to distinguish because they can occur sequentially in the same trajectory. Under our earliest-error criterion, the diagnoser must trace the trajectory to determine whether the initial failure occurred in query formulation or retrieval. Subsequent error propagation makes this attribution harder. The following two examples illustrate this distinction.

HotpotQA · Girl Talk2Wiki · Seven Women
Initial queryThe agent queried the Spice Girls without supporting evidence. The intended group was TLC.The agent correctly queried Seven Women (1944 Film).
Subsequent trajectoryRetrieved documents concerned the Spice Girls, and four subsequent queries retained the mistaken entity.Retrieval returned the 1966 film directed by John Ford. The agent then searched for Ford’s place of death.
Earliest errorSearch — the wrong entity was introduced before retrieval.Retriever — the wrong film first appeared in the retrieved evidence.
8B / Qwen3.5-Plus diagnosisRetriever / SearchRetriever / Search

The repair framework has some tolerance to this confusion because both operators can acquire additional evidence and regenerate the answer. Appendix D.2.4, Table 9 evaluates this through controlled rerouting on 587 misclassified trajectories, holding the repair model, retriever, and evidence context fixed.

Reference error typeMisclassified casesOriginal routingTeacher-label routing
Search26710.5%11.6%
Retriever17816.9%19.1%
All types58717.2%19.8%

Of all misclassifications, 62% occur between Search and Retriever. Correcting the routing improves repair rates for these two reference classes by 1.1 and 2.2 percentage points, respectively. These modest aggregate gains support partial tolerance to imperfect classification.

Response to W2. Repair Performance on MuSiQue

MuSiQue is indeed challenging. Our repair evaluation uses fixed failed trajectories generated by Qwen3-4B. Recovering these trajectories still requires obtaining missing evidence and completing multi-hop reasoning, so repair performance also depends on the capabilities of the diagnosis and repair models. Table 2 and Appendix D.1, Table 6 show that increasing their backbone from Qwen3-4B to Qwen3-8B improves MuSiQue repair rates across all three trajectory sources.

Trajectory sourceQwen3-4B repair rateQwen3-8B repair rate
ReAct7.4%7.7%
Search-o116.1%18.1%
Search-R16.7%8.3%

With Qwen3-8B, DR-RAG repairs 18.1% of failed Search-o1 trajectories on MuSiQue, compared with 10.8% for the strongest baseline. These results show that useful post-hoc recovery remains possible on MuSiQue and that a larger diagnosis and repair backbone can recover additional failures.

Response to W3. Impact of Diagnostic Accuracy on Repair

Diagnosis accuracy does affect repair quality. All main results use error types and locations predicted by the distilled diagnoser, so the reported gains already include its diagnostic errors. Table 3 shows that these predicted signals improve repair over removing either taxonomy-based routing or localization. The values below are ΔEM on the failed subset, averaged over the three agentic RAG baselines.

MethodHotpotQA2WikiMuSiQue
DR-RAG with predicted diagnosis28.923.811.4
Without taxonomy17.516.06.3
Without localization19.518.38.6

Appendix D.2.4, Table 9 further quantifies the benefit of improving error-type predictions: on 587 misclassified trajectories, replacing the predicted types with teacher labels increases repair success from 17.2% to 19.8%. Thus, more accurate diagnosis offers additional gains, while the existing predicted diagnosis already provides useful guidance for repair.

Reviewer_VTf4OpenReview ↗

Response to the rebuttal

Thank you to the authors for the detailed response of my concerns. Although the authors have address some of my concern, my concern regarding the limited recovery rates on complex multi-hop benchmark as well as the clarity of presentation. Therefore, I will maintan my current rating.

AuthorsOpenReview ↗

Follow-up on Recovery Rate and Presentation

Thank you for taking the time to consider our rebuttal and provide further feedback. We would like to clarify the practical significance of recovery on difficult tasks by considering both repair effectiveness and recomputation cost. On MuSiQue, Tables 2 and 4 report the following results with Qwen3-8B.

Trajectory sourceRerun: repair rate / timeDR-RAG: repair rate / time
ReAct3.8% / 927 s7.7% / 233 s
Search-o17.3% / 817 s18.1% / 250 s
Search-R17.2% / 773 s8.3% / 394 s

Repair rates are measured on initially failed trajectories. Times are totals for processing the failed trajectories in each setting, and DR-RAG includes both diagnosis and repair. These results show that reusing failed trajectories can recover more answers while requiring substantially less recomputation, including on MuSiQue.

Regarding presentation clarity, could you point us to the specific sections, definitions, or explanations that remain unclear? We would appreciate concrete examples so that we can provide targeted clarification and improve the manuscript accordingly.

Reviewer_m36HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DR-RAG addresses post-hoc failure repair in Agentic RAG: given a completed retrieval–reasoning trajectory that produces an incorrect answer, it aims to correct the failure without rerunning the entire pipeline. The framework comprises two modules. The diagnosis module uses a distilled language model to classify failures into four categories—Format, Reasoning, Retriever, and Search—and identify the earliest failure point. The repair module selects a specialized operator based on the diagnosis, seeking to reuse valid trajectory prefixes and previously retrieved evidence. Experiments on HotpotQA, 2Wiki, and MuSiQue report higher repair rates and greater computational efficiency than the selected baselines.

Critical strengths
  1. The paper addresses a meaningful research problem and presents a systematic framework for post-hoc repair of completed, unsuccessful Agentic RAG trajectories.

  2. The evaluation is relatively comprehensive, covering three QA datasets, three Agentic RAG baselines, and language models of different sizes. Additional experiments include component ablations, diagnosis robustness analyses, multiple random seeds, and comparisons under controlled retrieval budgets.

Critical weaknesses
  1. The main experiments provide the ground-truth answer as an input to the diagnosis model. Although Appendix D.6 introduces a deployment simulation in which an external verifier triggers repair and supplies a pseudo-ground-truth answer, this more realistic setting remains supplementary. The main evaluation therefore does not adequately establish performance under deployment conditions where ground-truth answers are unavailable.

  2. Format Repair generates at least ten candidate answers and counts a repair as successful if any candidate matches the ground truth. This candidate-set evaluation is not equivalent to the single-answer Exact Match protocol used for conventional Agentic RAG outputs, raising concerns about the comparability of the reported results. A fair comparison should require a single final answer selected without access to the ground truth, or apply the same candidate-generation and evaluation protocol to all methods.

  3. For Search Errors, the paper defines the earliest failure point as the first search query misdirected by flawed upstream reasoning. Truncating at this query may retain the erroneous assumption in the reused prefix, so the intervention does not necessarily remove the root cause. The authors should distinguish the point where an error originates, the point where it manifests in search behavior, and the repair starting point, and experimentally assess the risks of retaining erroneous reasoning in the prefix.

  4. Error classification and failure localization are central contributions, but the paper does not report a direct quantitative evaluation of localization accuracy. The classification results, localization ablation, and qualitative example do not establish whether the predicted failure locations are accurate. The authors should report localization accuracy separately, ideally against independently annotated reference locations.

  5. In Appendix D.4, the original EM/F1 scores for ReAct, Search-o1, and Search-R1 are identical across the Qwen3-8B and LLaMA-3.1-8B settings. If the original trajectories were generated independently using the respective backbones, this exact agreement requires clarification.

  6. Section 3 is unusually brief as a standalone task-formulation section, providing only a short description of the trajectory representation.

Questions for authors

Please address the concerns raised in the Weaknesses section and provide the relevant clarifications, explanations, and supporting evidence.

4 comments
AuthorsOpenReview ↗

Response to Reviewer m36H (Part 1/2)

Response to W1. Deployment Evaluation with an External Verifier

We conducted a verifier-driven deployment simulation (Appendix D.6) and provide its detailed results by framework and dataset below. DeepSeek-V4-Pro uses only the question, retrieved documents, and original prediction to trigger repair and generate a pseudo-reference answer, replacing the gold answer supplied to the diagnoser. The diagnosis and repair backbone is Qwen3-8B. Each framework–dataset setting is evaluated on all 500 trajectories, totaling 4,500, including verifier misses and false positives. EM/F1 are expressed as percentages.

FrameworkDatasetEM (original → after repair)F1 (original → after repair)
ReActHotpotQA27.2 → 44.834.7 → 54.1
ReAct2Wiki29.4 → 36.034.4 → 43.0
ReActMuSiQue13.2 → 20.219.9 → 28.2
Search-o1HotpotQA43.0 → 48.451.0 → 57.3
Search-o12Wiki33.2 → 38.840.2 → 47.0
Search-o1MuSiQue14.4 → 19.421.5 → 27.7
Search-R1HotpotQA37.0 → 40.845.4 → 51.2
Search-R12Wiki32.4 → 40.638.4 → 46.7
Search-R1MuSiQue13.0 → 17.419.9 → 25.3
Overall4,500 trajectories27.0 → 34.033.9 → 42.3

The pipeline corrects 419 initially incorrect answers and breaks 101 initially correct answers, yielding a net gain of 318 correct answers (+7.1 EM points). We will make the reference-assisted assumption explicit in the task formulation and highlight this deployment simulation in the main evaluation.

Response to W2. Single-Answer Evaluation of Format Repair

We revised Format Repair to return a single final answer, selected independently of the ground truth, and reevaluated all methods using the same single-answer EM protocol on the same failed trajectories. DR-RAG retains its overall advantage.

Repair backboneDR-RAG, single answerRerunStep-wise RetryRAG-Critic
Qwen3-8B19.96%12.78%8.40%12.99%
LLaMA-3.1-8B13.45%8.49%7.97%5.78

Response to W3. Defining Search Errors and Repair Starting Points

We will clarify the definition and localization of Search Error. It covers failures where an incorrect query-generation decision prevents the retrieval of necessary evidence. The erroneous assumption may first appear in an upstream reasoning step or in the query itself. Under the earliest-error criterion, repair should begin before the first erroneous action in the recorded trajectory. If the assumption appears in an earlier reasoning step, that step must also be discarded.

The following examples illustrate cases where the preceding reasoning states a valid plan, and the first observable error occurs in the query.

HotpotQA · GöringMuSiQue · Edith Humphrey
QuestionWhen did the war in which Göring served as a fighter pilot end?How many students attend the university where Edith Humphrey studied?
Preceding reasoningIdentify the war, then determine its end year.Identify her university, then find its enrollment.
Generated queryWhen did World War II end?Who attended the University of North Carolina at Chapel Hill
First errorThe query introduces World War II without support; the evidence identifies World War I.The query introduces UNC without support; the evidence identifies the University of Zurich.
Repair startBefore the first query, retaining the preceding plan.Before the first query, retaining the preceding plan.

These cases support distinguishing the error category from the action where it first occurs. Reasoning Error covers incorrect inference despite sufficient evidence, whereas Search Error involves a faulty search decision that leaves evidence missing. We will revise Section 4.2.2, Appendix E.6, and the diagnosis prompt description to make the corresponding truncation rule explicit.

AuthorsOpenReview ↗

Response to Reviewer m36H (Part 2/2)

Response to W4. Quantitative Evaluation of Failure Localization

Thank you for pointing out this omission. We now report exact localization accuracy for the Qwen3-8B distilled diagnoser by dataset and agentic RAG baseline, using Qwen3.5-Plus teacher annotations as the reference. An exact match requires the prediction and reference to identify the same action in the trajectory. Each cell gives the accuracy and the number of matches over evaluated cases.

BaselineHotpotQA2WikiMuSiQue
ReAct65.5% (171/261)58.6% (170/290)60.3% (228/378)
Search-o172.6% (159/219)79.1% (227/287)70.5% (261/370)
Search-R179.8% (193/242)77.6% (232/299)76.4% (305/399)

Pooled across all nine settings, exact localization accuracy is 70.9% (1,946/2,745). We evaluate non-Format cases from the original diagnosis evaluation set, excluding 10 reference locations that cannot be mapped to an action.

The archived labels identify the responsible query for Retriever errors and the faulty reasoning step or query for Search errors. We will clarify these localization targets in Sections 4.2.2 and E.6 and include this breakdown in the revision.

Response to W5. Shared Input Trajectories Across Repair Backbones

DR-RAG is a framework for post-hoc diagnosis and repair of completed Agentic RAG trajectories. ReAct, Search-o1, and Search-R1 serve as trajectory sources. We collect their failed executions and compare DR-RAG with repair methods—Rerun, Step-wise Retry, and RAG-Critic—on the same trajectories.

In Appendix D.4, the original trajectories were generated with Qwen3-4B and reused across both repair settings. The Qwen3-8B and LLaMA-3.1-8B headings identify the repair backbones. Keeping the input trajectories fixed allows us to compare repair effectiveness on the same failures, so the original EM/F1 scores are identical by construction. We will explicitly distinguish the trajectory-generation model, trajectory-generating frameworks, and repair backbones in the experimental setup and table descriptions.

Response to W6. Clarifying the Task Formulation

We will add clarification in Section 3 that our task is post-hoc repair of Agentic RAG trajectories. We collect failed trajectories generated by existing systems, diagnose their error types and locations, and apply targeted local repair.

Thank you for the detailed rebuttal and the additional experimental results. I appreciate the authors' efforts to address my concerns.

The external-verifier-based deployment simulation provides useful evidence regarding the practicality of DR-RAG. I also appreciate the revised single-answer evaluation protocol, the quantitative fault localization results, and the clarification regarding shared input trajectories across different repair backbones. These additions address several of my concerns and improve the clarity of the experimental evaluation.

However, some of my main concerns remain. First, although the deployment simulation no longer relies directly on ground-truth answers, it still depends on an external verifier to generate reference answers. The robustness of the framework when such references are unreliable or unavailable remains insufficiently established. The reported introduction of 101 new errors also highlights the potential risks of incorrect repair decisions.

Second, while the authors clarify the selection of repair starting points, the reliability of preserving earlier trajectory segments remains unclear, particularly when errors propagate across multiple reasoning and retrieval steps. The provided examples are helpful but do not fully establish the robustness of this strategy.

Third, I appreciate the newly reported fault localization accuracy. However, the overall accuracy of 70.9%, together with the reliance on model-assisted reference annotations, leaves some concerns about the reliability of the diagnosis component, which is central to the proposed framework.

Overall, the rebuttal strengthens the empirical evaluation and resolves some of my original concerns, particularly those related to evaluation consistency and experimental reporting. Nevertheless, I believe the remaining issues regarding deployment assumptions, fault localization reliability, and error propagation are important enough to maintain my original assessment. Therefore, I will keep my score at 2.

AuthorsOpenReview ↗

Follow-up Response to Reviewer m36H

Thank you for taking the time to review our responses and additional experiments. We address the remaining concerns with further analyses of the existing results below.

1. Reference reliability and incorrect repairs. The 101 newly introduced errors should be considered alongside the 419 corrected errors: under the Appendix D.6 protocol, the system yields a net gain of 318 correct answers across 4,500 trajectories. We further stratified the actually repaired cases by whether the verifier-generated reference matches the gold answer under EM:

Verifier referenceRepaired casesWrong → correctCorrect → wrongNet gain
Matches under EM3041623+159
Does not match under EM1,82625798+159

Thus, the positive net benefit persists in the subset with nonmatching references, after accounting for harmful repairs. This supports robustness to imperfect reference signals under the evaluated configuration.

2. Reliability of retained prefixes. To go beyond the qualitative examples, we aligned the original repair outcomes with the localization annotations across all nine framework–dataset combinations. We specifically examined predictions later than the reference failure position, where an earlier error may remain in the retained prefix:

Subset with late localizationSuccessfully repaired / casesRepair rate
All error types66 / 34219.3%

These results provide quantitative evidence that successful repair does not require exact localization in every case, supporting some robustness to localization errors.

3. Localization and annotation reliability. The reported 70.9% measures exact action-level agreement with the reference annotations. Table 3 further evaluates whether these predicted locations improve repair outcomes. Averaged across the three frameworks, the ΔEM results are:

DatasetWithout localization: repair from the beginningDR-RAG: repair using predicted locations
HotpotQA19.528.9
2Wiki18.323.8
MuSiQue8.611.4

Using predicted locations consistently improves repair performance across all three datasets. This demonstrates that the localization component provides useful guidance even with imperfect predictions.

Regarding annotation reliability, Appendix D.2.1 provides a complementary check for error classification: on the same subset, accuracy is 80.0% against human labels and 82.0% against teacher labels. This agreement supports consistency in the classification evaluation; the 70.9% localization result separately measures agreement with model-assisted location annotations.

Reviewer_fgdaOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents a framework for repairing RAG failures on an agent trajectory level. It provides a taxonomy of different RAG failure modes (format error, reasoning error, retriever error, search error), and provides a RAG repairing framework that:

  • For format errors, rewrites the answer to comply with the needed format.
  • For reasoning errors, it locates the problematic step in the thinking trajectory, and regenerates the outputs from there, but this time filtering out the problematic outputs.
  • For retriever errors, it rewrites the queries based on feedback, and executes the searches with the new queries.
  • For search errors, it regenerates all reasoning and searching performed after the retriever step. The framework is evaluated across three datasets (HotPotQA, 2Wiki, and MuSiQue), as well as three agentic baselines (ReAct, Search-o1, and Search-R1), and two models (Llama-3.1-8b and Qwen3-8b).
Critical strengths

The paper contributes a taxonomy for classifying RAG failures, separating errors with the agent having sufficient and insufficient knowledge, then separating the sufficient knowledge errors into format and reasoning failures, and separating the insufficient knowledge errors into retriever and search errors.

  • The paper provides a framework for solving such failures during the model’s trajectory, and tests it on two models across three datasets and three agentic baselines.
Critical weaknesses

RAG is not used as much nowadays, and has been replaced by other techniques like skills, MCPs, and memory layers.

  • Frontier models would likely not need this technique since they wouldn’t have many reasoning, formatting, retrieval, or search errors that can be solved by an agent but wouldn’t be generated correctly in the first place.
Questions for authors

What impact do you think this paper could have on current agentic systems? As of right now, I am not too sure on how this paper could help the field.

2 comments
AuthorsOpenReview ↗

Response to Reviewer fgda

Thank you for raising these concerns about the relevance of our work to current agentic systems. We address the two points below.

W1. Relationship to skills, MCP, and memory layers

Skills, MCP, and memory layers serve different roles and can coexist with retrieval-augmented generation. Skills provide task procedures, MCP provides interfaces to tools and data sources, and memory layers manage persistent information. These mechanisms do not inherently eliminate ineffective queries, missing evidence, or reasoning errors.

Our framework targets agentic trajectories involving multi-step reasoning, search, and tool interactions, which are also central to many current agent architectures. It addresses how to diagnose failures within these trajectories and select appropriate repair actions. Adopting skills, MCP, or memory therefore does not remove the underlying problem our work studies.

W2. Necessity of repair for frontier models

The ability to repair an error retrospectively does not imply that the error would necessarily have been avoided during initial generation. A repair process can inspect the complete trajectory, including tool observations and downstream evidence that were unavailable when an earlier decision was made. Consequently, even the same model may correct an error during retrospective diagnosis that it failed to avoid during its initial execution.

Moreover, higher initial accuracy does not imply that the remaining failures have no repair value. Frontier models can still make errors during multi-step reasoning and tool interactions, particularly when facing ambiguous queries, incomplete external evidence, or errors propagated from earlier decisions. Improvements in model capability do not, by themselves, eliminate the value of trajectory diagnosis and repair.

Beyond online recovery, trajectory repair also provides an application path for offline SFT data processing. By identifying erroneous steps, preserving valid portions, and repairing affected continuations, the framework can help recover failed trajectories that would otherwise be discarded. After verification, these repaired trajectories can serve as candidate SFT data. Its relevance therefore extends to both agent failure recovery and training-data construction.

Dear Authors

Thank you so much. it gives better insight into your paper and I really like your point of "higher initial accuracy does not imply that the remaining failures have no repair value". While I plan to keep my rating, I do have a better estimation of your paper. Thank you!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary for the Area Chair — Submission 29530

Dear Area Chair,

Thank you for reviewing our submission. We summarize the rating timeline, concerns, and responses below.

1. Rating timeline

ReviewerNov. 6, 20:27Nov. 6, 22:36 — no author responseCurrent (Nov. 19)
VTf4422
m36H222
fgda222

All dates are in 2026, China Standard Time (UTC+8). VTf4 lowered the rating from 4 to 2 within 2 hours and 9 minutes, before any author response.

2. Concerns and responses

W = initial weakness; F = post-rebuttal concern.

IDConcernResponse and evidence
VTf4-W1Search and Retriever errors are difficult to distinguish.Clarified the boundary with examples; misclassified trajectories remain repairable (17.2%, versus 19.8% with teacher-label routing; Table 9).
VTf4-W2Recovery on MuSiQue is limited.On MuSiQue/Search-o1, recovery is 18.1% vs. 7.3% for Rerun, at 250 s vs. 817 s (Tables 2, 4).
VTf4-W3Successful repair depends on accurate diagnosis.Predicted diagnosis outperforms removing classification or localization, despite diagnostic errors (Table 3).
VTf4-F1Recovery on complex multi-hop tasks remains limited.Across all three MuSiQue trajectory sources, DR-RAG improves recovery over Rerun while reducing processing time (Tables 2, 4).
VTf4-F2Presentation remains insufficiently clear.Requested specific passages or definitions needing clarification.
m36H-W1Gold-answer dependence.Expanded D.6 to nine settings: 419 corrections − 101 regressions = 318 net gains over 4,500 trajectories, using verifier references.
m36H-W2Candidate-set scoring is not comparable with single-answer EM.Single-answer evaluation retains the lead: 19.96% vs. 12.99% (Qwen) and 13.45% vs. 8.49% (LLaMA), against the strongest baselines.
m36H-W3Reused prefixes may retain upstream errors.Clarified truncation before the earliest erroneous action, including upstream reasoning, with concrete examples.
m36H-W4No direct localization evaluation.Added nine-setting exact localization agreement: 70.9% (1,946/2,745) against Qwen3.5-Plus annotations.
m36H-W5Identical base scores across repair backbones.Both repair backbones share Qwen3-4B input trajectories; identical base scores are expected.
m36H-W6Task formulation is too brief.Will clarify task inputs, outputs, and reference-access assumptions.
m36H-F1References may be unreliable or unavailable; the 101 newly introduced errors raise reliability concerns.Even with EM-nonmatching references, 257 corrections − 98 regressions = +159, supporting robustness to imperfect references under the D.6 configuration.
m36H-F2Preserved prefixes may propagate errors; examples alone do not establish robustness.Across nine settings, 66/342 (19.3%) late-localization cases are repaired: exact localization is not required for every successful repair.
m36H-F3Localization agreement of 70.9% and model-assisted reference annotations leave reliability concerns.Predicted locations improve repair on all three datasets (Table 3). Classification accuracy is 80% against human labels vs. 82% against teacher labels; localization agreement is separately model-referenced.
fgda-W1RAG has been replaced by skills, MCP, and memory layers.Skills, MCP, and memory complement retrieval; they do not eliminate evidence or reasoning failures.
fgda-W2Frontier models may not need repair; the impact on current agents is unclear.Retrospective repair uses later evidence; verified trajectories may support training. Frontier-model and training gains remain untested.

Discussion status: m36H acknowledged the deployment evidence, single-answer protocol, localization results, and shared-trajectory clarification, while retaining concerns F1–F3. fgda assigns soundness 3 (“I have no substantive concerns about the paper’s soundness”); the main objections concern research relevance.

We would be grateful if the AC could take the above clarifications and additional experiments into account in the final assessment. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.