- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies failure-aware self-training for adapting retrieval-augmented language models to specialized domains using unlabeled corpora. Existing approaches such as SimRAG rely on round-trip filtering to select synthetic question-answer pairs, but this process does not distinguish whether the student already knows the answer, can extract it from retrieved evidence, or fails because of the reader or retriever.
To address this limitation, the authors propose TriageRAG, which probes the student under closed-book and open-book conditions and categorizes synthetic examples into four groups: known, solved, context-failure, and retrieval-failure. Different categories are assigned different training strategies. In particular, the parametric route trains the model to produce correct answers even when the retrieved context lacks sufficient evidence.
The authors evaluate TriageRAG on 11 datasets across medical, scientific, and computer science domains using Qwen3-4B, with an additional Qwen3-8B evaluation on CS-Bench. Results show improvements over a budget-matched SimRAG implementation. Ablation studies further suggest that retaining evidence-absent training contexts is important for mitigating the negative effects of unreliable retrieval.
Critical strengths
S1: Problem Formulation and Method Design.
The paper addresses an important problem in domain-specific RAG adaptation by distinguishing retrieval failures from reader failures during self-training. The proposed failure-aware categorization is intuitive and well motivated. In particular, the idea of retaining training examples whose retrieved contexts lack the required evidence is interesting, as conventional round-trip filtering may encourage the model to over-rely on retrieval. This observation provides a useful perspective on improving the robustness of RAG systems when retrieval is imperfect.
S2: Experimental Evaluation and Mechanistic Analysis.
The experiments are generally well designed and cover 11 datasets across three domains using Qwen3-4B, with an additional evaluation using Qwen3-8B on CS-Bench. The authors include budget-matched comparisons with SimRAG, random selection, and RAFT-style training, as well as ablation studies and multi-seed evaluations. The parametric-route ablation in Table 4 provides evidence that training on evidence-absent contexts contributes substantially to the observed improvements. The analysis of retrieval failures and the negative results from retriever fine-tuning are also informative.
Critical weaknesses
W1: Limited Independent Contribution of Four-Way Triage.
TriageRAG's gains over random selection are small with Qwen3-4B (0.69, 0.11, and 0.61 points across the three domains), with confidence intervals including zero for science and CS. Moreover, random selection and RAFT-style training also benefit from evidence-absent contexts, while the parametric-route ablation changes evidence availability. Since the baselines differ in their proportions of evidence-absent contexts, the experiments do not fully isolate the benefit of four-way triage from that of evidence-absent training.
W2: Reliability of Failure Categorization and Synthetic Supervision.
The evidence-hit heuristic treats source-passage retrieval or answer-string overlap as sufficient evidence, although neither guarantees semantic support. Categorization also relies on only four probing samples per condition and a fixed threshold of 0.75, while the three training seeds share the initial probe. Furthermore, self-answerability filtering does not independently verify the correctness or unambiguity of synthetic QA pairs. These limitations raise concerns about the reliability of failure assignments and subsequent supervision.
W3: Insufficient Analysis of Retrieval-Induced Degradation.
Although retrieval-induced performance degradation is a central motivation, the paper does not quantify misleading or conflicting retrieved contexts or distinguish their effects from missing evidence. More importantly, examples answered correctly without retrieval but incorrectly with retrieval are classified as known, potentially overlooking a key failure mode. The reported evidence-hit statistics also concern synthetic training pools rather than evaluation questions. Finally, the parametric route is not evaluated for unsupported answering, abstention, or calibration, leaving its actual robustness benefits unclear.
Questions for authors
Q1: Independent Benefit of Triage.
Could the authors compare TriageRAG with a non-triaged baseline matched for evidence-absent context ratios, question formats, and training budget? Would simpler evidence-hit-based selection achieve comparable results?
Q2: Overlooked Retrieval-Induced Failures.
What proportion of examples have high closed-book but low open-book accuracy ((p_{cb}\geq\tau), (p_{ob}<\tau))? Since these are classified as known, how does TriageRAG address this failure mode?
Q3: Sources of Retrieval Degradation.
Could the authors quantify missing, irrelevant, misleading, and conflicting evidence on evaluation questions? How often does retrieval turn correct closed-book predictions into incorrect ones, and how does TriageRAG affect this rate?
Q4: Reliability of Failure Diagnosis.
Could the authors manually validate evidence-hit labels and synthetic QA correctness? How stable are the failure assignments under different probing sample counts, thresholds, and independent probing seeds?
Q5: Parametric Learning Versus Retrieval Robustness.
How does the parametric route compare with closed-book SFT on the same QA pairs? Under misleading or unanswerable contexts, does it improve accuracy without increasing unsupported answers or degrading abstention and calibration?