acceptodds
Under review as a conference paper at ICLR 2027

TriageRAG: Failure-Aware Self-Training for Adapting Retrieval-Augmented Language Models to Specialized Domains

Abstract

Retrieval-augmented generation (RAG) lets large language models (LLMs) answer knowledge-intensive questions from retrieved evidence, but adapting a general-purpose RAG system to a specialized domain such as medicine is difficult when only an unlabeled corpus is available. Self-training approaches such as SimRAG address this by letting the LLM write its own question-answer pairs from domain passages and fine-tuning on the pairs that survive a round-trip consistency filter. We observe that these approaches treat every retained example identically: they never ask whether the student already knows the answer, already extracts it correctly from the retrieved context, or fails—and if so, whether the failure lies with the reader or with the retriever. To tackle this, we propose TriageRAG, a failure-aware self-training approach that probes the student on each self-generated question with and without retrieved context, triages it into one of four categories (known, solved, context-failure, retrieval-failure), and routes each category to the training signal that addresses it, including a parametric route that teaches the reader to answer when retrieval misses. Probing is cheap and repeated every round, so the training mixture follows the student, and the triage histogram doubles as a label-free diagnostic of where the domain gap lies. Experiments on 11 datasets spanning three domains and two backbone sizes demonstrate that TriageRAG outperforms a budget-matched SimRAG by 1.2–6.4 points and is the only method that recovers when retrieval hurts the base model. A single ablation identifies the mechanism: round-trip filtering removes every training context that lacks the evidence, and a reader that never sees such contexts learns to over-trust noisy retrieval.

Primary area:
foundation or frontier models, including LLMs
Keywords:
retrieval-augmented generation, self-training, domain adaptation, synthetic data, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pn5FOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies failure-aware self-training for adapting retrieval-augmented language models to specialized domains using unlabeled corpora. Existing approaches such as SimRAG rely on round-trip filtering to select synthetic question-answer pairs, but this process does not distinguish whether the student already knows the answer, can extract it from retrieved evidence, or fails because of the reader or retriever.

To address this limitation, the authors propose TriageRAG, which probes the student under closed-book and open-book conditions and categorizes synthetic examples into four groups: known, solved, context-failure, and retrieval-failure. Different categories are assigned different training strategies. In particular, the parametric route trains the model to produce correct answers even when the retrieved context lacks sufficient evidence.

The authors evaluate TriageRAG on 11 datasets across medical, scientific, and computer science domains using Qwen3-4B, with an additional Qwen3-8B evaluation on CS-Bench. Results show improvements over a budget-matched SimRAG implementation. Ablation studies further suggest that retaining evidence-absent training contexts is important for mitigating the negative effects of unreliable retrieval.

Critical strengths

S1: Problem Formulation and Method Design.

The paper addresses an important problem in domain-specific RAG adaptation by distinguishing retrieval failures from reader failures during self-training. The proposed failure-aware categorization is intuitive and well motivated. In particular, the idea of retaining training examples whose retrieved contexts lack the required evidence is interesting, as conventional round-trip filtering may encourage the model to over-rely on retrieval. This observation provides a useful perspective on improving the robustness of RAG systems when retrieval is imperfect.

S2: Experimental Evaluation and Mechanistic Analysis.

The experiments are generally well designed and cover 11 datasets across three domains using Qwen3-4B, with an additional evaluation using Qwen3-8B on CS-Bench. The authors include budget-matched comparisons with SimRAG, random selection, and RAFT-style training, as well as ablation studies and multi-seed evaluations. The parametric-route ablation in Table 4 provides evidence that training on evidence-absent contexts contributes substantially to the observed improvements. The analysis of retrieval failures and the negative results from retriever fine-tuning are also informative.

Critical weaknesses

W1: Limited Independent Contribution of Four-Way Triage.

TriageRAG's gains over random selection are small with Qwen3-4B (0.69, 0.11, and 0.61 points across the three domains), with confidence intervals including zero for science and CS. Moreover, random selection and RAFT-style training also benefit from evidence-absent contexts, while the parametric-route ablation changes evidence availability. Since the baselines differ in their proportions of evidence-absent contexts, the experiments do not fully isolate the benefit of four-way triage from that of evidence-absent training.

W2: Reliability of Failure Categorization and Synthetic Supervision.

The evidence-hit heuristic treats source-passage retrieval or answer-string overlap as sufficient evidence, although neither guarantees semantic support. Categorization also relies on only four probing samples per condition and a fixed threshold of 0.75, while the three training seeds share the initial probe. Furthermore, self-answerability filtering does not independently verify the correctness or unambiguity of synthetic QA pairs. These limitations raise concerns about the reliability of failure assignments and subsequent supervision.

W3: Insufficient Analysis of Retrieval-Induced Degradation.

Although retrieval-induced performance degradation is a central motivation, the paper does not quantify misleading or conflicting retrieved contexts or distinguish their effects from missing evidence. More importantly, examples answered correctly without retrieval but incorrectly with retrieval are classified as known, potentially overlooking a key failure mode. The reported evidence-hit statistics also concern synthetic training pools rather than evaluation questions. Finally, the parametric route is not evaluated for unsupported answering, abstention, or calibration, leaving its actual robustness benefits unclear.

Questions for authors

Q1: Independent Benefit of Triage.

Could the authors compare TriageRAG with a non-triaged baseline matched for evidence-absent context ratios, question formats, and training budget? Would simpler evidence-hit-based selection achieve comparable results?

Q2: Overlooked Retrieval-Induced Failures.

What proportion of examples have high closed-book but low open-book accuracy ((p_{cb}\geq\tau), (p_{ob}<\tau))? Since these are classified as known, how does TriageRAG address this failure mode?

Q3: Sources of Retrieval Degradation.

Could the authors quantify missing, irrelevant, misleading, and conflicting evidence on evaluation questions? How often does retrieval turn correct closed-book predictions into incorrect ones, and how does TriageRAG affect this rate?

Q4: Reliability of Failure Diagnosis.

Could the authors manually validate evidence-hit labels and synthetic QA correctness? How stable are the failure assignments under different probing sample counts, thresholds, and independent probing seeds?

Q5: Parametric Learning Versus Retrieval Robustness.

How does the parametric route compare with closed-book SFT on the same QA pairs? Under misleading or unanswerable contexts, does it improve accuracy without increasing unsupported answers or degrading abstention and calibration?

Reviewer_D8hbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TriageRAG, a failure-aware self-training framework that adapts retrieval-augmented LLMs to specialized domains using an unlabeled corpus. Specifically, TriageRAG improves SimRAG in Stage 2 self-training by probing the student LLM’s ability on question-answer-document triplets, triaging them by failure type, and treating each type with a different approach. Experiments on multiple specialized domains, such as medical, computer science, and science, show negligible performance gains. Overall, the proposed TriageRAG may bring additional latency and complexity, yet bring little gain for adapting retrieval-augmented LMs to specialized domains.

Critical strengths
  1. The motivation of the paper is interesting and promising. The second stage of SimRAG, domain adaptation through self-training, indeed treats all training triplets equally. The round-trip consistency filter might not be optimal. Based on the limitations of SimRAG, the paper proposes to triage the generated training triplets and treat them differently, which is unique and interesting.
  2. This paper is generally easy to follow and is sound, with solid experiments and ablation studies.
Critical weaknesses
  1. Overall, the novelty and contributions of the paper are limited. The paper largely builds upon SimRAG and attempts to address one of its limitations: how training triplets are handled during the domain-adaptive self-training phase. Both probing the student LLM's knowledge through open-book and closed-book settings and classifying questions into known, unknown, context-failure, and retrieval-failure categories are not novel ideas. Therefore, the proposed method primarily combines existing techniques and represents an incremental improvement over SimRAG. Furthermore, the performance gains over the baselines (SimRAG, RAFT, and random selection) are marginal, particularly in the medical and scientific domains, raising concerns about the generalizability of the proposed method across different domains.
  2. In general, the random selection baseline achieves performance comparable to, or only slightly worse than, TriageRAG. Counterintuitively, multiple rounds of TriageRAG provide no additional improvement and sometimes even lead to worse performance compared to a single round. This suggests that the proposed method introduces additional complexity and computational overhead while offering only marginal performance gains. However, the paper lacks a detailed analysis or explanation of why multiple rounds fail to improve performance.
Questions for authors
  1. Why is Backbone + RAG worse than Backbone (closed-book) in the scientific and CS domain?
  2. Why do multiple rounds of TriageRAG provide no additional improvement and sometimes even lead to worse performance compared to a single round?
Reviewer_koY4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper builds on SimRAG, a self-training alternative to Retrieval-Augmented Finetuning (RAFT), whereby the model writes its own training question and answer pairs from passages in the corpus, and then passes them through a quality filter to keep only the pairs where the correct evidence was retrieved. The authors argue this is problematic because the model then never trains on cases where the evidence is poor, thus over-relying on the evidence, and perhaps handicapping the model in cases where it would have otherwise answered correctly. Specifically, in their implementation, TriageRAG, they drop this filter and split all generated pairs into four distinct families: known (i.e. no need for evidence), solved (i.e. evidence was the clear catalyst for answering), context failure (i.e. the model answered poorly despite sufficient evidence), and retrieval failure (i.e. the evidence was not retrieved and the model cannot answer without it), and utilize them during training with a different kind of training example and quota for each group. They then benchmark their approach versus both SimRAG and other simpler baselines on Qwen3-4B and Qwen3-8B models (the latter on computer science only) on 11 multiple-choice and yes/no datasets in medicine, science, and computer science.

Critical strengths
  1. This paper provides a good, well-structured argument about how SimRAG style self-training may underperform. All of the second-stage models have the same first-stage model, synthetic pool of questions, number of training examples and training method, so when the parametric route is removed from TriageRAG, it falls back to SimRAG's level in each of the three domains, and full TriageRAG beats this ablated version on all three seeds (Tables 4 and 5). I think this is the most interesting insight of the paper, as it demonstrates that excluding training contexts that lack the evidence will hurt performance.

  2. The paper has a straightforward, easy-to-follow presentation of the ideas presented in the paper, and the experiments are well controlled and honestly reported. The authors present their results fairly and do a good job in reporting on their comparisons of SimRAG, random sampling, and the ablation of the parametric pathway (i.e., the ablation of the parameteric route to get to TriageRAG), with three seeds and paired bootstrap intervals. In addition to clearly stating that they believe that adding a triage component will add little value beyond random sampling at 4B, the authors also report on a negative experiment on retriever tuning (Table 7), which is refreshing to see.

Critical weaknesses

My main concern is that the results do not show a clear benefit from the triage itself, as at 4B, TriageRAG is nearly equivalent to simpler baselines and most of its gain over SimRAG comes from removing SimRAG's round-trip filter. Namely, a baseline that simply picks training questions at random from the whole pool, without the filter, already recovers most of TriageRAG's gain over SimRAG on CS (Section 5.1). On top of that, triage adds +0.6 [-0.9, 2.1] on CS and +0.1 [-0.4, 0.6] in science, and both confidence intervals contain zero. The interval lies above zero only in medicine (+0.7) and on CS at 8B (+2.3), and CS is the only domain assessed at 8B. A RAFT-style baseline (Zhang et al., 2024) is similarly close to TriageRAG, but the authors ran it only for medicine and CS at 4B, with one seed. Furthermore, the abstract describes TriageRAG as the only method that reaches the closed-book level everywhere retrieval hurts, but this holds only on seed 0, by 0.2 to 0.3 points, and on the three-seed average for CS at 4B TriageRAG falls below closed book (67.47 vs. 67.99, Table 5). Finally, the random baseline is not a matched control, since its share of missing-evidence contexts ranges from 10% to 46% across domains while TriageRAG stays near 21% (Appendix C). Taken together, the data largely support a simpler statement than the title: also train on contexts that lack the evidence, similar to RAFT. Unless triage clearly outperforms a control matched on that share, I feel the paper would be more accurately described as a study of round-trip filtering and missing-evidence training contexts.

Questions for authors

I would like to see two additional ablations: random selection with the same share of missing-evidence contexts as TriageRAG in each domain, and the RAFT-style baseline extended to science and to CS at 8B. Please run both in all three domains at 4B and on CS at 8B with the three seeds of Table 5 and report the closed-book claim on three-seed means.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.