acceptodds
Under review as a conference paper at ICLR 2027

AppRAG: When Relevant Evidence Does Not Apply in Retrieval-Augmented Generation

Abstract

Retrieval-augmented generation (RAG) uses retrieved text passages as evidence to answer knowledge-intensive questions. However, relevant and factually correct evidence may not satisfy the conditions of the reasoning step it is used to support. When the context can include only a limited number of passages, applicable evidence may rank too low to be selected. Meanwhile, individually useful passages may not jointly support all necessary reasoning steps. To address these evidence-selection challenges, we introduce AppRAG, a three-stage framework for selecting applicable and complementary evidence. First, requirement-conditioned applicability modeling assesses whether evidence supports question-derived requirements under matching conditions, distinguishing valid partial support from condition mismatch while keeping unresolved evidence separate. Second, applicability-guided embedding calibration uses these assessments to update the query representation and rerank available candidates, verifying evidence replacements before updating the context. Third, applicability-aware evidence selection constructs complementary evidence sets under a per-context document budget and compares the answer drafts generated from each set. The answer generator remains frozen throughout. Our analysis characterizes when missing condition-match information imposes unavoidable evidence-selection loss. On evaluation sets from HotpotQA, MuSiQue, and 2WikiMultiHopQA, AppRAG improves Qwen3-8B's overall token-level F1 by 2.82-13.43 pp across all eight tested RAG integrations, including a 10.61-point gain over Vanilla RAG. Our code is available at https://anonymous.4open.science/r/AppRAG-C2E8

Primary area:
foundation or frontier models, including LLMs
Keywords:
Retrieval-Augmented Generation, Large Language Models, Applicability-Aware Evidence Selection, Multi-hop Question Answering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Ki4ROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies evidence applicability in RAG, where relevant and factually correct passages may not satisfy the conditions of their intended reasoning steps. The authors propose AppRAG, combining requirement-conditioned applicability modeling, query embedding calibration with verified context replacement, and complementary evidence selection through draft comparison and a risk gate. The paper also analyzes selection loss caused by missing condition information. Experiments on three multi-hop QA benchmarks show overall F1 improvements across eight RAG integrations, supported by ablations and evidence-ranking analyses

Critical strengths
  1. The paper addresses a meaningful limitation of relevance-based evidence selection. Its distinction between valid partial support, condition mismatch, and unresolved evidence is useful for multi-hop reasoning, where intermediate evidence may support only part of the required inference.

  2. AppRAG integrates applicability assessment, verified ranking updates, and complementary context construction into a coherent framework. Experiments show consistent overall F1 gains across eight RAG integrations. Fixed-candidate ranking analyses and verification ablations further support the effectiveness of its main components.

Critical weaknesses
  1. The independent benefit of applicability-specific modeling needs clearer validation. Appendix C.10 reports statistically unresolved gains over ordinary requirement-based selection, while generic multi-draft SetR-style selection performs comparably in Table 12. Since the main comparisons do not match total computation, it remains unclear how much improvement comes from applicability modeling versus additional verification and draft selection.

  2. Validation of the core IA distinction is incomplete. The human evaluation covers A/IR labels but excludes IA and U, and the judge used for Cov and answer-level IA-risk failed prespecified quality checks. Additionally, the context IA-hit comparison uses contexts of different lengths. These limitations weaken the applicability diagnostics, although the reference-based EM/F1 improvements remain informative.

Questions for authors
  1. Which controlled comparison best isolates the benefit of applicability-specific signals over ordinary requirement coverage and generic multi-draft selection? Please clarify the effect size and uncertainty under comparable candidate access and inference budgets.

  2. Can the authors provide human validation of IA labels, particularly for valid bridge evidence and passages that support one requirement but mismatch another? Which quality checks did the Cov/IA-risk judge fail, and how does this affect the reported diagnostics?

  3. Does the reduction in IA exposure remain under a common context-length budget? Please also clarify the complete inference cost relative to the strongest selection baseline, including scoring, verification, and draft generation.

3 comments
AuthorsOpenReview ↗

Responses to Reviewer Ki4R (Part 1/2)

Thank you for your valuable comments and insights. We have summarized the points of concern and would like to address them as follows.

Response to W1 & C1

The fixed-setting head ablation in Table 5 isolates the value of retaining the four applicability categories: the four-class head improves F1 by 1.62 and 1.58 pp over Generic and A-only. The heads share training examples, initialization, architecture, training steps, and coverage supervision. Candidate construction, verifier, and draft budget are fixed. Generic groups A/IA/U against IR; A-only separates A from the other labels.

ControlF1 (%) ↑AppRAG − control (pp) ↑ [95% CI]Holm p
Generic, fixed59.60+1.62 [0.65, 2.58]0.0063
A-only, fixed59.64+1.58 [0.63, 2.55]0.0096
Generic, independently tuned60.31+0.91 [-0.03, 1.83]0.2048
A-only, independently tuned59.88+1.34 [0.30, 2.35]0.0714

The fixed-setting gains support the value of retaining the four applicability categories and pass Holm correction across the 13 F1 comparisons. Independently tuned comparisons, using the same five development settings for each head, and the recomposition ablation remain statistically unresolved. The displayed intervals are unadjusted paired 95% bootstrap intervals.

The broader relevance/filter/SetR comparison addresses the separate question of the complete selection procedure. On the same held-out 1,500-question control study, all methods use Qwen3-8B generation, a common Qwen3.8-27B verifier/answer selector, nine drafts, and eight calls. AppRAG has the highest observed mean F1, with paired differences of +1.07/+0.90/+1.01 pp (Tables 4 and 21).

Response to W2 & C2

The new human study includes both cases. Two reviewers checked 1,950 requirement–passage units, including 1,770 natural and 180 enriched cases; 346 units with missing model labels were excluded. GPT and DeepSeek labels and explanations were visible during review. Figure 9 reports IA/U-specific Cohen κ:

ComparisonIA κ ↑U κ ↑
Human–human0.7790.664
GPT–DeepSeek0.5760.268
Human–GPT0.7890.700
Human–DeepSeek0.7860.525

Human–model entries average the two reviewers' κ values; Figure 9 provides question-cluster bootstrap intervals, and Table 10 reports four-category agreement. Within the 499 units with a model disagreement on at least one field, both reviewers confirmed 56 IA, 24 U, 38 partial-support, and 56 bridge-support cases, with overlap between groups.

Both also confirmed five passages as A for one requirement and IA for another, including three natural-sample cases. For example, a passage about Scarlett O'Hara identifies Vivien Leigh as the actress but lists the character's spouses. It supports identifying the actress, while using the listed spouses to answer whom the actress married mismatches the required person. These AI-assisted judgments extend validation to IA/U and intermediate support, while retaining the possibility of shared labeling errors (Appendix C.2).

Table 24 specifies the Cov/answer-level IA-risk judge failures:

Task / stageFailed checkRequiredObserved ↑
Cov, final holdoutTwo-view agreement≥0.900.875
Requirement coverage, final holdoutTwo-view agreement≥0.900.875
IA-risk, second calibrationValid parse/source≥0.980.9375
IA-risk, second calibrationTwo-view agreement≥0.900.875
IA-risk, second calibrationAtomic accuracy≥0.900.875
IA-risk, second calibrationRISK-class accuracy≥0.800.750

IA-risk passed its final holdout but failed the acceptance rule under the prespecified limit of two calibration revisions. Cov and answer-level IA-risk therefore remain descriptive model-based measures, not validated semantic measures. Reference-based EM/F1 and the human category evaluation are assessed separately.

AuthorsOpenReview ↗

Responses to Reviewer Ki4R (Part 2/2)

Response to W2 & C3

At the same four-passage budget, both human label sets show lower IA-hit point estimates on the 150-question sample (Table 20; Appendix C.15):

Human labelsAppRAG − Vanilla IA-hit (pp) ↓ [95% CI]
Human A−4.00 [−12.00, 4.67]
Human B−12.00 [−20.00, −4.00]

In this separate saved-output analysis, overall F1 improves by 10.12 pp [5.48, 15.21]. Holm adjustment across 32 evidence-level comparisons leaves the IA-hit reduction statistically uncertain; the displayed 95% intervals are unadjusted. The common budget fixes passage counts, with token lengths varying and missing labels left unresolved for both systems. Together, the analyses show higher answer quality and a consistent downward direction in IA-hit point estimates across annotators.

For the strongest selection controls in Table 4, all four methods use nine drafts and eight 27B calls per query:

Method27B tokens (k/query) ↓8B tokens (k/query) ↓p50 (s) ↓p95 (s) ↓
Relevance35382680
LLM filtering35382982
SetR35382656
AppRAG38382565

The separate timing study covers requirement generation, retrieval, local scoring, fresh draft generation, verification, and answer selection, excluding model loading and warmup. Token totals cover 27B and 8B input/output and exclude shared requirement generation and additional retries; offline training is separate from inference (Appendix C.12).

The trade-off is higher observed mean F1 with about 7.2% more 27B tokens and a similar median latency, at matched draft and call budgets. Appendix C.12 provides the timing protocol and detailed cost boundaries.

Table 1 separately reports online output throughput (token/s) for the main integrations. REARANK changes from 127.66 to 128.45 token/s with a 0.64-point F1 gain; HippoRAG2 and LightRAG change from 311.54 to 327.07 and 196.41 to 200.62 token/s, with F1 gains of 2.16 and 2.95 points. Token/s divides total LLM output tokens, including intermediate verification and comparison outputs, by total inference time, excluding index construction. Non-graph measurements use three runs and include estimates for requirement-generation and comparison outputs; graph methods are measured separately (Appendix E.2). This rate describes output throughput; the matched-budget analysis above reports per-query token use and latency.

The authors’ response has addressed most of my concerns, I will maintain my current rating.

Reviewer_TdWYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AppRAG, a method for selecting evidence that is applicable to a question’s specific conditions in retrieval-augmented generation. AppRAG combines a set on signals including requirement-conditioned applicability scoring, query-embedding calibration with verified context replacements, and coverage-oriented evidence-set selection. It generates candidate answers from different evidence sets, compares the answer–evidence pairs, and accepts an alternative only when comparison judgments agree and it passes an applicability-risk check. The answer generator remains frozen throughout. Experiments on three benchmarks report improvements across multiple RAG settings.

Critical strengths
  1. The paper addresses a meaningful problem: retrieved evidence can be relevant to a question but still fail to support the answer under the required conditions.

  2. AppRAG shows improvements in average F1 across different RAG settings, suggesting that the method can be useful.

Critical weaknesses
  1. The proposed method involves a complex pipeline, but it is unclear how much the additional stages contribute beyond reranking. A comparison with a strong reranker using the same candidate pool and a comparable inference budget would help justify this complexity. This is closely-related to my concern on the novelty. Without a clear explanation on the main technical contribution, AppRAG appears to be primarily a well-engineered combination of existing techniques.

  2. Several results in the main table are identical across different methods after adding AppRAG. Do these configurations select the same evidence and produce the same answers, or is there another explanation for the identical scores? Without clarification, it is difficult to determine whether the reported gains reflect improvements.

Questions for authors
  1. Can the authors explain why Table 1 contains many identical performance numbers across different methods after adding AppRAG?

  2. How much additional improvement does this complex pipeline provide over a strong reranker using the same candidate pool and a comparable inference budget?

  3. The Vanilla RAG comparison changes both the candidate pool and the evidence-selection method. How much of the reported gain comes from the larger candidate pool?

1 comment
AuthorsOpenReview ↗

Responses to Reviewer TdWY

Thank you for your valuable comments and insights. We have summarized the points of concern and would like to address them as follows.

Response to W2 & C1

Vanilla+AppRAG and RRF+AppRAG share the same selected contexts and reuse the same sampled answers. Their identical 41.43 overall F1 therefore comes from one shared selection and generation output set, evaluated against two baselines; Table 1 marks this sharing. For RankZephyr+AppRAG and REARANK+AppRAG, the outputs have high overlap: across three runs on 1,500 questions, 98.47% and 98.64% of their 4,500 answer strings match Vanilla+AppRAG, and the ordered contexts match on 93.67% and 94.09%.

Section 4.2 and Appendix E.2 now distinguish the shared Vanilla/RRF output set from the high overlap of the other integrations. AppRAG can update the ranking and final context within each interface's candidate access, which explains how different upstream rankings can lead to similar final answers.

Response to W1 & C2

With candidate access, nine drafts, and eight auxiliary-model calls matched, AppRAG has mean F1 61.22, compared with 60.15 for relevance ranking, 60.32 for LLM filtering, and 60.21 for SetR. This is a quality–cost trade-off: AppRAG uses about 7.2% more 27B tokens, with comparable observed median latency under matched calls and drafts. The paired F1 differences are +1.07, +0.90, and +1.01 pp, respectively (Tables 4 and 21).

All controls share Qwen3-8B generation with ACE prompting, the Qwen3.8-27B verifier/answer selector, and a separate held-out set of 500 questions per dataset. AppRAG uses retrained heads and recomposition with its risk gate disabled, under settings fixed on development data (Section 4.5; Appendix C.12).

For the main integrations, Table 1 also reports token/s: 45.58→89.84 for Vanilla and 127.66→128.45 for REARANK, alongside F1 gains of 8.46 and 0.64 pp. This rate divides total LLM output tokens, including intermediate verification and comparison outputs, by total inference time, excluding index construction. Non-graph measurements use three runs and include estimated requirement-generation and comparison tokens (Appendix E.2). It complements the matched-budget token and latency analysis above; a higher output rate alone does not establish lower per-query cost.

The technical contribution is requirement-conditioned applicability modeling: distinguishing valid intermediate support, a mismatch in the conditions of use, and unresolved evidence. Table 5 isolates the value of retaining these label distinctions within a fixed pipeline, holding coverage supervision, candidate construction, verifier, and draft budget fixed:

Fixed-setting controlAppRAG − control (pp) ↑ [95% CI]Holm p
Generic, fixed+1.62 [0.65, 2.58]0.0063
A-only, fixed+1.58 [0.63, 2.55]0.0096

These comparisons pass Holm correction across the 13 F1 contrasts. The clearest positive evidence is therefore for retaining applicability labels under fixed decision settings; the complete procedure has smaller observed mean gains over the strong selectors in Tables 4 and 21. Independently tuned head comparisons and the recomposition ablation also remain unresolved.

Response to C3

Table 6 holds each selection method fixed and changes candidate access:

Selection methodOriginal pool F1 ↑Expanded pool F1 ↑Pool gain (pp) ↑
Strong relevance22.6126.79+4.18
AppRAG21.4925.00+3.51
Relevance + applicability23.0627.42+4.36

AppRAG gains 3.51 F1 points from expansion in this study; strong relevance gains 4.18. Applicability alone remains below strong relevance in both pools. Combining relevance and applicability gives the highest point estimate in each pool, exceeding strong relevance by 0.45 and 0.63 pp. This pattern suggests using applicability alongside relevance as an additional selection signal. Paired intervals for the combination are unavailable, so these are descriptive differences.

This separate candidate-pool configuration quantifies an expansion effect without changing selection. It is not a decomposition of the new main-table +8.46 gain, whose generation settings differ. The fixed-setting head ablation in Table 5 provides the direct comparison of label information at fixed candidate access.

Reviewer_fiHmOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AppRAG, a three-stage evidence-selection framework for RAG (applicability scoring, query-embedding calibration with LLM-verified document exchange, and coverage-oriented multi-context construction with draft comparison and a risk gate) on top of a frozen generator. It reports F1 gains of 2.82–13.43 pp across eight RAG pipelines on HotpotQA, MuSiQue and 2Wiki, plus a theoretical appendix on selection loss and NP-hardness.

Critical strengths
  1. Separating "relevant" from "applicable", and keeping valid partial support, condition mismatch and unresolved (U) evidence distinct, is a useful framing.
  2. The authors report many negative or unresolved results and state what each experiment does not establish.
  3. The generator stays frozen, and the framework plugs into different retrieval interfaces.
Critical weaknesses
  1. Table 1 uses 90 questions per dataset (270 total; EM values are multiples of 1/90). At this scale, per-dataset F1 standard errors are several points, yet Table 1 reports no CIs or significance tests, while many "gains" in the 2–6 pp range are interpreted as consistent improvements

  2. The largest gain (+13.43) is on EntityLinkRAG, a custom baseline that is worse than Vanilla RAG (19.75 vs 25.21). ACE-RAG and EntityLinkRAG are the authors' own adaptations. Graph baselines index only 1,059 prefiltered documents (the union of BGE-M3 top-4 results), which is unrepresentative of HippoRAG2/LightRAG. On 2Wiki, Vanilla RAG (13.17 F1) is 27 points below LLM-only (40.48), which suggests the retrieval setup, not the method class, is the bottleneck. Gains are largest where baselines are weakest.

  3. AppRAG+Vanilla and AppRAG+RRF yield identical numbers (35.82), and AppRAG with RankZephyr/REARANK gives 35.85–36.04. So AppRAG largely replaces the upstream pipeline with its own BGE-M3∪Qwen3-Embedding union, three-reranker fusion, and Qwen3-32B verification/judging. Baselines receive top-4 passages from one retriever and one generation call. AppRAG uses a much larger model for verification and judging, ~2.9 drafts and ~4 LLM checks, with 11–18k tokens per query. The paper itself notes that the main comparison "fixes per-context counts, not total compute or auxiliary-model capacity". No compute-matched baseline is provided, such as a 32B reranker/filter, best-of-n with the same judge, or an equal-token SetR.

  4. Human validation (C.2) covers only the binary A/IR distinction; IA and U, the paper's central novelty, are explicitly excluded. Training labels come from Qwen3-32B, the same family as the online verifier/judge, so there is circularity and shared error. The Cov/IA-risk judge failed the authors' own quality checks. The gains are more plausibly explained by better multi-hop distractor filtering, bridge coverage and best-of-n selection than by "condition applicability".

  5. The motivating reimbursement scenario is never evaluated.

  6. The system has four trained modules and about 60 risk features, many thresholds and fallback rules, and many prompts, but there is no sensitivity analysis on the main benchmark.

  7. The text is extremely dense and heavily hedged, and the method is described through many abbreviations and nested rules, which hurts readability.

Questions for authors
  1. Please report CIs (paired bootstrap) for every row of Table 1, and results on larger evaluation sets (e.g., ≥500 questions per dataset, full dev sets). Do the gains hold for non-custom baselines?

  2. Why are AppRAG+Vanilla and AppRAG+RRF-ensemble numerically identical? How much of the final context actually depends on the upstream method? If AppRAG replaces the retrieval stage, in what sense is it a "complementary layer"?

  3. Can you provide compute-matched baselines using the same Qwen3-32B: LLM-based reranking/filtering, best-of-n drafts selected by the same judge (without applicability signals), and SetR with equal calls/tokens? Please report total tokens, calls and latency per method in the main table.

  4. Given Table 6/7/C.10 (applicability scoring ≈ relevance scoring; training objective differences unresolved; generic multi-draft SetR ≥ AppRAG-guided), what evidence isolates a contribution of applicability per se from (i) the 32B verifier and (ii) multi-draft judging?

  5. Since IA affects only ≈1–9% of items, how do you reconcile a ~4 pp IA reduction with a ~10 pp F1 gain? Please give a stratified analysis (questions with vs without detected IA in baseline contexts) and a length-matched IA-exposure comparison (same number of passages).

  6. Why does retrieval-augmented generation underperform LLM-only by ~27 F1 on 2Wiki in your setup? Is this due to prefiltered corpora or the benchmark's comparison-type questions? Would results hold with standard full-corpus retrieval for HippoRAG2/LightRAG?

  7. Is there any evaluation on a dataset that genuinely tests condition applicability (e.g., knowledge-conflict or enterprise-policy datasets like the Figure 1 example), and on out-of-domain data without in-distribution training?

  8. Which main-text configuration (verifier model, generator size, K, B, τ) is used for each table and figure? Please standardize it, or explain why Qwen3-4B (ablation) and Qwen3.6-35B-A3B (verification study) are used instead of the main system. Please also provide hyperparameter sensitivity on the main benchmark.

5 comments
AuthorsOpenReview ↗

Responses to Reviewer fiHm (Part 1/4)

Thank you for your valuable comments and insights. We have summarized the points of concern and would like to address them as follows.

Response to W1 & C1

The revised Table 1 evaluates the same 500 questions per dataset across all methods, for 1,500 questions in total. It reports three-run means, overall F1 confidence intervals for each method, and paired intervals for the AppRAG gain. Overall scores weight the three datasets equally. For the non-graph rows, we average each question's three scores, pairing baseline and AppRAG scores (Section 4.1; Appendix E.2).

PipelineBaseline F1 ↑ [95% CI]+ AppRAG F1 ↑ [95% CI]ΔF1 (pp) ↑ [95% CI]
Vanilla RAG32.97 [30.95, 35.09]41.43 [39.28, 43.55]+8.46 [6.68, 10.20]
HippoRAG232.18 [31.57, 34.30]34.35 [31.05, 35.13]+2.16 [1.78, 2.36]
LightRAG32.91 [30.46, 34.33]35.86 [32.13, 38.57]+2.95 [2.76, 3.15]
EntityLinkRAG28.97 [27.06, 30.98]41.33 [39.18, 43.45]+12.37 [10.38, 14.34]
ACE-RAG34.01 [31.93, 36.13]40.22 [38.06, 42.37]+6.20 [4.45, 7.98]
RRF37.60 [35.50, 39.64]41.43 [39.28, 43.55]+3.83 [2.22, 5.40]
RankZephyr37.03 [34.94, 39.11]41.43 [39.25, 43.56]+4.40 [2.81, 6.06]
REARANK40.69 [38.48, 42.81]41.33 [39.16, 43.45]+0.64 [-0.95, 2.24]

AppRAG improves mean F1 across all eight integrations. The Vanilla, RRF, and RankZephyr comparisons have positive paired gain intervals, while REARANK shows a smaller positive mean gain. EntityLinkRAG and ACE-RAG are task-specific adaptations, identified in Table 1 and Appendix E.1. All outputs are retained in scoring.

Response to W3 & C2

Vanilla+AppRAG and RRF+AppRAG share one set of selected contexts and reuse the same sampled answers. This shared selection and generation output explains their identical Table 1 scores: one AppRAG output set is compared against two different baselines. Across the three runs, the output comparison is:

Compared with Vanilla + AppRAGIdentical ordered contexts (%)Identical answer strings (%)
RRF + AppRAG100.00100.00
RankZephyr + AppRAG93.6798.47
REARANK + AppRAG94.0998.64

These percentages use 4,500 outputs per method from 1,500 independent questions. Section 4.2 and Appendix E.2 now explain the shared outputs and high answer overlap; Table 1 marks the shared Vanilla/RRF result.

AppRAG operates through the candidate-pool interface and can update the upstream ranking and final context. Standalone AppRAG expands Vanilla's candidates, while the named reranking integrations use their admitted candidate pool and rankings (Appendix E.1). Here, 'complementary' means adding applicability-guided selection to a retrieval interface. The results measure the combined procedure within each interface, with the shared outputs and overlap reported above.

AuthorsOpenReview ↗

Responses to Reviewer fiHm (Part 2/4)

Response to W3 & C3

We added relevance ranking, LLM filtering, and SetR-style selection with the same candidate access, nine drafts, and eight auxiliary-model calls. Table 4 reports answer quality and inference cost together:

MethodF1 (%) ↑27B tokens (k/query) ↓8B tokens (k/query) ↓Latency p50 (s) ↓Latency p95 (s) ↓
Relevance60.1535382680
LLM filtering60.3235382982
SetR60.2135382656
AppRAG61.2238382565

Table 4 shows the quality–cost trade-off: AppRAG achieves the highest observed mean F1 at a similar median latency, using about 7.2% more 27B tokens under matched candidate access, draft counts, and auxiliary-model calls. Paired comparisons are reported in Table 21.

All methods use Qwen3-8B generation with ACE prompting and Qwen3.8-27B verification and answer selection on a separate held-out sample from the same three datasets. AppRAG uses retrained applicability heads and recomposition, with the risk gate disabled and settings fixed on development data (Section 4.5).

Tokens count model input and output; latency is measured end to end in a separate timing study, covering requirement generation, retrieval, local scoring, drafting, verification, and answer selection. Appendix C.12 gives the timing protocol and token-accounting boundaries.

Table 1 also reports online output throughput (Token/s), defined as total LLM output tokens divided by total inference time, excluding index construction. Non-graph measurements span three runs and include token estimates for requirement generation and comparison outputs; graph methods are measured separately (Appendix E.2):

PipelineΔF1 (pp)Baseline token/s+ AppRAG token/s
Vanilla RAG+8.4645.5889.84
HippoRAG2+2.16311.54327.07
LightRAG+2.95196.41200.62
EntityLinkRAG+12.371.6097.82
ACE-RAG+6.2019.55112.86
RRF+3.8343.66103.35
RankZephyr+4.4068.06123.01
REARANK+0.64127.66128.45

The graph integrations gain 2.16–2.95 F1 points with small changes in token/s, while Vanilla RAG and EntityLinkRAG show larger quality gains and rate changes. Token/s includes intermediate verification and comparison outputs, so a higher rate alone does not imply lower latency or fewer tokens per query. The per-query token and latency results above provide the complementary cost comparison under matched draft and call budgets.

Response to W4 & C4

Table 5 directly tests the value of retaining the four applicability categories within a fixed selection pipeline. The four-class head improves F1 by 1.62 and 1.58 pp over Generic and A-only, respectively, with both comparisons passing Holm correction. Candidate construction, coverage supervision, the verifier, and draft budget are fixed. The heads share training examples, initialization, architecture, and training steps. Generic merges A/IA/U against IR; A-only separates A from the remaining labels.

ControlF1 (%) ↑AppRAG − control (pp) ↑ [95% CI]Holm p
Generic, fixed59.60+1.62 [0.65, 2.58]0.0063
A-only, fixed59.64+1.58 [0.63, 2.55]0.0096
w/o Recomp., fixed60.81+0.41 [-0.79, 1.58]0.4920
Generic, independently tuned60.31+0.91 [-0.03, 1.83]0.2048
A-only, independently tuned59.88+1.34 [0.30, 2.35]0.0714
w/o Recomp., independently tuned60.32+0.90 [-0.21, 2.00]0.2114

These gains support retaining finer applicability distinctions under fixed decision settings, without adding verifier capacity or drafts. When each head independently selects among the same five development settings, the differences are no longer statistically resolved; recomposition gains are also unresolved. Tables 4 and 21 compare the complete procedure with strong selectors.

AuthorsOpenReview ↗

Responses to Reviewer fiHm (Part 3/4)

Response to W4

The added human study covers IA and U directly. Figure 9 reports label-specific agreement on 1,950 requirement–passage units, treating each category against the remaining categories:

ComparisonIA κ ↑U κ ↑
Human–human0.7790.664
GPT–DeepSeek0.5760.268
Human–GPT0.7890.700
Human–DeepSeek0.7860.525

Two human reviewers checked all 1,950 units, including 1,770 natural and 180 enriched cases. GPT and DeepSeek labels and explanations were visible, so this is AI-assisted review. Table 10 separately reports four-category agreement. Among the 499 units with a model disagreement on at least one annotation field, both reviewers confirmed 56 IA, 24 U, 38 valid partial-support, and 56 bridge-support cases, with overlapping groups. They also confirmed five passages that are A for one requirement and IA for another, including three natural-sample cases (Appendix C.2).

This extends validation beyond the original A/IR comparison. U agreement is lower than IA agreement, and the visible model annotations leave shared-error bias possible. Table 24 reports the Cov/answer-level IA-risk evaluator's failed checks on presentation agreement, parsing/source attribution, and risk recognition. Those measures remain descriptive model-based measurements; reference-based EM/F1 and the human category study use separate evaluations.

Response to C5

Answer F1 and context IA-hit measure different outcomes, so their percentage-point changes need not match. F1 gains have positive point estimates in both detected-IA and resolved no-IA strata. They reflect a procedure that also changes support coverage, evidence combinations, and draft selection; this analysis does not isolate the contribution of IA removal.

Table 20 (Appendix C.15) analyzes saved outputs on 150 human-reviewed multi-hop questions, 50 per dataset, using four passages per system and strata defined by detected IA in the Vanilla context. This evaluation is separate from the three-run main experiment and the earlier model-judged IA study. Overall F1 rises from 17.76 to 27.88, a paired gain of 10.12 pp [5.48, 15.21].

Human labelsBaseline IA: N; ΔF1 ↑ [95% CI]Resolved no IA: N; ΔF1 ↑ [95% CI]
Human A77; +13.59 [6.51, 21.54]35; +6.80 [−1.79, 16.57]
Human B81; +10.67 [3.99, 18.09]34; +9.90 [1.73, 19.99]

The remaining 38/35 questions under Human A/B have unresolved or missing baseline labels. Human A's no-IA interval crosses zero, and these exploratory intervals do not test a difference between strata.

With the same four-passage budget, IA-hit changes are:

Human labelsAppRAG − Vanilla IA-hit (pp) ↓ [95% CI]
Human A−4.00 [−12.00, 4.67]
Human B−12.00 [−20.00, −4.00]

Both annotators give lower IA-hit point estimates under the common four-passage budget. The displayed intervals are unadjusted; the 32-comparison Holm analysis leaves statistical uncertainty about the reduction. Passage counts are fixed, token lengths vary, and missing labels remain unresolved for both systems.

The answer-quality and IA-exposure panels in Figure 1(b) summarize separate evaluations, with IA exposure using the protocol in Section 4.4 and Appendix E.5. Percentage-point differences are calculated before rounding.

Response to W2 & C6

In the rerun, 2Wiki F1 is 32.64 for LLM-only, 24.53 for Vanilla, and 30.93 for Vanilla+AppRAG (Table 1). The Vanilla deficit is now 8.11 pp, while AppRAG remains 1.71 pp below LLM-only. The non-graph methods use the shared Qwen3-8B decoding settings in Section 4.1 and Appendix E.2. Non-answer rates for these three 2Wiki systems are 0.13%, 0.80%, and 0.27%, respectively. No outputs are discarded or resampled on that basis.

Table 23 provides a separate corpus replay and support analysis under the earlier prompt requesting abstention when evidence is insufficient. Full-corpus access improves AppRAG, while Vanilla selects the same ordered top-four passages in both corpus settings. In the saved-context analysis, complete support coverage is 10.00% for Vanilla and 17.20% for AppRAG. Neither retrieves the full annotated support set for any of the 212 bridge-comparison questions, and both score 3.26 F1 on that subset. These observations identify incomplete support as a possible contributor; they do not isolate its effect from the prompt or establish the cause of the remaining gap in the rerun.

We devoted substantial time to rebuilding the HippoRAG2 and LightRAG indexes over the full benchmark corpora using their standard indexing pipelines. This included entity/relation extraction and graph construction across the full corpora, with each baseline and its AppRAG variant sharing the rebuilt index for the main evaluation (Appendix E.1). Their 2Wiki F1 values are 37.04→38.19 for HippoRAG2 and 37.41→40.32 for LightRAG, above the current LLM-only score of 32.64. Their reported overall F1 gains are +2.16 and +2.95 pp.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.