- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies evidence applicability in RAG, where relevant and factually correct passages may not satisfy the conditions of their intended reasoning steps. The authors propose AppRAG, combining requirement-conditioned applicability modeling, query embedding calibration with verified context replacement, and complementary evidence selection through draft comparison and a risk gate. The paper also analyzes selection loss caused by missing condition information. Experiments on three multi-hop QA benchmarks show overall F1 improvements across eight RAG integrations, supported by ablations and evidence-ranking analyses
Critical strengths
-
The paper addresses a meaningful limitation of relevance-based evidence selection. Its distinction between valid partial support, condition mismatch, and unresolved evidence is useful for multi-hop reasoning, where intermediate evidence may support only part of the required inference.
-
AppRAG integrates applicability assessment, verified ranking updates, and complementary context construction into a coherent framework. Experiments show consistent overall F1 gains across eight RAG integrations. Fixed-candidate ranking analyses and verification ablations further support the effectiveness of its main components.
Critical weaknesses
-
The independent benefit of applicability-specific modeling needs clearer validation. Appendix C.10 reports statistically unresolved gains over ordinary requirement-based selection, while generic multi-draft SetR-style selection performs comparably in Table 12. Since the main comparisons do not match total computation, it remains unclear how much improvement comes from applicability modeling versus additional verification and draft selection.
-
Validation of the core IA distinction is incomplete. The human evaluation covers A/IR labels but excludes IA and U, and the judge used for Cov and answer-level IA-risk failed prespecified quality checks. Additionally, the context IA-hit comparison uses contexts of different lengths. These limitations weaken the applicability diagnostics, although the reference-based EM/F1 improvements remain informative.
Questions for authors
-
Which controlled comparison best isolates the benefit of applicability-specific signals over ordinary requirement coverage and generic multi-draft selection? Please clarify the effect size and uncertainty under comparable candidate access and inference budgets.
-
Can the authors provide human validation of IA labels, particularly for valid bridge evidence and passages that support one requirement but mismatch another? Which quality checks did the Cov/IA-risk judge fail, and how does this affect the reported diagnostics?
-
Does the reduction in IA exposure remain under a common context-length budget? Please also clarify the complete inference cost relative to the strongest selection baseline, including scoring, verification, and draft generation.
3 comments
Responses to Reviewer Ki4R (Part 2/2)
Response to W2 & C3
At the same four-passage budget, both human label sets show lower IA-hit point estimates on the 150-question sample (Table 20; Appendix C.15):
| Human labels | AppRAG − Vanilla IA-hit (pp) ↓ [95% CI] |
|---|---|
| Human A | −4.00 [−12.00, 4.67] |
| Human B | −12.00 [−20.00, −4.00] |
In this separate saved-output analysis, overall F1 improves by 10.12 pp [5.48, 15.21]. Holm adjustment across 32 evidence-level comparisons leaves the IA-hit reduction statistically uncertain; the displayed 95% intervals are unadjusted. The common budget fixes passage counts, with token lengths varying and missing labels left unresolved for both systems. Together, the analyses show higher answer quality and a consistent downward direction in IA-hit point estimates across annotators.
For the strongest selection controls in Table 4, all four methods use nine drafts and eight 27B calls per query:
| Method | 27B tokens (k/query) ↓ | 8B tokens (k/query) ↓ | p50 (s) ↓ | p95 (s) ↓ |
|---|---|---|---|---|
| Relevance | 35 | 38 | 26 | 80 |
| LLM filtering | 35 | 38 | 29 | 82 |
| SetR | 35 | 38 | 26 | 56 |
| AppRAG | 38 | 38 | 25 | 65 |
The separate timing study covers requirement generation, retrieval, local scoring, fresh draft generation, verification, and answer selection, excluding model loading and warmup. Token totals cover 27B and 8B input/output and exclude shared requirement generation and additional retries; offline training is separate from inference (Appendix C.12).
The trade-off is higher observed mean F1 with about 7.2% more 27B tokens and a similar median latency, at matched draft and call budgets. Appendix C.12 provides the timing protocol and detailed cost boundaries.
Table 1 separately reports online output throughput (token/s) for the main integrations. REARANK changes from 127.66 to 128.45 token/s with a 0.64-point F1 gain; HippoRAG2 and LightRAG change from 311.54 to 327.07 and 196.41 to 200.62 token/s, with F1 gains of 2.16 and 2.95 points. Token/s divides total LLM output tokens, including intermediate verification and comparison outputs, by total inference time, excluding index construction. Non-graph measurements use three runs and include estimates for requirement-generation and comparison outputs; graph methods are measured separately (Appendix E.2). This rate describes output throughput; the matched-budget analysis above reports per-query token use and latency.
The authors’ response has addressed most of my concerns, I will maintain my current rating.
Responses to Reviewer Ki4R (Part 1/2)
Thank you for your valuable comments and insights. We have summarized the points of concern and would like to address them as follows.
Response to W1 & C1
The fixed-setting head ablation in Table 5 isolates the value of retaining the four applicability categories: the four-class head improves F1 by 1.62 and 1.58 pp over Generic and A-only. The heads share training examples, initialization, architecture, training steps, and coverage supervision. Candidate construction, verifier, and draft budget are fixed. Generic groups A/IA/U against IR; A-only separates A from the other labels.
The fixed-setting gains support the value of retaining the four applicability categories and pass Holm correction across the 13 F1 comparisons. Independently tuned comparisons, using the same five development settings for each head, and the recomposition ablation remain statistically unresolved. The displayed intervals are unadjusted paired 95% bootstrap intervals.
The broader relevance/filter/SetR comparison addresses the separate question of the complete selection procedure. On the same held-out 1,500-question control study, all methods use Qwen3-8B generation, a common Qwen3.8-27B verifier/answer selector, nine drafts, and eight calls. AppRAG has the highest observed mean F1, with paired differences of +1.07/+0.90/+1.01 pp (Tables 4 and 21).
Response to W2 & C2
The new human study includes both cases. Two reviewers checked 1,950 requirement–passage units, including 1,770 natural and 180 enriched cases; 346 units with missing model labels were excluded. GPT and DeepSeek labels and explanations were visible during review. Figure 9 reports IA/U-specific Cohen κ:
Human–model entries average the two reviewers' κ values; Figure 9 provides question-cluster bootstrap intervals, and Table 10 reports four-category agreement. Within the 499 units with a model disagreement on at least one field, both reviewers confirmed 56 IA, 24 U, 38 partial-support, and 56 bridge-support cases, with overlap between groups.
Both also confirmed five passages as A for one requirement and IA for another, including three natural-sample cases. For example, a passage about Scarlett O'Hara identifies Vivien Leigh as the actress but lists the character's spouses. It supports identifying the actress, while using the listed spouses to answer whom the actress married mismatches the required person. These AI-assisted judgments extend validation to IA/U and intermediate support, while retaining the possibility of shared labeling errors (Appendix C.2).
Table 24 specifies the Cov/answer-level IA-risk judge failures:
IA-risk passed its final holdout but failed the acceptance rule under the prespecified limit of two calibration revisions. Cov and answer-level IA-risk therefore remain descriptive model-based measures, not validated semantic measures. Reference-based EM/F1 and the human category evaluation are assessed separately.