- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies answer-supervised optimization of a retriever–reranker–generator cascade without gold supporting-document labels. StageRAG logs document retention, verifier-attributed support, and answer correctness, converts the resulting states into within-query preferences, and separately trains the retriever and reranker using capped preference sampling. The generator and inference topology remain fixed. Experiments on HotpotQA, 2WikiMultiHopQA, and RAGBench-TATQA show improvements in retrieval recall and answer F1, with additional component, supervision, verifier, and backbone analyses.
Critical strengths
- The problem is practically relevant. Improving both retrieval stages from question–answer pairs avoids task-specific evidence annotation, and the method adds no inference-time verifier or generation calls.
- The main improvements are substantial and consistent across the three evaluated datasets. The component ablations in Table 2 support complementary benefits from retriever and reranker optimization. The evidence-transition analysis is particularly useful: on HotpotQA, complete gold-evidence coverage in the final context increases from 30.0% to 41.8%.
- The experimental analysis goes beyond the main comparison. Table 3 controls the training budget when changing supervision, and the appendices examine the ambiguous failure state, verifier substitution, and a Qwen-based pipeline. I appreciate the explicit distinction between observable support attribution and causal evidence use, as well as the stated limitations of the theoretical assumptions.
Critical weaknesses
- The additional answer-quality benefit of the full state construction over simpler supervision needs stronger support. In Table 3, binary z4 supervision already reaches 0.5216 F1, compared with 0.5273 for the full method: a difference of 0.57 percentage points. The recall improvements are clearer, but this central comparison is reported only on HotpotQA with one training seed. The development-set confidence intervals in Appendix A.7 concern alternative treatments of z3, rather than this comparison.
- The theory motivates downstream-aware ranking but does not establish that the proposed state preferences estimate the derived pipeline value. Appendix A.7 correctly acknowledges that even the core ordering is an empirical surrogate. In particular, retained but unattributed documents are preferred over filtered documents despite neither having verified support. The contribution should therefore be judged primarily by its empirical evidence, rather than as a theoretically validated preference-learning rule.
- The stronger-backbone evidence is narrower than the main results might suggest. On Qwen-based TAT-QA, full optimization reaches 0.5040 F1, while reranker-only optimization reaches 0.5038 (Table 10). This supports compatibility, but provides little evidence of additional answer-quality benefit from retriever optimization in that setting. Likewise, the verifier audit reports only 46.74% gold precision for the original attributed support set on TAT-QA. Gold mismatch is not necessarily a verifier error, but representative disagreements would help interpret the supervision on this dataset.
Questions for authors
- Can you report paired uncertainty for the full-state versus binary-z4 comparison, and provide the corresponding comparison on one additional dataset? This would help establish whether the finer state distinctions reliably improve answer quality.
- How much of the gain comes from transferring the original reranker’s retention preferences, versus verifier-derived answer support? A retention-only control using all logged queries and the same training budget would clarify the value of answer supervision beyond ranker distillation; the existing answer-only control uses only successful trajectories.
- For Qwen-based TAT-QA, is the near equality between full and reranker-only F1 stable across runs? For the TAT-QA verifier audit, could you show a few support/gold disagreements indicating whether they reflect alternative valid evidence, incomplete annotations, or attribution errors?
2 comments
Response to Reviewer YBTy - Part 2
W3 & Q3: Qwen TAT-QA multi-seed stability and verifier audit disagreements
1. Three-seed check on Qwen TAT-QA (Table 11 in Revised Paper):
We repeated the Full-stage vs. Reranker-only test on TAT-QA test@500 across three training seeds (42, 43, and 44):
| Seed | Full-stage R@3 / F1 ↑ | Reranker-only R@3 / F1 ↑ | ΔR@3 (pp) | ΔF1 (pp) |
|---|---|---|---|---|
| 42 | 0.4768 / 0.5040 | 0.4528 / 0.5038 | +2.40 | +0.02 |
| 43 | 0.4668 / 0.5013 | 0.4478 / 0.4998 | +1.90 | +0.15 |
| 44 | 0.4738 / 0.5056 | 0.4518 / 0.5004 | +2.20 | +0.52 |
| Average | 0.4725 / 0.5036 | 0.4508 / 0.5013 | +2.17 | +0.23 |
Across all three seeds, Full-stage beats Reranker-only in every run, raising Recall@3 by +2.17 pp (0.4725 vs. 0.4508) and F1 by +0.23 pp (0.5036 vs. 0.5013) on average. Compared with the unoptimized Qwen cascade (Table 10), StageRAG lifts Recall@3 by +5.47 pp and F1 by +1.77 pp. This confirms stable gains on Qwen3, while end-task F1 over Reranker-only stays modest due to the fixed Qwen3-8B reader.
2. Audit examples of TAT-QA support/gold mismatches (Table 20 in Revised Paper):
For reference, Table 20 in the Revised Paper (Table 19 in the original submission) summarizes the cross-model audit on 200 queries per dataset:
| Dataset | Answer agreement (%) | Support confirmation (%) | Gold precision (%) |
|---|---|---|---|
| HotpotQA | 99.50 | 92.25 | 89.80 |
| 2Wiki | 100.00 | 77.38 | 88.78 |
| TAT-QA | 97.00 | 95.29 | 46.74 |
Note that 46.74% (43/92) measures overlap with the annotated gold set, not verifier accuracy. Using a fixed random seed, we checked 20 of the 49 non-matching records (across 18 queries) and found three clear cases:
- Valid evidence outside the gold set (): Asked for the principal of senior notes due June 2021, the system answered USD 800 million. The cited non-gold passage explicitly states that same USD 800 million principal for those notes.
- Correct answer with wrong citation (): The system correctly answered 2019 as having the highest diluted net EPS. However, it cited a non-gold table of weighted-average share counts instead of the gold EPS table.
- Literal support with a scope mismatch (): Asked for the change in total R&D expenses, the cited passage reports a 22% increase excluding impairment charges. It literally supports the stated number, but misses the question's scope—and the verifier properly marked the answer as wrong ().
Hence, gold mismatches on TAT-QA come from a mix of valid unannotated passages, citation errors, and scope mismatches caught by answer verification. Notably, Case 1 shows that even benchmarks with evidence annotations can miss valid supporting passages. By learning directly from answer-supervised trajectories instead of static gold labels, StageRAG naturally discovers and uses this unannotated evidence.
Q2: Retention-only control versus answer-supervised stages
As suggested, we added a Retention-only baseline under the same training budget (Table 3 and Appendix A.6 in Revised Paper). It uses all logged training queries without answer filtering or verifier support, and matches optimizer update steps separately for the retriever and reranker. It learns only to rank initial top-3 retained documents above filtered ones:
| Method (HotpotQA test@500) | Recall@20 ↑ | Recall@3 ↑ | EM ↑ | F1 ↑ |
|---|---|---|---|---|
| Retention-only | 0.7170 | 0.6570 | 0.3960 | 0.4929 |
| StageRAG (Full-stage) | 0.7310 | 0.6770 | 0.4340 | 0.5273 |
| StageRAG Gain (pp; 95% paired CI) | +1.40 [0.10, 2.70] | +2.00 [0.60, 3.50] | +3.80 [1.20, 6.40] | +3.44 [0.89, 6.03] |
Overall, StageRAG significantly beats Retention-only on all four metrics (+3.44 F1, 95% CI [0.89, 6.03]). This demonstrates that StageRAG does more than copy the initial reranker. Turning answer correctness and citation support into stage-aware preferences improves both retrieval and answer quality without gold labels.
We hope these additional analyses address your remaining concerns and further strengthen your support for our work.
Response to Reviewer YBTy - Part 1
Dear Reviewer YBTy,
Thank you for your careful review and for noting the complementary stage gains, the difference between support attribution and causal use, and our clear limitation section. Below we report the requested statistical tests, new controls, and audit examples, and note the updates in our revised manuscript.
W1 & Q1: Paired uncertainty and a second dataset (Full-stage vs. Binary-z4)
To check if fine-grained trajectory states improve retrieval and answer quality over Binary z4, we tested both on 1,000 fixed queries on HotpotQA and 1,000 fixed queries on 2WikiMultiHopQA. We report 10,000-sample paired bootstrap 95% CIs for Full-stage minus Binary-z4 (in percentage points). To check if the fixed reader (GPT-4o-mini) limits answer gains, we kept retrieved contexts fixed and also ran GPT-5.6 Luna:
HotpotQA (test@1,000):
2WikiMultiHopQA (test@1,000):
(Note: Within each dataset, both readers receive the exact same retrieved contexts, so the recall scores and gains are identical.)
Taken together, StageRAG beats Binary z4 in recall on both datasets (all four recall 95% CIs above zero, up to +6.68 pp). With GPT-4o-mini, HotpotQA answers improve slightly (+0.70 EM, +0.91 F1) while 2Wiki does not, because the reader fails to use the extra evidence, not our retrieval pipeline. To test this, we kept the retrieved contexts unchanged and switched only to GPT-5.6 Luna. On the exact same contexts, StageRAG improves both EM and F1 on both datasets (+2.70 EM and +1.60 F1 on HotpotQA; +2.50 EM and +2.43 F1 on 2Wiki, all four 95% CIs above zero).
W2: Link between the theory and empirical preferences
We agree with your view and state this boundary clearly in Section 3.5 of the Revised Paper: our theory proves the necessity of cascading optimization of RAG retrieval system and motivates pipeline-aware ranking, rather than directly deriving the empirical state order. However, directly solving this objective requires heavy online RL or repeated counterfactual LLM rollouts, which is often not practical. StageRAG offers an effective design that bridges theory and practice: it avoids complex training algorithms (such as RL), keeps annotation costs low, and delivers strong empirical gains.
Specifically, viπ(q,d)=piπ(q,d)r(q,d) separates potential contribution r from downstream survival probability p. Here, z2≻z1 serves as a practical proxy for p: neither document is cited, yet z2 passed reranking while z1 was filtered. Even without numerically estimating v, Table 3 shows this proxy is essential: removing z2≻z1 pairs drops Recall@3 by 2.80 pp (67.70% → 64.90%).
Thus, the theory establishes why upstream retrieval must account for downstream survival, and StageRAG turns that insight into a simple, low-cost, and effective design across the cascade.