- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies evidence selection for retrieval-augmented question answering with a fixed candidate context and reader. It separates optimization on a predicted support graph from the effect of compression on answer quality. The proposed CRESS-S selector combines additive and shared-cost greedy selection, retaining the additive solution as a safeguard. A validated flow relaxation provides an instance-specific certificate of graph optimization quality. The framework also includes a common budget-aware repair search and calibration over a fixed family of compression-and-fallback policies. The reader-level target is paired answer degradation relative to the full-context baseline, rather than absolute answer correctness. Evaluation consists of controlled graph experiments and calibration simulations. A public-data reader study is specified but has not been executed.
Critical strengths
- Careful separation of optimization quality and reader reliability. The paper clearly distinguishes coverage of predicted requirements, near-optimality under the declared graph objective, and preservation of the reader’s answer. The counterexample in Section 4.1 explains why locally calibrated support scores need not remain valid after cost-based selection. Calibrating the complete admission-and-reading policy addresses the relevant selection effect, and the discussion of paired degradation explicitly avoids interpreting the guarantee as absolute answer correctness. These distinctions make the formulation useful for understanding what evidence-selection guarantees do and do not establish.
- The controlled experiments make optimization and support-prediction errors distinguishable. The graph comparisons use matched candidates, costs, and corrupted support memberships, together with MILP reference optima and independently validated lower bounds. Evaluation against latent support labels is kept separate from predicted coverage. The paper also reports certificate tails, search failures, and fallback-only calibration outcomes. This experimental design makes the controlled results interpretable and provides useful evidence about the behavior and limitations of the proposed components.
Critical weaknesses
-
The value of the combined approach for question answering has not been demonstrated. Section 5.4 and Appendix G explicitly state that the public-data reader study has not been run. The completed studies evaluate graph optimization and policy calibration separately. They therefore do not establish whether contexts selected using learned support predictions can be certified at a useful admission rate while reducing measured deployment cost. In particular, the calibration simulation uses 16 threshold policies with a risk target of 0.10, whereas the proposed reader study uses 42 policies with a target of 0.02; the simulation does not establish the operating behavior of that reader pipeline. The authors acknowledge this limitation, and I do not regard it as a contradiction of the stated theorem. Nevertheless, because the paper identifies the interface between selection, certification, and reader calibration as its main contribution, the absence of a joint evaluation leaves a central part of its value unresolved.
-
The incremental contribution beyond simpler combinations is not sufficiently established. The manuscript appropriately identifies the weighted-cover guarantee and simultaneous finite-family calibration as established ingredients. This makes the additional value of the safeguard, graph certificate, and common repair particularly important. Table A.4 reports an 18.5% mean cost reduction relative to additive CRESS, but a more modest 2.3% reduction relative to union-cost greedy, which already accounts for shared resources. After common repair, the differences narrow further: at budget 48, CRESS-S and union-cost greedy both admit all 120 graphs, with fallback-inclusive normalized graph costs of 0.073 and 0.074, respectively (Table A.7). These results show a benefit, but do not yet establish whether the additional selection machinery provides a consequential advantage once computation is included. Furthermore, Appendix G specifies that the reader policy does not call the graph LP. The certificate is useful as an optimization diagnostic, but its role in the proposed interface needs a clearer account. Taken together, the current evidence does not sufficiently establish the significance of the full combination beyond a simpler shared-cost selector with the same repair and calibration.
Questions for authors
-
What is the principal additional capability provided by the proposed interface? Beyond combining shared-cost selection with finite-family risk calibration, which methodological or theoretical result do the authors consider the main new contribution? In particular, is the graph certificate intended solely for offline optimization auditing, or does it inform the construction or selection of reader policies? Clarifying this would help assess the contribution independently of the unexecuted reader study.
-
Why retain omission penalties when admission requires complete predicted coverage? Section 3.3 and Appendix F.3 acknowledge that omission penalties can cause the core selector to return a context that fails the full-coverage admission gate. What motivates retaining these penalties rather than treating all predicted requirements as mandatory before applying the budget gate? Would that alternative change the interpretation of the gains attributed to common repair? This question concerns omission-induced rejections, separately from cases in which greedy selection misses a budget-feasible solution.
-
What computational overhead accompanies the safeguard’s remaining benefit after repair? For the matched comparisons in Table A.7, what additional selection and repair time does CRESS-S require relative to union-cost greedy? Since their reported graph costs after repair are close, matched computational measurements would materially affect the assessment of whether the safeguard provides a useful net benefit.
Thank you for distinguishing the soundness of the analysis from the unresolved application value. Your questions identify where the individual components need to justify their place in the complete policy.
1. What the graph certificate adds. In the implemented reader protocol, the graph certificate is an offline optimization diagnostic. It neither constructs the selected context nor gates reader admission. Appendix G's admission rules use coverage, confidence, and budget, not the LP ratio. It would be incorrect to attribute a reader-quality improvement to that certificate.
Its additional capability is an instance-specific upper bound on suboptimality against the shared-cost graph optimum, rather than only an additive-fragment guarantee. With a validated positive lower bound L, F(H)/OPTG≤F(H)/L. This can identify an inefficient selected graph without a uniform fragment-completeness assumption. It cannot identify missing semantic evidence. The rational residual correction makes the numerical lower bound checkable, but we do not claim that primal-dual certification is a new optimization principle.
The broader contribution is the separation of quantities that can otherwise be conflated. Predicted coverage, graph optimality, and reader degradation are different targets. Section 4.1 shows that even calibrated local scores need not remain valid after cost-based selection. C-RAG (Kang et al., ICML 2024) controls generation loss, whereas Principled Context Engineering (Chakraborty et al., ECIR 2026) controls relevant-snippet retention. Our use of finite-family testing follows Learn then Test (Angelopoulos et al., 2025). We are not claiming a new statistical framework or that every component is necessary in every deployment.
2. Omission penalties versus mandatory coverage. We agree that a full-coverage gate makes mandatory coverage a particularly important control. We ran it on the same 120 graphs. At budget 48, all 43 original CRESS-S rejections arise from missing predicted requirements. None is over budget. Removing omission choices raises initial admission from 77 to 113. The seven remaining rejections exceed the budget. Common repair brings both variants to 120 admissions.
The mandatory variant needs seven searches instead of 43. Its aggregate normalized resource cost after repair is 0.07930, versus 0.07314 for the penalized variant, while latent true-support recall is 0.9236 versus 0.9222. Same-host selection-plus-conditional-search time is 1.46 ms versus 5.97 ms per graph. Thus penalties are not intrinsically preferable. Here they produce more search and somewhat lower retained-resource cost, not a demonstrated reader benefit. This control also shows why the recovered admission cannot be credited solely to better budget search. Much of it repairs omissions introduced by the penalized objective itself.
3. Overhead behind the remaining safeguard gain. For the matched budget-48 comparison, union-cost greedy takes 0.37 ms for selection and 5.11 ms for conditional search, totaling 5.48 ms per graph. CRESS-S takes 0.67 ms and 5.29 ms, totaling 5.97 ms. The approximately 0.49 ms overhead accompanies aggregate normalized resource costs of 0.07390 and 0.07314. Selector timings are warmed seven-repeat medians per instance, then averaged. Search is timed once per graph-budget and shared between matched comparisons. These one-host CPU measurements exclude scoring and reader inference. They do not establish net deployment savings.
Checking this comparison exposed a caption error in Table A.7. The displayed costs are ratios of total deployed to total full-context resource cost, not means of per-instance ratios. The corresponding per-instance means are 0.06616 and 0.06531. Both summaries preserve your conclusion that the repaired methods are close. The accompanying results give both aggregations explicitly.
Joint validation and the calibration settings. We agree that the 16-policy, 0.10-risk simulation cannot establish behavior of the unrun 42-policy reader study at 0.02. Even with no observed paired failures, the latter requires 334 admitted calibration questions at confidence error 0.05. The direct-search and mandatory-coverage alternatives are controlled ablations only. Incorporating them into a reader comparison requires fixing the enlarged family and its simultaneous correction before calibration.
The public-reader result remains missing. Your requested joint evaluation must include absolute accuracy, all four paired outcomes, admission, and fallback-inclusive computation. The new controls narrow the empirical claim rather than manufacture a large gain. We respectfully ask that the paper's value be assessed in its validated separation of optimization and statistical claims, together with the documented tradeoffs and failure cases, without treating the modest safeguard improvement as proof of end-to-end efficacy.