- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper uses a learned critic to predict whether retrieval-based revisions will reduce hallucination while preserving useful content. Conformal calibration determines which revisions to accept and provides a marginal risk guarantee across revision rounds under exchangeability. The evaluation focuses on hallucination reduction, utility preservation, and intervention risk.
Critical strengths
-
The problem is worth studying. A correction can introduce errors even when average accuracy improves, and measuring both factuality gains and lost information is a useful way to evaluate revision systems.
-
The authors account for candidate selection and the states produced by earlier revisions. The maximum-residual score and separate policy-matched folds give a coherent procedure under the stated assumptions.
-
The experiments cover several tasks, including medical reports. The shared-critic controls help separate prediction from calibration, while the retrieval perturbations and calibration-size study examine relevant sources of sensitivity.
Critical weaknesses
-
The certificate can approve revisions that leave every existing error intact. For the medical loss in Eq. (16), consider two false, reference-unmatched entries and one correctly matched entry. Adding five correctly matched entries lowers the loss from to while preserving both errors. Utility can also improve, so both acceptance conditions can hold without any error repair. CHAIR has the same dilution problem. The human study evaluates final outputs without validating these revision-level changes. The experiments therefore do not establish that certification corresponds to actual error repair.
-
The calibration setup does not establish the exchangeability required by the theorem. Table 2 leaves the task composition and benchmark-specific quantile fitting unclear. Disjoint data are insufficient, and a guarantee for a pooled task mixture does not establish risk control on each benchmark.
-
The critic ablations do not isolate why risk control deteriorates. Figure 12 reports PVRs of 16.9% and 12.4% against a 5% target, without specifying whether each variant was recalibrated. Since the theorem applies to any fixed critic after proper calibration, these results do not establish that weaker prediction explains the higher violation rates.
-
I find the methodological contribution limited. Retrieval gating [1], retrieval-gain prediction [2], and policy-aware calibration [3] have already been studied. PICD mainly extends these ideas to selecting revision locations across multiple rounds. The ablations establish that its calibration choices affect violation rates, but do not establish a substantial advance over existing approaches. Comparisons with simpler gates under matched supervision and coverage are also missing.
Questions for authors
-
Among confidently wrong answers and perceptual failures, what fraction does the critic successfully repair, leave unchanged, or make worse? What evidence distinguishes failures inherited from the generator's representations from failures caused by retrieval? The rare-concept analysis combines these sources and does not establish that the critic can detect errors the generator itself fails to recognize.
-
Why does enabling round two change coverage when the first-round policy is held fixed? A later round cannot change whether an input receives its first revision. Which other setting differs between the one-round and two-round results, and what does this imply for the attribution of gains to the second round?
-
How sensitive are coverage and calibration margins to the eight-candidate cap? Does denser candidate selection recover errors missed by the current boundary rule?
-
After changing the retrieval corpus or deployment distribution, how much new calibration data is needed to recover the risk target? Is recalibration enough, or does the critic need retraining too?
[1] Self-Knowledge Guided Retrieval Augmentation for Large Language Models. Findings of EMNLP, 2023.
[2] Predicting Retrieval Utility and Answer Quality in Retrieval-Augmented Generation. Advances in Information Retrieval (ECIR), 2026.
[3] Look Again Before You Abstain: Budgeted Conformal Evidence Acquisition for Reliable Vision-Language Models. 2026.
7 comments
Response (2/3): Methodological contribution and distinction from existing literature
(Continued from 1/3.)
W3. The critic ablations do not isolate why risk control deteriorates. We agree with the reviewers point, and we corrected the attribution. Theorem 1 holds for any fixed critic after trajectory-level, round-matched calibration, so a weaker critic that is properly calibrated loses coverage, not validity. The figure therefore cannot support the attribution we made fully. The 16.9% row is the entropy-threshold selector of Figure 4, at the identical operating point (39.5% coverage, 15.8% HIR): a threshold rule with no conformal calibration. For the 12.4% row we no longer draw any conclusion about risk control from its PVR; we cite it only for prediction quality (AUROC 0.754 versus 0.842), but our future work will be to work on the risk control as a natural extension to this paper. The revision relabels them as prediction ablations, removes the sentence that tied them to PVR, and states that under PICD's calibration prediction quality determines coverage. The rows that do isolate risk control are the shared-critic controls, where the critic is identical and only calibration differs: pointwise 7.1%, trajectory-only 5.6%, reused first-round margins 7.6%, PICD 4.4%.
W4. Limited methodological contribution; comparison with [1–3] and with simpler gates. Thank you for the references; the revision cites and discusses all three. The differences are specific.
- [1] decides whether retrieval is needed from the model's self-knowledge. It is a current-state signal with no guarantee. Our Section 4.1 shows that such signals predict the outcome of a revision worse than paired supervision (best 0.752 versus 0.842).
- [2] predicts retrieval utility and answer quality for text-only RAG by regression on retriever- and reader-side features. It gives point predictions without finite-sample risk control.
- [3] (arXiv, June 2026) is the closest. It calibrates an assert-or-abstain threshold on the post-acquisition score and bounds the selective risk of asserted claims. We share the principle that calibration must see the deployed policy. PICD differs in the certified quantity (the paired change of a revision relative to the response it replaces, for two objectives), in the selection problem (an argmax over branch points, handled by the trajectory maximum), and in the sequential structure (accepted revisions change later states, handled by disjoint folds replayed through the frozen earlier policies). PICD never abstains; it keeps the original response.
On simpler gates under matched conditions, the submission contains three such comparisons. The binary utility router has the same capacity as the critic and, at nearly the same coverage as PICD (37.8% versus 36.9%), has a PVR of 12.1% versus 4.4%. The entropy threshold at 39.5% coverage has 16.9%. Figure 1b plots PVR against coverage for all selectors, and the proxy gates lie above the frontier at every coverage. We also state the limit of our own claim: against the uncalibrated version of the same critic, calibration provides control of the risk level, not a better trade-off at matched coverage (Section 4.2).
[1] Wang et al. Self-Knowledge Guided Retrieval Augmentation for Large Language Models. [2] Tian et al. Predicting Retrieval Utility and Answer Quality in Retrieval-Augmented Generation. [3] Xu et al. Look Again Before You Abstain: Budgeted Conformal Evidence Acquisition for Reliable Vision-Language Models.
Response (3/3): Perceptual failure and Sensitivity to the eight-candidate cap
(Continued from 2/3.)
Q1. Repair / unchanged / worse among confidently wrong answers and perceptual failures; generator-inherited versus retrieval-caused failures.
- Over all 48,216 candidate revisions: 43.7% acceptable, 21.5% leave the loss unchanged, 34.8% harmful.
- The critic is not limited to what the generator recognises. The generator's own confidence predicts the outcome poorly (entropy 0.662, maximum token probability 0.641), whereas the critic, which also receives the visual representation and the retrieved evidence, reaches 0.842. The gap is the information that the generator's confidence does not carry.
- Retrieval is separated from the generator by intervention. The retrieval perturbations hold the generator and critic fixed and change only the evidence. Coverage falls from 36.9% to 5.9% under contradictory evidence while PVR stays between 4.4% and 4.9%: when retrieval cannot support a repair, PICD keeps the response.
- Among analysed residual failures of accepted revisions, 23% are attributed to missing or contradictory retrieval, 31% to critic overestimation and 8% to unstable regeneration.
We agree that the rare-concept analysis combines both sources, and the revision states that now.
Q2. Why does enabling round two change coverage when the first-round policy is fixed? Your reasoning is correct: under the stopping rule a second round cannot change whether an input receives its first revision. The one-round row is a separately configured and calibrated run, not the main policy truncated after round one. With the error budget and the calibration data are assigned to a single round, so its first-round margins are not those of the two-round policy, and the premise of a fixed first-round policy does not hold between the two rows. The two rows therefore do not attribute any gain to the second round, and the revision says this and relabels the row. The second round's own contribution is given by the round-wise rates within the single two-round policy (round-one PVR 2.6%, round-two PVR 1.9%, pathwise 4.4%).
Q3. Sensitivity to the eight-candidate cap; does denser selection recover missed errors? We did not sweep the cap, so we report what the submission determines. On MIMIC-CXR, where every input is long-form, Table 2 gives a mean of 5.7 candidate states per test study (18,704 over 3,269), against a cap of eight. The direction of the trade-off is fixed by the construction: the calibrated score is a maximum over candidates, so a denser set can only widen the margin, while it offers more places to branch. The failure analysis bounds the possible benefit: a branch point lying after the unsupported claim is the primary source of 12% of analysed residual failures. That share is the most that denser candidates could recover. The revision reports these facts and marks the sweep as open.
Q4. After changing the retrieval corpus or the deployment distribution, how much new calibration data is needed? Recalibration or retraining? Recalibration is sufficient for the guarantee; retraining is needed only to recover coverage. Theorem 1 holds for any fixed critic once the folds are exchangeable with the new deployment and are produced by the new pipeline, so a changed corpus or distribution requires new folds but no change to the critic or the VLM. A stale critic receives wider margins and revises less. On the amount: a finite margin requires inputs per fold at our main setting. In-distribution, coverage rises from 27.4% at to 36.9% at and 38.1% at , so about 2,000 inputs per fold reach the plateau.
We hope all of your concerns are resolved with our revision and we would like to request you to consider the revised submission for the final assessment. Regards.
Response (1/3): Certificate and calibration setup
Thank you for the detailed and technically precise review. Your points W1–W3 led to specific updates in our paper. Here are our responses:
W1. The certificate can approve revisions that leave every existing error intact (dilution). The arithmetic is correct. HCXR (Eq. 16) and HCHAIR are rates, so a revision that adds supported content lowers them without deleting an existing error. We now state this next to the theorem. Our response has three parts.
What is claimed. Theorem 1 certifies that an accepted revision does not degrade the response under the fixed (H,U). It does not claim that each accepted revision deletes a particular error, and the Scope paragraph already limits the guarantee to the evaluators. The revision makes the non-degradation reading explicit in that paragraph.
A loss that cannot be diluted. The method and proof use H only as a bounded function fixed before calibration. With the count-based loss Hcnt(Y)=min{1,∣A(Y)∖A(Y⋆)∣/c} for a fixed cap c, ΔH>0 holds only if the number of unsupported atoms strictly decreases, and Theorem 1 applies verbatim. The revision adds this option.
Evidence in the submission that the gains are not dilution.
We agree that the human study rates final outputs and does not validate revision-level changes; the revision says so and keeps revision-level human calibration as the stated extension.
W2. The calibration setup does not establish exchangeability; pooled versus per-benchmark. We agree on both principles. Disjointness is not what the theorem uses; it requires each fold and the test input to be exchangeable given the earlier data, under an identical pipeline (Section B.1). And a guarantee for a pooled mixture does not transfer to its components, which is why the Scope paragraph excludes subgroup coverage. Exchangeability is an assumption of Theorem 1, as in every split-conformal result, and we do not claim that the data prove it. What the protocol does to support it, is: all partitions are fixed before any fitting, the retriever, critic, evaluators and decoding settings are identical in calibration and deployment, and each later-round fold is replayed through the frozen earlier policies. On the level of the guarantee we accept the reading by reviewer and now state it in the paper: Table 2 organises the folds by domain, the theorem applies to the population from which a fold is drawn, and we do not claim a separate guarantee for each benchmark. The benchmark–backbone rates we report are empirical results, computed within each group before macro-averaging. Consistent with the assumption, the realised rate follows the requested one at all four targets (0.9, 4.4, 9.1, 18.6%).