acceptodds
Under review as a conference paper at ICLR 2027

Certify Before You Intervene: Pathwise Intervention-Certified Decoding for Multimodal Generation

Abstract

Vision-language models (VLMs) describe objects, attributes and findings that the image does not show. Inference-time corrections, such as retrieving evidence and regenerating part of the response, reduce these hallucinations on average, but they also turn correct responses into wrong ones and delete useful content. Whether a correction helps depends on the specific revision, which uncertainty and evidence relevance measure only indirectly. We introduce Pathwise Intervention-Certified Decoding (PICD), which revises a response only when a calibrated lower bound on the benefit of that revision is positive. A critic, trained on paired continuations that share an input and a prefix, predicts how a revision changes the hallucination loss and the utility of the completed response. PICD calibrates the largest overestimation of this critic along each response, so that its bounds hold for whichever candidate the decoder selects, and recalibrates every round on the states that earlier rounds produce. Under exchangeability, with a probability the operator chooses, every accepted revision on a response's path lowers hallucination loss and stays within a utility tolerance. Across three VLMs and five general-domain and medical benchmarks, the critic predicts which revisions lower hallucination loss better than uncertainty and relevance signals, and PICD keeps the share of responses with a violating revision below each requested rate we test, at the cost of revising fewer responses than an uncalibrated critic.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
vision-language models, hallucination mitigation, conformal prediction, uncertainty quantification, inference-time intervention, trustworthy multimodal generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_83vMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper uses a learned critic to predict whether retrieval-based revisions will reduce hallucination while preserving useful content. Conformal calibration determines which revisions to accept and provides a marginal risk guarantee across revision rounds under exchangeability. The evaluation focuses on hallucination reduction, utility preservation, and intervention risk.

Critical strengths
  1. The problem is worth studying. A correction can introduce errors even when average accuracy improves, and measuring both factuality gains and lost information is a useful way to evaluate revision systems.

  2. The authors account for candidate selection and the states produced by earlier revisions. The maximum-residual score and separate policy-matched folds give a coherent procedure under the stated assumptions.

  3. The experiments cover several tasks, including medical reports. The shared-critic controls help separate prediction from calibration, while the retrieval perturbations and calibration-size study examine relevant sources of sensitivity.

Critical weaknesses
  1. The certificate can approve revisions that leave every existing error intact. For the medical loss in Eq. (16), consider two false, reference-unmatched entries and one correctly matched entry. Adding five correctly matched entries lowers the loss from 2/32/3 to 2/82/8 while preserving both errors. Utility can also improve, so both acceptance conditions can hold without any error repair. CHAIR has the same dilution problem. The human study evaluates final outputs without validating these revision-level changes. The experiments therefore do not establish that certification corresponds to actual error repair.

  2. The calibration setup does not establish the exchangeability required by the theorem. Table 2 leaves the task composition and benchmark-specific quantile fitting unclear. Disjoint data are insufficient, and a guarantee for a pooled task mixture does not establish risk control on each benchmark.

  3. The critic ablations do not isolate why risk control deteriorates. Figure 12 reports PVRs of 16.9% and 12.4% against a 5% target, without specifying whether each variant was recalibrated. Since the theorem applies to any fixed critic after proper calibration, these results do not establish that weaker prediction explains the higher violation rates.

  4. I find the methodological contribution limited. Retrieval gating [1], retrieval-gain prediction [2], and policy-aware calibration [3] have already been studied. PICD mainly extends these ideas to selecting revision locations across multiple rounds. The ablations establish that its calibration choices affect violation rates, but do not establish a substantial advance over existing approaches. Comparisons with simpler gates under matched supervision and coverage are also missing.

Questions for authors
  1. Among confidently wrong answers and perceptual failures, what fraction does the critic successfully repair, leave unchanged, or make worse? What evidence distinguishes failures inherited from the generator's representations from failures caused by retrieval? The rare-concept analysis combines these sources and does not establish that the critic can detect errors the generator itself fails to recognize.

  2. Why does enabling round two change coverage when the first-round policy is held fixed? A later round cannot change whether an input receives its first revision. Which other setting differs between the one-round and two-round results, and what does this imply for the attribution of gains to the second round?

  3. How sensitive are coverage and calibration margins to the eight-candidate cap? Does denser candidate selection recover errors missed by the current boundary rule?

  4. After changing the retrieval corpus or deployment distribution, how much new calibration data is needed to recover the risk target? Is recalibration enough, or does the critic need retraining too?

[1] Self-Knowledge Guided Retrieval Augmentation for Large Language Models. Findings of EMNLP, 2023.

[2] Predicting Retrieval Utility and Answer Quality in Retrieval-Augmented Generation. Advances in Information Retrieval (ECIR), 2026.

[3] Look Again Before You Abstain: Budgeted Conformal Evidence Acquisition for Reliable Vision-Language Models. 2026.

7 comments
AuthorsOpenReview ↗

Response (1/3): Certificate and calibration setup

Thank you for the detailed and technically precise review. Your points W1–W3 led to specific updates in our paper. Here are our responses:

W1. The certificate can approve revisions that leave every existing error intact (dilution). The arithmetic is correct. HCXRH_{\text{CXR}} (Eq. 16) and HCHAIRH_{\text{CHAIR}} are rates, so a revision that adds supported content lowers them without deleting an existing error. We now state this next to the theorem. Our response has three parts.

What is claimed. Theorem 1 certifies that an accepted revision does not degrade the response under the fixed (H,U)(H,U). It does not claim that each accepted revision deletes a particular error, and the Scope paragraph already limits the guarantee to the evaluators. The revision makes the non-degradation reading explicit in that paragraph.

A loss that cannot be diluted. The method and proof use HH only as a bounded function fixed before calibration. With the count-based loss Hcnt(Y)=min⁡{1, ∣A(Y)∖A(Y⋆)∣/c}H_{\text{cnt}}(Y)=\min\{1,\,|\mathcal{A}(Y)\setminus\mathcal{A}(Y^\star)|/c\} for a fixed cap cc, ΔH>0\Delta_H>0 holds only if the number of unsupported atoms strictly decreases, and Theorem 1 applies verbatim. The revision adds this option.

Evidence in the submission that the gains are not dilution.

  • Three of the five benchmarks (POPE, HallusionBench, MMHal-Bench) use binary per-response losses, which added content cannot lower. PICD improves on each of them (e.g. MMHal hallucination rate 0.55 lowered to 0.35).
  • CHAIRS\text{CHAIR}_S, the share of captions containing at least one hallucinated object, is invariant to dilution: adding correct objects never makes a caption clean. It falls from 35.5% (base) to 15.9% (PICD), and from 20.7% for CEBC. This requires hallucinated objects to be removed.

We agree that the human study rates final outputs and does not validate revision-level changes; the revision says so and keeps revision-level human calibration as the stated extension.

W2. The calibration setup does not establish exchangeability; pooled versus per-benchmark. We agree on both principles. Disjointness is not what the theorem uses; it requires each fold and the test input to be exchangeable given the earlier data, under an identical pipeline (Section B.1). And a guarantee for a pooled mixture does not transfer to its components, which is why the Scope paragraph excludes subgroup coverage. Exchangeability is an assumption of Theorem 1, as in every split-conformal result, and we do not claim that the data prove it. What the protocol does to support it, is: all partitions are fixed before any fitting, the retriever, critic, evaluators and decoding settings are identical in calibration and deployment, and each later-round fold is replayed through the frozen earlier policies. On the level of the guarantee we accept the reading by reviewer and now state it in the paper: Table 2 organises the folds by domain, the theorem applies to the population from which a fold is drawn, and we do not claim a separate guarantee for each benchmark. The benchmark–backbone rates we report are empirical results, computed within each group before macro-averaging. Consistent with the assumption, the realised rate follows the requested one at all four targets (0.9, 4.4, 9.1, 18.6%).

AuthorsOpenReview ↗

Response (2/3): Methodological contribution and distinction from existing literature

(Continued from 1/3.)

W3. The critic ablations do not isolate why risk control deteriorates. We agree with the reviewers point, and we corrected the attribution. Theorem 1 holds for any fixed critic after trajectory-level, round-matched calibration, so a weaker critic that is properly calibrated loses coverage, not validity. The figure therefore cannot support the attribution we made fully. The 16.9% row is the entropy-threshold selector of Figure 4, at the identical operating point (39.5% coverage, 15.8% HIR): a threshold rule with no conformal calibration. For the 12.4% row we no longer draw any conclusion about risk control from its PVR; we cite it only for prediction quality (AUROC 0.754 versus 0.842), but our future work will be to work on the risk control as a natural extension to this paper. The revision relabels them as prediction ablations, removes the sentence that tied them to PVR, and states that under PICD's calibration prediction quality determines coverage. The rows that do isolate risk control are the shared-critic controls, where the critic is identical and only calibration differs: pointwise 7.1%, trajectory-only 5.6%, reused first-round margins 7.6%, PICD 4.4%.

W4. Limited methodological contribution; comparison with [1–3] and with simpler gates. Thank you for the references; the revision cites and discusses all three. The differences are specific.

  • [1] decides whether retrieval is needed from the model's self-knowledge. It is a current-state signal with no guarantee. Our Section 4.1 shows that such signals predict the outcome of a revision worse than paired supervision (best 0.752 versus 0.842).
  • [2] predicts retrieval utility and answer quality for text-only RAG by regression on retriever- and reader-side features. It gives point predictions without finite-sample risk control.
  • [3] (arXiv, June 2026) is the closest. It calibrates an assert-or-abstain threshold on the post-acquisition score and bounds the selective risk of asserted claims. We share the principle that calibration must see the deployed policy. PICD differs in the certified quantity (the paired change of a revision relative to the response it replaces, for two objectives), in the selection problem (an argmax over branch points, handled by the trajectory maximum), and in the sequential structure (accepted revisions change later states, handled by disjoint folds replayed through the frozen earlier policies). PICD never abstains; it keeps the original response.

On simpler gates under matched conditions, the submission contains three such comparisons. The binary utility router has the same capacity as the critic and, at nearly the same coverage as PICD (37.8% versus 36.9%), has a PVR of 12.1% versus 4.4%. The entropy threshold at 39.5% coverage has 16.9%. Figure 1b plots PVR against coverage for all selectors, and the proxy gates lie above the frontier at every coverage. We also state the limit of our own claim: against the uncalibrated version of the same critic, calibration provides control of the risk level, not a better trade-off at matched coverage (Section 4.2).

[1] Wang et al. Self-Knowledge Guided Retrieval Augmentation for Large Language Models. [2] Tian et al. Predicting Retrieval Utility and Answer Quality in Retrieval-Augmented Generation. [3] Xu et al. Look Again Before You Abstain: Budgeted Conformal Evidence Acquisition for Reliable Vision-Language Models.

AuthorsOpenReview ↗

Response (3/3): Perceptual failure and Sensitivity to the eight-candidate cap

(Continued from 2/3.)

Q1. Repair / unchanged / worse among confidently wrong answers and perceptual failures; generator-inherited versus retrieval-caused failures.

  • Over all 48,216 candidate revisions: 43.7% acceptable, 21.5% leave the loss unchanged, 34.8% harmful.
  • The critic is not limited to what the generator recognises. The generator's own confidence predicts the outcome poorly (entropy 0.662, maximum token probability 0.641), whereas the critic, which also receives the visual representation and the retrieved evidence, reaches 0.842. The gap is the information that the generator's confidence does not carry.
  • Retrieval is separated from the generator by intervention. The retrieval perturbations hold the generator and critic fixed and change only the evidence. Coverage falls from 36.9% to 5.9% under contradictory evidence while PVR stays between 4.4% and 4.9%: when retrieval cannot support a repair, PICD keeps the response.
  • Among analysed residual failures of accepted revisions, 23% are attributed to missing or contradictory retrieval, 31% to critic overestimation and 8% to unstable regeneration.

We agree that the rare-concept analysis combines both sources, and the revision states that now.

Q2. Why does enabling round two change coverage when the first-round policy is fixed? Your reasoning is correct: under the stopping rule a second round cannot change whether an input receives its first revision. The one-round row is a separately configured and calibrated run, not the main policy truncated after round one. With R=1R=1 the error budget and the calibration data are assigned to a single round, so its first-round margins are not those of the two-round policy, and the premise of a fixed first-round policy does not hold between the two rows. The two rows therefore do not attribute any gain to the second round, and the revision says this and relabels the row. The second round's own contribution is given by the round-wise rates within the single two-round policy (round-one PVR 2.6%, round-two PVR 1.9%, pathwise 4.4%).

Q3. Sensitivity to the eight-candidate cap; does denser selection recover missed errors? We did not sweep the cap, so we report what the submission determines. On MIMIC-CXR, where every input is long-form, Table 2 gives a mean of 5.7 candidate states per test study (18,704 over 3,269), against a cap of eight. The direction of the trade-off is fixed by the construction: the calibrated score is a maximum over candidates, so a denser set can only widen the margin, while it offers more places to branch. The failure analysis bounds the possible benefit: a branch point lying after the unsupported claim is the primary source of 12% of analysed residual failures. That share is the most that denser candidates could recover. The revision reports these facts and marks the sweep as open.

Q4. After changing the retrieval corpus or the deployment distribution, how much new calibration data is needed? Recalibration or retraining? Recalibration is sufficient for the guarantee; retraining is needed only to recover coverage. Theorem 1 holds for any fixed critic once the folds are exchangeable with the new deployment and are produced by the new pipeline, so a changed corpus or distribution requires new folds but no change to the critic or the VLM. A stale critic receives wider margins and revises less. On the amount: a finite margin requires n≥2R/αpath−1=79n \ge 2R/\alpha_{\text{path}}-1 = 79 inputs per fold at our main setting. In-distribution, coverage rises from 27.4% at n=250n=250 to 36.9% at n=2,000n=2{,}000 and 38.1% at n=4,000n=4{,}000, so about 2,000 inputs per fold reach the plateau.

We hope all of your concerns are resolved with our revision and we would like to request you to consider the revised submission for the final assessment. Regards.

Reviewer_z2zGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to reduce hallucinations in large vision-language models through inference-time corrections. During LVLM generation, current-state signals such as entropy provide limited guidance on whether a revision will improve the final answer. However, by comparing the final outputs of the original and revised paths from the same prefix, the quality difference—termed "terminal gain"—becomes straightforward to measure and learn. Building on this observation, the authors propose Pathwise Intervention-Certified Decoding (PICD), which applies a revision only when a calibrated lower bound on the terminal gain is positive. Specifically, PICD scores candidate revisions with a learned critic and selects the branch with the largest certified gain. Evaluated on four hallucination benchmarks with LLaVA-1.5-7B, Qwen2-VL-7B, and InternVL2.5-8B, PICD demonstrates consistent improvements over existing baselines.

Critical strengths

This article proposes using terminal gain instead of the current-state signal to determine whether the current step of the LVLM needs correction. Experiments then demonstrated that, given the current uncertainty of the model, the highest AUROC for the relevance prediction of the search results to correct the current step was only 0.752, while the AUROC for direct prediction using the critic was 0.842. This shows that the current uncertainty of the model and the improved response after correction are not highly correlated. This discovery provides a valuable perspective for LVLM hallucination correction.

Critical weaknesses

The experiments lack confidence intervals or the mean results from multiple experimental runs. Since this method relies heavily on conformal prediction, it is essential to conduct multiple experiments and report confidence intervals or mean values to verify that the observed improvements stem from the method itself rather than from random noise.

End-to-end time consumption assessment is relatively weak: In the field of LVLM hallucination relief, especially training-free, time consumption is a very important indicator. However, the paper only provides a median of 512ms for a mixed dataset, without providing a comparison of the time taken with the base model for generating lengths of 256/512/1024/2048.

Questions for authors

It would be beneficial to include additional experimental results for different values of MmaxM_{max}. In particular, it remains unclear whether the outcomes are consistent across various MmaxM_{max} values or whether Mmax=8M_{max}=8 is indeed the optimal choice.

A more detailed comparison of different ways to define "branch points" would strengthen the paper. In particular, it would be helpful to see comparative experiments between sentence-level and paragraph-level branch point definitions, along with a justification for why the sentence-level approach was ultimately chosen.

3 comments
AuthorsOpenReview ↗

Response (1/2): Confidence intervals and end-to-end time

Thank you for the review and for recognising the terminal-gain perspective.

W1. No confidence intervals or means over multiple runs. We agree that intervals belong in the paper. First, the sources of randomness. The VLMs decode greedily and the retriever, critic and thresholds are frozen, so PICD's decoding and selection are deterministic: repeating a run returns the same outputs. Seeds matter only for the stochastic baselines, which are averaged over three seeds (Section C). The remaining randomness is which inputs fall in the calibration and test sets, and this is exactly the randomness over which Theorem 1 holds.

Second, the conclusions do not rest on one number:

  • The control result holds at all four requested targets (0.9, 4.4, 9.1, 18.6% at 1, 5, 10, 20%), while pointwise calibration exceeds all four (2.3, 7.1, 12.8, 23.6%) and trajectory-only calibration exceeds all four (1.5, 5.6, 10.8, 21.3%). The ordering is the same at every target.
  • The differences are large relative to any plausible noise: harmful interventions fall from 8.9% to 2.8% with the identical critic, and AUROC rises from 0.752 to 0.842 over 48,216 candidates.
  • PICD has the lowest harmful-intervention rate on each of the three backbones separately.
  • The calibration-size study measures the stability of the calibrated margin directly: its variability falls from 0.031 at n=250n=250 to 0.009 at n=2,000n=2{,}000.

The camera-ready will add 95% paired-bootstrap intervals (10,000 input-level resamples, the procedure already specified in Section C) to Figures 1c and 4. These are computed from stored outputs and change no point estimate.

W2. End-to-end time, and latency at 256/512/1024/2048 tokens. The submission reports more than the median. Per example on an A100:

Methodp50 (ms)p95 (ms)Relative cost
Base decoder4106121.00×
Entropy-gated RAG4787181.17×
PICD5127711.25×
Always-retrieve RAG5428231.32×
Reranker + revision5588461.36×

PICD is cheaper than both always-on retrieval baselines. Its components are 44 ms retrieval per example, 2.8 ms critic time per candidate, and 108 ms per accepted suffix.

On length: every experiment uses the benchmarks' 256-token output limit. The scaling follows from the design, and the revision adds it as a cost model. Selection cost does not grow with length, because PICD scores at most 8 branch points per round for at most 2 rounds regardless of response length. Decoding cost grows only through the one accepted suffix per round, which is never longer than the response, and only 36.9% of inputs are revised at all. The overhead is therefore bounded by a length-independent selection cost plus at most RR suffix regenerations, whereas always-retrieve methods regenerate on every input. For transparency: PICD keeps the VLM frozen but is not training-free; the critic is fitted once offline (42 GPU-hours per backbone).

AuthorsOpenReview ↗

Response (2/2): Parameter sweeps and Sentence-level versus paragraph-level branch points

(Continued from 1/2.)

Q1. Results for different values of the parameter (the symbol did not render in the review). We report sweeps for each quantity the operator sets, and the behaviour is monotone and consistent in each:

  • Requested rate αpath\alpha_{\text{path}} at 1, 5, 10, 20%: realised PVR 0.9, 4.4, 9.1, 18.6%; coverage 16.9, 36.9, 48.6, 61.8% (Figure 1c).
  • Utility tolerance δtot\delta_{\text{tot}} at 0, 0.01, 0.03, 0.05, 0.10: hallucination loss 0.146, 0.139, 0.132, 0.126, 0.119; utility 0.690, 0.688, 0.684, 0.674, 0.651 (Figure 3f).
  • Calibration size nn from 250 to 4,000 (Figure 3e).

We do not claim that αpath=0.05\alpha_{\text{path}}=0.05 or δtot=0.03\delta_{\text{tot}}=0.03 is optimal. They are risk and tolerance levels chosen by the operator, not tuned hyperparameters or hand-picked. The purpose of the certificate is that the realised rate follows whichever level is requested, which the four targets show. The revision makes this explicit now.

Q2. Sentence-level versus paragraph-level branch points. Sentence boundaries were chosen for three reasons. (a) A sentence is the smallest unit that carries a complete claim, so branching there keeps the longest correct prefix, which is what the utility constraint protects. (b) The sentence-level set contains the coarser one: every internal paragraph or section boundary is also a sentence boundary. Calibration covers all candidates of a response simultaneously, so the selector remains free to pick a coarse boundary whenever its certified gain is largest. For MIMIC-CXR the candidates explicitly include report-section boundaries. (c) Outputs are at most 256 tokens, so a paragraph-only rule would leave few or no internal branch points. The cost of finer candidates is a wider margin, since the calibrated score is a maximum over candidates; the cap of eight candidates bounds this cost. We did not run a paragraph-only variant and the revision states the rationale above instead.

Ethics flag- A request for clarification. The review sets "Flag For Ethics Review: Yes" but does not state a reason, and neither the weaknesses nor the questions mention an ethics concern. Could you please tell us which aspect prompted the flag? To be useful in the meantime, here is the status of the three aspects we think are most likely:

  • Reader study. The submitted Ethics Statement contained an unfilled field for the institutional determination of the reader study. This was our error and it is corrected in the revision. Because the identifier names our institution, we can provide it to the AC and the ethics reviewers in a private comment if required.
  • Clinical data. MIMIC-CXR was used under its data-use agreement, all medical partitions are patient-disjoint, no patient data or generated report is released, and no output was used for patient care. The paper states that the medical results do not support autonomous clinical use.

If the concern is a different one, we will respond to it as soon as you let us know. Thank you for your detailed review and engagement. We hope our responses clarify your concerns and request you to consider our revised version for your final assessment. Regards.

AuthorsOpenReview ↗

Summary of our response and revision

Thank you again for your review. In brief, the revised manuscript (changes highlighted in yellow) addresses each of your points:

  • Robustness of results. PICD's decoding is deterministic, and it holds the requested violation rate at all four targets (1–20%), ahead of pointwise and trajectory-only calibration at each target. We will add 95% bootstrap intervals to Figures 1c and 4 in the final version.
  • End-to-end time. PICD runs at 1.25× the base decoder, faster than always-retrieve RAG and reranking. Its selection cost is independent of response length (Section 4.4).
  • Parameter choice. The violation target and the utility tolerance are operator-chosen levels, and the realised rate follows each requested level (Section 4).
  • Branch points. Sentence boundaries include every paragraph and section boundary and keep the longest correct prefix (Section 3.1).
  • Ethics. The Ethics Statement now records the reader-study determination.

Your feedback helped us sharpen the paper, and we hope you will consider the revised version in your final assessment. Thank you for your time and engagement. Regards.

Reviewer_U7gNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Pathwise Intervention-Certified Decoding (PICD), a framework for reducing hallucinations in multimodal generation by deciding when a model should revise its response. The main idea is to make a revision only when there is sufficient evidence that it will reduce hallucination loss without causing too much loss in utility. To do this, the authors train a gain critic on pairs of original and revised continuations to predict how a revision would affect hallucination loss and utility. They then use conformal calibration to account for prediction errors across possible revision points and successive rounds, providing a pathwise guarantee under exchangeability. Experiments on three VLMs and five benchmarks, including MIMIC-CXR report generation, show that PICD keeps the observed violation rates below the tested targets while improving the quality of the final responses compared with the evaluated baselines.

Critical strengths

The paper addresses an important problem by predicting whether a revision will help rather than relying only on uncertainty or retrieval relevance. The calibration method accounts for the selection of revision points and the effects of earlier revisions, with a theoretical guarantee. Empirical results are solid across diverse settings, including a medical domain, and the analysis is careful about costs and limitations.

Critical weaknesses

The guarantees depend on the hallucination and utility evaluators. In the medical experiments, claims are checked against reference reports rather than the images themselves, so it is unclear how well the results reflect actual visual grounding. The method assumes exchangeability between calibration and test data. Under distribution shift, the violation rate increases from 4.4% to 11.9%, which limits how reliably the guarantees can be applied in practice. The experiments do not examine how sensitive the results are to the critic's architecture or input features. Since the method relies on predicting the benefit of revisions, a more detailed analysis of the critic would be useful.

Questions for authors

The experiments allow at most two revision rounds. How would the method perform with more rounds, particularly for longer responses? Would the calibration become too conservative as the number of rounds increases? How are the error budgets for hallucination loss and utility allocated across revision rounds? Did the authors use equal splits, and how sensitive are the observed violation rates and revision coverage to this allocation?

3 comments
AuthorsOpenReview ↗

response (1/1): Guarantees, exchangeability and sensitivity to the critic's architecture

Thank you for the careful and constructive review.

W1. Guarantees depend on the evaluators; MIMIC-CXR is checked against reference reports. We agree, and Theorem 1 is stated relative to the fixed evaluators HH and UU. Three facts bound the concern. (a) The proof uses the evaluator only as a bounded function fixed before calibration, so an image-grounded or expert-labelled HH can replace ours with no change to the method or the theorem. (b) Only the medical loss is based on reference text. The four general-domain benchmarks score responses against image-level annotations or ground-truth answers. (c) The blinded study (300 pairs, medical cases rated by radiologists) is independent of the automatic evaluators: factuality is 4.31 for PICD versus 3.42 for the base decoder, with 78.0% preference and Krippendorff's α\alpha of 0.69–0.74. We also quantify the gap rather than hide it: evaluator–human disagreement is the primary source of 26% of the analysed residual failures (Section 4.5). The revision repeats this scope next to the theorem.

W2. Exchangeability; PVR rises from 4.4% to 11.9% under shift. Exchangeability is the assumption of every split-conformal guarantee. We report the behaviour outside it deliberately: 11.9% is the severe external-shift condition and 6.8% the moderate one, and the paper states that the guarantee does not cover either. Two properties make this manageable in practice. (a) Restoring validity needs recalibration only. Theorem 1 holds for any fixed critic, so neither the VLM nor the critic has to be retrained for the guarantee; retraining the critic affects coverage only. (b) The data requirement is small. A finite margin needs n≥2R/αpath−1=79n \ge 2R/\alpha_{\text{path}} - 1 = 79 inputs per fold at our main setting, and in the calibration-size study coverage at n=2,000n=2{,}000 (36.9%) is within 1.2 points of n=4,000n=4{,}000 (38.1%). We did not run post-shift recalibration, and the revision says so and lists it with weighted conformal methods as the next step.

W3. Sensitivity to the critic's architecture and input features. The submission isolates the factors that drive prediction quality:

  • Supervision target at equal capacity. The binary utility router has the same capacity as our critic and reaches AUROC 0.752, versus 0.842 for the terminal-gain critic. The gain comes from the target, not from the architecture.
  • Paired supervision. Training without paired continuations gives 0.754; removing the sign loss gives 0.823.
  • Input signals used alone. Entropy 0.662, retrieval similarity 0.691, cross-encoder relevance 0.731, current-output risk 0.748. No single input explains the critic's accuracy.
  • Backbones. Three VLMs with different vision encoders and hidden states; PICD has the lowest harmful-intervention rate on each.

The critic is deliberately minimal (a two-hidden-layer MLP). Architecture cannot affect validity: the theorem holds for any fixed critic, and a weaker critic receives a wider margin and revises fewer inputs.

Q1. More than two rounds, longer responses, conservativeness. PICD runs with any RR; each extra round needs one more disjoint fold and a share of the budget. Under an equal split each objective in each round receives αpath/(2R)\alpha_{\text{path}}/(2R): 1.25% at R=2R=2, 0.83% at R=3R=3, 0.63% at R=4R=4. So yes, margins widen and per-round coverage falls as RR grows. This is the cost of a union bound that assumes nothing about dependence between rounds. Two mechanisms limit it: any prespecified allocation summing to αpath\alpha_{\text{path}} is valid, so the budget can be front-loaded; and rounds are self-limiting, because PICD stops at the first round without a certified revision. In our results the second round already contributes less than the first (round-wise PVR 1.9% versus 2.6%). All experiments use a 256-token output limit with at most two rounds.

Q2. Allocation of the error budgets and sensitivity to it. The paper specifies the equal split (Section B, risk allocation): αH,r=αU,r=αpath/(2R)\alpha_{H,r}=\alpha_{U,r}=\alpha_{\text{path}}/(2R), which is 1.25% per objective per round at R=2R=2, the values used in the worked example of Figure 6. The split is prespecified, not tuned. This is required, not a convenience: Theorem 1 needs prespecified budgets, and choosing the split from observed violation rates would break the guarantee. For that reason we have no allocation sweep. Validity does not depend on the split, since any allocation summing to αpath\alpha_{\text{path}} satisfies the theorem; only coverage does, through the margins qH,rq_{H,r} and qU,rq_{U,r}. The revision states the split in the setup.

We hope our responses clarify your concerns and we thank you for your time and feedback. Regards.

Thank you for the detailed response. I appreciate the authors' efforts, and most of my concerns have been addressed. I would like to keep my original rating.

AuthorsOpenReview ↗

Thank you!

Thank you for your follow-up and for your positive assessment. We are glad the response addressed most of your concerns, and we appreciate your constructive review, which made the scope of the guarantee, the error-budget allocation and the behaviour under shift clearer in the revised manuscript. Regards.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.