acceptodds
Under review as a conference paper at ICLR 2027

Guiding Selective Self-Correctionin Vision-Language Modelsthrough Evidence Auditing

Abstract

Vision-language models can produce coherent reasoning, but misread visual information may persist through later reasoning and self-checking. Revision can also turn correct answers into wrong ones. We propose PACT, a training-free framework with context-isolated evidence auditing and selective revision. Perception, option-level auditing, and task-adapted consistency testing run without the initial answer or its reasoning. Fusion hides earlier answer identities and combines the image with diagnostic reports. A deterministic gate accepts or rejects its candidate based on consistency and contradiction. All stages share one frozen model, without a separately trained verifier or reference answers at inference time. Across 19 open-source model configurations on MMStar and RealWorldQA, PACT scores higher than six ablations in all 38 model–dataset pairs. Its mean differences from the listed Baseline reference scores are 3.09 and 3.12 percentage points, respectively. On both benchmarks, perception has the highest single-module mean score, while removing auditing causes the largest mean loss. These results show that a module's effect depends on the full configuration.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vision-language models, visual reasoning, context isolation, evidence auditing, selective revision

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_dkhwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes PACT, a training-free self-correction framework for vision-language models. A frozen model runs three separate diagnostics: perception, option auditing, and option permutation. A fusion call and a deterministic gate then decide whether to keep the initial answer or adopt the revision. Experiments cover 19 open-source model configurations on MMStar and RealWorldQA. PACT outperforms six ablations in all 38 model–dataset pairs, and its mean scores exceed the listed public baseline references by 3.09 and 3.12 points.

Critical strengths
  1. The paper targets a real failure mode of self-correction, namely that revisions can turn correct answers into wrong ones. Separating revision proposal from revision acceptance is a sensible design choice.

  2. The evaluation covers 19 model configurations from three model families, which gives the experiments reasonable breadth.

Critical weaknesses
  1. The main improvement claim is not based on a clean comparison. The Baseline column uses public reference scores rather than paired reruns of the initial solve. The paper also notes that the Qwen3.5 public scores use Thinking settings and that RealWorldQA is converted into a multiple-choice format. The reported gains of roughly three points therefore mix the effect of PACT with differences in evaluation protocol. The authors themselves state that paired improvements require shared initial and final predictions, yet these numbers remain the headline result.

  2. The empirical evidence is still not enough to show that the gain comes from PACT itself. Even among the authors' own variants, the ablation pattern is hard to interpret. On MMStar, the variant that keeps both auditing and permutation testing scores 71.59. This is lower than auditing alone at 72.12 and lower than permutation testing alone at 72.04. Yet adding perception to this combination raises the score by about three points, and the paper does not explain why the branches interact this way. PACT also uses up to five calls per sample, but there is no matched-budget comparison with self-consistency or existing self-correction methods. The paper additionally defines wrong-to-correct and correct-to-wrong rates in Equation 13 but never reports them, and there are no variance estimates or significance tests.

Questions for authors
  1. Can you provide paired results for the baseline and PACT under exactly the same evaluation setup?

  2. How does PACT compare with self-consistency or prior self-correction methods at the same call budget?

  3. Why does the variant combining auditing and permutation testing score lower than either branch alone, while adding perception to it yields about three points?

  4. Can you report the wrong-to-correct and correct-to-wrong rates from Equation 13?

Reviewer_EkL3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

PACT addresses two problems in visual self-correction: a model may repeat an earlier visual mistake, or change a correct answer into a wrong one. It uses one frozen model to generate an initial answer and then run separate checks of visual evidence, answer options, and consistency under option reordering. These checks do not receive the initial answer or its reasoning. A fusion step produces a candidate answer from the image and diagnostic reports, and a fixed rule decides whether to accept it. The paper evaluates 19 model configurations on MMStar and a multiple-choice version of RealWorldQA. The full method scores higher than six ablations in all 38 reported model–dataset pairs, with mean differences of 3.09 and 3.12 percentage points from the listed public reference scores.

Critical strengths
  1. The method has a clear motivation and explicit decision rules. Separating evidence checking from the initial answer is a reasonable way to reduce its direct influence on later judgments. The distinction between support, contradiction, and uncertainty is also useful. Sections 3.2–3.6 clearly describe what each stage receives and how the final answer is chosen.
  2. The approach requires no additional training and is tested across several model families. All stages use the same frozen model, without a separately trained verifier or access to reference answers at inference. Table 1 covers 19 configurations from three model families. Within the reported comparisons, the full method consistently scores higher than its six ablations.
Critical weaknesses
  1. The main comparison does not establish the actual correction gain. Table 1 uses public reference scores as the Baseline, rather than the accuracy of the initial predictions used by PACT. The reported differences therefore do not show whether the method improves upon its own initial answers on the same questions. For RealWorldQA, converting all questions to multiple-choice format also limits comparison with scores obtained under the original format. The authors acknowledge these differences in Sections 4.2–4.4, but the paired evaluation remains missing. This is a central requirement for judging a self-correction method.
  2. The effect of the acceptance gate is not measured separately. Equation 13 explains that the net accuracy change comes from correcting initial errors minus changing correct answers into wrong ones. However, the experiments do not report these two rates or provide a separate comparison with and without the gate. All ablations retain fusion and gating, while removing auditing or consistency testing changes both the diagnostic information and the gate’s behavior. The paper also notes that stability protection can preserve wrong answers, but does not measure how often this happens. The results therefore leave the benefit of selective revision unclear.
  3. Comparisons under the same inference budget are missing. When permutation testing applies and no retries are needed, full PACT uses five model calls. The reported ablations use fewer calls, so they change both the diagnostic process and the computation available. There is no comparison with repeated answering or self-consistency under the same call and generation budgets. The two-call Core configuration defined in Section 3.8 is also absent from the results table. These controls are needed to show whether the proposed workflow offers an advantage over simpler ways of using the same budget.
Questions for authors
  1. Can you report the accuracy of the actual initial answers, fusion candidates, and final answers on the same evaluation samples? Please include the rates of correcting wrong answers and changing correct answers into wrong ones, together with paired confidence intervals. For RealWorldQA, please also specify the evaluated sample counts, option-construction procedure, and answer mapping.
  2. Does PACT still improve over simpler methods when the inference budget is matched? A comparison with repeated answering or self-consistency would be useful. Keeping the diagnostic reports and fusion candidates fixed, please also test the effect of removing stability protection and contradiction-based rejection separately.
Reviewer_Kf9sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PACT, a training-free framework for selective self-correction in multiple-choice visual question answering. Starting from an initial answer, PACT runs three diagnostic steps with separate prompts to the same frozen model: describing the image without answer options, checking whether the visual evidence supports each option, and testing answer consistency after reordering the options. The model then revisits the question using these diagnostic outputs, and a rule-based gate determines whether to accept the revision. The paper also defines correction gains and harmful changes to analyze the effects of revision. The framework is evaluated on MMStar and a multiple-choice adaptation of RealWorldQA using 19 open-source model configurations.

Critical strengths
  1. PACT separates answer revision from revision acceptance. The framework distinguishes generating a potentially better answer from deciding whether the change should be adopted, using a simple and interpretable decision rule.

  2. The paper discusses the limitations of its components, including interactions between modules, cases where stability checks preserve incorrect answers, and the difficulty of attributing improvements to individual modules from the ablations.

Critical weaknesses
  1. The evaluation lacks compute-matched baselines. Full PACT requires up to five model calls per question, while all ablations retain a two-call backbone. There is no comparison with five-sample self-consistency, the two-call Core variant from Section 3.8 is absent from Table 1, and existing methods such as Self-Refine, CoVe, and Critic-V are not included. Without these comparisons, the reported 3-point improvement cannot be separated from the effect of additional computation.

  2. The main improvements are calculated against publicly reported reference scores rather than paired evaluations. Table 2, which should provide these reference scores, is missing from the PDF. Appendix A, containing the complete algorithm, is also missing. These omissions prevent readers from verifying the main comparisons and implementation details.

  3. The proposed decomposition of accuracy changes is not evaluated directly. Although the paper defines correction gains and harmful-change losses, it reports only net accuracy without the corresponding correction rate or correct-to-wrong flip rate.

Questions for authors
  1. A comparison against five-sample majority voting under the same call budget would help establish whether PACT improves on repeated sampling. How does PACT perform in this setting?

  2. Table 2 and Appendix A appear to be missing from the PDF. Could the authors include them so that the reference scores and algorithm can be checked?

  3. Eq. (13) distinguishes successful corrections from harmful changes, but the experiments report only net accuracy. What are the corresponding correction and harmful flip rates?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.