- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes PACT, a training-free self-correction framework for vision-language models. A frozen model runs three separate diagnostics: perception, option auditing, and option permutation. A fusion call and a deterministic gate then decide whether to keep the initial answer or adopt the revision. Experiments cover 19 open-source model configurations on MMStar and RealWorldQA. PACT outperforms six ablations in all 38 model–dataset pairs, and its mean scores exceed the listed public baseline references by 3.09 and 3.12 points.
Critical strengths
-
The paper targets a real failure mode of self-correction, namely that revisions can turn correct answers into wrong ones. Separating revision proposal from revision acceptance is a sensible design choice.
-
The evaluation covers 19 model configurations from three model families, which gives the experiments reasonable breadth.
Critical weaknesses
-
The main improvement claim is not based on a clean comparison. The Baseline column uses public reference scores rather than paired reruns of the initial solve. The paper also notes that the Qwen3.5 public scores use Thinking settings and that RealWorldQA is converted into a multiple-choice format. The reported gains of roughly three points therefore mix the effect of PACT with differences in evaluation protocol. The authors themselves state that paired improvements require shared initial and final predictions, yet these numbers remain the headline result.
-
The empirical evidence is still not enough to show that the gain comes from PACT itself. Even among the authors' own variants, the ablation pattern is hard to interpret. On MMStar, the variant that keeps both auditing and permutation testing scores 71.59. This is lower than auditing alone at 72.12 and lower than permutation testing alone at 72.04. Yet adding perception to this combination raises the score by about three points, and the paper does not explain why the branches interact this way. PACT also uses up to five calls per sample, but there is no matched-budget comparison with self-consistency or existing self-correction methods. The paper additionally defines wrong-to-correct and correct-to-wrong rates in Equation 13 but never reports them, and there are no variance estimates or significance tests.
Questions for authors
-
Can you provide paired results for the baseline and PACT under exactly the same evaluation setup?
-
How does PACT compare with self-consistency or prior self-correction methods at the same call budget?
-
Why does the variant combining auditing and permutation testing score lower than either branch alone, while adding perception to it yields about three points?
-
Can you report the wrong-to-correct and correct-to-wrong rates from Equation 13?