- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the prediction inconsistency of vision-language models (VLMs) between atomic and joint visual tasks. The authors observe that a VLM may correctly identify individual visual attributes when queried separately but fail to recognize the same attributes when multiple judgments are combined into a joint query. Based on this observation, they propose PRIDE, a label-free self-evolution framework that exploits internal atomic–joint prediction divergence as a source of self-supervision. Experiments across multiple VLM backbones and additional GQA and ChartQA benchmarks demonstrate improvements in downstream accuracy. The paper also provides ablation studies, prompt robustness evaluations, theoretical analysis, and long-horizon self-evolution experiments.
Critical strengths
- The paper addresses a meaningful problem concerning prediction inconsistency in VLMs. The observation that a VLM can correctly answer individual visual questions but fail when these questions are combined is interesting and practically relevant. The paper further investigates this phenomenon through prompt paraphrasing, schema matching, length controls, and repeated sampling, providing empirical evidence that some atomic–joint disagreements cannot be explained solely by prompt variations or stochastic decoding.
- The experimental evaluation is relatively comprehensive. The authors evaluate PRIDE against various industrial anomaly detection approaches and report results across four VLM backbones. They also investigate transfer to GQA and ChartQA and provide multiple ablation studies, parameter sensitivity analyses, and long-horizon experiments. The use of source-disjoint adaptation and evaluation datasets for the main MMAD experiments is another positive aspect.
- The paper provides useful analyses of the optimization mechanism and its limitations. The theoretical discussion of local recoverability, the trust-region analysis, and the experiments investigating preservation damage and nonlinear residuals help explain the behavior of the proposed optimization method. The authors also acknowledge performance saturation and degradation over longer self-evolution trajectories rather than reporting only positive results.
Critical weaknesses
- The reliability of disagreement-driven self-supervision is not sufficiently justified. Stable atomic predictions are not necessarily correct, and incorrect pseudo-labels may be reinforced during self-evolution. The long-horizon experiments also show declining target precision and performance degradation, raising concerns about the reliability of sustained self-improvement.
- The necessity of Jacobian/Gram reconciliation is not sufficiently demonstrated. Although Table A2 includes Atomic SFT and PCGrad baselines, these comparisons do not fully isolate the contribution of the Jacobian-based optimization. Comparisons with direct Joint SFT and SFT with KL-based preservation are needed to justify the additional complexity.
- The experiments do not directly demonstrate a reduction in the compositionality gap. The reported results mainly focus on downstream accuracy improvements, without sufficiently analyzing changes in atomic–joint disagreement before and after training. This makes it difficult to determine whether the improvements primarily result from recovering suppressed atomic capabilities.
- The preservation mechanism does not guarantee the retention of general VLM reasoning capabilities. The proposed constraints only preserve selected output scores rather than the full autoregressive generation behavior. The paper lacks direct evaluations of whether the model's original multi-step reasoning and chain-of-thought capabilities are maintained after self-evolution.
Questions for authors
- What ensures that disagreement-derived pseudo-labels remain reliable throughout self-evolution? Prediction stability does not necessarily indicate correctness. How does PRIDE prevent confidently incorrect atomic predictions from being reinforced over successive training rounds? Could the authors provide an analysis separating the effects of incorrect pseudo-label accumulation, repeated-data training, and parameter drift? An oracle-label analysis or controlled pseudo-label corruption experiment would help clarify the robustness of the proposed mechanism.
- Could the authors compare PRIDE against direct Joint SFT and Joint SFT with KL-based preservation, using identical pseudo-labels, repair/preserve sets, trainable parameters, and optimization budgets? Additionally, could the same repair/preservation objective be optimized using conventional gradient descent to isolate the contribution of the closed-form Jacobian solver? Please also clarify the implementation differences between Atomic SFT and Elicitation-only in Table A2.
- The preservation constraints operate on selected output scores rather than the full autoregressive distribution. Could the authors evaluate the impact of PRIDE on multi-step visual reasoning and chain-of-thought generation? Furthermore, how does the fixed-prefix local Jacobian approximation account for changes to the autoregressive context after earlier output predictions are modified?