acceptodds
Withdrawn from ICLR 2027

PRIDE: Prediction Reconciliation via Internal Divergence Elicitation for Self-Evolving Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models can answer a visual attribute correctly in isolation yet fail on the same attribute inside a compound query. Because such disagreement may arise from prompt form or sampling noise, we first establish a controlled premise: a subset of atomic-joint disagreements persists under paraphrase, length/schema matching, and repeated rollouts, and is enriched for recoverable joint errors. We introduce PRIDE, a label-free self-evolution framework that filters this stable task-conditioned divergence and converts accepted output corrections into visual-parameter updates through a preservation-aware Jacobian/Gram solve. Under a controlled Qwen3.5-4B setting trained on the full ADOPD-Dataset-6K across twelve disjoint 500-image rounds, with MMAD reserved exclusively for evaluation, PRIDE establishes a new state of the art on the MMAD AD/DC subtasks, reaching 85.57/74.40 in 0-shot and 87.16/76.02 in 1-shot evaluation. The gains further transfer across Qwen3-VL-4B, LLaVA-OneVision-1.5-4B, and Gemma-3n-E4B, and extend beyond industrial anomaly reasoning to open-ended GQA and ChartQA. We additionally report the complete seven-task MMAD profile to verify that improvements on AD/DC do not come at the expense of localization and other non-target capabilities. These results support atomic–joint task-conditioned divergence as a reliable endogenous supervision signal for self-evolving vision-language models when signal stability, output coupling, and local update validity are explicitly controlled.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision Language Models, Self-Evolution

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vDzkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the prediction inconsistency of vision-language models (VLMs) between atomic and joint visual tasks. The authors observe that a VLM may correctly identify individual visual attributes when queried separately but fail to recognize the same attributes when multiple judgments are combined into a joint query. Based on this observation, they propose PRIDE, a label-free self-evolution framework that exploits internal atomic–joint prediction divergence as a source of self-supervision. Experiments across multiple VLM backbones and additional GQA and ChartQA benchmarks demonstrate improvements in downstream accuracy. The paper also provides ablation studies, prompt robustness evaluations, theoretical analysis, and long-horizon self-evolution experiments.

Critical strengths
  1. The paper addresses a meaningful problem concerning prediction inconsistency in VLMs. The observation that a VLM can correctly answer individual visual questions but fail when these questions are combined is interesting and practically relevant. The paper further investigates this phenomenon through prompt paraphrasing, schema matching, length controls, and repeated sampling, providing empirical evidence that some atomic–joint disagreements cannot be explained solely by prompt variations or stochastic decoding.
  2. The experimental evaluation is relatively comprehensive. The authors evaluate PRIDE against various industrial anomaly detection approaches and report results across four VLM backbones. They also investigate transfer to GQA and ChartQA and provide multiple ablation studies, parameter sensitivity analyses, and long-horizon experiments. The use of source-disjoint adaptation and evaluation datasets for the main MMAD experiments is another positive aspect.
  3. The paper provides useful analyses of the optimization mechanism and its limitations. The theoretical discussion of local recoverability, the trust-region analysis, and the experiments investigating preservation damage and nonlinear residuals help explain the behavior of the proposed optimization method. The authors also acknowledge performance saturation and degradation over longer self-evolution trajectories rather than reporting only positive results.
Critical weaknesses
  1. The reliability of disagreement-driven self-supervision is not sufficiently justified. Stable atomic predictions are not necessarily correct, and incorrect pseudo-labels may be reinforced during self-evolution. The long-horizon experiments also show declining target precision and performance degradation, raising concerns about the reliability of sustained self-improvement.
  2. The necessity of Jacobian/Gram reconciliation is not sufficiently demonstrated. Although Table A2 includes Atomic SFT and PCGrad baselines, these comparisons do not fully isolate the contribution of the Jacobian-based optimization. Comparisons with direct Joint SFT and SFT with KL-based preservation are needed to justify the additional complexity.
  3. The experiments do not directly demonstrate a reduction in the compositionality gap. The reported results mainly focus on downstream accuracy improvements, without sufficiently analyzing changes in atomic–joint disagreement before and after training. This makes it difficult to determine whether the improvements primarily result from recovering suppressed atomic capabilities.
  4. The preservation mechanism does not guarantee the retention of general VLM reasoning capabilities. The proposed constraints only preserve selected output scores rather than the full autoregressive generation behavior. The paper lacks direct evaluations of whether the model's original multi-step reasoning and chain-of-thought capabilities are maintained after self-evolution.
Questions for authors
  1. What ensures that disagreement-derived pseudo-labels remain reliable throughout self-evolution? Prediction stability does not necessarily indicate correctness. How does PRIDE prevent confidently incorrect atomic predictions from being reinforced over successive training rounds? Could the authors provide an analysis separating the effects of incorrect pseudo-label accumulation, repeated-data training, and parameter drift? An oracle-label analysis or controlled pseudo-label corruption experiment would help clarify the robustness of the proposed mechanism.
  2. Could the authors compare PRIDE against direct Joint SFT and Joint SFT with KL-based preservation, using identical pseudo-labels, repair/preserve sets, trainable parameters, and optimization budgets? Additionally, could the same repair/preservation objective be optimized using conventional gradient descent to isolate the contribution of the closed-form Jacobian solver? Please also clarify the implementation differences between Atomic SFT and Elicitation-only in Table A2.
  3. The preservation constraints operate on selected output scores rather than the full autoregressive distribution. Could the authors evaluate the impact of PRIDE on multi-step visual reasoning and chain-of-thought generation? Furthermore, how does the fixed-prefix local Jacobian approximation account for changes to the autoregressive context after earlier output predictions are modified?
Reviewer_8euoOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

PRIDE uses disagreements between atomic and joint VLM predictions for label-free adaptation. It selects stable atomic answers as pseudo-targets, then applies a regularized Jacobian update to correct joint predictions while preserving agreed outputs. Experiments cover MMAD, four approximately 4B-scale backbones, GQA, and ChartQA. The idea has merit, but the threshold inconsistency and unclear implementation details weaken confidence in the results. Stronger evidence is also needed to attribute the gains to composition recovery and justify the added computational cost.

Critical strengths
  1. The problem is interesting, and using internal prediction discrepancies as supervision is a reasonable idea.
  2. The repair/preserve objective is clear. Damping, trust-region control, and nonlinear checks address practical risks of local updates.
  3. The paper reports substantial gains and includes useful prompt controls, matched ablations, and multi-seed results.
Critical weaknesses
  1. The stability-threshold results appear inconsistent with the definition. With eight rollouts, thresholds of 0.65 and 0.75 should accept the same predictions, yet Tables A5/A6 report different coverage and performance. This needs explanation.
  2. Stable atomic answers can still be wrong. Neither the theory nor the recoverability and nonlinear checks establish target correctness. Recovery/harm rates, target precision, and net correction gains before and after filtering should be reported across model scales. The paper should also distinguish suppressed atomic capability from cases where joint context provides useful disambiguation.
  3. The experiments do not clearly separate composition recovery from general domain adaptation. Before/after atomic and joint accuracies, together with tests across different task combinations, sizes, and orders, would better support the claimed mechanism.
  4. Computational cost and scalability remain unclear. The paper should specify whether each solve covers an image, batch, or shard, and report costs for target generation, Jacobian acquisition, communication, solving, and validation. Comparisons with self-training and repair/preserve distillation under the same GPU-hour budget are needed, especially for stronger models with fewer recoverable errors.
  5. Preservation covers only selected local outputs, not historical examples or broader capabilities. The paper should clarify whether validation regenerates the full joint response, since fixed-prefix scoring may miss effects from changes to earlier output slots.
  6. Evaluation details need clarification. Table A10 appears to conflate AD accuracy with AD-BAcc. The GQA/ChartQA experiments should specify adaptation/evaluation splits, image overlap, and whether evaluation uses individual or joint questions.
Questions for authors

see above comments

Reviewer_Hf4mOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces PRIDE, a label-free self-evolution framework for vision-language models (VLMs). It targets the "compositional gap" where a VLM can answer an attribute query correctly in isolation but fails when composed inside a joint query. PRIDE filters stable atomic-joint disagreements to define a repair region, and consistent outputs to define a preserve region. It maps the output corrections into local parameters using a preservation-aware Jacobian/Gram solve on trainable adapters (LoRA). Evaluating on ADOPD-Dataset-6K and MMAD, PRIDE demonstrates significant improvements on anomaly discrimination (AD) and decision consistency (DC) subtasks.

Critical strengths

Clever leveraging of internal asymmetry: Using atomic-joint divergence as an endogenous training signal during label-free adaptation is a highly creative and original contribution.

Rigorous preservation constraints: Solving a local preservation-aware ridge regression to update LoRA parameters while protecting consistent coordinates is a mathematically sound alternative to naive SFT.

Critical weaknesses

High Jacobian/Autograd Cost: Performing row-wise Jacobian acquisition over multiple query dimensions and output tokens is computationally expensive. This significantly limits the framework's scalability to larger models (e.g., 72B VLMs) or complex multi-turn dialogs.

Sensitivity to Elicitation Gating Thresholds: Gating thresholds such as stability τ_c and recoverability τ_r are highly critical. If the thresholds are set too low, the model suffers from late-round drift and collateral damage on protected capabilities; if too high, model updates are frequently rejected, leading to training stagnation.

Questions for authors

Since Jacobian acquisition is the main computational bottleneck, have you explored cheap first-order approximations to scale PRIDE to larger backbones?

How does PRIDE handle situations where both the atomic response and joint response are stable but incorrect (i.e., when they agree on a false prediction)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.