acceptodds
Withdrawn from ICLR 2027

When Good Verifiers Go Bad: Silent Negative Transfer in Verifier-Guided VLM Training

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Verifier reliability is not portable across tasks. A verifier-guided self-DPO pipeline that produces genuine held-out gains on MathVista +9.6 points on the self-training data and +8.0 points held out) can become harmful on MMMU. Crucially, the failure is not visible from the target-task self-training signal: averaged across six learner-verifier configurations, performance on the MMMU self-training data still improves by +3.52 points, while held-out performance drops by 1.42 points. We term this silent negative transfer:a verifier previously validated as useful can continue to exhibit the signatures of successful self-training after its induced update ceases to transfer to unseen data. This raises a more basic question: how can we tell whether verifier supervision is reliable on the target task? A downstream failure alone cannot answer this question, because poor generalization may arise from two distinct sources: the verifier may induce an incorrect learning direction, or the correct learning direction itself may fail to generalize beyond the self-training distribution. We formalize these effects through gradient fidelity F, which measures alignment between the verifier-induced and correct training directions, and gradient transferability T, which measures alignment between the correct training and held-out directions. This decomposition yields a conservative Safe-Transfer Margin: positive first-order held-out alignment is certified when Motivated by this diagnosis, we introduce a simple yet effective precision-first filtering method, Asymmetric Acceptance Gating (AAG), which selects preference pairs according to the verifier's absolute confidence in the response receiving the positive update. Gradient-alignment analyses on MMMU show that AAG increases gradient fidelity from 0.29 to 0.42. In a separate same-cell analysis, the correct training direction remains positively aligned with the held-out direction T=0.31, yet raw verification rotates the induced update to negative held-out alignment -0.13; AAG restores positive alignment +0.22. Across all six MMMU learner–verifier configurations, AAG outperforms raw verifier-guided training on held-out evaluation. These results redefine what it means for a verifier to be reliable: not merely that it has worked before, but that it induces the right update on the task at hand—and that this update remains useful beyond the data that generated it.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
verifier, self-improving, VLM

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_pqXNOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies silent negative transfer in verifier-guided self-training for vision-language models, where improvements on self-training data do not translate into gains on held-out data. The authors distinguish two potential sources of failure through gradient fidelity and gradient transferability, and derive a Safe-Transfer Margin. They further propose Asymmetric Acceptance Gating, which filters preference pairs based on the verifier's confidence in the chosen response. Experiments on MathVista and MMMU show that AAG improves preference quality and generally outperforms raw verifier-guided DPO on held-out evaluation.

However, I find the motivation insufficiently supported by the empirical evidence, particularly regarding the prevalence and underlying causes of the claimed negative transfer phenomenon.

Critical strengths
  1. The paper provides a useful conceptual distinction between gradient fidelity and gradient transferability.

  2. AAG is straightforward to implement.

Critical weaknesses
  1. The motivation relies primarily on the MathVista–MMMU comparison. However, the average held-out degradation on MMMU is modest (−1.42 points), and only four of six configurations exhibit negative transfer. This raises concerns about whether the phenomenon is sufficiently systematic and significant.

  2. MathVista and MMMU differ in task difficulty, domain composition, and data distributions. It is unclear whether the degradation results from verifier unreliability or general self-training limitations. Notably, even ground-truth preference supervision yields only +0.80 points on held-out MMMU, suggesting that poor transferability may be the primary bottleneck.

  3. Although the paper includes ground-truth preference and matched-budget filtering controls, it lacks systematic comparisons with verifier-independent self-training strategies. Consequently, the extent to which negative transfer is specifically caused by verifier errors rather than generic self-training overfitting remains unclear.

  4. The gradient fidelity/transferability framework provides an interesting interpretation, but AAG is essentially a confidence-based filtering strategy that does not directly depend on this theory. In particular, the theoretical results do not establish that higher chosen-response precision necessarily leads to better gradient alignment or held-out generalization.

  5. The MMMU held-out set contains only 200 examples, and the reported performance differences are relatively small. Multi-seed experiments are limited to Qwen3-VL-4B, while larger learners primarily use a single seed. The lack of held-out confidence intervals makes it difficult to assess the statistical reliability of the findings.

Questions for authors
  1. Can the authors provide stronger evidence that silent negative transfer is systematic across tasks and model configurations?

  2. How can the authors distinguish verifier-induced failures from poor self-training generalization, given the limited gains from ground-truth preferences?

  3. How would verifier-independent self-training baselines perform under matched training budgets?

  4. Can the authors clarify the theoretical connection between AAG, gradient fidelity, and held-out generalization?

  5. Are the reported held-out differences statistically significant and robust across data splits and random seeds?

Reviewer_khzgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper examines the generalization of verifier-guided self-DPO for vision-language models. The authors report gains on both self-training and held-out prompts in a MathVista setting, but a different pattern on MMMU: across six learner–verifier configurations, Raw improves transductive accuracy by 3.52 points while reducing held-out accuracy by 1.42 points. They distinguish gradient fidelity FF, which compares verifier-induced and correct-preference training directions, from gradient transferability TT, which compares correct-preference training and held-out directions. An angular bound gives a sufficient condition for positive first-order transfer, and a separate analysis characterizes preference reversals in DPO. Asymmetric Acceptance Gating (AAG) retains pairs with high chosen-response confidence. It outperforms Raw on held-out MMMU in all six configurations, changing the mean gain over Frozen from −1.42 to +0.98 points. Gradient probes, three-seed 4B experiments, alternative training/filtering baselines and a ScienceQA replication support the analysis.

Critical strengths

1. A useful diagnostic distinction with appropriately scoped theory. Separating verifier-induced distortion from limited transfer of correct supervision helps organize the possible causes of self-training failure. Theorem 1, Corollary 1, Proposition 1 and Theorem 2 are correct within their stated scope. The authors explicitly distinguish a conservative first-order certificate from a prediction of final accuracy.

2. Consistent mitigation evidence with informative controls. AAG exceeds Raw in all six held-out MMMU configurations (Table 1) and all six paired 4B seed comparisons (Table 5). Appendix A.5 includes rDPO, confidence weighting, dual-threshold filtering, AAG-SFT and correct preferences on the AAG subset. These controls strengthen the empirical case. The authors also report AAG's remaining regression below Frozen for 8B/G2.5 (.510 versus .533), appropriately limiting the mitigation claim.

Critical weaknesses

1. The empirical claims need clearer evaluation protocols and uncertainty estimates.

  • Section 5.1 describes two complementary evaluation settings, but does not clearly link the panels of Table 1 to the same trained models. A common-run comparison would directly establish the claimed divergence. The MMMU transductive evaluation size and aggregation metric also need to be specified.
  • The held-out metric macro-averages 200 questions over 30 subjects, yet no test-sample uncertainty is reported for the mean Raw regression of 1.42 points. The seed standard deviations in Table 5 describe optimization variability; the sign test (p=.031p = .031) concerns AAG−Raw\mathrm{AAG} - \mathrm{Raw}, not Raw−Frozen\mathrm{Raw} - \mathrm{Frozen}. Paired, subject-aware intervals are needed to assess the negative-transfer claim.
  • The training-pool accounting is unresolved. Under ordinary pair-level precision, Table 3's τ=.90\tau = .90 condition contains 77 correct chosen responses among 121 retained pairs. If Table 14 uses the same 230-pair pool and ranking, its top 50% has precision at most 77115=.67\frac{77}{115} = .67, rather than .84. Different pools, reruns or weighting could explain this; Section 5.1 distinguishes Tables 3 and 13, but does not identify Table 14's final training pool. Similarly, reversal rates of .046 over 121 pairs or .120 over 230 pairs cannot arise from integer event counts rounded to three decimals. Explicit counts and aggregation rules would resolve these ambiguities. Table 7 also omits the retained count, configuration, evaluation regime and optimization exposure needed to assess its matched-data control.

2. The mechanistic attribution and methodological increment need sharper support.

  • Tables 11–12 provide useful initialization and training-trajectory evidence, but only one MMMU configuration reports TT. Table 8 also shows Correct-pref. changing from .512 on all pairs to .548 on the AAG subset. This supports a role for selection even under correct supervision, but does not quantify selection versus label-quality effects without fixed pair identities and optimization exposure. The small full-pool oracle gain alone cannot establish transferability as the dominant bottleneck across MMMU.
  • Table 11 explicitly shares initialization checkpoints and reference pools. However, correct-reference pairing, gradient parameter space, sample counts and aggregation remain insufficiently specified, as does their relationship to the actual Adafactor step. These details are essential to reproduce and interpret the reported directional improvement.
  • Chosen-side quality filtering already appears in fDPO (2024), and ROPO (2024) analyzes the amplification of noisy preferences in DPO. AAG's absolute chosen-confidence criterion differs from fDPO's policy-relative quality filter, but the distinction and its value need to be established. The strongest potential contribution is the VLM transfer diagnosis, whose evidence remains limited by the issues above.
Questions for authors
  1. Do the Table 1 panels evaluate the same trained models? Please specify the transductive evaluation size and metric, show train/held-out results for a common run, and provide paired, subject-aware uncertainty estimates for Raw−Frozen\mathrm{Raw} - \mathrm{Frozen} and AAG−Raw\mathrm{AAG} - \mathrm{Raw}.
  2. Section 5.1 distinguishes the pools in Tables 3 and 13. Which pool, seed and eligibility/filtering order produced Table 14, and how many pairs actually entered DPO? Please reconcile its .84 precision at 50% coverage with Table 3's 77121\frac{77}{121}, and give reversal counts and denominators.
  3. Table 11 specifies shared initialization checkpoints and reference pools. How are correct-reference pairs defined when both candidates have the same correctness, and which parameters, samples and aggregation define the gradients? For Table 12's later measurements, are Raw/AAG evaluated at common parameter checkpoints, with references recomputed there? How do these measurements relate to actual Adafactor steps, and what evidence supports extending the transferability diagnosis beyond the measured pool/configuration?
  4. Please clarify AAG's methodological increment over fDPO-style chosen-side filtering and identify any existing comparison that accounts for data, verifier-query and optimization budgets. For Table 7, what are NN, the configuration, evaluation regime and training budget, and how does equal-budget Raw differ from Random-NN?

A common-run paired analysis, reconciled pool statistics, reproducible gradient measurements and clearer positioning against prior filtering methods would materially strengthen my assessment.

Reviewer_TXHfOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates silent negative transfer in verifier-guided self-training, where a verifier that improves held-out performance on one task can degrade generalization on another. To explain this phenomenon, the authors introduce a gradient-based framework that distinguishes gradient fidelity, measuring the alignment between verifier-induced and ground-truth preference updates, from gradient transferability, measuring whether correct training updates generalize to held-out data. Based on this analysis, this paper propose Asymmetric Acceptance Gating (AAG), a simple confidence-based preference-pair filtering strategy that prioritizes the correctness of chosen responses.

Experiments on MathVista, MMMU, and ScienceQA demonstrate the negative-transfer phenomenon. The results show that AAG improves chosen-response precision, enhances gradient alignment, and outperforms standard verifier-guided DPO on held-out MMMU evaluations.

Critical strengths

1. Simple and practical method. The proposed AAG method is straightforward and easy to understand, requiring only confidence-based filtering of preference pairs without modifying the standard DPO objective.

2. Meaningful problem identification and empirical validation. The paper identifies silent negative transfer as a failure mode in verifier-guided self-training. Experiments demonstrate the effectiveness of AAG, while gradient analyses and ablation studies provide additional evidence.

Critical weaknesses

1. Limited novelty and weak theory–method connection. AAG is essentially a simple confidence-based filtering strategy for DPO. The proposed gradient metrics are only used for post-hoc analysis and do not guide training. Moreover, the theoretical results do not directly establish that AAG improves gradient alignment, substantially weakening the paper's methodological contribution.

2. Limited experimental scope and generalizability. The evaluation is primarily limited to VLMs trained with offline self-DPO, using only two verifiers from the Gemini family. Moreover, the MMMU held-out set contains only 200 examples, and larger models are evaluated with a single reported seed. These limitations raise concerns about the robustness of the results and the applicability of AAG to more diverse verifiers and broader training paradigms, such as RL and online self-improvement.

Questions for authors

Q1. Connection between theory and method. The proposed gradient metrics are only used for post-hoc analysis and do not directly guide AAG training. Can the authors clarify how the theoretical framework supports the specific design of AAG?

Q2. Relationship between chosen precision and gradient fidelity. Does improving chosen-response precision necessarily lead to better gradient alignment? Can the authors provide stronger theoretical and empirical evidence supporting this relationship?

Q3. Statistical significance of held-out improvements. AAG achieves only a 0.98 percentage-point average improvement over the frozen baseline on MMMU, with just 200 held-out examples. Can the authors provide statistical significance tests to establish the reliability of these gains?

Q4. Dependence on verifier confidence quality. AAG assumes that verifier confidence reliably ranks the correctness of chosen responses. How robust is the method when verifiers are overconfident or their confidence rankings are unreliable?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.