acceptodds
Under review as a conference paper at ICLR 2027

When Does Data Leakage Translate into Evaluation Gains? A Study of VLM Post-Training

Abstract

Why does evaluation-data exposure yield substantial score gains in some post-training settings but only limited average changes in others? We study train–evaluation overlap in vision–language models (VLMs) across training objectives and task settings. On natural tasks, increasing overlap from 0% to 100% raises fixed-set accuracy by 21.2 percentage points for supervised fine-tuning and by 14.0 and 14.2 points for DPO and PPO, but by only about 3.7 points for both GRPO and RLOO. In the adaptive visual-token setting, the five post-training objectives instead show fixed-set changes from -0.7 to +1.3 percentage points, including a near-zero response for supervised fine-tuning. Limited average responses therefore occur in distinct post-training settings and cannot be assigned to an optimizer category alone. To analyze this variation, we combine local propagation analysis with an exact finite-horizon decomposition connecting exposure-induced training differences, their propagation through later updates, and whether the resulting behavioral changes affect the evaluation score. This framework distinguishes a change in training from an observable evaluation benefit without assuming that similar terminal responses share one mechanism. Together, the results show that exposure responses must be interpreted in relation to the training setting instead of being inferred from an optimization label when considered in isolation.

Primary area:
reinforcement learning
Keywords:
Data leakage, Vision-language models, Post-training, Train-evaluation overlap, Exposure-response analysis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_X23KOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how evaluation-data leakage affects VLM post-training across different optimization objectives and task settings. Through experiments with four VLM backbones, ten multimodal benchmarks, and five training objectives (SFT, DPO, PPO, GRPO, and RLOO), the authors observe substantially different evaluation gains under controlled train–evaluation overlap. To explain these differences, the paper develops a theoretical framework connecting exposure-induced training updates, their propagation through subsequent optimization, and the observability of resulting behavioral changes in evaluation scores.

Critical strengths
  • Clear motivation and a relevant research question. The paper investigates an important issue in model evaluation: whether exposure to evaluation samples necessarily translates into inflated benchmark performance. The distinction between training-data exposure and observable evaluation gains is meaningful, particularly for understanding the reliability of VLM post-training evaluations.

  • Broad experimental coverage. The study includes four VLM backbones, ten multimodal benchmarks, five post-training objectives, and two task settings. The use of multiple overlap ratios and uncontaminated holdout sets provides a reasonably comprehensive characterization of exposure responses. The detailed model- and benchmark-level results also help reveal variations that are obscured by aggregate statistics.

  • A unified analytical framework. The paper distinguishes exposure-induced update differences, their propagation through subsequent training, and the sensitivity of evaluation metrics to behavioral changes. The accompanying local analysis and exact finite-horizon decomposition provide a coherent mathematical perspective on why training exposure need not produce measurable evaluation improvements.

Critical weaknesses
  • Insufficient validation of the theoretical framework. Section 3 attributes exposure responses to update differences, their propagation, and evaluation-score sensitivity. However, the experiments only measure final accuracy without directly analyzing (d_t), training dynamics, or behavioral distribution changes. Consequently, the proposed mechanisms remain theoretical possibilities rather than empirically verified explanations.
  • Confounding factors in experimental comparisons. As acknowledged in Section 5, training-data composition, sample ordering, and evaluation settings differ across conditions. Moreover, the natural and adaptive settings change both the learning target and feedback pathway. These uncontrolled factors make it difficult to attribute the observed differences to optimization objectives or task characteristics.
  • Missing ablations and sensitivity analyses. The paper lacks systematic analyses of learning rate, training duration, batch size, regularization, sampling configurations, and contamination frequency or timing. In contrast, the Memorization–Exploitation Analysis of Magar and Schwartz [1] already investigates several such factors. Without these experiments, the robustness of the observed differences across post-training objectives remains unclear.
  • Limited generalizability of the analysis. The paper mainly reports differences in final accuracy without establishing predictive relationships between the proposed theoretical factors and exposure responses. Furthermore, the experiments focus on exact sample overlap, without considering paraphrased, partial, or near-duplicate contamination. It remains unclear whether the findings generalize beyond the evaluated protocols.

References

[1] Magar, I., & Schwartz, R. (2022). Data Contamination: From Memorization to Exploitation. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL 2022), Volume 2: Short Papers, 157–165.

Questions for authors
  • Direct mechanism validation. Could the authors measure exposure-induced update differences and their evolution across training checkpoints, using gradient/update norms, parameter differences, or prediction-distribution changes? Can these measurements explain the differences between strong and weak exposure responses, rather than merely being consistent with the proposed decomposition?

  • Behavioral analysis. In the adaptive visual-token setting, does exposure significantly change the visual-selection policy despite nearly unchanged answer accuracy? Could the authors measure selected-region overlap, token-selection distributions, or policy divergence and correlate these changes with final evaluation responses? This would directly test the proposed distinction between behavioral changes and score observability.

  • Controlled comparisons. Could the authors provide architecture- and protocol-matched experiments that hold training-data composition, optimization budget, data ordering, and evaluation procedures fixed wherever possible? Additionally, can the effects of task structure and feedback pathway be isolated rather than changing them simultaneously?

  • Sensitivity and contamination dynamics. How stable are the main observations under different learning rates, training durations, batch sizes, regularization strengths, and sampling configurations? Could the authors investigate exposure frequency and timing, particularly whether early contamination effects persist or disappear after further clean-data training? Such experiments would also clarify whether the differences across objectives reflect stable phenomena.

  • Generalizability and statistical robustness. Can the authors validate the resulting insights on held-out model/task combinations or more realistic contamination settings, such as paraphrased and near-duplicate samples? Could they also provide paired confidence intervals or statistical tests for the main between-method differences and examine whether the conclusions remain consistent under alternative aggregation schemes?

7 comments

Author Rebuttal (1/4)

W1, Q1: Can the proposed mechanisms be measured directly rather than inferred from final accuracy?

Answer: Thank you for suggesting direct mechanism measurements and controlled comparisons. The measurements show that exposure can leave parameter and visual-selection differences after its score advantage disappears. They directly test how exposure changes an update, how later training changes that effect, and whether changed behavior changes the scored answer.

To measure the local update effect, we branch from identical parameters, AdamW state and random-number state on Natural Qwen3-VL-8B / ChartQA. We test 32 exposed/clean question pairs at updates 200, 1,000 and 1,800. Two methods, two overlap conditions and three seeds give 1,152 paired diagnostics. To test persistence, Adaptive compares 50 exposed updates followed by 85 clean updates with a clean counterpart. Repeated mask sampling separately measures the final visual-policy difference.

Table 1. Direct update, continuation and policy measurements. ± is across-seed standard deviation. Brackets give 95% confidence intervals where available. n/r means not reported.

ExperimentMeasurementPPOGRPO
Natural: same-state step, 100% overlap‖dθ‖₂0.00631 ± 0.0002320.000737 ± 0.000520
Natural: same-state step, 100% overlapPaired Fixed gain (pp)+11.50 ± 3.04−0.33 ± 1.53
Adaptive: 50 exposed → 85 clean stepsFixed difference (pp)−0.5 → +0.5+6.5 → −1.5
Adaptive: 50 exposed → 85 clean stepsParameter distancen/r0.1223 → 0.1718
Adaptive: endpointExcess mask distance (pp)0.03182.6188
Adaptive: endpointFixed difference (pp) [95% CI]−4.5 [−10.5, +1.5]+3.5 [−2.5, +9.5]

At 100% overlap, PPO's local parameter difference is approximately 8.6 times GRPO's. Only one of 288 exposed GRPO groups has mixed binary rewards, identifying why relative-advantage updates are rare. In Adaptive, clean continuation lowers GRPO's Fixed effect by 8.0 points, from +6.5 to −1.5, while parameter distance grows from 0.1223 to 0.1718. The experiment therefore separates retention of a learned change from retention of a score advantage. Q3 then tests the feedback explanation by changing the reward directly.

Q2: Does exposure change visual selection despite nearly unchanged answer accuracy?

Answer: Yes. In 83.59% of paired GRPO observations, the selected masks change but the scored outcome does not. The measured policy shift remains after subtracting the selector's ordinary resampling variation.

To test this distinction, we compare 0% and 100% overlap on the same 32 Fixed and 32 Holdout questions, sampling each condition 16 times per model. Excess mask distance is cross-condition mean mask disagreement minus within-condition mean disagreement. Question-paired bootstrap gives the 95% confidence intervals.

Table 2. Visual-policy differences and scored answers on Qwen3-VL-8B / ChartQA.

Policy / score measurementPPOGRPO
Excess mask distance (pp) [95% CI]0.0318 [0.0177, 0.0460]2.6188 [2.2683, 2.9937]
Mean action-probability difference (pp)0.1554.718
Fixed score difference (pp) [95% CI]−4.5 [−10.5, +1.5]+3.5 [−2.5, +9.5]
Changed masks with unchanged scores (%)n/r83.59
Mask/score Spearman r [95% CI]n/r0.150 [−0.097, 0.397]
Mask/score relationshipSmall policy shift, score CI crosses zeroMasks can change without score changes

GRPO's excess mask distance is 2.6188 points [2.2683, 2.9937], and its mean action-probability difference is 4.718 points. Its Fixed effect is +3.5 [−2.5, +9.5], and the mask/score Spearman correlation is 0.150 [−0.097, 0.397]. The policy interval lies entirely above zero while the score interval spans zero. Together with the 83.59% unchanged-outcome rate, this directly shows what final accuracy misses: exposure can change the visual information selected without changing the evaluation's scored answer.

Author Rebuttal (2/4)

W2, Q3: Can protocol differences and the feedback pathway be controlled separately?

Answer: Yes. With the architecture fixed, changing task-score reward to equal-weight joint reward shifts GRPO's DocVQA exposure response by +20.07 percentage points. Matched Natural training also yields exposed-set gains with complete-answer feedback.

For the Natural control, PPO and GRPO share initialization, data composition/order, exposed positions, budget, four complete-answer rollouts, answer extraction and reward. Both use all rollouts. PPO has an independent value head.

Table 3. Matched Natural effects. Holdout columns give scores and changes. The macro row gives changes only.

Model / taskPPO Fixed Δ (pp)GRPO Fixed Δ (pp)PPO Holdout: 0% → 100% (Δ pp)GRPO Holdout: 0% → 100% (Δ pp)
Qwen3-VL-8B / ChartQA+9.00+2.0060.50→59.50 (−1.00)63.00→62.00 (−1.00)
Qwen3-VL-8B / DocVQA+1.02+5.4040.44→36.23 (−4.21)38.43→39.64 (+1.20)
Qwen3-VL-8B / TextVQA+4.30+6.6059.45→59.35 (−0.10)63.00→63.05 (+0.05)
Qwen2.5-VL-7B / ChartQA+3.50+4.0064.00→67.50 (+3.50)69.50→65.50 (−4.00)
Qwen2.5-VL-7B / DocVQA+9.23+11.7645.63→45.21 (−0.42)47.61→48.24 (+0.63)
Qwen2.5-VL-7B / TextVQA+4.85+8.2067.95→67.80 (−0.15)70.80→70.15 (−0.65)
Six-cell macro-average+5.32+6.33−0.40−0.63

GRPO's Fixed/Holdout effects of +6.33/−0.63 give a 6.96-point contrast. PPO's +5.32/−0.40 give 5.72 points. Both show exposure-specific gains with complete-answer training. The substantial GRPO gain connects exposure sensitivity to sampling and feedback conditions.

To isolate Adaptive feedback, we fix Qwen3-VL, the selector, 540 presentations and 135 updates. All conditions remove latency penalties. Rewards are task score, continuous complete-reference-answer probability, or their equal-weight combination.

Table 4. Feedback intervention: Fixed scores and paired exposure effects.

Task / rewardMethodFixed: 0% → 100%Fixed Δ, pp [95% CI]
ChartQA / task scorePPO35.50→36.50+1.00 [−3.50, +5.50]
ChartQA / task scoreGRPO37.50→40.50+3.00 [−3.00, +9.00]
ChartQA / answer probabilityPPO35.50→36.50+1.00 [−3.50, +5.50]
ChartQA / answer probabilityGRPO41.50→42.00+0.50 [−5.50, +6.50]
ChartQA / equal-weight jointPPO35.50→36.50+1.00 [−3.50, +5.50]
ChartQA / equal-weight jointGRPO35.50→41.50+6.00 [+0.00, +12.00]
DocVQA / task scorePPO38.64→39.34+0.70 [−3.35, +4.76]
DocVQA / task scoreGRPO60.68→46.82−13.85 [−20.30, −7.51]
DocVQA / answer probabilityPPO38.40→37.33−1.07 [−5.13, +2.99]
DocVQA / answer probabilityGRPO39.41→43.13+3.72 [−2.54, +9.98]
DocVQA / equal-weight jointPPO37.15→35.99−1.16 [−4.92, +2.56]
DocVQA / equal-weight jointGRPO41.70→47.91+6.21 [−0.78, +13.19]

On DocVQA, GRPO's Fixed effect changes from −13.85 to +3.72 and +6.21 points. Relative to task-score reward, the reported exposure-response interactions are +17.57 [8.25, 26.75] and +20.07 [10.11, 30.00], with Holm-adjusted p=0.0182 and 0.0030. These interactions compare exposure effects against each reward's own 0%-overlap baseline, not absolute accuracy across rewards.

Table 5. The same feedback intervention: all Holdout results.

Task / rewardMethodHoldout: 0% → 100%Holdout Δ, pp [95% CI]
ChartQA / task scorePPO38.50→37.50−1.00 [−7.00, +4.50]
ChartQA / task scoreGRPO40.00→39.50−0.50 [−6.50, +5.50]
ChartQA / answer probabilityPPO37.50→37.00−0.50 [−5.50, +5.00]
ChartQA / answer probabilityGRPO46.50→51.00+4.50 [−1.50, +10.50]
ChartQA / equal-weight jointPPO39.00→36.00−3.00 [−8.50, +2.50]
ChartQA / equal-weight jointGRPO41.50→46.00+4.50 [−3.00, +12.00]
DocVQA / task scorePPO29.16→28.27−0.88 [−5.34, +3.55]
DocVQA / task scoreGRPO52.01→39.42−12.59 [−18.96, −6.28]
DocVQA / answer probabilityPPO24.77→31.74+6.97 [+2.72, +11.47]
DocVQA / answer probabilityGRPO32.48→39.23+6.75 [+0.62, +12.78]
DocVQA / equal-weight jointPPO27.57→27.25−0.32 [−4.16, +3.55]
DocVQA / equal-weight jointGRPO30.66→43.21+12.56 [+7.21, +18.04]

The corresponding GRPO Holdout changes are −12.59, +6.75 and +12.56 points. Feedback therefore changes both exposed-set response and transfer. On DocVQA, task-score reward produces non-identical group rewards in 45.00% and 48.52% of updates at 0% and 100% overlap. Answer-probability and joint rewards do so in all 135 updates. Thus changing reward changes the learning signal and exposure response without changing architecture.

Author Rebuttal (3/4)

W3, Q4: How stable are the results under hyperparameter changes, repeated exposure and contamination timing?

Answer: The tests separate exposed-set gains, unseen performance and policy change. Hyperparameters affect magnitude. Repetition and timing show how policy differences accumulate while score advantages disappear.

Following your suggestion, 40 Qwen3-VL-8B / ChartQA Adaptive configurations vary one setting at a time. With 540 presentations fixed, batch size also changes the update count.

Table 6. All hyperparameter results, in pp [95% CI], comparing 100% with 0% overlap.

Varied settingPPO Fixed ΔPPO Holdout ΔGRPO Fixed ΔGRPO Holdout Δ
Learning rate 5e−5+1.0 [−2.50, 4.50]+2.5 [−1.00, 6.50]+5.5 [0.50, 10.50]+1.5 [−4.00, 7.00]
Learning rate 2e−4−3.0 [−8.50, 2.50]−0.5 [−6.50, 5.01]+4.0 [−1.50, 9.50]0.0 [−5.50, 5.50]
Batch size 2+1.0 [−3.00, 5.00]+3.0 [−2.00, 8.00]0.0 [−6.00, 6.00]0.0 [−6.00, 6.00]
Batch size 8−1.5 [−4.50, 1.50]−1.0 [−4.00, 2.00]+2.5 [−3.00, 8.00]+3.0 [−3.00, 9.00]
Rollouts 2+2.0 [−2.00, 6.50]+1.5 [−3.50, 6.50]−4.0 [−10.00, 1.50]+5.0 [−0.50, 10.50]
Rollouts 8+2.0 [−2.00, 6.00]+1.5 [−3.50, 6.50]+6.0 [1.00, 11.50]−0.5 [−6.00, 5.00]
Temperature 0.30.0 [−4.50, 4.50]+0.5 [−5.00, 6.00]+0.5 [−4.50, 6.00]0.0 [−5.00, 5.00]
Temperature 1.0+3.0 [−1.00, 7.00]+1.0 [−4.00, 6.00]+2.5 [−3.50, 8.00]−1.5 [−6.50, 3.50]
KL coefficient 0.01+1.5 [−2.50, 5.50]−2.0 [−6.50, 2.50]−4.0 [−9.00, 0.50]−5.0 [−10.50, 0.50]
KL coefficient 0.10.0 [−5.00, 5.00]−1.5 [−6.50, 3.50]+6.5 [1.00, 12.50]−3.5 [−9.00, 2.00]

PPO's Fixed effects remain between −3.0 and +3.0 points. GRPO reaches +5.5 at learning rate 5e−5, +6.0 with eight rollouts and +6.5 at KL=0.1. Their Holdout changes are +1.5, −0.5 and −3.5. The KL=0.1 setting gives a 10.0-point Fixed/Holdout contrast, separating exposed-set gains from unseen performance.

To test repetition, a separate study fixes 1,080 presentations and 270 updates. Extra exposed visits replace clean presentations.

Table 7. Repetition effects on scores and policy differences. Entries are pp [95% CI].

MethodVisits / questionFixed ΔHoldout ΔExcess mask distance
PPO1+2.0 [−2.50, 6.50]−1.5 [−5.50, 2.50]0.070 [0.05, 0.10]
PPO2+2.0 [−2.00, 6.01]0.0 [−4.50, 4.50]0.106 [0.07, 0.14]
PPO40.0 [−4.50, 5.00]−2.0 [−7.00, 3.00]0.277 [0.21, 0.35]
GRPO1−2.0 [−8.00, 4.00]+0.5 [−5.00, 6.00]0.576 [0.45, 0.72]
GRPO2+0.5 [−5.00, 6.50]+1.5 [−4.50, 7.50]0.993 [0.81, 1.20]
GRPO40.0 [−6.50, 6.00]−2.0 [−7.50, 3.00]5.294 [4.58, 6.05]

Four versus one GRPO visits raise excess mask distance from 0.576 to 5.294 points, about 9.2 times the original value, an increase of 4.718 points. Fixed changes from −2.0 to 0.0, with both intervals spanning zero. PPO mask distance also rises from 0.070 to 0.277. Repetition enlarges the policy difference without a comparable score increase.

To test timing, we expose the first or last 50 updates of a 135-update Adaptive run, then track the early-exposure run during clean continuation.

Table 8. Early and late exposure. Changes and mask distances are in pp.

MethodExposure windowFixed: 0% → 100%Fixed Δ [95% CI]Holdout ΔExcess mask distance [95% CI]
PPOEarly, steps 1–5038.5→39.0+0.5 [−3.5, 4.5]0.00.003 [−0.005, 0.011]
PPOLate, steps 86–13533.0→35.0+2.0 [−2.0, 6.0]+1.50.044 [0.026, 0.064]
GRPOEarly, steps 1–5038.0→36.5−1.5 [−6.5, 3.5]−1.01.670 [1.400, 1.964]
GRPOLate, steps 86–13535.0→40.0+5.0 [−0.5, 10.5]+1.03.119 [2.760, 3.501]

GRPO's late-exposure mask distance exceeds its early-exposure value by 1.449 points, 3.119 versus 1.670. The corresponding Fixed estimates are +5.0 and −1.5, with both intervals spanning zero. Table 9 traces how the early score advantage changes during clean training.

Table 9. Paired Fixed differences during early exposure and clean continuation, in pp.

MethodStep 50Step 75Step 100Step 135
PPO−0.5−1.0+1.0+0.5
GRPO+6.5+3.5+3.0−1.5

GRPO's Fixed advantage falls from +6.5 through +3.5 and +3.0 to −1.5 points, an 8.0-point decline over 85 clean updates. Final parameter distance remains 0.1718 and excess mask distance 1.670 [1.400, 1.964]. Thus a score advantage can disappear while exposure-induced policy differences remain. Timing and clean continuation explain how an exposure effect becomes hidden from the final score.

Reviewer_8AFBOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors study the problem of how model performance changes under different training paradigms when we add evaluation data to the training data. They develop a theoretical framework to explain this phenomenon. Then they conduct experiments using different multimodal LLMs under SFT, DPO, PPO, GRPO and RLOO, by adding evaluation data into the training cohort at different ratios. They show that SFT, DFP and PPO are sensitive such that the model performance gets a great boost when trained on the evaluation data, while GRPO and RLOO are not sensitive.

Critical strengths
  • The paper is well motivated, and the problem is important.
  • The results are useful and interesting.
Critical weaknesses
  • My main concern is the significance of this work.
    • The theory is just theory, and it is not connected to the experimental observations.
    • The authors present many results, but did not provide much explanation.
  • The details of the experiment setup are not provided, thus it is difficult to check whether the experiment design is sound or not.
Questions for authors
  • How to connect the theory to the experimental observations? How do we calculate ∇Mτ(wT0)\nabla M_{\tau}(w_{T}^{0}), Pt→TP_{t\rightarrow T} and dtd_{t} in the experiments?
  • What is the reason that GRPO and RLOO are barely affected while adding the evaluation data into the training data?
  • What are the details of the experiment setup (e.g., training configuration, data preprocessing, implementation details, etc.)?
5 comments

Author Rebuttal (1/3)

W1: What is the scientific contribution beyond showing that leakage can inflate benchmark scores?

Answer: Thank you for your questions about the contribution, its experimental support and the training setup. Our contribution is to establish exposure-specific evaluation effects under matched training protocols and explain how feedback, later training and score sensitivity shape those effects. Across all 18 matched model/task/method comparisons, the Fixed score change exceeds the Holdout change, with an average gap of +5.60 percentage points.

To test persistence under matched sampling and rewards, we compare PPO, GRPO and RLOO on Qwen3-VL-8B and InternVL3.5-8B across ChartQA, DocVQA and AI2D. Within each model/task, we match initialization, data order, exposure positions, four complete-answer samples, reward, evaluation and 2,000 updates. All sampled answers enter training, and PPO uses an independent value head.

Fixed is eligible for exposure, while Holdout is excluded from training. Each score change compares 100% with 0% overlap. The exposure-specific gap is Fixed change minus Holdout change.

Table 1. Matched Natural exposure effects. Each entry is Fixed change / Holdout change in percentage points. Results are paired point estimates from full-set evaluation.

ModelTaskPPOGRPORLOO
Qwen3-VLChartQA+9.00 / −1.00+2.00 / −1.00+2.00 / +0.50
Qwen3-VLDocVQA+1.02 / −4.21+5.40 / +1.20+6.14 / +1.97
Qwen3-VLAI2D−1.50 / −4.00+3.50 / −0.50+3.50 / −0.50
InternVL3.5ChartQA+10.00 / +4.00+6.50 / +2.50+6.50 / +0.50
InternVL3.5DocVQA+1.01 / −0.50+5.95 / −2.61+7.54 / −2.60
InternVL3.5AI2D+7.00 / −1.50+6.50 / −3.00+5.00 / −3.00

Fixed improves in 17 of 18 comparisons, averaging +4.84 points versus −0.76 on Holdout. All 18 exposure-specific gaps are positive, ranging from +1.50 to +10.14 points. The effect persists across both models, three tasks and three methods after matching.

The practical distinction is between a benchmark gain concentrated on exposed questions and improvement on unseen questions. Our mechanism tests below explain how training feedback changes this gap and how learned changes can persist after a score advantage disappears. Together, these results connect exposure-specific evaluation effects to measurable learning and behavior.

W3: What explains the different exposure responses?

Answer: The results distinguish exposed-set gains, unseen-question transfer and learned behavior. Original Natural SFT gains +21.2 points on Fixed but only +0.7 on Holdout, a 20.5-point gap. Adaptive trains a visual selector with the VLM frozen, so a small answer-score response can coexist with a changed selection policy.

To separate these explanations, the matched comparison above holds sampling and rewards common across methods. The measurements below test update differences, retention during clean training and policy changes relative to answer scores. This connects each response to its learning target and feedback.

Original Natural uses normalized binary exact-match evaluation. Original Adaptive uses task-native scores, including ChartQA numerical tolerance, DocVQA ANLS and TextVQA consensus scoring. Adaptive applies RGB and EXIF orientation correction, resizes images to 448×448 and forms 196 spatial patches. Four sequential selection stages retain 137 patches. Unselected areas are whitened before the image enters the standard VLM processor. These image patches differ from internal visual tokens.

The models are Qwen2.5-VL-7B-Instruct, Qwen3-VL-8B-Instruct, InternVL3.5-8B-HF and LLaVA-OneVision-7B. The benchmarks are AI2D, ChartQA, DocVQA, MMBench-EN, MMMU, MMStar, MMVet, MathVista, OCRBench and TextVQA.

To test persistence on Adaptive Qwen3-VL-8B / ChartQA, we follow 50 exposed GRPO updates with 85 clean updates and compare with a clean counterpart. The Fixed advantage falls from +6.5 to −1.5 points, an 8.0-point decline. Parameter distance instead rises from 0.1223 to 0.1718, and final excess mask distance remains 1.670 [1.400, 1.964] points. Parameter and policy differences remain despite the lost score advantage.

To test whether changed selections affect scored answers, a separate Adaptive endpoint experiment samples each policy 16 times on the same 32 Fixed and 32 Holdout questions. Excess mask distance subtracts within-condition disagreement from cross-condition disagreement. GRPO's policy difference is 2.6188 [2.2683, 2.9937] points, while its Fixed effect is +3.5 [−2.5, +9.5], using question-paired bootstrap 95% intervals. In 83.59% of paired observations, masks change but scores do not. This directly demonstrates the behavioral changes that final accuracy misses.

Author Rebuttal (2/3)

W2, Q1: How are the theoretical quantities measured and connected to the observed score responses?

Answer: We measure update differences directly, test their persistence through clean training and compare visual-policy changes with scored answers. These experiments distinguish a changed update, a retained policy difference and an evaluation gain.

Following your suggestion, we compare exposed-question and clean-question updates from identical parameters, AdamW state and random-number state on Natural Qwen3-VL-8B / ChartQA. dθ is the difference between updated parameters. δg measures the exposed-minus-clean difference in mean log-likelihood of the complete reference answer, including EOS, on the same exposed question. We test 32 predetermined pairs at updates 200, 1,000 and 1,800 across two methods, two overlap conditions and three seeds.

Table 2. Original Natural update diagnostics and paired final effects. Values are mean ± standard deviation across training seeds. Local measurements use 100% overlap. Final score changes compare 100% with 0% overlap in the same runs.

MethodMean ‖dθ‖₂Mean δg (nats/token)Paired Fixed Δ (pp)Paired Holdout Δ (pp)
PPO0.00631 ± 0.000232+0.0176 ± 0.0133+11.50 ± 3.04+2.00 ± 2.29
GRPO0.000737 ± 0.000520+0.0106 ± 0.0184−0.33 ± 1.53−1.50 ± 1.80

PPO's local parameter difference is approximately 8.6 times GRPO's. Its +11.50-point Fixed gain exceeds its Holdout change by 9.50 points. These measurements place the local update difference and the final exposure-specific effect in the same experimental comparison.

To distinguish update magnitude from its immediate answer-likelihood effect, we repeat same-state branching on DocVQA/GRPO with G=4 and 100% overlap. Eight question pairs at updates 500, 1,000 and 1,500 compare the original training reward with a reward without the 0.5 edit-similarity cutoff.

Table 3. Local update measurements under the DocVQA/GRPO training-reward intervention.

Training rewardMean LoRA parameter distanceReference-answer log-likelihood difference (nats/token)
Original reward G40.007818+0.005437
Adjusted reward G40.010407−0.000535

Parameter distance increases by 33.1%, while the immediate reference-answer likelihood difference moves from +0.005437 to −0.000535 nats/token. A larger exposure-induced update therefore need not favor the reference answer more strongly. Measuring both quantities tests the distinction between update magnitude and answer relevance.

Q2: Why are GRPO and RLOO weakly affected in the original Natural protocol?

Answer: The original sampling and binary rewards often give all answers in a group the same reward, producing zero relative advantages. Matched complete-answer training and reward interventions show that the weak response depends on this feedback, while exposure-specific gains persist under the matched alternatives.

To test this explanation, we measure nonzero-gradient frequency in the original runs. GRPO produces nonzero gradients in 4.8% of updates, and RLOO in 6.7% to 7.0%. To test the feedback mechanism across settings, Table 4 measures the matched complete-answer runs from Table 1. Each entry counts 600 exposed visits, with 200 questions presented three times and four answers sampled per visit. Visits with different rewards also produce nonzero immediate policy gradients in these runs.

Table 4. Exposed visits with different rewards and nonzero immediate policy gradients.

ModelTaskGRPORLOO
Qwen3-VLChartQA26/600 (4.33%)26/600 (4.33%)
Qwen3-VLDocVQA137/600 (22.83%)134/600 (22.33%)
Qwen3-VLAI2D17/600 (2.83%)17/600 (2.83%)
InternVL3.5ChartQA43/600 (7.17%)44/600 (7.33%)
InternVL3.5DocVQA103/600 (17.17%)102/600 (17.00%)
InternVL3.5AI2D30/600 (5.00%)25/600 (4.17%)

Usable feedback occurs in 2.83% to 22.83% of exposed GRPO visits and 2.83% to 22.33% of RLOO visits. Sparse feedback can still yield a large exposure-specific effect: InternVL3.5/DocVQA RLOO has mixed rewards on 17.00% of exposed visits and a +10.14-point Fixed-versus-Holdout gap. Feedback availability identifies when exposure can drive learning, while paired scores measure the evaluation effect. The corresponding intervention results are in Rebuttal (3/3).

Author Rebuttal (3/3)

Q2 (continued): How do reward and sampling changes affect the exposure response?

Answer: Adjusting rewards increases the exposure-specific gap by 1.41 to 6.00 points across four GRPO/RLOO comparisons. The gap stays positive under G=8 sampling.

To test feedback through intervention, we change the training reward on Qwen3-VL-8B / ChartQA and DocVQA. Initialization, data order, exposure positions, four complete-answer samples and 2,000 updates are matched. ChartQA uses relative numerical accuracy with revised parsing and nonnumeric normalization. DocVQA removes the 0.5 edit-similarity cutoff. Both take the maximum across references. The final task evaluator stays unchanged.

Table 5. G=4 reward intervention, original → adjusted. The gap is Fixed Δ minus Holdout Δ. Scores are paired estimates in pp. Percentages count sampled answers with nonzero advantage across all questions in the 100%-overlap run.

TaskMethodNonzero-advantage answers (%)Fixed Δ (pp)Holdout Δ (pp)Exposure-specific gap (pp)
ChartQAGRPO7.350 → 23.925+2.000 → +7.000−1.000 → −2.000+3.000 → +9.000
ChartQARLOO7.250 → 23.350+2.000 → +5.000+0.500 → +0.500+1.500 → +4.500
DocVQAGRPO20.812 → 40.150+5.403 → +6.146+1.205 → +0.536+4.198 → +5.609
DocVQARLOO21.950 → 41.837+6.143 → +6.337+1.974 → +0.349+4.169 → +5.989

Changing the reward increases the exposure-specific gap by 1.41 to 6.00 points in all four comparisons and the nonzero-advantage frequency by 16.10 to 19.89 percentage points. ChartQA/GRPO's Fixed gain rises from +2.00 to +7.00 points, and its gap triples from +3.00 to +9.00. With evaluation unchanged, this intervention links stronger training feedback to stronger exposure-specific effects.

With the original reward, increasing G to 8 on ChartQA/GRPO raises Fixed gain by 0.50 points and Holdout change by 1.50, moving the gap from +3.00 to +2.00. The gap remains positive in all 12 tested conditions.

W4, Q3: What are the exact training, preprocessing and evaluation configurations?

Answer: Natural trains attention-projection LoRA parameters. Adaptive trains a separate visual selector while freezing the entire VLM. Tables 6 and 7 specify the original configurations and data schedules. The matched Natural controls use common complete-answer sampling and feedback across the compared methods.

Table 6. Original training configurations.

ConfigurationOriginal NaturalOriginal Adaptive
Trainable parametersAttention-projection LoRA, frozen backbonePolicy MLP 110→64→1, frozen VLM, PPO also has value MLP
Parameter typeLoRA rank=64, alpha=128, dropout=0Policy/value MLP float32, frozen VLM bfloat16
Optimizer and learning rateAdamW, 1e−5, weight decay=0.01Adam, 1e−4, weight decay=0
Update budgetOne question/update, 2,000 updatesFour questions/update, 540 presentations and 135 updates
Rollout groupGRPO/RLOO use G=4GRPO/RLOO use G=4
Loss constantsPPO/GRPO clipping=0.2, DPO β=0.1PPO/GRPO clipping=0.2, DPO β=0.1
Explicit KL in main lossNoneNone
Gradient clippingGlobal L2 norm ≤1.0None in the original main loop
EvaluationFull Fixed and Holdout after 2,000 updatesFull Fixed and Holdout after 135 updates

Table 7. Data splits and exposure schedules. The last two columns list 0% / 50% / 100% overlap in that order.

Dataset groupBase / Fixed / HoldoutNatural poolAdaptive exposed presentations
Eight standard datasets600 / 200 / 200600 / 700 / 8000 / 100 / 200
MMMU540 / 180 / 180540 / 630 / 7200 / 90 / 180
MMVet132 / 43 / 43132 / 154 / 1750 / 66 / 129

For the eight standard datasets, Natural's pool grows from 600 to 700 or 800 questions within the same 2,000-update budget. Adaptive replaces clean presentations with 100 or 200 exposed presentations within its fixed budget of 540. Overlap counts distinct Fixed question IDs, and Holdout IDs are excluded from training.

Original Natural SFT supervises reference-answer tokens plus EOS. Original PPO uses teacher-forced target-token rewards and a policy-derived value signal. Original DPO uses first-target-token policy scores with a stop-gradient offset. Most original GRPO/RLOO configurations use four first-answer-token samples and binary rewards. The Qwen2.5-VL / GRPO / OCRBench exception samples four answers of up to 16 tokens.

The matched Natural controls use complete-answer samples, common answer extraction and rewards, a common evaluator within each task, and all sampled rollouts in the objectives. PPO has an independent value head. Original Adaptive reward combines answer performance, deviation from the patch-retention target, latency and shaping terms. These distinguish original runs from matched controls.

Reviewer_5NGHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper presents a study of how evaluation data leakage impacts downstream evaluation across various post-training objectives and settings in Vision Language Models. The study uses four VLMs and compares SFT, DPO, PPO, GRPO, RLOO across several datasets on three seeds. The evaluation data is deliberately leaked and the final accuracies are measured. The two experimental settings tested are natural and adaptive visual token setting. The key takeaway is that the training setting matters more than the objective - although SFT seems to be vulernable to leakage in natural setting, it doesnt show a large leakage effect in the adaptive setting. A theoretical framework is provided to understand this phenomenon.

Critical strengths
  1. The problem of data leakage and the influence of training objectives and settings is very timely and important. The overall formulation of the problem is clean and sound.

  2. The evaluation tests several models and training objectives across multiple seeds. I think the results are useful for practioners, for e.g. smaller leakage in outcome-reward RL methods (although its somewhat expected because the model samples from its own distribution) compared to SFT.

  3. Overall I think the empirical contributions are reasonable, transparent and valuable to the community.

Critical weaknesses
  1. The writing can be improve substantially. Especially focus on clarifying the experimental adaptive experimental setting (see questions below). I was unable to fine the training/implementation details in the paper or the appendix (L 56-57). Missing details make this paper hard to read and interpret the results.

1a. Most importantly, implementation details and training settings can significantly change the results. There should be a sensitivity analysis on the hyperparameters chosen to improve the quality of the paper. For e.g. the most relevant factor is how many times each leaked item is seen (epochs or training steps), since memorization depends strongly on repetition. For RL, add the KL coefficient and the GRPO group size.

  1. Can any term in the theory be measured? For example the update difference dtd_t or its alignment with the score gradient to show which link explains the small GRPO/RLOO response? I think this analysis would make the paper significantly stronger in terms of its contributions.

  2. Some runs have very high seed variance (in datasets like OCRBench, MMStar). Examples: InternVL3.5 natural DPO on MMStar at 50% overlap is 33$\pm$46.8 (Table 2); LLaVA adaptive DPO on OCRBench is 56.2$\pm$44.0 (Table 7); InternVL3.5 natural GRPO on DocVQA at 100% overlap collapses to 32.$\pm$27.9 (Table 2). Please clarify if the conclusions would hold without the unstable runs.

Questions for authors
  1. What is the selection policy in the adaptive setting (a separate small network or part of the VLM?)
  2. How many tokens does the adaptive setting select?
  3. It is unclear if the VLM is update or the selection policy or both?
  4. In the SFT experiments, is the entire trajectory leaked to the model or just the final answer? Presumably, the model is trained on QA pairs but please clarify if any Chain of Thought was leaked? What would happen if more information than just the final answer is leaked?
  5. Does the overlap replace samples or adds more samples? Which samples are replaced?
7 comments

Author Rebuttal (1/3)

W1, Q1–Q3: What is the Adaptive selection policy, how many regions does it retain, and which parameters are updated?

Answer: Thank you for your careful review and questions about implementation, sensitivity and mechanism measurements. Adaptive trains a separate visual-selection policy that retains 137 of 196 spatial patches while the entire VLM remains frozen. These patches are image regions, not the internal visual tokens produced by the VLM.

To clarify the implementation, Table 1 specifies the selector, supervision and budget. Unselected regions are whitened, and the reconstructed image enters the standard VLM image processor.

Table 1. Adaptive architecture, supervision and main-experiment budget.

ComponentOriginal Adaptive implementation
VLMEntire model frozen, no update to text generation parameters
SelectorPolicy MLP 110→64→1 using local visual, question, and selection-state features
PPO-specific moduleSeparate value MLP
Image preparationRGB and EXIF orientation correction, 448×448 image, 196 spatial patches
Selection procedureFour sequential decisions, retain 137 spatial patches
Unselected areaWhiten in place, recompose before the standard VLM processor
SFTSupervise the selector with high-reward selection trajectories
DPOTrain the selector from high-/low-reward trajectory preferences
PPO, GRPO, RLOOUse trajectory-level rewards and their respective advantages
Budget per main-experiment cell540 question presentations and 135 policy updates
RolloutsFour selection trajectories per question, the original PPO loss uses only the first

All five objectives train the visual selector. Holding the answer generator fixed lets us measure whether exposure changes what the model sees and whether those selections affect its scored answers.

W1a: How sensitive are the results to group size, KL regularization and repeated exposure?

Answer: Group size and KL regularization change the exposure gain. Repeated exposure can strengthen the visual-policy difference without a comparable score increase, separating learning from score inflation.

Following your suggestion, Table 2 tests sampling and KL regularization in the matched complete-answer Natural control on Qwen3-VL-8B / ChartQA. All conditions use 2,000 single-question updates, and PPO has an independent value head. The reference uses G=4 complete-answer samples, KL=0 and three visits per exposed question. Fixed contains questions eligible for exposure, while Holdout is excluded from training. Each Δ compares 100% with 0% overlap.

Table 2. Matched Natural sampling and KL sensitivity. pp denotes percentage points. A dash denotes an unreported result.

ConfigurationPPO Fixed Δ (pp)GRPO Fixed Δ (pp)
Reference: G=4, KL=0, three exposures+9.0+2.0
Group size G=2-+0.5
Group size G=8-+2.5
KL coefficient 0.1+3.5+2.0

Increasing GRPO's group size from 2 to 8 raises its Fixed gain from +0.5 to +2.5 points. Increasing PPO's KL coefficient from 0 to 0.1 reduces its gain from +9.0 to +3.5 points, a 5.5-point reduction. Exposure sensitivity therefore changes with sampling and regularization within the same training setting.

To test repetition, we fix 1,080 presentations and 270 Adaptive updates on Qwen3-VL-8B / ChartQA. Extra exposed visits replace clean presentations. Alongside scores, we measure excess mask distance, which subtracts within-condition mask disagreement from cross-condition disagreement to account for ordinary resampling variation.

Table 3. Adaptive repetition sensitivity at a fixed presentation budget. Values are pp [95% confidence interval].

MethodVisits per exposed questionFixed ΔHoldout ΔExcess mask distance
PPO1+2.0 [−2.50, +6.50]−1.5 [−5.50, +2.50]0.070 [0.05, 0.10]
PPO2+2.0 [−2.00, +6.01]0.0 [−4.50, +4.50]0.106 [0.07, 0.14]
PPO40.0 [−4.50, +5.00]−2.0 [−7.00, +3.00]0.277 [0.21, 0.35]
GRPO1−2.0 [−8.00, +4.00]+0.5 [−5.00, +6.00]0.576 [0.45, 0.72]
GRPO2+0.5 [−5.00, +6.50]+1.5 [−4.50, +7.50]0.993 [0.81, 1.20]
GRPO40.0 [−6.50, +6.00]−2.0 [−7.50, +3.00]5.294 [4.58, 6.05]

Increasing GRPO exposure from one to four visits raises excess mask distance from 0.576 to 5.294 points, approximately 9.2 times the original value. The corresponding Fixed effects are −2.0 and 0.0 points. PPO's mask distance also rises from 0.070 to 0.277. Repetition strengthens the learned selection difference even when final scores show little response.

Author Rebuttal (2/3)

W2: Can the update difference and its evaluation-relevant effect be measured?

Answer: We directly measure the parameter difference between exposed and clean updates and the resulting reference-answer likelihood difference. These diagnostics identify sparse relative-advantage feedback in the original Natural GRPO implementation: 287 of 288 exposed groups have identical rewards and zero relative advantages.

Following your suggestion, we start both updates from identical model parameters, AdamW state and random-number state on Qwen3-VL-8B / ChartQA. dθ is the difference between the updated parameters. δg measures the exposed-minus-clean difference in mean log-likelihood of the complete reference answer, including EOS, on the same exposed question. We test 32 predetermined question pairs at updates 200, 1,000 and 1,800.

Table 4. Same-state update measurements and final Natural accuracy. Values are mean ± standard deviation across training seeds.

MethodOverlapFixed (%)Holdout (%)Mean ‖dθ‖₂Mean δg (nats/token)
PPO0%55.83 ± 1.6147.00 ± 0.870.00864 ± 0.000234+0.0713 ± 0.0165
PPO100%67.33 ± 1.4449.00 ± 1.730.00631 ± 0.000232+0.0176 ± 0.0133
GRPO0%57.83 ± 1.8948.50 ± 0.870.000428 ± 0.000399+0.0000796 ± 0.0000864
GRPO100%57.50 ± 0.8747.00 ± 2.500.000737 ± 0.000520+0.0106 ± 0.0184

At 100% overlap, PPO's parameter difference is approximately 8.6 times GRPO's. Their paired Fixed gains are +11.50 ± 3.04 and −0.33 ± 1.53 points. The single mixed-reward GRPO diagnostic group produces a +3.0619 nats/token likelihood change. Original RLOO runs likewise produce nonzero gradients in only 6.7% to 7.0% of updates. These measurements identify a specific bottleneck: exposure often provides no relative-advantage learning signal under the original sampling and rewards.

W3: Do the conclusions hold after excluding high-variance runs?

Answer: Yes. Excluding the identified high-variance combinations preserves the original Natural contrast and the small Adaptive macro-average response. Following your suggestion, we remove complete model/task combinations, retain all seeds of the remaining combinations and recompute the paired exposure effects.

Table 5. Fixed exposure effects after the specified exclusions, in percentage points.

Setting / exclusionMethod or aggregateFixed Δ (pp)
Natural: exclude InternVL3.5 / MMStarDPO+14.38
Natural: exclude InternVL3.5 / DocVQAGRPO+4.20
Natural: 38 common stable model/task combinationsSFT+21.13
Natural: 38 common stable model/task combinationsDPO+14.62
Natural: 38 common stable model/task combinationsPPO+14.48
Natural: 38 common stable model/task combinationsGRPO+4.36
Natural: 38 common stable model/task combinationsRLOO+3.74
Adaptive: exclude LLaVA / OCRBenchRange across all five methods−0.73 to +1.33

Natural SFT retains a +21.13-point Fixed gain on the 38 common stable combinations, while all five Adaptive effects remain between −0.73 and +1.33 points. The high-variance LLaVA / OCRBench / DPO case has paired exposure differences of only 0, 0 and −1 points across seeds. The reported contrast survives these exclusions, separating absolute-score variance from the paired exposure pattern.

Author Rebuttal (3/3)

Q4: Does SFT expose final answers or reasoning traces, and what happens when explanations are added?

Answer: Original Natural SFT exposes short reference answers plus EOS, without reasoning traces. Adding teacher explanations reduces the Fixed exposure gain by 11.0 to 25.0 points across the four tested student/task comparisons. Adaptive SFT instead supervises patch-selection actions while keeping the answer generator frozen.

To test the richer supervision you suggested, we use a fixed Qwen3-VL-32B teacher with two students on ChartQA and MathVista. The 24 configurations cover two students, two tasks, two supervision targets and 0%, 50% and 100% overlap. Initialization, exposed-example positions and presentation budget are matched. Explanation and answer losses are normalized separately, testing the complete supervision change, including content, target length and answer context.

Table 6. Fixed exposure gains under the two supervision targets.

Student / taskAnswer-only Δ (pp)Explanation + answer Δ (pp)
Qwen3-VL-8B / ChartQA+17.5+2.5
Qwen3-VL-8B / MathVista+22.5+4.0
Qwen2.5-VL-7B / ChartQA+17.5+6.5
Qwen2.5-VL-7B / MathVista+26.0+1.0

Adding explanations reduces the exposure gain by 15.0, 18.5, 11.0 and 25.0 points, respectively. To check the same contrast with longer decoding, the Qwen3-VL-8B / MathVista comparison allows 1,024 tokens and gives +22.5 versus −0.5 points, with an explanation-minus-answer-only interaction of −23.0 [−31.5, −14.5]. Richer leaked supervision yields a smaller exposure gain in these comparisons, demonstrating that the supervision target matters alongside the amount of overlap.

Q5: Does overlap append or replace training examples, and which examples are affected?

Answer: Natural appends exposed Fixed questions to the base training pool while retaining 2,000 updates. Adaptive replaces clean question presentations with exposed Fixed questions within its fixed presentation budget. Overlap counts the fraction of distinct Fixed questions eligible for training, not the fraction of updates. Holdout question IDs are excluded throughout.

Table 7. Data sizes and exposure counts. The last three columns list 0% / 50% / 100% overlap in that order.

Dataset groupBase / Fixed / HoldoutNatural poolNatural exposed visitsAdaptive exposed presentations
Eight standard benchmarks600 / 200 / 200600 / 700 / 8000 / 300 / 6000 / 100 / 200
MMMU540 / 180 / 180540 / 630 / 7200 / 360 / 5400 / 90 / 180
MMVet132 / 43 / 43132 / 154 / 1750 / 286 / 5160 / 66 / 129

For the eight standard benchmarks, 50% and 100% overlap expose 100 and 200 distinct Fixed questions. Natural schedules three visits per exposed question, giving 300 and 600 exposed visits. Adaptive uses 100 or 200 exposed presentations within 540 total presentations and 135 updates. Multiple rollouts of one question count as one presentation. The original Qwen2.5-VL-7B / GRPO / OCRBench Natural exception has 0 / 250 / 504 exposed visits.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.