acceptodds
Under review as a conference paper at ICLR 2027

Exploring Unsupervised Multimodal Reward Modeling

Abstract

Vision-language reward models are trained on annotated preference pairs and ranked by benchmark accuracy; the pairs are costly and noisy to curate from human annotations, and the number is trusted more than it is understood. We present CRB, a reward model that uses no human preference label: a linear head read out of an intermediate layer of a frozen Qwen3-VL backbone, trained to prefer a web caption over a same-image corruption of it. The readout layer is the decisive choice, worth up to 5.8 points over the last layer. With it, one recipe transferred without re-tuning matches GPT-4o at 2B and 8B on all three benchmarks we test, by 0.5, 6.6 and 2.4 points on Multimodal RewardBench, VL-RewardBench and MMRB2 reasoning, from a few thousand trained parameters at about 0.1% of fine-tuning compute. At an equal pair budget, corruptions teach a frozen head 3 to 5 points more than curated RLAIF-V pairs. We then show what the number does not certify. The objective is convex over frozen features, so no training-side change moves its plateau. In Best-of-N the probe beats majority vote by 7 to 11 points only where the generator is confidently wrong. Two probes with the same score, trained on halves of the data split by whether CLIP can see the corruption, improve by 6.4 points on hallucination judging. What did improve deployment was correctness labels minted by cheap verifiers from the policy's own candidates: one bootstrap turn reduces vision hallucination by 5.1 CHAIR points and lifts image-generation pass rate by 5.4. With the combination of corruption, read-out and bootstraping, out method makes unsupervised MM reward learning cheaper and more efficient.

Primary area:
reinforcement learning
Keywords:
Reward Model, Unsupervised Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Th6jOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper trains a linear reward head on intermediate features of a frozen vision-language model using web captions paired with synthetic perturbations. Human preference labels are excluded from head training, although a labeled benchmark selects the probe layer. The study evaluates multiple backbone sizes and reward benchmarks, analyzes the effect of the negative sampler, and tests Best-of-N selection and one round of policy updating. Its main empirical finding is that preference-benchmark accuracy can disagree with downstream selection quality: perturbation-trained probes score well on benchmarks, whereas adaptation to verifier-labeled policy outputs improves caption selection despite reducing some benchmark scores.

Critical strengths
  1. A frozen backbone and a small linear head provide a practical way to exploit synthetic comparisons. The intermediate-layer ablations and the second backbone family strengthen the result beyond a single checkpoint. The 32B probe reaches 72.4% on VL-RewardBench versus the quoted 65.8% for GPT-4o (Table 1), while the paper explicitly treats the layer-selection benchmark as development data.
  2. The CLIP-split probes and the Best-of-N controls show why similar benchmark scores need not imply similar behavior. Including majority vote, log-likelihood scoring, random selection, and verifier-based adaptation makes the negative results useful. The reported held-out COCO-2 improvement is also stronger evidence than evaluating only the tasks used to build the adaptation pairs (Table 7).
Critical weaknesses
  1. The density-ratio interpretation is applied beyond the conditions established by the theory. Proposition 1 applies to independent cross-entry comparisons with common support. The actual objective includes dependent diagonal terms, and Appendix I explicitly states that the implemented sampler does not satisfy the support assumption. More importantly, Section 3.4 invokes Proposition 1 to claim that verifier-ordered policy pairs learn the ratio needed for Best-of-N, although Appendix I.1 notes that judge-ordered pairs are governed by their joint comparison law, not their marginals alone. The fixed-penalty convexity argument also needs separation from AdamW’s changing preconditioner (Appendix J). These qualifications do not invalidate the experiments, but they weaken a central explanation of why the method should work.
  2. CHAIR supplies both adaptation labels and captioning evaluation, so the gain establishes improvement on that proxy without showing preserved informativeness or broader factuality; Appendix A acknowledges this limitation. The original perturbation probes do not improve captioning over greedy decoding, and their hard-QA gains are not clearly better than the training-free log-likelihood control. The actor experiments likewise do not establish a selector-specific benefit over random pairing (Appendix G). Thus, the paper offers convincing diagnostic observations, but less evidence for a broadly useful reward-learning and self-improvement procedure.
Questions for authors
  1. Which conclusions about the trained head remain valid for the full dependent-pair objective and the implemented AdamW optimizer? Can the application of Proposition 1 to verifier-ordered pairs be justified, or explicitly restricted to a heuristic interpretation?
  2. Are adaptation training pairs and all reported downstream evaluations disjoint at the image, prompt, and candidate levels? Please clarify this separately for the main Table 7 cells and the leave-one-task-out result, rather than only for the original perturbation corpus.
  3. Does the caption-selection improvement persist under an independent factuality assessment that also checks retained information? A small, clearly held-out evaluation would help distinguish general improvement from learning the CHAIR criterion.
4 comments
AuthorsOpenReview ↗

Response Q1/W1

We thank the reviewer for reading Appendices I and J closely, and the insightful suggestions on the theory section. Below we address each question in turn/

1. Proposition 1 for verifier-ordered pairs (Q1, W1)

Q1. "… Can the application of Proposition 1 to verifier-ordered pairs be justified, or explicitly restricted to a heuristic interpretation?"

We restrict it, and concede all three points. Dependent diagonal terms and the support violation are why App. I analyses the linear head through the fixed-penalty problem (Corollary 1) and not through Eq. (ratio). We changed the text accordingly:

  • §3.4 now cites Corollary 1 for verifier-ordered pairs.
  • The ratio reading of judge-ordered pairs is stated as heuristic, because winner and loser are dependent after labelling (App. I.1).
  • The common-support caveat moves into the main text.

2. What remains valid under the dependent-pair objective and AdamW (Q1, W1)

Q1. "Which conclusions about the trained head remain valid for the full dependent-pair objective and the implemented AdamW optimizer?"

The surviving result concerns regularized linear pairwise learning under an arbitrary fixed comparison law. Let dd be the difference between the frozen feature vectors of the preferred and non-preferred entries, and let μ\mu be the distribution of dd. This law can include dependent pairs, diagonal and cross-entry comparisons, or verifier-ordered pairs; it need not factor into independent winner and loser distributions. The same formulation covers a finite dataset by taking μ\mu to be the dataset's weighted empirical distribution.

For a linear score, the objective is

\qquad \ell(z)=\log(1+e^{-z}).$$ With $\lambda>0$, fixed $D\succ0$, and finite feature moments, this objective is strongly convex and has a unique minimiser. Dependence and lack of common support do not invalidate this conclusion: convexity holds for each individual comparison and is preserved by averaging. Its first-order condition is $$\lambda D w^\star=\beta\,\mathbb{E}_{d\sim\mu}[\sigma(-\beta w^{\star\top}d)\,d],$$ which is the characterization in Corollary 1. Thus **the full comparison law and the regulariser determine the optimum**. More generally, convexity holds for any convex loss of a linear pairwise margin; a strongly convex regulariser supplies uniqueness. The displayed first-order condition is specific to the logistic loss and quadratic penalty. **Three conclusions must remain separate.** First, this optimization result characterizes the stated objective without requiring a density-ratio interpretation. Second, identifying its optimum with a density ratio requires the additional assumptions of Proposition 1; the general pairwise result alone gives neither that identification nor a guarantee of downstream Best-of-N quality. Third, uniqueness of an optimum does not establish that a particular training algorithm reaches it. Gradient descent converges under suitable smoothness and step-size conditions, but AdamW's adaptive preconditioning and decoupled weight decay do not generally correspond to optimization of this same fixed-penalty objective. For the implemented AdamW head, ranking stability and selection quality therefore remain empirical findings. We tested the distinction in a controlled cached-feature experiment (App. E.1): 2B and 8B heads, 7,200 training pairs, and a constant learning rate rather than the full trainer's cosine schedule. The measurements below describe this experiment, not a direct convergence audit of the paper's full training runs: | Quantity | Value | |---|---| | Agreement on held-out in-distribution orderings, across seeds / LR / steps / batch / pool size | 97 to 100% | | Cosine to the seed-0 AdamW head | 0.81 to 1.00 (2B), 0.78 to 1.00 (8B) | | Regularised gradient norm at the final iterate | 0.14 to 0.64 (2B), 0.03 to 0.14 (8B) | | Cosine to the unique minimiser (L-BFGS = plain GD from two inits, cosine 1.000) | 0.69 to 0.82 | | Benchmark pairs where AdamW heads and minimiser disagree | 13 to 20% (2B), 11 to 13% (8B) | | Benchmark accuracy, AdamW heads vs. minimiser | 8B .656 vs .652; 2B .625 vs .561 | In this experiment, **97–100% agreement on held-out in-distribution rankings coexists with substantial disagreement with the fixed-penalty minimiser**. Thus ranking stability does not establish convergence to that minimiser. We retain ranking stability as an empirical observation and remove convexity as its explanation; the theoretical characterization applies only to the stated fixed-penalty problem. Reviewer NqgQ raised the AdamW point as well.
AuthorsOpenReview ↗

Response Q2, Q3

3. Disjointness for Table 7 and the leave-one-task-out result (Q2)

Q2. "Are adaptation training pairs and all reported downstream evaluations disjoint at the image, prompt, and candidate levels? Please clarify this separately for the main Table 7 cells and the leave-one-task-out result …"

App. F.1 now states it for every Table 7 cell, the leave-one-task-out arms, the image arm and the DPO arms.

  • Table 7, in-pool cells: share prompts and candidates with the verified head's training pairs.
  • Leave-one-task-out (held-out COCO-2 column): disjoint; −4.1 ± 2.9.
  • New, fresh candidate pools: none of the 8,000 candidates per pool appears in a training pair; −6.1 ± 0.9 (3 seeds), every seed's CI excluding zero (held-out head −7.0 ± 0.6, 3 seeds, on fresh COCO-2).
  • New, 500 COCO images in no prompt set: −7.6 ± 0.5 (3 seeds), while the perturbation probes give −2.5 ± 3.2 (2B) and +0.1 (8B).
  • DPO cells: the corpora draw on all four tasks' prompts, COCO-2 included (App. G), so no DPO cell is prompt-disjoint; the table marks them in-prompt.

The two new disjoint evaluations in detail:

Setup: (a) fresh pools: 16 new samples per prompt (new sampling seed) on the 500 COCO and 500 COCO-2 prompts, 2B and 8B generators; none of the 8,000 candidates per pool occurs in any training pair. (b) COCO-3: 500 new COCO val2014 images used by no prompt set. Verified heads retrained with 3 seeds; probes and log-likelihood are single runs. Δ in CHAIRs and coverage points vs. each pool's own greedy decode (lower CHAIRs and higher coverage are better).

(a) Fresh candidate pools (pooled over 4 pools, 2,000 prompt-cells)

SelectorCHAIRsΔ CHAIRsΔ coverage
Greedy.387
Random.394+0.7 ± 2.2−0.2 ± 0.8
Log-likelihood 2B (training-free).359−2.8 ± 2.2−0.4 ± 0.8
Perturbation probe 2B.370−1.7 ± 2.2−1.5 ± 0.8
Perturbation probe 8B.393+0.6 ± 2.1+1.4 ± 0.8
Verified head (3 seeds).326 ± .009−6.1 ± 0.9−4.2 ± 0.6
Held-out head, COCO-2 pools only (3 seeds).318 ± .006−7.0 ± 0.6−4.2 ± 0.3

(b) COCO-3: new images and prompts (pooled over 2B and 8B, 1,000 prompt-cells)

SelectorCHAIRsΔ CHAIRsΔ coverage
Greedy.399
Random.406+0.7 ± 3.2−0.7 ± 1.1
Log-likelihood 2B (training-free).393−0.6 ± 3.0−1.3 ± 1.1
Perturbation probe 2B.374−2.5 ± 3.2−1.7 ± 1.1
Perturbation probe 8B.400+0.1 ± 2.9+0.1 ± 1.1
Verified head (3 seeds).323 ± .005−7.6 ± 0.5−4.3 ± 0.9

Key takeaways

  1. The gain is not an in-pool artifact. On candidates and images the head never saw, the verified head still cuts CHAIRs by 6.1 and 7.6 points (in-pool Table 7: 5.1), and every seed's CI excludes zero. Each pool is compared with its own greedy decode, so the three numbers are not directly comparable to each other.
  2. The coverage cost travels with it (−4.2 to −4.3 points everywhere), matching point 4.
  3. The perturbation probes stay at greedy level, as in the paper, and are no better than the training-free log-likelihood judge.

4. Independent factuality that also checks retained information (Q3)

Q3. "Does the caption-selection improvement persist under an independent factuality assessment that also checks retained information?"

Yes under GPT-6-Astra's independent factuality assessment, with an informativeness tradeoff. We evaluated the head trained without COCO-2 pairs on 200 held-out COCO-2 prompt instances per generator. The evaluation images, prompt instances and candidates are excluded from that head's adaptation training. Astra sees the image and the selected and greedy captions, without selector identities or CHAIR scores, and answers separately which caption has fewer false statements, which conveys more true, specific information, and which is better overall.

Each comparison is judged in both A/B orders; the table uses only position-consistent verdicts. Net preference is win minus loss in percentage points, with ties included in the denominator; brackets are 95% bootstrap confidence intervals. After filtered calls, 199 comparisons per generator have both orders available; informativeness judgments are order-inconsistent in 13% / 19% of comparisons at 2B / 8B, and factuality judgments in 18% at both sizes.

Held-out head vs. greedy2B generator8B generator
Factuality+15.9 [ +1.8, +30.5 ]+15.9 [ +0.6, +30.5 ]
Informativeness−28.9 [−42.8, −14.5]−16.1 [−30.5, −0.6]
Overall quality+4.7 [−11.6, +19.8]+6.7 [−7.9, +21.3]
AuthorsOpenReview ↗

Response Q3 cont. + W2

The factuality gain therefore extends beyond CHAIR under this independent judge at both generator sizes. Astra assesses false statements against the image rather than COCO-object mentions alone. As a control, a random candidate is judged worse than greedy overall: −15.3 [−29.9, −1.7] at 2B and −17.4 [−32.6, −2.3] at 8B. The selected captions have positive overall-quality point estimates.

The retained-information check also identifies the cost: Astra judges the selected captions less informative at both sizes. These net preferences measure how often one caption is preferred, not the percentage of information lost. Separately, the object-coverage analysis finds 0.25 fewer ground-truth COCO categories mentioned per caption and a coverage decrease of 4.9 ± 0.9 points, despite approximately unchanged length. We therefore claim held-out factuality improvement under Astra with reduced informativeness, rather than preserved information or an established overall-quality gain. The independent assessment and coverage measurements are reported in App. F.2.

5. Perturbation probes, hard QA and the actor experiments (W2)

W2. "The original perturbation probes do not improve captioning over greedy decoding, and their hard-QA gains are not clearly better than the training-free log-likelihood control. The actor experiments likewise do not establish a selector-specific benefit over random pairing (Appendix G)."

Agreed on all three, and §4 reports them:

  • On hard QA, probe-2B minus the training-free log-likelihood judge is +1.6 ± 3.0 points (8B generator), and every mined-negative probe stays at greedy level on captioning.
  • In DPO, RM-picked minus random pairs is −.022 ± .033 CHAIRs; the actor side is marked open.

The paper's claim about the perturbation probes and actor experiments remains diagnostic: benchmark rank does not predict selection gain, and selector-specific actor improvement is not established. For the verifier-adapted selector, point 4 adds held-out factuality improvement under Astra, with reduced informativeness and no established overall-quality gain.

Thank you again for your insightful and constructive feedback; it really helps strengthen the work! As the discussion period ends November 18, we would be grateful to know whether our responses address your concerns, and we are happy to clarify any remaining point.

Reviewer_fZmdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a human-preference-label-free recipe for training multimodal reward heads: a vision-language backbone is frozen, and a small linear head on an intermediate layer is trained to rank web captions above same-image perturbations. The authors treat the negative sampler as the source of supervision and connect the in-batch Bradley–Terry objective to a regularized approximation of a density ratio.

Across several Qwen3-VL sizes and a second backbone family, the probes are competitive on Multimodal RewardBench, VL-RewardBench, and MMRB2. More importantly, the paper shows that similar preference-benchmark scores can conceal different downstream behavior: perturbation-trained probes do not consistently improve Best-of-N selection, whereas retraining on verifier-labeled policy outputs improves caption selection despite lower benchmark scores. The paper also studies image-generation selection and one DPO update.

Its main contribution is both a simple reward-modeling recipe and an empirical argument that reward models should be evaluated against their intended deployment, not only by benchmark rank.

Critical strengths

(1) The paper makes explicit that the negative sampler determines what the reward head is trained to distinguish, rather than treating label-free pairs as generic supervision. The density-ratio interpretation, together with the experiment in which probes with similar benchmark scores rank hallucinations differently, provides a consequential lesson for reward-model evaluation and selection.

(2) The study covers multiple Qwen3-VL scales, a second backbone family, three reward benchmarks, matched-pair comparisons, negative-sampler ablations, and downstream selection for both captioning and image generation. The paper also reports limitations and contamination checks, making the evidence easier to interpret and the work potentially reproducible.

Critical weaknesses

(1) CHAIR supplies both the captioning labels and the reported captioning metric; in the image-generation experiments, the OWLv2 verifier appears to serve both as the source of labels and as the evaluator. The held-out COCO-2 prompts reduce example overlap, but they do not remove this metric coupling.

The results therefore establish improvements on the verifier's target attributes, more directly than they establish general factuality, informativeness, or semantic quality. This matters because the strongest practical result depends on verifier-minted training pairs.

(2) The Best-of-N cells are single-seed, the image-generation study uses 160 hard prompts, and the verifier-based adaptation is evaluated for only one round. In addition, the DPO control with random pairs achieves a similar reduction in CHAIR to the verifier-pair condition, so the policy-update experiment does not yet show a clear selector-specific benefit.

The layer is also selected using a labeled Multimodal RewardBench development split; thus, the base head avoids human preference labels during head training, but the full pipeline is not entirely label-free. This distinction should be kept explicit when discussing the scope of the method.

Questions for authors
  1. Can you evaluate the verifier-trained captioning and image-generation selectors with an independent metric or human adjudication, and report a measure of caption informativeness or coverage? This would help determine whether the gains extend beyond the specific properties measured by CHAIR and OWLv2.

  2. Can you report multi-seed results for the main Best-of-N experiments, especially image generation, and a statistical comparison of reward-model-selected versus random pairs in the DPO experiment? Given the single-seed results and the similar DPO gains from random pairs, these checks would materially strengthen the claim of robust downstream utility.

  3. Could you clarify the intended scope of “unsupervised” and provide a sensitivity check using a fixed probe layer or a separate layer-selection split? The current setup selects layer depth using labeled Multimodal RewardBench data, while the deployment-improving bootstrap uses verifier labels. Separating these roles would make the supervision requirements and the generality of the main claim easier to assess.

4 comments
AuthorsOpenReview ↗

Response Q1/W1

We thank the reviewer for the precise questions. All three asked for experiments, and we ran them. Below we address each question in turn

1. Independent metric and caption coverage (Q1, W1)

Q1. "Can you evaluate the verifier-trained captioning and image-generation selectors with an independent metric ..., and report informativeness or coverage?"

W1. "… improvements on the verifier's target attributes, more directly than they establish general factuality, informativeness, or semantic quality."

We agree that CHAIR and OWLv2 couple label and metric. Three new checks bound the claim: coverage (i), an independent VLM judge of factuality, informativeness and overall semantic quality on both tasks (ii), and independent image scorers (iii).

(i) Coverage. The verified pick mentions only 0.25 fewer real objects per caption, out of an average of 4.6 ground-truth COCO objects per image (coverage −4.9 ± 0.9 points; held-out head −4.6). The head trades a few mentions for precision.

(ii) Independent judge, both tasks. GPT-6-Astra sees the image(s) and the two candidates in both orders and answers three questions that never mention the verifier; only position-consistent verdicts count. Captions: the 200 held-out COCO-2 prompts per generator, pick vs. greedy, on fewer false statements, more information, and which caption is the better description overall (two COCO images blocked by the judge's content filter drop out, leaving 199 of 200 prompts per cell). Images: the 160 hard prompts, pick vs. a random sample of the same pool, on prompt faithfulness, visual quality, and which is the better generation overall.

Captions: pick vs. greedy (net = win % − loss % vs. greedy, ± 75% CI half-width)

Pick vs. greedyFewer false statements (2B / 8B)More information (2B / 8B)Overall better description (2B / 8B)
Held-out verified head+16 ± 9 / +16 ± 9−29 ± 8 / −16 ± 9+5 ± 9 / +7 ± 9
Verified head+14 ± 8 / +15 ± 9−30 ± 8 / −22 ± 8+3 ± 8 / +9 ± 9
Random candidate−18 ± 9 / −13 ± 8−11 ± 9 / −10 ± 9−15 ± 9 / −17 ± 9

Key takeaways (captions)

  1. The picks are more factual and less informative than greedy, the coverage cost of (i).
  2. On overall quality they are rated on par with greedy, and a random candidate clearly below it. The verified head trades some information for fewer absent objects without a measurable loss in overall description quality.

Images: pick vs. a random sample of the same pool (net = win % − loss % vs. random)

SelectorPrompt faithfulnessVisual qualityOverall
Verified head 8B+9.5 ± 6.5+9.8 ± 17.7+17.9 ± 17.5
Perturbation image head 8B+5.7 ± 6.8+22.6 ± 17.9+24.8 ± 18.4
CLIP ViT-L/14+8.2 ± 6.9−0.8 ± 18.5+6.5 ± 17.0
Text-trained 2B probe−5.6 ± 7.7+16.5 ± 18.3+7.4 ± 16.6

Key takeaways (images)

  1. The verified head is the most prompt-faithful selector, and the only trained head whose faithfulness gain excludes zero. Most faithfulness verdicts are ties (about 80%), since both images often satisfy the prompt. On quality and overall preference, the perturbation-trained 8B head leads, and the verified head's overall gain ranks second. The verified head selects for what its labels encode, prompt faithfulness, and we state the image result that way.

(iii) Image arm with an independent evaluator.

Setup: hard prompts of Fig. 1c, Best-of-12 Qwen-Image samples per prompt (1,920 images). Every pick is re-scored by two scorers that is disjoint in training or selection: the official GenEval evaluator (Mask2Former detector), and VQAScore (CLIP-FlanT5-XXL, the probability that the image answers "yes" to the prompt), a semantic prompt–image alignment score.

Score of each selector's pick (Δ in points vs. random)

SelectorGenEval passΔ GenEvalVQAScoreΔ VQAScoreOWLv2 pass (training verifier)Δ OWLv2
Random (pool mean)70.988.376.5
CLIP ViT-L/1470.6−0.3 ± 4.388.6+0.3 ± 1.476.9+0.4 ± 4.6
Text-trained 2B probe66.2−4.6 ± 4.787.4−1.0 ± 1.470.0−6.5 ± 5.0
Perturbation image head 2B71.3+0.4 ± 4.588.30.0 ± 1.477.5+1.0 ± 4.5
Perturbation image head 8B74.4+3.5 ± 4.887.5−0.8 ± 1.880.0+3.5 ± 4.5
Verified head 2B74.4+3.5 ± 4.289.5+1.2 ± 1.278.1+1.6 ± 4.2
Verified head 8B75.0+4.1 ± 4.589.5+1.1 ± 1.381.9+5.4 ± 4.9
OWLv2 oracle (first passing sample)74.4+3.5 ± 4.489.7+1.4 ± 1.295.6+19.1 ± 4.1
AuthorsOpenReview ↗

Response Q1/W1 cont. + Q2/Q3

Key takeaways

  1. The ordering survives two independent scorers. Under both GenEval and VQAScore the verified heads are the top two non-oracle selectors, and CLIP and the text-trained probe stay at or below random, as in Fig. 1c. Under VQAScore the verified heads gain as much as the OWLv2 oracle does (+1.1 to +1.2 against +1.4).
  2. The evaluators disagree often. OWLv2 and GenEval agree on only 78% of images (κ = 0.45): the OWLv2 oracle gains +19.1 under its own verifier but only +3.5 under GenEval.

2. Multi-seed Best-of-N results (Q2, W2)

Q2. "Can you report multi-seed results for the main Best-of-N experiments, especially image generation …?"

Three head seeds per Table 7 cell (CHAIRs@16, lower is better):

HeadCOCO / 2BCOCO / 8BCOCO-2 / 2BCOCO-2 / 8B
Verified, paper (1 seed).312.332.294.344
Verified, 3 seeds.305 ± .012.333 ± .009.294 ± .008.349 ± .010
Held-out, paper (1 seed).324.358
Held-out, 3 seeds.319 ± .010.365 ± .012

Pooled, the gain is −5.1 ± 0.7 (3 seeds) against greedy with every seed's CI excluding zero, and −4.0 ± 1.1 (3 seeds) for the held-out head with two of three seeds' CIs excluding zero. The coverage cost recurs in every seed.

For image generation we retrained both verified image heads with three seeds (seed = prompt-level split, initialization and data order) on the same 160 hard prompts;

Verified image headOWLv2 pass (training verifier)GenEval pass (official)VQAScore
8B+5.2 ± 0.4+3.1 ± 1.0+0.6 ± 0.5
2B+1.6 ± 0.6+3.5 ± 1.9+1.0 ± 0.5

The paper's single-seed 8B gain (+5.4) is reproduced across seeds, and under both independent scorers, every seed keeps a positive gain.

3. More than one round of adaptation (W2)

W2. "… the verifier-based adaptation is evaluated for only one round."

We ran four turns of the loop (App. G.1).

Setup: 2B policy, single seed; each turn generates new pools, mints about 1,300 verifier-labelled pairs, retrains the head and runs one DPO update. A random-pair chain (about 2,000 pairs per turn) is the control.

Greedy COCO CHAIRs of the updated policy (Δ in points vs. the base policy, .446)

TurnVerified pairsVerified chainRandom-pair chainHead MMRB1
11,381−4.4 ± 3.2−2.6 ± 3.3.652
21,314−3.8 ± 3.2−4.0 ± 3.0.591
31,290−2.4 ± 3.1−1.8 ± 3.4.563
41,250−4.6 ± 3.4−2.4 ± 3.4.524

Key takeaways

  1. One turn is the whole effect at this scale. CHAIRs drops at turn 1 and stays there; no later turn improves on it.
  2. The label does not separate the chains. The random-pair chain reaches the same level by turn 2, consistent with the DPO result in point 4.
  3. QA is unaffected and the head degrades as a judge. MathVista and MMMU stay within −1.8 to +2.8 points on every turn, while heads trained on later turns' pairs fall from .652 to .524 on MMRB1.

4. RM-selected vs. random pairs in DPO (Q2, W2)

Q2. "… and a statistical comparison of reward-model-selected versus random pairs in the DPO experiment?"

RM-picked minus random pairs is −.022 ± .033 CHAIRs, which is not significant, and §4 now states so. The claim is about the selector; the actor side is marked open. Reviewer Th6j raised the same point.

AuthorsOpenReview ↗

Response Q3 cont.

5. Scope of "unsupervised" (Q3)

Q3. "Could you clarify the intended scope of 'unsupervised' …? The current setup selects layer depth using labeled Multimodal RewardBench data, while the deployment-improving bootstrap uses verifier labels."

We agree the distinction should be explicit. The abstract states it:

"We build one with no human preference label […]; held-out perturbation loss can replace benchmark lables for layer selection"

§4.1 treats Multimodal RewardBench as development data throughout, and VL-RewardBench and MMRB2, which no choice in the recipe has seen, as the transfer evidence.

The verifier labels used by the bootstrap are named as such wherever the verified head appears.

6. Fixed probe layer or a separate layer-selection split (Q3)

Q3. "… and provide a sensitivity check using a fixed probe layer or a separate layer-selection split?"

We ran both (App. D.1).

Setup: the layer sweep behind Fig. 3: a linear head on the mean-pooled hidden states of every second layer (every fourth at 32B), the paper's recipe, 5 head seeds per layer. These short sweep runs sit a few points below the full pipeline, so only differences within a column are meaningful.

Fixed probe depth: MMRB1 trainable accuracy, cost in points vs. the dev-selected layer

Layer rule2B (L=28)4B (L=36)8B (L=36)32B (L=64)
Dev pick (paper)N−4N−6N−12N−12
Fixed N−4 at every size0−0.2−2.1−1.6
Fixed relative depth (0.857 L)00−2.0−0.4
Last layer−4.9−4.4−6.9−3.3

Seed standard deviation is 0.1 to 0.2 points in every cell.

Label-free selection: layer picked by held-out perturbation loss (no benchmark label)

2B4B8B32B
Layer pickedN−8N−2N−2N−12 (= dev pick)
Cost vs. dev pick, MMRB1 overall (points)−1.0−0.4+0.30

Key takeaways

  1. The depth choice is cheap to fix. A single rule (N−4) at every size costs at most 2.1 points; the last layer costs 3.3 to 6.9, so intermediate depth is what matters, not the exact layer.
  2. A label-free rule finds the right band. Selecting by held-out perturbation loss, with no benchmark label at all, lands within about 1 point of the dev pick at every size, and recovers it exactly at 32B.

Thank you, reviewer, for taking the time to thoroughly read our work and for the thoughtful comments and constructive feedback! As the discussion period ends November 18, we would be grateful to know whether our responses address your concerns, and we are happy to clarify any remaining point.

Reviewer_NqgQOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper trains a linear reward head on a frozen vision-language model using original captions and automatically edited versions, without human preference labels for training. It studies how the choice of layer and training pairs affects the reward. The main finding is that strong preference-benchmark scores do not necessarily lead to better Best-of-N selection. Training on verifier-labelled outputs from the target policy improves caption selection, even when benchmark scores drop.

Critical strengths
  1. The method is simple and works well on preference benchmarks. The 32B probe reaches 72.4% on VL-RewardBench, compared with the reported 65.8% for GPT-4o. Results across several model sizes and a second backbone family support the usefulness of the approach.

  2. The gap between benchmark scores and actual selection quality is worth studying. Two probes with similar development scores differ by 6.4 points on hallucination judging. The comparisons with majority voting, log-likelihood scoring, and random-pair DPO are useful, and the paper reports negative results clearly.

Critical weaknesses
  1. Some adaptation results reuse training data for evaluation. The verified head is trained on pairs from the candidate pools used for its main selection results, and the DPO experiment uses the same prompt sets for training and evaluation. The COCO-2 leave-out result does show transfer, but the other results do not establish how well the gains, or the lack of QA degradation, carry over to unseen prompts.

  2. The explanation of the training plateau is not fully supported. Convexity guarantees a unique optimum for a fixed regularized objective, but the actual training uses AdamW with a changing preconditioner. Similar benchmark scores across settings do not show that the heads reached the same optimum. This matters because the paper itself shows that similar scores can hide different reward behavior.

Questions for authors
  1. Which adaptation and DPO results are evaluated on prompts, images, and responses excluded from training? Please separate those from results that reuse the training pools.
  2. Apart from stable benchmark scores, what evidence shows that the heads have converged to the claimed optimum? Do you have loss or gradient measurements that support this?
1 comment

We thank the reviewer for the positive assessment and for the two questions, which we answer with tables below.

1. Results on prompts, images and responses excluded from training (Q1, W1)

Q1. "Which adaptation and DPO results are evaluated on prompts, images, and responses excluded from training? Please separate those from results that reuse the training pools."

The new disjoint evaluations confirm the earlier caption-selection finding. The verified head reduces CHAIRs by 6.1 ± 0.9 points on fresh candidates and 7.6 ± 0.5 points on 500 new COCO-3 images, consistent with the earlier 5.1-point in-pool reduction. The leave-out head also achieves 7.0 ± 0.6 points on fresh COCO-2 pools, disjoint at the image, prompt-instance and candidate levels. Here ± is standard deviation across three training seeds; every seed's 95% bootstrap interval excludes zero. These are separate evaluations with their own greedy baselines, so agreement is in the direction of improvement. The coverage tradeoff also persists.

For clarity, the main Table 7 verified-head cells reuse adaptation pools; fresh candidates alone retain the original images/prompts. The COCO-2/MMMU leave-out evaluations, COCO-3 and the image-generation test arm are disjoint at all three levels. DPO reuses training prompts/images, including COCO-2, so these new selector results do not establish held-out actor improvement. App. F.1 gives the per-cell audit.

2. Ranking stability and comparison with the fixed-penalty optimum (Q2, W2)

Q2. "Apart from stable benchmark scores, what evidence shows that the heads have converged to the claimed optimum? Do you have loss or gradient measurements that support this?"

We now provide a direct optimization diagnostic beyond aggregate benchmark scores (App. E.1). The strongest finding is that the learned rankings are stable across training variations, even when the head weights differ. We measure losses, regularised gradients, weight-vector similarity and pairwise ranking agreement, and compare AdamW heads against a numerical solution of the fixed-penalty objective.

The controlled experiment uses frozen cached features at 2B and 8B, 7,200 training pairs, and a constant learning rate rather than the full trainer's cosine schedule. Across seeds, learning rate, training steps, batch size and pool size:

  • Held-out in-distribution ranking agreement is 97–100%. This checks the individual pairwise decisions, rather than merely whether aggregate accuracies happen to match, and supports the empirical robustness of the learned ranking on this distribution.
  • Ranking stability does not require identical weights. Cosine similarity to the reference AdamW head ranges from 0.78 to 1.00. The gradient norm of the fixed-penalty objective at the AdamW endpoints is 0.03–0.64; comparison with its numerical minimiser confirms that these endpoints are distinct from that solution.
  • Reaching the fixed-penalty optimum does not improve benchmark accuracy in this comparison. L-BFGS and plain gradient descent from two initialisations agree to cosine 1.000, providing a numerical reference. AdamW heads have cosine 0.69–0.82 to that reference. Benchmark accuracy is similar at 8B (.656 for AdamW versus .652 for the reference), while AdamW is 6.4 points higher at 2B (.625 versus .561). These are results within the cached-feature experiment, not the paper's full-pipeline accuracies.

These measurements strengthen the evidence for stable in-distribution ranking while identifying the correct scope of the theory. The unique-minimiser guarantee concerns the stated fixed-penalty objective; AdamW's adaptive preconditioning and decoupled weight decay do not generally optimize that same objective. We therefore distinguish empirical ranking stability from convergence to its minimiser. The contribution does not depend on the two being identical: the ranking behavior is measured directly, and downstream usefulness is assessed separately. We have revised the explanation of the plateau accordingly; the detailed measurements also appear in our reply to Th6j, point 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.