- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper trains a linear reward head on intermediate features of a frozen vision-language model using web captions paired with synthetic perturbations. Human preference labels are excluded from head training, although a labeled benchmark selects the probe layer. The study evaluates multiple backbone sizes and reward benchmarks, analyzes the effect of the negative sampler, and tests Best-of-N selection and one round of policy updating. Its main empirical finding is that preference-benchmark accuracy can disagree with downstream selection quality: perturbation-trained probes score well on benchmarks, whereas adaptation to verifier-labeled policy outputs improves caption selection despite reducing some benchmark scores.
Critical strengths
- A frozen backbone and a small linear head provide a practical way to exploit synthetic comparisons. The intermediate-layer ablations and the second backbone family strengthen the result beyond a single checkpoint. The 32B probe reaches 72.4% on VL-RewardBench versus the quoted 65.8% for GPT-4o (Table 1), while the paper explicitly treats the layer-selection benchmark as development data.
- The CLIP-split probes and the Best-of-N controls show why similar benchmark scores need not imply similar behavior. Including majority vote, log-likelihood scoring, random selection, and verifier-based adaptation makes the negative results useful. The reported held-out COCO-2 improvement is also stronger evidence than evaluating only the tasks used to build the adaptation pairs (Table 7).
Critical weaknesses
- The density-ratio interpretation is applied beyond the conditions established by the theory. Proposition 1 applies to independent cross-entry comparisons with common support. The actual objective includes dependent diagonal terms, and Appendix I explicitly states that the implemented sampler does not satisfy the support assumption. More importantly, Section 3.4 invokes Proposition 1 to claim that verifier-ordered policy pairs learn the ratio needed for Best-of-N, although Appendix I.1 notes that judge-ordered pairs are governed by their joint comparison law, not their marginals alone. The fixed-penalty convexity argument also needs separation from AdamW’s changing preconditioner (Appendix J). These qualifications do not invalidate the experiments, but they weaken a central explanation of why the method should work.
- CHAIR supplies both adaptation labels and captioning evaluation, so the gain establishes improvement on that proxy without showing preserved informativeness or broader factuality; Appendix A acknowledges this limitation. The original perturbation probes do not improve captioning over greedy decoding, and their hard-QA gains are not clearly better than the training-free log-likelihood control. The actor experiments likewise do not establish a selector-specific benefit over random pairing (Appendix G). Thus, the paper offers convincing diagnostic observations, but less evidence for a broadly useful reward-learning and self-improvement procedure.
Questions for authors
- Which conclusions about the trained head remain valid for the full dependent-pair objective and the implemented AdamW optimizer? Can the application of Proposition 1 to verifier-ordered pairs be justified, or explicitly restricted to a heuristic interpretation?
- Are adaptation training pairs and all reported downstream evaluations disjoint at the image, prompt, and candidate levels? Please clarify this separately for the main Table 7 cells and the leave-one-task-out result, rather than only for the original perturbation corpus.
- Does the caption-selection improvement persist under an independent factuality assessment that also checks retained information? A small, clearly held-out evaluation would help distinguish general improvement from learning the CHAIR criterion.
4 comments
Response Q2, Q3
3. Disjointness for Table 7 and the leave-one-task-out result (Q2)
Q2. "Are adaptation training pairs and all reported downstream evaluations disjoint at the image, prompt, and candidate levels? Please clarify this separately for the main Table 7 cells and the leave-one-task-out result …"
App. F.1 now states it for every Table 7 cell, the leave-one-task-out arms, the image arm and the DPO arms.
- Table 7, in-pool cells: share prompts and candidates with the verified head's training pairs.
- Leave-one-task-out (held-out COCO-2 column): disjoint; −4.1 ± 2.9.
- New, fresh candidate pools: none of the 8,000 candidates per pool appears in a training pair; −6.1 ± 0.9 (3 seeds), every seed's CI excluding zero (held-out head −7.0 ± 0.6, 3 seeds, on fresh COCO-2).
- New, 500 COCO images in no prompt set: −7.6 ± 0.5 (3 seeds), while the perturbation probes give −2.5 ± 3.2 (2B) and +0.1 (8B).
- DPO cells: the corpora draw on all four tasks' prompts, COCO-2 included (App. G), so no DPO cell is prompt-disjoint; the table marks them in-prompt.
The two new disjoint evaluations in detail:
Setup: (a) fresh pools: 16 new samples per prompt (new sampling seed) on the 500 COCO and 500 COCO-2 prompts, 2B and 8B generators; none of the 8,000 candidates per pool occurs in any training pair. (b) COCO-3: 500 new COCO val2014 images used by no prompt set. Verified heads retrained with 3 seeds; probes and log-likelihood are single runs. Δ in CHAIRs and coverage points vs. each pool's own greedy decode (lower CHAIRs and higher coverage are better).
(a) Fresh candidate pools (pooled over 4 pools, 2,000 prompt-cells)
| Selector | CHAIRs | Δ CHAIRs | Δ coverage |
|---|---|---|---|
| Greedy | .387 | ||
| Random | .394 | +0.7 ± 2.2 | −0.2 ± 0.8 |
| Log-likelihood 2B (training-free) | .359 | −2.8 ± 2.2 | −0.4 ± 0.8 |
| Perturbation probe 2B | .370 | −1.7 ± 2.2 | −1.5 ± 0.8 |
| Perturbation probe 8B | .393 | +0.6 ± 2.1 | +1.4 ± 0.8 |
| Verified head (3 seeds) | .326 ± .009 | −6.1 ± 0.9 | −4.2 ± 0.6 |
| Held-out head, COCO-2 pools only (3 seeds) | .318 ± .006 | −7.0 ± 0.6 | −4.2 ± 0.3 |
(b) COCO-3: new images and prompts (pooled over 2B and 8B, 1,000 prompt-cells)
| Selector | CHAIRs | Δ CHAIRs | Δ coverage |
|---|---|---|---|
| Greedy | .399 | ||
| Random | .406 | +0.7 ± 3.2 | −0.7 ± 1.1 |
| Log-likelihood 2B (training-free) | .393 | −0.6 ± 3.0 | −1.3 ± 1.1 |
| Perturbation probe 2B | .374 | −2.5 ± 3.2 | −1.7 ± 1.1 |
| Perturbation probe 8B | .400 | +0.1 ± 2.9 | +0.1 ± 1.1 |
| Verified head (3 seeds) | .323 ± .005 | −7.6 ± 0.5 | −4.3 ± 0.9 |
Key takeaways
- The gain is not an in-pool artifact. On candidates and images the head never saw, the verified head still cuts CHAIRs by 6.1 and 7.6 points (in-pool Table 7: 5.1), and every seed's CI excludes zero. Each pool is compared with its own greedy decode, so the three numbers are not directly comparable to each other.
- The coverage cost travels with it (−4.2 to −4.3 points everywhere), matching point 4.
- The perturbation probes stay at greedy level, as in the paper, and are no better than the training-free log-likelihood judge.
4. Independent factuality that also checks retained information (Q3)
Q3. "Does the caption-selection improvement persist under an independent factuality assessment that also checks retained information?"
Yes under GPT-6-Astra's independent factuality assessment, with an informativeness tradeoff. We evaluated the head trained without COCO-2 pairs on 200 held-out COCO-2 prompt instances per generator. The evaluation images, prompt instances and candidates are excluded from that head's adaptation training. Astra sees the image and the selected and greedy captions, without selector identities or CHAIR scores, and answers separately which caption has fewer false statements, which conveys more true, specific information, and which is better overall.
Each comparison is judged in both A/B orders; the table uses only position-consistent verdicts. Net preference is win minus loss in percentage points, with ties included in the denominator; brackets are 95% bootstrap confidence intervals. After filtered calls, 199 comparisons per generator have both orders available; informativeness judgments are order-inconsistent in 13% / 19% of comparisons at 2B / 8B, and factuality judgments in 18% at both sizes.
| Held-out head vs. greedy | 2B generator | 8B generator |
|---|---|---|
| Factuality | +15.9 [ +1.8, +30.5 ] | +15.9 [ +0.6, +30.5 ] |
| Informativeness | −28.9 [−42.8, −14.5] | −16.1 [−30.5, −0.6] |
| Overall quality | +4.7 [−11.6, +19.8] | +6.7 [−7.9, +21.3] |
Response Q3 cont. + W2
The factuality gain therefore extends beyond CHAIR under this independent judge at both generator sizes. Astra assesses false statements against the image rather than COCO-object mentions alone. As a control, a random candidate is judged worse than greedy overall: −15.3 [−29.9, −1.7] at 2B and −17.4 [−32.6, −2.3] at 8B. The selected captions have positive overall-quality point estimates.
The retained-information check also identifies the cost: Astra judges the selected captions less informative at both sizes. These net preferences measure how often one caption is preferred, not the percentage of information lost. Separately, the object-coverage analysis finds 0.25 fewer ground-truth COCO categories mentioned per caption and a coverage decrease of 4.9 ± 0.9 points, despite approximately unchanged length. We therefore claim held-out factuality improvement under Astra with reduced informativeness, rather than preserved information or an established overall-quality gain. The independent assessment and coverage measurements are reported in App. F.2.
5. Perturbation probes, hard QA and the actor experiments (W2)
W2. "The original perturbation probes do not improve captioning over greedy decoding, and their hard-QA gains are not clearly better than the training-free log-likelihood control. The actor experiments likewise do not establish a selector-specific benefit over random pairing (Appendix G)."
Agreed on all three, and §4 reports them:
- On hard QA, probe-2B minus the training-free log-likelihood judge is +1.6 ± 3.0 points (8B generator), and every mined-negative probe stays at greedy level on captioning.
- In DPO, RM-picked minus random pairs is −.022 ± .033 CHAIRs; the actor side is marked open.
The paper's claim about the perturbation probes and actor experiments remains diagnostic: benchmark rank does not predict selection gain, and selector-specific actor improvement is not established. For the verifier-adapted selector, point 4 adds held-out factuality improvement under Astra, with reduced informativeness and no established overall-quality gain.
Thank you again for your insightful and constructive feedback; it really helps strengthen the work! As the discussion period ends November 18, we would be grateful to know whether our responses address your concerns, and we are happy to clarify any remaining point.
Response Q1/W1
We thank the reviewer for reading Appendices I and J closely, and the insightful suggestions on the theory section. Below we address each question in turn/
1. Proposition 1 for verifier-ordered pairs (Q1, W1)
We restrict it, and concede all three points. Dependent diagonal terms and the support violation are why App. I analyses the linear head through the fixed-penalty problem (Corollary 1) and not through Eq. (ratio). We changed the text accordingly:
2. What remains valid under the dependent-pair objective and AdamW (Q1, W1)
The surviving result concerns regularized linear pairwise learning under an arbitrary fixed comparison law. Let d be the difference between the frozen feature vectors of the preferred and non-preferred entries, and let μ be the distribution of d. This law can include dependent pairs, diagonal and cross-entry comparisons, or verifier-ordered pairs; it need not factor into independent winner and loser distributions. The same formulation covers a finite dataset by taking μ to be the dataset's weighted empirical distribution.
For a linear score, the objective is
\qquad \ell(z)=\log(1+e^{-z}).$$ With $\lambda>0$, fixed $D\succ0$, and finite feature moments, this objective is strongly convex and has a unique minimiser. Dependence and lack of common support do not invalidate this conclusion: convexity holds for each individual comparison and is preserved by averaging. Its first-order condition is $$\lambda D w^\star=\beta\,\mathbb{E}_{d\sim\mu}[\sigma(-\beta w^{\star\top}d)\,d],$$ which is the characterization in Corollary 1. Thus **the full comparison law and the regulariser determine the optimum**. More generally, convexity holds for any convex loss of a linear pairwise margin; a strongly convex regulariser supplies uniqueness. The displayed first-order condition is specific to the logistic loss and quadratic penalty. **Three conclusions must remain separate.** First, this optimization result characterizes the stated objective without requiring a density-ratio interpretation. Second, identifying its optimum with a density ratio requires the additional assumptions of Proposition 1; the general pairwise result alone gives neither that identification nor a guarantee of downstream Best-of-N quality. Third, uniqueness of an optimum does not establish that a particular training algorithm reaches it. Gradient descent converges under suitable smoothness and step-size conditions, but AdamW's adaptive preconditioning and decoupled weight decay do not generally correspond to optimization of this same fixed-penalty objective. For the implemented AdamW head, ranking stability and selection quality therefore remain empirical findings. We tested the distinction in a controlled cached-feature experiment (App. E.1): 2B and 8B heads, 7,200 training pairs, and a constant learning rate rather than the full trainer's cosine schedule. The measurements below describe this experiment, not a direct convergence audit of the paper's full training runs: | Quantity | Value | |---|---| | Agreement on held-out in-distribution orderings, across seeds / LR / steps / batch / pool size | 97 to 100% | | Cosine to the seed-0 AdamW head | 0.81 to 1.00 (2B), 0.78 to 1.00 (8B) | | Regularised gradient norm at the final iterate | 0.14 to 0.64 (2B), 0.03 to 0.14 (8B) | | Cosine to the unique minimiser (L-BFGS = plain GD from two inits, cosine 1.000) | 0.69 to 0.82 | | Benchmark pairs where AdamW heads and minimiser disagree | 13 to 20% (2B), 11 to 13% (8B) | | Benchmark accuracy, AdamW heads vs. minimiser | 8B .656 vs .652; 2B .625 vs .561 | In this experiment, **97–100% agreement on held-out in-distribution rankings coexists with substantial disagreement with the fixed-penalty minimiser**. Thus ranking stability does not establish convergence to that minimiser. We retain ranking stability as an empirical observation and remove convexity as its explanation; the theoretical characterization applies only to the stated fixed-penalty problem. Reviewer NqgQ raised the AdamW point as well.