- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether hidden preferences can be transferred between vision–language models solely through the selection of near-duplicate images. The authors train target-preferring and neutral curators to select among eight visually similar variants of each source photograph using caption likelihood, while keeping the original images, human-written captions, and training configurations fixed. Students trained on target-curated images exhibit increased target preferences under image-free evaluation across four VLM families. Additional experiments examine target specificity, JPEG-matched controls, training configurations, and second-generation transfer through generated captions. The paper identifies a new, restricted channel for subliminal preference transfer, although its underlying mechanisms and practical implications remain incompletely established.
Critical strengths
-
A well-motivated and more constrained setting for subliminal preference transfer. The paper extends prior research on subliminal learning by isolating near-duplicate image selection as a transmission channel. Unlike previous approaches involving generated content, target-guided dataset selection, or preference labels, the proposed setting fixes source membership and human-written captions and uses target-free caption likelihood for selection. This provides a meaningful new perspective on potential risks in model-based data curation.
-
Careful experimental controls and reasonable cross-model coverage. The experiments cover four VLM families and multiple preference targets. The matched target, neutral, and second-neutral arms, together with independent curator ensembles, student-seed repetitions, and target-rotation experiments, strengthen the evidence that the observed preference shifts are related to the curation process rather than arbitrary student-training variation.
-
Extensive empirical characterization of the phenomenon and its limitations. The paper investigates JPEG-quality confounding, dataset size, training updates, learning rates, alternative evaluation prompts, and cross-checkpoint transfer. It also explores whether preferences acquired through image selection can propagate to a second generation through generated captions. These experiments provide useful evidence about the conditions under which transfer occurs and fails.
Critical weaknesses
-
Incomplete isolation of the factors responsible for preference transfer. The JPEG-matched controls only account for JPEG-quality distributions, leaving other transformation parameters and their interactions uncontrolled. Moreover, target–neutral and neutral–neutral comparisons differ in their image-selection disagreement rates, potentially weakening the interpretation of the neutral–neutral sham as a comparable reference.
-
Limited understanding of the underlying transfer mechanism. Although the paper demonstrates that curator-dependent image selection can induce downstream preference shifts, it provides limited evidence explaining which visual features carry the signal or how the resulting training updates produce target-specific behavioral changes. The exploratory gradient analyses do not sufficiently establish the proposed transmission pathway.
-
Limited statistical validation and independence of experimental repetitions. The transfer criterion relies on heuristic thresholds without calibrated statistical significance or multiple-comparison control. Moreover, curator blocks share the same source bank, student-seed repetitions reuse selected images, and prompt-level bootstrap intervals do not capture uncertainty from data sampling or curator training. These dependencies limit the strength of the robustness claims.
Questions for authors
-
Can the authors provide stronger selection-matched controls that account for all image transformation parameters, their joint distributions, and target–neutral selection disagreement rates? Would the observed transfer persist under these more strictly matched controls?
-
How does curator preference strength affect downstream transfer? Can the authors systematically vary the curator's preference strength and analyze its relationship with selection disagreement and student preference shifts?
-
How well do the observed likelihood shifts generalize to actual generated behavior? Could the authors evaluate open-ended preference questions, alternative candidate sets, and more diverse prompt formulations to establish the practical significance of the transfer?
-
How representative is the experimental setting of real-world data curation? Can the authors evaluate naturally occurring near-duplicate images and weaker or naturally emerging curator preferences to demonstrate that the effect extends beyond strongly biased curators and synthetic image transformations?
4 comments
Response to Reviewer 3jPM (2/2)
Statistical validation and independence (W3). See G4.
- Because a single three-block cohort trains 36 curators, scores 64,000 candidates per curator, and fine-tunes nine VLM students, compute limits the number of independent end-to-end replications, so we use §2.3 as a decision rule rather than a significance test: 2h comes from a single neutral–neutral difference per block and does not control false-positive rates or multiple comparisons.
- The adjusted 98.75% and 99.17% intervals in G1-1 and G3-2 apply only to those follow-up analyses and do not modify the §2.3 rule.
- To test beyond prompt-level resamples, a single source bank, and one set of curator trainings, we report block-and-question-family bootstrap intervals, leave-one-family-out checks, and a full replication on a second image bank from new source photographs with newly trained curators and students, where all four models meet the criterion again. Within each bank, source photographs shared across blocks, student-seed repeats that reuse fixed selections, bootstrap intervals conditioned on trained checkpoints, and a neutral–neutral sham not rebuilt to the target–neutral disagreement count remain limitations.
Curator preference strength and selection breadth (Q2). See G5.
- Curators trained on 50%-target answers yield smaller student contrasts than target-only curators across all 12 blocks (four models, three blocks each) without meeting this test's dispersion criterion in any model, though testing one weaker level does not rule out weak-preference transfer [App. B.4]. Here 50% refers to the curator training answers rather than measured preference: on qualification prompts, the 50%-target curators' log-odds preference against neutral curators is 5% to 8% of the target-only curators' preference in every block, and their selections disagree with neutral selections on 14% to 38% of sources versus 22% to 51% for target-only curators.
- Within each block, the student seeds share one selection (giving 12 block-level comparisons of two strength levels), and this run did not include a second-neutral selection as a disagreement baseline.
- Separately, in LLaVA-1.5 development runs [App. B.16], candidate pool size K = 2, 8 and 16 gives mean contrasts of 0.036, 0.105 and 0.126 nats, and curating 100%, 50% and 25% of pairs gives 0.109, 0.050 and 0.046. A finer sweep linking preference strength to disagreement remains open.
Generated behaviour, alternative candidate sets, and prompt diversity (Q3). See G3.
- Under eight-option greedy decoding, owl answers rise in Qwen3-VL (+10.4 pp) and oak answers in Gemma 4 (+7.8 pp) [§7, App. B.6]. Without listed options [App. E.4, E.5], only Qwen3-VL oak rises consistently, on both held-out open-ended sets, while the remaining cells show no detectable shift, so we state the claim at the likelihood level (G3).
- On prompt wording and spelling [App. E.1–E.3], a second 128-prompt bank stays positive in all three blocks across all 13 owl and camel conditions, and rescoring surface forms (capitalized, plural, or article-prefixed) stays positive in all 18 evaluable cases (+0.053 to +0.607 nats, 10 meeting the criterion).
- Replacing all seven non-target animals without retraining (set A: sheep, shark, penguin, rabbit, cat, turtle, goat, and set B: deer, parrot, squirrel, dog, wolf, snake, lion) meets the criterion in 7 of 8 pairs [App. E.2]. The table reports mean owl contrasts in nats (Gemma 4 is 0.341 in this run vs 0.326 in Table 2):
| Model | Original | Set A | Set B |
|---|---|---|---|
| Qwen3-VL | 0.362 | 0.369 (Met) | 0.367 (Met) |
| Qwen2.5-VL | 0.198 | 0.170 (Met) | 0.204 (Met) |
| Gemma 4 | 0.341 | 0.272 (Met) | 0.314 (Met) |
| LLaVA-1.5 | 0.089 | 0.046 (Not met) | 0.119 (Met) |
- On LLaVA-1.5 set A, all three blocks are positive and owl leads the profile, though 0.046 falls below twice the sham (0.053). Testing two sets does not prove full invariance.
Distinct real photographs, weaker preferences, and natural near-duplicates (Q4). See G5.
- To move closer to candidate-selection steps in curation workflows without synthetic perturbations, we paired each of 8,000 fixed captions with eight distinct retrieved real photographs and selected one per caption by ordinary caption likelihood (§3.3, [App. B.3, Fig. 7]). With trained curators, owl transfer meets the §2.3 criterion across all three student seed sets in Gemma 4, Qwen3-VL and LLaVA-1.5, while Qwen2.5-VL shows positive owl gains (0.10–0.16, 99.29% intervals above zero) that fall below twice the sham and peak on other animals (on one retrieval bank with partial caption support).
- For weaker curator preferences, see Q2 above. For naturally occurring near-duplicates, we screened three candidate sources (PhotoTriage, MSR-VTT, and DiDeMo, G5-2), but none yielded a viable full-scale shared-caption bank, so natural near-duplicates, natural curator biases, and deployment-scale curation remain untested [§7].
Thank you for the detailed rebuttal and additional experiments. The stronger transformation-matched controls, independent replication on a second image bank, and gradient analyses substantially address my concerns and strengthen the central findings.
However, the limited changes in open-ended generated behavior and the lack of validation on naturally occurring near-duplicates and curator biases still constrain the generalizability of the results.
Overall, the rebuttal has increased my confidence in the paper, but I will maintain my rating of 3.
Response to Reviewer 3jPM
Thank you for the follow-up comment and for reviewing the additional experiments. We agree on both points regarding scope. Without listed options, generated answers shift consistently only for oak in Qwen3-VL, and naturally occurring near-duplicates and natural curator biases remain untested [§7, App. E.4–E.5].
As one final update, the LLaVA-1.5 per-transformation ablation is now complete in the updated PDF [§3.3, App. B.10]. Its all-transformation reference meets the criterion (0.089 nats), and among single transformations only brightness meets it (0.064 nats, 71%), with cropping close behind but below twice its sham (0.057 nats, 64%), so across all four models, which transformation suffices depends on the model.
Response to Reviewer 3jPM (1/2)
Thank you for the careful review and the constructive questions.
Transformation controls, per-transformation ablations, and disagreement rates (W1, Q1). See G1 and its update.
Mechanism across one-step and 375-update diagnostics (W2). See G5-1 and §3.3.
References