acceptodds
Under review as a conference paper at ICLR 2027

Many Captioners or One? The Price and Payout of Mixing Frozen VLMs

Abstract

Recaptioning pipelines often mix several vision–language models (VLMs), on the premise that different models see different things. We test this premise against its obvious alternative, resampling one good captioner, at a matched number of captions. To this end, we build a controlled corpus in which 30 frozen VLMs caption the same M ImageNet training and k validation images under one prompt and one decoding specification, with resamples at identical settings. We find that mixing buys protection, not quality. In six of seven uses, from training a text encoder to auditing labels, the mix never reliably beats resampling the captioner that is best for that use, and it is worse in three, including text-encoder training, where much of the gap goes with caption length. Against a typical captioner, however, the mix gains in the same six uses: one captioner drawn at random per image trains a better text encoder than 29 of the 30 models used alone. This protection matters because the best captioner is hard to know in advance. It changes with the use and the prompt, its zero-shot accuracy says little about its value as a training source, and in the seventh use, a concept bottleneck, the captioner we declared best in advance is the weakest source we tested, so the mix wins. Mixing is therefore insurance against a poor choice rather than an improvement on a good one, at about twice the cost of generation. The code is available at https://anonymous.4open.science/r/ICRL_2027_34944_code.

Primary area:
foundation or frontier models, including LLMs
Keywords:
synthetic captions, recaptioning, vision-language models, caption diversity, ensembles, zero-shot classification, ImageNet

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_grQWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates whether mixing captions from multiple frozen vision–language models offers benefits beyond repeatedly sampling a single captioner. The authors construct a controlled corpus using 30 VLMs on ImageNet and compare the two strategies at matched caption counts across seven uses, including text-encoder training, zero-shot classification, label auditing, test-time caption reading, and concept bottlenecks. Mixing generally improves over a typical captioner but does not consistently outperform a well-chosen single captioner, with several comparisons remaining inconclusive. Captioner rankings vary across uses, and prototype accuracy is an unreliable predictor of downstream training utility. The authors therefore characterize mixing primarily as protection against poor captioner selection, while noting limitations related to dataset scope, caption length, and selection and generation costs.

Critical strengths
  1. A carefully controlled and statistically qualified comparison. Matching caption counts addresses an important confound in comparisons between mixing and resampling. The factorial analysis and extensive follow-up checks help identify when their benefits differ. The paper also distinguishes equivalence from inconclusive results and reports multiplicity correction and exceptions to pre-registration, making its conclusions appropriately qualified.

  2. Practical analysis of captioner selection and cost. The paper goes beyond performance comparisons by measuring selection accuracy and regret for several evaluation uses, as well as generation costs. It shows that the best captioner depends on the use and provides a substantial controlled corpus with a stated release plan, offering a useful resource for further study.

Critical weaknesses
  1. Limited generalizability. The controlled comparisons are informative, but all experiments use ImageNet-1k, and the main training evidence uses a frozen CLIP image encoder on 300k images. The small-scale image-encoder training and alternative-prompt checks leave it unclear whether the observed trade-offs extend to the broader recaptioning pipelines motivating the paper.
Questions for authors
  1. How strong is the evidence for a remaining resampling advantage under common token caps? The gap decreases from 3.0 points to 1.4 and 0.8 points at 64 and 48 tokens, respectively, while the seed-level intervals include zero. Please clarify how the class-bootstrap and seed-level uncertainty should inform the interpretation of these smaller gains.
  2. Does the mixing–resampling trade-off persist beyond ImageNet class-name classification? One representative comparison on another image–text dataset or a retrieval task would help assess whether the practical recommendation extends to broader recaptioning settings.
Reviewer_ZuFqOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper is an empirical study on captioning images using an ensemble of VLMs. The message is not entirely clear to me personally. However, I believe the authors try to claim that ensembling multiple VLMs is not as good as simply resampling a single VLM.

Critical strengths

Code is provided.

Critical weaknesses
  • The introduction is unclear. For example:

    • what is "12.2 points"? What is "mix minus a single captioner"? Are we talking about percentage points? If so, what is the metric? The "premium" and "payout" verbiage is unintelligible to me.
    • Same comment for Figure 1. What do these numbers mean? What is the unit?
    • What does "pre-registered" mean?
    • Figure 1's caption is not comprehensible to me. What is "per use"? What is "multiplicity correction"?
    • Figure 1: What is "equivalence margin". The figure style seems professional, but if you look closely, it is difficult to understand. What are these grey bars? What does "AUROC and precision ×100" mean?
  • The contributions are not sentences. Additionally, they appear incomprehensible.

  • Section 4 "Main Comparison" section reads like a mechanical discussion that adds no value.

  • What does "arm" mean. This word is used repeatedly throughout the paper.

I do not mean this review to be dismissive, but there are many more examples where the paper is not coherent. It is hard to understand the content.

Questions for authors

I would ask the authors to clarify the objective of the paper.

Reviewer_Fn7kOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper talks about whether mixing captions from many VLMs beats resampling one good captioner at the same caption count, using 30 frozen VLMs on ImageNet. Main takeaway – mixing never reliably beats the best captioner resampled, but it does beat a typical one, so it works more as an insurance rather than an improvement.

Critical strengths
  1. A controlled 30-VLM ImageNet corpus with true resamples is a useful contribution.
  2. Matched-caption-count design and pre-registered comparisons isolate the question better than prior work.
  3. The authors provide code, which is a great plus for reproducibility.
Critical weaknesses
  1. The hypothesis isn’t tested very broadly. The training result depends on one resampled captioner (Gemma 4 12B), and the only other one tested loses to the mix. Caption length also explains much of the gap.
  2. Narrow scope: ImageNet only, one prompt, frozen image encoder.
  3. Hard to follow in places. I found some figures confusing because of flipped signs between figures.
Questions for authors
  1. Can you resample one or two more top captioners at training scale to show the result isn’t specific to Gemma 4 12B?
  2. Could you extend the main comparison to another image-text dataset?
2 comments
Senior_Area_ChairsOpenReview ↗

late reviews available

Dear authors and reviewers: Please note that late reviews have been received for this submission. The authors are welcome to write rebuttals by replying to this comment, if they are not able to do so in the regular way.

Thank you for the review. Please note that two revisions have been uploaded during the discussion, the first in response to the other two reviews and the second in response to yours. The current PDF contains all changes.

"The hypothesis isn't tested very broadly. The training result depends on one resampled captioner (Gemma 4 12B), and the only other one tested loses to the mix." Gemma 4 12B was declared the best captioner before any training run, by a different measure, its prototype accuracy (Section 3). It then also turned out to be the best of the 30 as a training source. It is the only captioner that, used alone, trains better than a mix with one random captioner per image (41.5% against 38.9%, Table 2). Extra captions help about equally whether they are resamples or come from other captioners (Table 2), so a captioner that alone is weaker than this mix is unlikely to beat the mix after resampling. This is what we observe for the second-best source, Kimi-VL-A3B (Table 7). In other uses the best captioner is a different one, and the mix does not beat it after resampling either, in test-time reading (Table 3) or on Visual Genome with Molmo2-8B at full caption length (Appendix J).

"Caption length also explains much of the gap." We agree, and Section 4 quantifies it. With every caption cut to the same 64 tokens, the resampled captioner still leads by 1.5 points over five seeds. At 48 tokens, the remaining lead of 0.5 points is no longer statistically established (Table 7).

"Narrow scope: ImageNet only, one prompt, frozen image encoder." and "Could you extend the main comparison to another image-text dataset?" We did not repeat the training comparison on another dataset, and image-encoder training at the scale of recaptioning pipelines remains untested (Section 8). However, the paper has two smaller checks. With a ResNet-50 image encoder trained from scratch, the resampled captioner still leads, by 2.6 points (Table 7). A check with two other prompts shows that which captioner is best depends on the prompt (Appendix C). We also added two measurements outside ImageNet class names (Appendix J). On COCO retrieval, the text encoders trained on ImageNet captions show no resampling lead. Together, these results agree with the practical rule of the paper. Resampling pays when the captioner is chosen for the use, as on Visual Genome, and not when it was chosen for another use, as on COCO. Whether this holds when training at the scale of recaptioning pipelines remains open.

"Hard to follow in places. I found some figures confusing because of flipped signs between figures." In the submitted version and the first revision, Figure 1 showed the mix minus the resampled captioner and Figure 2b the resampled captioner minus the mix. In the second revision, both figures show the mix minus the best captioner resampled, so negative values favour resampling in both, and the Figure 2 caption is rewritten. Tables 2 and 7 and Section 4 report the lead of the resampled captioner (E − D6), and their row labels state the direction. The first revision had already rewritten the introduction and the Figure 1 caption.

"Can you resample one or two more top captioners at training scale to show the result isn't specific to Gemma 4 12B?" We did not resample further training sources. Each additional source needs five resampling passes over 300k images and three paired training runs, which did not fit in the discussion period. In training, the evidence therefore rests on Gemma 4 12B and the Kimi-VL-A3B check. The reasoning above leads us to expect the same outcome as for Kimi-VL-A3B for the lower-ranked sources, but we have not tested it. We state this limitation in Section 8.

We would be glad to know whether these answers and the revised figures address your concerns.

Comments on the paper

1 comment

We have uploaded a second revision in response to Reviewer Fn7k. The changes relevant to all reviewers are the following. Figure 2b now uses the same sign as Figure 1, the mix minus the best captioner resampled, so negative values favour resampling in both figures. Section 4 is shorter, and its results table (Table 2) is now in the main text. The captions of Figure 2 and of this table are rewritten. No result changed.

Moving the table renumbers three tables. The training results are now Table 2 (previously Table 6), test-time reading is Table 3 (previously Table 2), and the premium and payout per use are Table 6 (previously Table 5). Our reply to Reviewer grQW referred to Tables 5 and 6 under the old numbering, and we have updated it. All other references in our replies are unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.