- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates whether mixing captions from multiple frozen vision–language models offers benefits beyond repeatedly sampling a single captioner. The authors construct a controlled corpus using 30 VLMs on ImageNet and compare the two strategies at matched caption counts across seven uses, including text-encoder training, zero-shot classification, label auditing, test-time caption reading, and concept bottlenecks. Mixing generally improves over a typical captioner but does not consistently outperform a well-chosen single captioner, with several comparisons remaining inconclusive. Captioner rankings vary across uses, and prototype accuracy is an unreliable predictor of downstream training utility. The authors therefore characterize mixing primarily as protection against poor captioner selection, while noting limitations related to dataset scope, caption length, and selection and generation costs.
Critical strengths
-
A carefully controlled and statistically qualified comparison. Matching caption counts addresses an important confound in comparisons between mixing and resampling. The factorial analysis and extensive follow-up checks help identify when their benefits differ. The paper also distinguishes equivalence from inconclusive results and reports multiplicity correction and exceptions to pre-registration, making its conclusions appropriately qualified.
-
Practical analysis of captioner selection and cost. The paper goes beyond performance comparisons by measuring selection accuracy and regret for several evaluation uses, as well as generation costs. It shows that the best captioner depends on the use and provides a substantial controlled corpus with a stated release plan, offering a useful resource for further study.
Critical weaknesses
- Limited generalizability. The controlled comparisons are informative, but all experiments use ImageNet-1k, and the main training evidence uses a frozen CLIP image encoder on 300k images. The small-scale image-encoder training and alternative-prompt checks leave it unclear whether the observed trade-offs extend to the broader recaptioning pipelines motivating the paper.
Questions for authors
- How strong is the evidence for a remaining resampling advantage under common token caps? The gap decreases from 3.0 points to 1.4 and 0.8 points at 64 and 48 tokens, respectively, while the seed-level intervals include zero. Please clarify how the class-bootstrap and seed-level uncertainty should inform the interpretation of these smaller gains.
- Does the mixing–resampling trade-off persist beyond ImageNet class-name classification? One representative comparison on another image–text dataset or a retrieval task would help assess whether the practical recommendation extends to broader recaptioning settings.
late reviews available
Dear authors and reviewers: Please note that late reviews have been received for this submission. The authors are welcome to write rebuttals by replying to this comment, if they are not able to do so in the regular way.