- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a multimodal generative active learning setting in which the learner actively selects multimodal inputs whose ground-truth textual responses should be acquired under a limited annotation budget. To address the high computational cost of conventional generation-based uncertainty estimation, the authors propose an uncertainty-dynamics score computed from token-level entropy at two language-model layers during the prefill stage. The resulting score is used to construct a candidate set, followed by a diversity-aware coreset selection step based on multimodal hidden representations. The method is evaluated on VQAv2, ChartQA, and MMBench using SmolVLM-256M and, in an additional experiment, SmolVLM-2.2B. The reported results suggest improved annotation efficiency and reduced selection time compared with random, diversity-only, and generation-entropy-based uncertainty baselines.
Critical strengths
- Extending active learning from classification-oriented multimodal tasks to multimodal response generation is practically relevant, since acquiring high-quality free-form multimodal annotations can be substantially more expensive than collecting raw image--text inputs. The distinction between selecting conventional class labels and acquiring generative textual supervision is meaningful and could motivate further work in data-efficient multimodal learning.
- Estimating uncertainty during the prefill stage avoids autoregressive decoding for every unlabeled example, and the subsequent diversity selection is restricted to a reduced candidate pool. The runtime experiments indicate that this design can substantially reduce acquisition cost relative to a generation-based uncertainty baseline, which is important for active learning with multimodal language models.
Critical weaknesses
- The proposed ``uncertainty dynamics'' is not yet sufficiently justified as a measure of uncertainty for multimodal generation. The acquisition signal is constructed from the entropy of at two intermediate/deep layers during prefill, followed by a finite-difference approximation, $ u_i' = -\frac{U_i^{(l)} - U_i^{(l-h)}}{h}. $ However, these entropies are computed on representations of the observed input/prompt tokens rather than on the future autoregressively generated response. It is therefore unclear why this quantity should reliably capture uncertainty about the unknown ground-truth answer rather than difficulty in representing the input sequence itself. The reported correlation with average layer-wise entropy mainly establishes agreement with another internal entropy statistic; it does not establish that the score is better aligned with generation error, expected model improvement, or annotation utility. This is consequential because uncertainty dynamics is the principal methodological contribution of the paper.
- The main experiments compare against only three relatively simple baselines: random selection, k-center diversity selection, and a mean-token-entropy uncertainty baseline. The proposed method itself is essentially a combination of uncertainty-based screening and standard coreset selection, making comparison with stronger hybrid or batch active learning approaches particularly important. Moreover, the main results are averaged over only four runs, several improvements appear relatively small compared with the reported variation, and no statistical significance analysis is provided. The claim of up to annotation savings is based on matching performance across different acquisition iterations rather than on a statistically supported sample-efficiency criterion. Consequently, it is difficult to determine whether the observed gains reflect a robust methodological improvement or a favorable combination of the selected baselines and hyperparameters.
Questions for authors
- Can the authors provide evidence that the proposed prefill-based uncertainty-dynamics score is directly related to uncertainty of the generated answer? For example, how strongly does correlate with per-example generation loss, answer correctness, sequence-level entropy, or expected performance improvement after the example is annotated? Such an analysis would materially strengthen the interpretation of the proposed acquisition signal.
- Why is the average entropy across all layers treated as the reference ``full uncertainty dynamics''? Averaging layer-wise uncertainty removes the actual trajectory of uncertainty across depth and does not necessarily represent dynamics. Would ranking samples using the full vector or its slope/variation statistics lead to the same conclusions?
- Could the authors compare against stronger hybrid batch active-learning methods adapted to this setting, rather than only uncertainty-only and diversity-only baselines? In particular, it would be important to determine whether the improvement comes from the proposed layer-wise uncertainty signal or simply from combining uncertainty filtering with a standard k-center selection stage.
- Are the reported annotation savings statistically significant across random runs? Please report confidence intervals or appropriate paired statistical tests at matched annotation budgets. This is important because several differences between methods appear modest relative to the run-to-run variation.
4 comments
Response to Question 1: Thank you for raising this important question. Please refer to our response to Weakness 1, where we provide a detailed discussion from both motivational and empirical perspectives.
Response to Question 2: Thank you for raising this important question. As discussed in the main text, our algorithm aims to select samples with high uncertainty across layers. Motivated by this, we use the average entropy across all layers as a proxy for overall uncertainty dynamics and design our uncertainty-dynamics score to efficiently approximate it. We acknowledge that average entropy does not fully capture the layer-wise uncertainty trajectory. Investigating finer-grained dynamics, such as the rate of uncertainty changes across layers, is an important direction for our future work.
To further investigate whether alternative characterizations of layer-wise uncertainty dynamics lead to the same conclusions, we conducted additional experiments using slope and variation statistics for sample ranking on the VQAv2-6K dataset using the SmolVLM-256M model. The experimental results are reported below.
| Methods | Round 0 | Round 1 | Round 2 | Round 3 | Round 4 | Round 5 | Round 6 |
|---|---|---|---|---|---|---|---|
| Ours with slope | 0.56 | 0.56 | 0.56 | 0.57 | 0.60 | 0.63 | 0.71 |
| Ours with variation | 0.56 | 0.56 | 0.56 | 0.57 | 0.59 | 0.62 | 0.68 |
| Original Ours | 0.57 | 0.57 | 0.56 | 0.57 | 0.59 | 0.64 | 0.73 |
The results show that our original method achieves the better final performance, outperforming both slope-based and variation-based alternatives. Although these alternative statistics capture finer-grained uncertainty dynamics, they do not lead to better performance in our experiments. These findings support our design choice of using overall layer-wise uncertainty as an effective and computationally efficient proxy for sample selection. Nevertheless, more sophisticated characterizations of layer-wise uncertainty dynamics may further improve performance, which we leave for future work.
Response to Question 3: Thank you for raising this important concern. As discussed in Section 2.2, existing hybrid AL methods, such as BADGE, rely on classification-oriented gradient representations, making their direct adaptation to multimodal generation non-trivial. Our work aims to establish an initial hybrid acquisition baseline for this setting.
For the empirical perspective, we have already investigated whether the improvements arise merely from combining uncertainty filtering with k-center selection. Specifically, setting (=0) in Section 4.3.5 provides a standard hybrid baseline that retains uncertainty filtering and k-center selection while excluding the proposed layer-wise uncertainty dynamics signal. As shown in Figure 6 (in our manuscript), incorporating this signal improves performance from 0.70 to 0.73, providing empirical evidence of its additional contribution beyond the hybrid combination itself.
Response to Question 4: Thank you for raising this important question. Please refer to our response to Weakness 2, where we provide a detailed discussion supported by additional statistical significance analyses.
Dear Reviewer R6RY,
Thank you for your valuable feedback and thoughtful suggestions. We have submitted our rebuttal and provided further clarifications in response to your concerns.
As the discussion period is coming to a close, we would be grateful if you could revisit our responses and share any remaining concerns. We hope that our clarifications and additional evidence have adequately addressed your questions, and we would sincerely appreciate your reconsideration of the evaluation and scores if you find our responses satisfactory.
Thank you again for your time and consideration. We truly appreciate your efforts in reviewing our work!
I thank the authors for their constructive response and additional experiments, which have improved the empirical presentation, especially the confidence-interval and paired-comparison analyses. However, the main concerns remain insufficiently addressed: the prefill-based uncertainty-dynamics score is still not directly validated against generation loss, answer correctness, sequence-level entropy, or per-example expected improvement; the justification for average layer-wise entropy as a proxy for “full uncertainty dynamics” remains limited; stronger adapted hybrid/batch active-learning baselines are missing; and the statistical analysis covers only a narrow set of comparisons and runs. I therefore recommend keep the original score.
Dear reviewer,
We greatly appreciate your thoughtful feedback and constructive suggestions. Below, we provide detailed responses and clarifications. If our responses address your concerns, we would be grateful if you would consider updating your scores. Please let us know if further clarification is needed.
Response to Weakness 1: Thank you for this insightful comment. From an algorithmic perspective, our method is not designed to establish a direct connection between the uncertainty-dynamics score and the quality of future autoregressively generated responses. Instead, it is motivated by the trade-off between computational efficiency and final performance. By focusing on signals available during the prefill phase, we may sacrifice some information available during generation in exchange for substantially improved efficiency while maintaining strong performance.
From an empirical perspective, we further investigate the relationship between our uncertainty-dynamics score and the expected performance improvement from annotating each example, despite relying solely on prefill-phase information during data selection. Specifically, we conduct experiments on ChartQA-8K using SmolVLM-256M. Starting from the same model checkpoint, we use our method and Uncertainty_gmtes to select samples independently, fine-tune the model on the selected samples, and compare the resulting models' performance on the same validation set. Although the ideal setup would involve selecting and training on a single sample at a time, the performance changes induced by individual samples can exhibit substantial variance. Therefore, to obtain more stable and reliable results, we select 16 samples per method in each experiment. (At Step 1, we select 16 samples to fine-tune the model, while at Step 2, we select an additional 16 samples and continue fine-tuning from the model obtained at Step 1.) The results are presented below:
Conceptually, our method selects the sample with the highest uncertainty-dynamics score (si) for model fine-tuning, which we extend to the top 16 samples in our experiments. If si is positively correlated with the expected performance improvement after annotation, the selected samples should, on average, yield greater improvements in validation performance. The results show that our method yields greater performance gains than the uncertainty-based baseline, suggesting that our score better captures potential training utility.
Response to Weakness 2: Thank you for raising this important point. As discussed in Section 2 of our manuscript, existing hybrid active learning methods cannot be directly applied to our multimodal generative setting; to the best of our knowledge, our method is among the first hybrid approaches developed for this setting. Furthermore, in Section 4.3.5, we introduce an additional hybrid baseline that combines uncertainty-based screening using only final-layer uncertainty with diversity-based selection, and our method consistently outperforms this baseline, demonstrating the benefit of incorporating layer-wise uncertainty dynamics.
To quantify the variability across random runs, we additionally report 95% confidence intervals (CIs) for all methods on MMBench at matched annotation budgets over four runs, as shown below. Results are presented as a mean ± 95% CI margin of error.
Furthermore, to examine the reported 33.33% annotation savings, we conduct a paired statistical comparison across four matched random runs. Our method achieves a mean score of 0.703 using only 5% of annotated data, outperforming Uncertainty_gmtes (0.690) using 7.5%. The mean paired difference is 0.0133, with a 95% CI of [0.00003, 0.02661], excluding zero. This indicates a statistically significant performance improvement at the 5% level for this comparison, providing further evidence of the annotation efficiency of our method.