- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies whether a vision-language model can improve its visual question-generation ability using its own supervision while preserving question-answering performance. An evolving model proposes questions, while the initial checkpoint rewrites and filters them. The retained pairs support joint QG/QA supervised fine-tuning. The paper reports three rounds on three Qwen backbones, component ablations, a one-round InternVL experiment, and downstream QA training with questions from the improved questioner. Evaluation combines four model-judged perception/reasoning dimensions with embedding-based question-set diversity.
Critical strengths
-
The framework is coherent and reasonably complete. Rewriting, filtering, and dual-format supervision have clear roles, and the component ablations support their usefulness. Evaluation measures the trained model's raw questions rather than questions improved by test-time rewriting or filtering.
-
The main experiments show substantial improvements over the base questioners across the reported QG dimensions, with broadly preserved average QA performance. The third-round results also expose diminishing gains rather than relying exclusively on a single adaptation round.
-
The paper provides several useful validation checks: four-person human evaluation, image-level bootstrap variability, a frozen-versus-adapted rewriter comparison, one-round transfer to InternVL3.5-4B, cross-image-pool evaluation, and diversity measurements with multiple embedding models.
-
The downstream supervision experiment provides practical supporting evidence. With the image source and answer-generation model held fixed, questions from the improved questioner yield higher average QA accuracy than questions from the base model.
Critical weaknesses
-
Evaluator selection and reliability. The four-person human study in Table 12 is useful, and the reported GPT-human correlations provide a meaningful reference beyond the scores alone. However, the choice of GPT-5-mini as the main evaluator is not sufficiently motivated, and robustness to an independent judge, such as Gemini, remains unclear. The comparison also uses GPT versus mean human scores but mean pairwise correlations for human-human agreement, so these values should not be interpreted as a direct human-level performance comparison. Spatial reasoning retains a noticeable agreement gap. Could the authors explain the evaluator choice and, if feasible, check whether the main ranking is preserved by a second judge on a representative subset? This is a request to validate the specific judging task, not an assumption that one model family is universally the better judge.
-
The relationship between rubric scores, empirical difficulty, and training value remains unclear. The rubrics describe visual inspection and reasoning requirements, but the paper does not establish whether higher scores correspond to questions that answering models actually find harder. Repeated rollouts from a fixed model, or answering success rates across several models with independently verified answers, would provide an empirical difficulty reference. The relationship with training benefit is also only partially supported: Table 7 compares two question collections and shows a useful downstream gain, but does not establish that higher rubric scores predict greater training benefit. These two relationships should be distinguished, and the interpretation of the scores should remain appropriately limited.
-
The frozen-checkpoint ablation supports the rewriter choice but only partially explains the mechanism. Table 5 directly shows that the frozen base rewriter outperforms the adapted rewriter in the tested second-round Qwen2.5-VL-3B setting. This is valuable evidence. However, it does not isolate the role of freezing the filter or establish that the benefit specifically arises from maintaining a complementary generation distribution. Please clarify which remaining components are held fixed in this comparison and distinguish the observed checkpoint advantage from the broader explanation about avoiding coupling during self-evolution.
-
The limits of iterative improvement need clearer characterization. The third-round results are informative, but QG averages are nearly flat after the second round, some dimensions decrease, and QA does not improve monotonically. What limits further evolution, and how should a practitioner decide when to stop? The current results support improvement over the base model and diminishing later gains, rather than indefinite improvement or an established convergence mechanism. Table 13 also does not cover the third round, and its bootstrap variability does not measure variation across training seeds or repeated judge calls. A more explicit discussion of the observed boundary would strengthen the central self-evolution claim.
-
The potential tension between diversity and question quality is not directly examined. A given image may support only a small number of particularly valuable questions. Requiring different questions could eventually push generation toward less informative or less well-supported content. Table 16 usefully checks that the diversity trend is not specific to one embedding model, but this addresses measurement robustness rather than the quality-diversity trade-off. The InternVL diversity decrease and the third-round Qwen2.5-VL-7B decrease further suggest that the two objectives need not improve together. How does the method accommodate images with limited useful question space, and is there evidence that increasing diversity preserves per-question quality or training value?
-
Cross-backbone generalization remains preliminary. The InternVL3.5-4B experiment in Table 14 is an important non-Qwen result, so the evidence is not confined entirely to Qwen. Nevertheless, it covers one round, yields a smaller QG-average improvement, and slightly reduces diversity; corresponding multi-round and QA-preservation results are not reported. The main evidence therefore still comes from three relatively small Qwen models. Broader replication, including a more recent comparable-size vision-language backbone, would clarify how far the method generalizes. I regard this as a limitation of the scope of the claim rather than a requirement to evaluate every recent model.
Questions for authors
See above
6 comments
W2. Rubric difficulty, empirical answering difficulty, and training value
Our additional experiments connect the rubric to empirical answering difficulty and demonstrate reproducible downstream training utility. We evaluate these relationships separately: answering success provides an empirical difficulty reference, and the three-seed QA study measures the training value of the generated question collection.
Empirical answering difficulty. To examine whether higher rubric scores correspond to more challenging questions, we evaluated 180 questions stratified by rubric score, with eight answering samples per question. For SmolVLM-256M and Qwen2.5-VL-3B (pooled), answering success decreases from 63.65% to 48.65% to 44.48% across increasing score strata (Spearman ρ = −0.235, image-clustered 95% CI [−0.379, −0.072]). This provides evidence that higher rubric scores are associated with greater empirical answering difficulty.
Downstream training utility. With GPT-5.4 held fixed as the answer generator, the improvement in Table 7 reproduces across three training seeds, increasing the five-benchmark average from 61.58 ± 0.24 to 62.89 ± 0.10. This provides direct evidence that the improved question collection yields a reproducible training benefit. The answering-success analysis and the training study thus provide separate empirical support for question difficulty and supervision utility.
We will clarify the distinction between rubric-based quality, empirical difficulty, and downstream supervision utility in the revised manuscript.
W3. Isolating the frozen mechanism and the proposer's contribution
The new proposer-only intervention complements Table 5 by isolating the value of updating the proposer while keeping refinement fixed. Table 5 establishes the advantage of the frozen base rewriter in the tested configuration; the additional experiment below directly tests the contribution of the evolving proposer.
To investigate this, we conducted an additional proposer-only intervention. Both conditions use the same image pool, static RAG bank, frozen base rewriter and filter, candidate budget, per-image training counts, and optimization steps. The only difference in data construction is the proposer checkpoint: the frozen base model M0 versus the updated model M1. Both conditions use the same M1 checkpoint for subsequent training initialization.
| Metric | Frozen proposer | Updated proposer |
|---|---|---|
| Search | 0.3813 | 0.3738 |
| Coverage | 0.5011 | 0.5067 |
| Context | 0.4207 | 0.5449 |
| Spatial | 0.2840 | 0.2822 |
| Diversity | 0.3127 | 0.3289 |
| QG Avg. (5 dimensions) | 0.3800 | 0.4073 |
Updating the proposer improves the five-dimension QG average by +0.0273 (paired-image 95% CI [0.0170, 0.0375]), while diversity increases from 0.3127 to 0.3289.
The gain is driven primarily by contextual reasoning, with smaller increases in coverage and diversity. In this single-seed controlled experiment, updating the proposer improves aggregate QG quality under otherwise matched conditions.
Co-adapting the refinement components produces a different outcome. When rewriting and filtering are also updated, acceptance decreases from 14.71% to 8.75%, while the redundancy share of rejections increases from 39.3% to 64.8%.
Together, these experiments support two observations: updating the proposer improves overall question-generation quality when refinement remains fixed, whereas the tested co-adaptation strategy increases redundancy and reduces retention.
These findings support the design of an evolving proposer paired with frozen refinement and are consistent with complementary supervision as an explanation. The proposer-only intervention isolates the proposer checkpoint, while the co-adaptation experiment evaluates rewriting and filtering jointly. We will make these controlled components and their implications explicit in the revised manuscript.
W4. Saturation boundary and practical stopping criterion
The additional analysis characterizes a practical evolution boundary: substantial gains over Base followed by diminishing returns in later rounds. The extended evaluation and filtering logs provide concrete signals for deciding when to stop.
Extended uncertainty analysis. We have now added Third-round results to Table 15 in the revised manuscript, including image-level bootstrap standard deviations for all five QG dimensions across the three backbones. The updated results confirm that performance remains substantially above Base, while later-round changes are generally smaller and dimension-dependent.
| Backbone | R2 QG Avg. | R3 QG Avg. | R2 → R3 |
|---|---|---|---|
| Qwen2.5-VL-3B | 0.50 | 0.51 | +0.01 |
| Qwen2.5-VL-7B | 0.51 | 0.51 | 0.00 |
| Qwen3-VL-4B | 0.57 | 0.58 | +0.01 |
To characterize the observed saturation, we additionally analyzed filtering behavior across rounds. On Qwen2.5-VL-3B, acceptance decreases from 15.94% to 14.71% to 10.86%, while the redundancy share of rejections increases from 30.1% to 39.3% to 44.7%. Meanwhile, M₀'s conditional rejection rate remains relatively stable (69.8%/69.5%/73.6%).
These observations suggest that increasing redundancy within the fixed image pool contributes to the declining supply of distinct accepted questions.
Practical stopping criterion. We propose monitoring acceptance and redundancy alongside held-out QG or QA performance. A sustained decline in acceptance, accompanied by increasing redundancy and flattening validation performance, provides an empirical signal that further rounds may offer limited benefits.
| Backbone | Acceptance R1 → R3 | R3 redundancy share |
|---|---|---|
| Qwen2.5-VL-3B | 15.94% → 10.86% | 44.7% |
| Qwen3-VL-4B | 17.63% → 11.32% | 34.43% |
On both logged backbones, acceptance approaches approximately 11% by R3, coinciding with flattening held-out QG performance.
We emphasize that this value is an empirical reference point rather than a validated universal stopping threshold. The appropriate stopping criterion may depend on model capacity, image pool, and filtering configuration.
We will include the complete Third-round bootstrap results in Table 15 and clarify that these uncertainty estimates capture evaluation-sample variability, not training-seed or repeated-judge variability. We will present the proposed stopping criterion as a practical heuristic requiring calibration in new settings.
W5. Diversity–quality trade-off
Our per-image analysis shows that diversity and rubric-based quality improve together on most images over the full evolution horizon. We analyzed the two measures separately across the same 100 evaluation images, defining quality as the mean of the four rubric dimensions, excluding diversity.
-
Diversity and quality improve together on most images. From Base to R3, both diversity and quality increase on 79/100 images for Qwen2.5-VL-3B and 93/100 images for Qwen3-VL-4B. However, the relationship is less consistent between adjacent rounds, where diversity and quality sometimes move in opposite directions. Across the evaluated checkpoints and round-to-round changes, their per-image correlations remain relatively weak (|ρ| ≤ 0.28), indicating that the two objectives are not tightly coupled.
-
The filtering mechanism does not enforce diversity at the expense of quality. Each image receives a fixed budget of five seed proposals across different visual intents. The pipeline does not relax its acceptance criteria or generate additional candidates to meet a diversity quota. Images with fewer acceptable questions therefore contribute fewer training pairs, rather than forcing the inclusion of lower-quality candidates.
These results show that diversity and rubric-based quality can improve together under the evaluated settings, with joint improvements on most images from Base to R3. The per-image analysis captures variation between adjacent rounds, while the acceptance policy accommodates images with fewer useful questions. We will report these results alongside the separate correctness and downstream-utility evaluations in the revised manuscript.
We thank the reviewer for the detailed and constructive feedback. Our additional experiments strengthen the evidence in three areas: QG gains that persist under an independent judge, reproducible downstream training benefits, and a controlled demonstration of the evolving proposer's contribution. We also characterize later-round saturation and the relationship between diversity and quality.
W1. Evaluator selection, independent judging, and human agreement
An independent judge reproduces the main QG improvement, and calibration improves judge–human agreement across all four dimensions. The Gemini evaluation and the human-rating analysis provide complementary validation of the evaluation protocol.
Evaluator choice and independent validation. GPT-5-mini was selected for its cost and throughput at our evaluation scale. To assess whether our conclusions depend on this evaluator, we additionally evaluated 400 matched questions over the same 100 images using Gemini-2.5-Pro.
All three trained checkpoints outperform Base under the independent judge, showing that the main improvement persists across judge families. Gemini assigns its highest score to R2, with R3 also remaining above Base, consistent with the later-round saturation observed in our main evaluation.
Human-rating calibration and absolute agreement. To further assess evaluator reliability, we calibrated GPT-5-mini's ratings against the mean human ratings using three methods: linear equating (matching the mean and standard deviation), ordinary least-squares regression, and equipercentile matching. We used leave-one-out calibration to avoid evaluating on the same items used to fit the mapping, and computed confidence intervals with 2,000 item-level bootstrap resamples, refitting the calibration within each resample.
Among the three approaches, linear equating consistently achieves the highest absolute agreement, measured by ICC(2,1).
Calibration improves absolute agreement across all four dimensions, with paired bootstrap confidence intervals excluding zero. The improvement is particularly relevant for spatial reasoning, where GPT-5-mini exhibits the largest initial positive scoring bias relative to the human mean (+0.53). Linear equating reduces this systematic mismatch, increasing spatial ICC from 0.498 to 0.558.
All reported QG benchmark results retain the original, uncalibrated evaluation protocol. The calibration analysis separately measures absolute agreement with human ratings. The original judge–human Spearman correlations are 0.570, 0.527, 0.637, and 0.473 across the four dimensions, providing a complementary measure of rank-based agreement.
Together, the independent-family evaluation reproduces the main QG improvement, and calibration improves absolute agreement with human ratings across all four dimensions. We report these complementary validation results in Tables 12–14 and Appendix G of the revised manuscript, with rank-based and absolute agreement reported separately.