acceptodds
Under review as a conference paper at ICLR 2027

Self-Evolving Visual Questioner

Abstract

Vision-language models (VLMs) are typically trained as passive answerers, while their ability to actively ask diverse, non-trivial, and visually grounded questions remains underexplored. Existing visual questioners' performance is bottlenecked by the availability of high-quality training data and the cost of curating it. We show that a VLM can iteratively improve itself as a visual questioner without external teachers or new human annotations. We propose a self-evolving framework that uses a VLM itself as a proposer, rewriter, and filter to produce harder, more informative, and visually grounded questions, while maintaining diversity across visual intents. These questions are then used to train the VLM in both questioner and answerer modes, with the updated questioner proposing questions for the next round. To evaluate the questioner, we introduce an evaluation protocol that assesses questions along perception, reasoning, and diversity dimensions. Experiments across three backbone VLMs show that our method substantially enhances question quality, increasing the mean question-generation score by approximately 78% after two rounds. Under the same training-data budget, our self-supervision is more effective than training on static source annotations. Moreover, the self-evolving questioner preserves its answering ability.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
VLM, self-evolving

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1tXJOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether a vision-language model can improve its visual question-generation ability using its own supervision while preserving question-answering performance. An evolving model proposes questions, while the initial checkpoint rewrites and filters them. The retained pairs support joint QG/QA supervised fine-tuning. The paper reports three rounds on three Qwen backbones, component ablations, a one-round InternVL experiment, and downstream QA training with questions from the improved questioner. Evaluation combines four model-judged perception/reasoning dimensions with embedding-based question-set diversity.

Critical strengths
  1. The framework is coherent and reasonably complete. Rewriting, filtering, and dual-format supervision have clear roles, and the component ablations support their usefulness. Evaluation measures the trained model's raw questions rather than questions improved by test-time rewriting or filtering.

  2. The main experiments show substantial improvements over the base questioners across the reported QG dimensions, with broadly preserved average QA performance. The third-round results also expose diminishing gains rather than relying exclusively on a single adaptation round.

  3. The paper provides several useful validation checks: four-person human evaluation, image-level bootstrap variability, a frozen-versus-adapted rewriter comparison, one-round transfer to InternVL3.5-4B, cross-image-pool evaluation, and diversity measurements with multiple embedding models.

  4. The downstream supervision experiment provides practical supporting evidence. With the image source and answer-generation model held fixed, questions from the improved questioner yield higher average QA accuracy than questions from the base model.

Critical weaknesses
  1. Evaluator selection and reliability. The four-person human study in Table 12 is useful, and the reported GPT-human correlations provide a meaningful reference beyond the scores alone. However, the choice of GPT-5-mini as the main evaluator is not sufficiently motivated, and robustness to an independent judge, such as Gemini, remains unclear. The comparison also uses GPT versus mean human scores but mean pairwise correlations for human-human agreement, so these values should not be interpreted as a direct human-level performance comparison. Spatial reasoning retains a noticeable agreement gap. Could the authors explain the evaluator choice and, if feasible, check whether the main ranking is preserved by a second judge on a representative subset? This is a request to validate the specific judging task, not an assumption that one model family is universally the better judge.

  2. The relationship between rubric scores, empirical difficulty, and training value remains unclear. The rubrics describe visual inspection and reasoning requirements, but the paper does not establish whether higher scores correspond to questions that answering models actually find harder. Repeated rollouts from a fixed model, or answering success rates across several models with independently verified answers, would provide an empirical difficulty reference. The relationship with training benefit is also only partially supported: Table 7 compares two question collections and shows a useful downstream gain, but does not establish that higher rubric scores predict greater training benefit. These two relationships should be distinguished, and the interpretation of the scores should remain appropriately limited.

  3. The frozen-checkpoint ablation supports the rewriter choice but only partially explains the mechanism. Table 5 directly shows that the frozen base rewriter outperforms the adapted rewriter in the tested second-round Qwen2.5-VL-3B setting. This is valuable evidence. However, it does not isolate the role of freezing the filter or establish that the benefit specifically arises from maintaining a complementary generation distribution. Please clarify which remaining components are held fixed in this comparison and distinguish the observed checkpoint advantage from the broader explanation about avoiding coupling during self-evolution.

  4. The limits of iterative improvement need clearer characterization. The third-round results are informative, but QG averages are nearly flat after the second round, some dimensions decrease, and QA does not improve monotonically. What limits further evolution, and how should a practitioner decide when to stop? The current results support improvement over the base model and diminishing later gains, rather than indefinite improvement or an established convergence mechanism. Table 13 also does not cover the third round, and its bootstrap variability does not measure variation across training seeds or repeated judge calls. A more explicit discussion of the observed boundary would strengthen the central self-evolution claim.

  5. The potential tension between diversity and question quality is not directly examined. A given image may support only a small number of particularly valuable questions. Requiring different questions could eventually push generation toward less informative or less well-supported content. Table 16 usefully checks that the diversity trend is not specific to one embedding model, but this addresses measurement robustness rather than the quality-diversity trade-off. The InternVL diversity decrease and the third-round Qwen2.5-VL-7B decrease further suggest that the two objectives need not improve together. How does the method accommodate images with limited useful question space, and is there evidence that increasing diversity preserves per-question quality or training value?

  6. Cross-backbone generalization remains preliminary. The InternVL3.5-4B experiment in Table 14 is an important non-Qwen result, so the evidence is not confined entirely to Qwen. Nevertheless, it covers one round, yields a smaller QG-average improvement, and slightly reduces diversity; corresponding multi-round and QA-preservation results are not reported. The main evidence therefore still comes from three relatively small Qwen models. Broader replication, including a more recent comparable-size vision-language backbone, would clarify how far the method generalizes. I regard this as a limitation of the scope of the claim rather than a requirement to evaluate every recent model.

Questions for authors

See above

6 comments

We thank the reviewer for the detailed and constructive feedback. Our additional experiments strengthen the evidence in three areas: QG gains that persist under an independent judge, reproducible downstream training benefits, and a controlled demonstration of the evolving proposer's contribution. We also characterize later-round saturation and the relationship between diversity and quality.

W1. Evaluator selection, independent judging, and human agreement

An independent judge reproduces the main QG improvement, and calibration improves judge–human agreement across all four dimensions. The Gemini evaluation and the human-rating analysis provide complementary validation of the evaluation protocol.

Evaluator choice and independent validation. GPT-5-mini was selected for its cost and throughput at our evaluation scale. To assess whether our conclusions depend on this evaluator, we additionally evaluated 400 matched questions over the same 100 images using Gemini-2.5-Pro.

CheckpointSearchCoverageContextSpatialFour-dim mean
Base0.43000.64200.47400.28200.4570
R10.58200.65000.61200.46800.5780
R20.60200.67400.60600.48400.5915
R30.58800.67600.58800.47800.5825

All three trained checkpoints outperform Base under the independent judge, showing that the main improvement persists across judge families. Gemini assigns its highest score to R2, with R3 also remaining above Base, consistent with the later-round saturation observed in our main evaluation.

Human-rating calibration and absolute agreement. To further assess evaluator reliability, we calibrated GPT-5-mini's ratings against the mean human ratings using three methods: linear equating (matching the mean and standard deviation), ordinary least-squares regression, and equipercentile matching. We used leave-one-out calibration to avoid evaluating on the same items used to fit the mapping, and computed confidence intervals with 2,000 item-level bootstrap resamples, refitting the calibration within each resample.

Among the three approaches, linear equating consistently achieves the highest absolute agreement, measured by ICC(2,1).

DimensionUncalibrated ICCLinear-equated ICC [95% CI]Improvement [95% CI]
Visual Search0.5770.597 [0.47, 0.72]+0.021 [+0.007, +0.080]
Evidence Coverage0.6700.713 [0.61, 0.81]+0.043 [+0.024, +0.093]
Contextual Reasoning0.7630.792 [0.71, 0.88]+0.030 [+0.016, +0.073]
Spatial Reasoning0.4980.558 [0.37, 0.72]+0.060 [+0.032, +0.129]

Calibration improves absolute agreement across all four dimensions, with paired bootstrap confidence intervals excluding zero. The improvement is particularly relevant for spatial reasoning, where GPT-5-mini exhibits the largest initial positive scoring bias relative to the human mean (+0.53). Linear equating reduces this systematic mismatch, increasing spatial ICC from 0.498 to 0.558.

All reported QG benchmark results retain the original, uncalibrated evaluation protocol. The calibration analysis separately measures absolute agreement with human ratings. The original judge–human Spearman correlations are 0.570, 0.527, 0.637, and 0.473 across the four dimensions, providing a complementary measure of rank-based agreement.

Together, the independent-family evaluation reproduces the main QG improvement, and calibration improves absolute agreement with human ratings across all four dimensions. We report these complementary validation results in Tables 12–14 and Appendix G of the revised manuscript, with rank-based and absolute agreement reported separately.

W2. Rubric difficulty, empirical answering difficulty, and training value

Our additional experiments connect the rubric to empirical answering difficulty and demonstrate reproducible downstream training utility. We evaluate these relationships separately: answering success provides an empirical difficulty reference, and the three-seed QA study measures the training value of the generated question collection.

Empirical answering difficulty. To examine whether higher rubric scores correspond to more challenging questions, we evaluated 180 questions stratified by rubric score, with eight answering samples per question. For SmolVLM-256M and Qwen2.5-VL-3B (pooled), answering success decreases from 63.65% to 48.65% to 44.48% across increasing score strata (Spearman ρ = −0.235, image-clustered 95% CI [−0.379, −0.072]). This provides evidence that higher rubric scores are associated with greater empirical answering difficulty.

Downstream training utility. With GPT-5.4 held fixed as the answer generator, the improvement in Table 7 reproduces across three training seeds, increasing the five-benchmark average from 61.58 ± 0.24 to 62.89 ± 0.10. This provides direct evidence that the improved question collection yields a reproducible training benefit. The answering-success analysis and the training study thus provide separate empirical support for question difficulty and supervision utility.

We will clarify the distinction between rubric-based quality, empirical difficulty, and downstream supervision utility in the revised manuscript.

W3. Isolating the frozen mechanism and the proposer's contribution

The new proposer-only intervention complements Table 5 by isolating the value of updating the proposer while keeping refinement fixed. Table 5 establishes the advantage of the frozen base rewriter in the tested configuration; the additional experiment below directly tests the contribution of the evolving proposer.

To investigate this, we conducted an additional proposer-only intervention. Both conditions use the same image pool, static RAG bank, frozen base rewriter and filter, candidate budget, per-image training counts, and optimization steps. The only difference in data construction is the proposer checkpoint: the frozen base model M0 versus the updated model M1. Both conditions use the same M1 checkpoint for subsequent training initialization.

MetricFrozen proposerUpdated proposer
Search0.38130.3738
Coverage0.50110.5067
Context0.42070.5449
Spatial0.28400.2822
Diversity0.31270.3289
QG Avg. (5 dimensions)0.38000.4073

Updating the proposer improves the five-dimension QG average by +0.0273 (paired-image 95% CI [0.0170, 0.0375]), while diversity increases from 0.3127 to 0.3289.

The gain is driven primarily by contextual reasoning, with smaller increases in coverage and diversity. In this single-seed controlled experiment, updating the proposer improves aggregate QG quality under otherwise matched conditions.

Co-adapting the refinement components produces a different outcome. When rewriting and filtering are also updated, acceptance decreases from 14.71% to 8.75%, while the redundancy share of rejections increases from 39.3% to 64.8%.

Together, these experiments support two observations: updating the proposer improves overall question-generation quality when refinement remains fixed, whereas the tested co-adaptation strategy increases redundancy and reduces retention.

These findings support the design of an evolving proposer paired with frozen refinement and are consistent with complementary supervision as an explanation. The proposer-only intervention isolates the proposer checkpoint, while the co-adaptation experiment evaluates rewriting and filtering jointly. We will make these controlled components and their implications explicit in the revised manuscript.

W4. Saturation boundary and practical stopping criterion

The additional analysis characterizes a practical evolution boundary: substantial gains over Base followed by diminishing returns in later rounds. The extended evaluation and filtering logs provide concrete signals for deciding when to stop.

Extended uncertainty analysis. We have now added Third-round results to Table 15 in the revised manuscript, including image-level bootstrap standard deviations for all five QG dimensions across the three backbones. The updated results confirm that performance remains substantially above Base, while later-round changes are generally smaller and dimension-dependent.

BackboneR2 QG Avg.R3 QG Avg.R2 → R3
Qwen2.5-VL-3B0.500.51+0.01
Qwen2.5-VL-7B0.510.510.00
Qwen3-VL-4B0.570.58+0.01

To characterize the observed saturation, we additionally analyzed filtering behavior across rounds. On Qwen2.5-VL-3B, acceptance decreases from 15.94% to 14.71% to 10.86%, while the redundancy share of rejections increases from 30.1% to 39.3% to 44.7%. Meanwhile, M₀'s conditional rejection rate remains relatively stable (69.8%/69.5%/73.6%).

These observations suggest that increasing redundancy within the fixed image pool contributes to the declining supply of distinct accepted questions.

Practical stopping criterion. We propose monitoring acceptance and redundancy alongside held-out QG or QA performance. A sustained decline in acceptance, accompanied by increasing redundancy and flattening validation performance, provides an empirical signal that further rounds may offer limited benefits.

BackboneAcceptance R1 → R3R3 redundancy share
Qwen2.5-VL-3B15.94% → 10.86%44.7%
Qwen3-VL-4B17.63% → 11.32%34.43%

On both logged backbones, acceptance approaches approximately 11% by R3, coinciding with flattening held-out QG performance.

We emphasize that this value is an empirical reference point rather than a validated universal stopping threshold. The appropriate stopping criterion may depend on model capacity, image pool, and filtering configuration.

We will include the complete Third-round bootstrap results in Table 15 and clarify that these uncertainty estimates capture evaluation-sample variability, not training-seed or repeated-judge variability. We will present the proposed stopping criterion as a practical heuristic requiring calibration in new settings.

W5. Diversity–quality trade-off

Our per-image analysis shows that diversity and rubric-based quality improve together on most images over the full evolution horizon. We analyzed the two measures separately across the same 100 evaluation images, defining quality as the mean of the four rubric dimensions, excluding diversity.

  • Diversity and quality improve together on most images. From Base to R3, both diversity and quality increase on 79/100 images for Qwen2.5-VL-3B and 93/100 images for Qwen3-VL-4B. However, the relationship is less consistent between adjacent rounds, where diversity and quality sometimes move in opposite directions. Across the evaluated checkpoints and round-to-round changes, their per-image correlations remain relatively weak (|ρ| ≤ 0.28), indicating that the two objectives are not tightly coupled.

  • The filtering mechanism does not enforce diversity at the expense of quality. Each image receives a fixed budget of five seed proposals across different visual intents. The pipeline does not relax its acceptance criteria or generate additional candidates to meet a diversity quota. Images with fewer acceptable questions therefore contribute fewer training pairs, rather than forcing the inclusion of lower-quality candidates.

These results show that diversity and rubric-based quality can improve together under the evaluated settings, with joint improvements on most images from Base to R3. The per-image analysis captures variation between adjacent rounds, while the acceptance policy accommodates images with fewer useful questions. We will report these results alongside the separate correctness and downstream-utility evaluations in the revised manuscript.

Reviewer_xFVqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets visual question generation, where VLMs are usually trained only as answerers and existing questioners depend on human annotations, curated datasets, or stronger external models for supervision. Such supervision is costly to scale and tends to reinforce familiar question patterns and salient-object content. The authors propose a self-evolving loop in which the current model proposes questions under several visual intents, a frozen base model rewrites them into harder visually grounded questions and filters out ungrounded or non-improving pairs, and the retained pairs train the model in both question-generation and question-answering formats, with the updated model becoming the next proposer. The difference from prior work is that the questioner itself is the target of self-improvement, with no external teacher, new annotation, or reward model in the loop. The method is validated on three Qwen backbones with a new rubric-based evaluation of perception difficulty, reasoning difficulty, and question-set diversity, showing roughly 78% higher QG scores after two rounds with QA accuracy largely preserved, plus ablations on supervision format, rewriting and filtering, rewriter choice, and data source, a downstream QA training study, human agreement for the judge, and transfer to InternVL3.5 and RealWorldQA images.

Critical strengths
  1. The QG gains are large, consistent across three backbones of different strength, and come with no external teacher, annotation, or reward model. Base-to-Round-2 improvements of 0.21 to 0.25 in QG average are well above the reported bootstrap variability, so even with the evaluation caveats the main effect is not a marginal one, and the method is cheap enough to be reused.

  2. The matched-budget comparison in Table 6 is the most convincing experiment: with the same SAT images and 10K examples, refined self-supervision beats the original human annotations on both QG and QA, and the original annotations actually hurt QA. This shows the refinement step is adding real value rather than the gain coming from simply training on more data.

Critical weaknesses

Weaknesses

  1. The verifier is frozen throughout. Only the proposer Mt is trained; rewriting and filtering are always done by the base M0. This raises the question of how far iterative gains can continue when the model that selects supervision never improves, and Table 1 is consistent with early saturation: 3B goes 0.45 to 0.50 to 0.51 across rounds, and 7B stays at 0.51 in rounds two and three. Table 5 shows that simply swapping in M1 as rewriter hurts, so the paper offers no path to a co-evolving verifier. Reporting per-round filter acceptance rates, and testing whether a stronger external verifier recovers further gains, would help separate “the method saturates” from “M0’s judging saturates.”

  2. Evidence that harder questions make better supervision is thin. The rewriting prompts, the M0 filtering rules, and the GPT-5-mini evaluation rubric all share the same four author-defined difficulty dimensions, so higher QG scores show the model learned to match the rubric rather than that its questions are more useful in a broader sense. The only downstream check is Table 7, a 1.42-point QA gain concentrated on CVBench-3D with one benchmark decreasing. Appendix F validates the judge against humans, but no human check of the correctness or grounding of the retained training QA pairs is reported, even though M0 both writes and grades them.

  3. The comparison in Tables 2 and 8 has mismatched training conditions. LOVA3-7B and GPT-5.4 are prompted directly, while the proposed 3B questioner is fine-tuned on 10K examples constructed with prompts that target the evaluation dimensions. The result supports that the fine-tuned model scores higher under the authors’ protocol, not that it is a better questioner than GPT-5.4 in general. A fairer reference would be GPT-5.4 passed through the same rewriting and filtering step, or a baseline fine-tuned with a matched data budget.

  4. Some round-to-round claims outrun the reported uncertainty. Appendix G.1 states that several Round-1 to Round-2 differences are comparable to the bootstrap sampling variability and that the bootstrap excludes training-seed and judge variance, yet Section 4.2 still presents changes such as 3B diversity rising from 0.35 to 0.37 as improvements. The main Base-to-Round-2 gains are clearly larger than this noise; the smaller secondary claims in 4.2 and 4.3.3 should be stated as trends.

Minor

  1. The question proposal prompt in Figure 2 opens with topic and subtopic sampling instructions that make no reference to the image, followed by image-dependence constraints. Please clarify how these two parts interact and confirm that this is the prompt actually used.

  2. The filtering prompt in Figure 7 requires a JSON schema with three top-level fields, but the decision rules below it reference fields (“decision”, “single_focused_question”, “improves_visual_perception”, “improves_text_perception”, “improves_reasoning”) that are not in that schema; please clarify whether they are defined elsewhere. Model naming is also inconsistent (“Qwen3VL-4B” vs. “Qwen3-VL”).

Questions for authors

Clarification:

  1. What fraction of rewritten candidates survives filtering in each round, and does it drop across rounds?

  2. Has any human check been done on the correctness and image grounding of the retained training QA pairs, as opposed to the judge agreement study in Appendix F?

  3. Is Table 7 from a single training run? Please report seed variance, since the gain is driven mostly by CVBench-3D.

  4. Is the topic-sampling text at the start of Figure 2 the exact prompt used for all proposals? It never references the image.

  5. The decision rules in Figure 7 reference fields that are absent from the required JSON schema. Are they produced by a separate call or omitted from the figure?

Suggestions for future work:

  1. Running GPT-5.4 through the same rewriting and filtering step would make Table 2 more informative.

  2. Replacing M0 with a stronger verifier for one analysis round would show whether saturation comes from the method or from the frozen verifier.

4 comments

We thank the reviewer for the thoughtful assessment and for highlighting the value of the matched-budget comparison. Our additional experiments strengthen the evidence for useful self-generated supervision, reproducible downstream gains, and a measurable boundary to iterative improvement. Below, we present the new filtering analysis, three-seed replication, empirical difficulty evaluation, and stronger-verifier diagnostic.

W1. Frozen verifier and iterative saturation

The three-round filtering analysis reveals increasing redundancy across both tested backbones. We tracked acceptance rates, rejection reasons, and verifier behavior to characterize the diminishing returns observed in the main results.

BackboneAcceptance R1 / R2 / R3 (%)Redundancy share R1 / R2 / R3 (%)
Qwen2.5-VL-3B15.94 / 14.71 / 10.8630.1 / 39.3 / 44.7
Qwen3-VL-4B17.63 / 15.03 / 11.328.17 / 10.02 / 34.43

Increasing candidate redundancy. On both backbones, acceptance declines across rounds while the redundancy share of rejections increases. This pattern is consistent with the proposer increasingly generating questions similar to those already represented within the fixed image pool.

Stable verifier behavior. For Qwen2.5-VL-3B, M₀ rejects 69.8%, 69.5%, and 73.6% of candidates reaching its evaluation stage across R1–R3. Thus, the decline in overall acceptance is not accompanied by a comparable increase in its conditional rejection rate.

We additionally evaluated a variant in which both rewriting and filtering are updated alongside the proposer. Under this configuration, acceptance decreases to 8.75%, with redundancy accounting for 64.8% of rejections. This extends the frozen-rewriter comparison in Table 5, where using the base rewriter already outperforms the adapted rewriter (0.50 vs. 0.42 in QG Avg.).

These results suggest that increasing redundancy within the fixed image pool contributes to diminishing returns: the pattern appears across both backbones, and the tested co-adaptation strategy further reduces acceptance while increasing redundancy. The stronger-verifier diagnostic in Q7 provides an independent check of filtering behavior across rounds, complementing this analysis of candidate-pool saturation.

W2. Question difficulty versus useful supervision

The new experiments provide complementary evidence for question difficulty and supervision value. We examine rubric-based quality, empirical answering difficulty, and downstream learning benefit separately, with a human audit of the retained training pairs.

Rubric-based quality. Our four dimensions—visual search, evidence coverage, contextual reasoning, and spatial reasoning—represent the targeted visual-perception demands. The evaluation judge remains outside the training loop and provides no feedback to question generation or filtering.

Empirical answering difficulty. We evaluated 180 questions stratified by rubric scores, with eight answering samples per question. Answering success decreases from 63.65% to 48.65% to 44.48% across increasing score strata for two pooled answerers (Spearman ρ = −0.235, image-clustered 95% CI [−0.379, −0.072]). A Qwen2.5-VL-7B extension shows a similar overall direction (ρ = −0.144, 95% CI [−0.291, +0.010]), although its confidence interval includes zero.

These results establish an aggregate association between rubric scores and measured answering difficulty across the evaluated questions.

Downstream training utility. Table 6 shows that refined self-supervision outperforms the original annotations on both QG and QA under matched image and data budgets. The downstream benefit in Table 7 also reproduces across three training seeds, with positive paired gains of +1.39, +1.60, and +0.94 points (mean +1.31). These matched-budget and multi-seed results establish the practical training value of the improved question collection.

Human assessment of retained QA pairs. We manually reviewed 1,000 retained R1 QA pairs from the self-evolution pipeline, finding that 736/1,000 (73.6%) were judged acceptable. This audit directly measures the quality of the training supervision, complementing Appendix G's separate study of rubric-score agreement. The main self-evolution results and the matched-budget comparison in Table 6 establish the utility of the retained supervision. Separately, the three-seed experiment in Table 7 uses GPT-5.4-generated answers and demonstrates the downstream value of the improved question collection.

Together, the matched-budget comparison, empirical difficulty analysis, human audit, and seed replication provide converging evidence for the usefulness of the self-evolving questioner.

W3. Unequal adaptation in external-model comparisons

Self-evolution produces a strong direct questioner. In Tables 2 and 8, our adapted 3B model generates questions from a simple prompt without test-time rewriting or filtering, achieving higher QG averages than directly prompted GPT-5.4 and LOVA3-7B under the reported evaluation protocol. The models have different adaptation histories; these comparisons assess the resulting questioners' direct generation performance at deployment.

Table 6 provides the controlled comparison of supervision sources. With images and training-data budgets matched, refined self-supervision outperforms the original annotations on both QG and QA. This demonstrates that the refinement process produces more effective supervision under a fixed data budget, complementing the direct-generation comparisons in Tables 2 and 8.

Together, these experiments establish strong direct questioning performance and the training value of refined self-supervision under matched image and data budgets. We will make the complementary purposes of these comparisons explicit in the revision.

W4. Round-to-round improvements and statistical uncertainty

The main empirical finding is the substantial improvement over Base, followed by diminishing gains in later rounds. We have now extended the original Table 13 (Table 15 in the revised manuscript) to include third-round results and their image-level bootstrap standard deviations across all three backbones. The revision will distinguish these substantial Base-to-trained gains from smaller adjacent-round trends in Sections 4.2 and 4.3.3.

For example, on Qwen2.5-VL-3B:

DimensionR2R3
Visual Search0.49 ± 0.0120.48 ± 0.013
Contextual Reasoning0.64 ± 0.0140.64 ± 0.015
Spatial Reasoning0.40 ± 0.0160.46 ± 0.014
Diversity0.37 ± 0.0110.37 ± 0.010
QG Avg. (five dimensions)0.500.51

Although spatial reasoning increases from R2 to R3, the overall QG average changes only slightly, while visual search decreases and contextual reasoning and diversity remain approximately unchanged. Similarly, the Qwen2.5-VL-7B QG average remains at 0.51, while Qwen3-VL-4B increases slightly from 0.57 to 0.58.

The overall improvement over Base remains robust. The Base-to-R3 gain remains +0.0929 after score-blind question-length matching (95% CI [0.0644, 0.1206]). Under an independent-family judge, R3 also outperforms Base (0.5825 vs. 0.4570), although R2 scores higher than R3 (0.5915).

The results demonstrate sustained gains over Base and a later-round plateau, with variation across dimensions. We will report small adjacent-round changes as trends and label the bootstrap standard deviations explicitly as evaluation-sample variability. The three-seed downstream replication in Q3 separately assesses the reproducibility of the QA training benefit.

M1. Figure 2 and image-conditioned topic sampling

We thank the reviewer for identifying this ambiguity. Figure 2 shows the proposal prompt used in our experiments, with the image supplied separately as a multimodal input alongside the text prompt.

The initial topic and subtopic instructions encourage diversity, while subsequent constraints require generated questions to depend on the image. Both components operate on the same multimodal input rather than forming an image-independent question-generation procedure.

We will clarify the image input and the roles of the two instruction components in the figure caption.

M2. Figure 7 schema and model naming

We thank the reviewer for identifying the inconsistency between the displayed filtering prompt and the decision rules.

The prompt shown in Figure 7 of the original submission was carried over from an earlier pipeline version, in which the model directly generated accept/reject decisions through fields such as decision, single_focused_question, and the improves_* flags.

In the implemented pipeline, the filtering decision is applied programmatically based on the model's output, rather than directly generated through those legacy decision fields.

Figures 6–8 in the revised manuscript show the implemented validity and difficulty measurements and their output schemas, and explain how these outputs determine acceptance in code. We will also standardize model naming throughout the manuscript.

Q3. Training-seed variance in Table 7

The downstream gain reproduces across three training seeds. We extended Table 7 from one run per condition to three seeds (63, 123, 777) for both training recipes, yielding the following results.

BenchmarkBase questionsImproved questionsDifference (pp)
CV-Bench-2D68.32 ± 0.1468.56 ± 0.12+0.24
CV-Bench-3D67.92 ± 0.2574.44 ± 0.13+6.53
SparBench29.73 ± 0.2730.09 ± 0.18+0.36
VStarBench75.39 ± 0.9175.92 ± 0.52+0.52
RealWorldQA66.54 ± 0.2365.45 ± 0.27−1.09
Five-benchmark average61.58 ± 0.2462.89 ± 0.10+1.31

The overall gain reproduces across all three training seeds, with paired differences of +1.39, +1.60, and +0.94 points.

The strongest gain is on CV-Bench-3D (+6.53 points), with smaller gains on three other benchmarks and an approximately one-point decrease on RealWorldQA. The small observed standard deviations show that this benchmark-specific pattern is consistent across the tested seeds.

These results establish a reproducible downstream benefit from the improved question collection under the evaluated recipes. We will update Table 7 with the three-seed results and per-benchmark differences.

Q6–Q7. Suggestions for future work

We thank the reviewer for these constructive suggestions, which identify two valuable directions for extending the framework with stronger external models.

6. Applying the refinement pipeline to GPT-5.4 proposals. Passing GPT-5.4-generated questions through the same rewriting and filtering pipeline would provide an informative pipeline-level comparison. It would also test how much our refinement process benefits questions from a stronger external proposer, extending the direct question-generation comparison in Table 2. We view this as a useful direction for further evaluating the applicability of the refinement pipeline.

7. Exploring a stronger external verifier. We have taken an initial step in this direction by conducting an offline diagnostic using GPT-5.5. On the evaluated backbone, we sampled 150 candidates accepted and 150 rejected by M₀ at the difficulty gate in each round, then rescored them using the same prompts and decision rule.

RoundDecision agreement with GPT-5.5
R10.542
R20.557
R30.583

Agreement does not deteriorate across rounds. The observed R3–R1 difference is +0.041 (image-clustered 95% CI [−0.040, +0.121]), with the interval including zero.

M₀'s decisions retain predictive value under the external scorer. In R3, candidates accepted by M₀ are 16.7 percentage points more likely to pass the external gate than rejected candidates (95% CI [+5.1, +27.8]).

Together with the increasing redundancy rates in W1, these results support candidate-pool limitations as a contributor to saturation. Building on this diagnostic, integrating a stronger verifier into the self-evolution loop is a promising next step for directly testing whether additional gains can be recovered in later rounds. We appreciate both suggestions as concrete directions for extending the current study.

As the discussion period is coming to a close, we wanted to briefly follow up on our rebuttal. If any of your concerns remain, we’d be happy to clarify. If our response addresses your main concerns, we’d appreciate it if you would consider revisiting your rating. Thank you!

Reviewer_UvuWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Proposes a self-evolving visual questioner in which a VLM generates and refines its own questions. The current model (M_t) proposes questions, and a frozen base model (M_0) rewrites them to be more challenging and filters them to build training data. This data is used to train the model in both QG and QA formats, and the updated model becomes the question proposer for the next round. Introduces an evaluation protocol for question quality consisting of perception (Search, Coverage), reasoning (Context, Spatial), and diversity. Across three Qwen-family backbones, the average QG score improves by about 78% after two rounds, while QA performance is largely maintained.

Critical strengths
  • The problem formulation is novel: it treats the VLM as a questioner rather than a passive answerer, and designs a self-evolving loop without an external teacher by separating proposal (M_t) from rewriting and filtering (M_0).
  • Proposes an evaluation protocol that decomposes question quality into perception (Search, Coverage), reasoning (Context, Spatial), and set-level diversity, and partially validates judge reliability with human annotation (4 annotators, 100 questions).
Critical weaknesses
  • The training objective and evaluation criteria overlap. The attributes required by the rewriting prompt (multiple objects, implicit reasoning, spatial comparison) are exactly what the judge rubric rewards. It is unclear whether the model has improved its questioning ability or simply learned a style that matches the rubric.
  • There is no verification of the correctness of the generated QA pairs. Since M_0 handles both answer generation and filtering, the pipeline is subject to self-verification bias.
  • The effect of question length is not controlled for.
  • Training and evaluation are limited to spatially oriented data and benchmarks, so generalization to general VQA settings cannot be assessed.
Questions for authors
  • Are there any human verification results on whether the generated questions and answers are actually correct?
  • Is Fig. 2 the prompt that was actually used in the experiments?
4 comments

We thank the reviewer for the constructive feedback. We have added length-controlled evaluation, independent-family judging, human verification of retained QA pairs, and multi-seed downstream experiments. These results strengthen the evidence for both improved questioning ability and the practical value of self-generated supervision.

W1. Rubric self-alignment versus questioning ability

The QG gains are supported by evidence beyond the original rubric scores. They persist after question-length matching and under an independent-family judge, while higher-scoring questions are empirically harder to answer. These checks directly test the robustness and meaning of the measured improvement.

The evaluation judge is independent of the training loop. Question proposal, rewriting, filtering, and training receive no scores or feedback from the evaluation judge. Although the refinement and evaluation stages share conceptual quality dimensions, the questioner is not optimized against evaluator-specific outputs.

Empirical answering difficulty provides complementary evidence. To investigate whether higher rubric scores reflect more demanding questions rather than stylistic alignment alone, we evaluated 180 questions stratified by rubric score, with eight answering samples per question from two answerers. Answering success decreases from 63.65% to 48.65% to 44.48% across increasing score strata (Spearman ρ = −0.235, image-clustered 95% CI [−0.379, −0.072]).

These results connect higher rubric scores to greater empirical answering difficulty. Together with length matching and independent-family judging, they provide converging support for the improvement in questioning ability. The human review and downstream experiments in W2 further establish the quality and practical utility of the generated supervision.

W2. Correctness of self-generated QA pairs and self-verification bias

We directly assessed retained supervision through human review of 1,000 R1 QA pairs. Of these, 736/1,000 (73.6%) were judged acceptable. For open-ended answers, we considered a response correct when no identifiable error was found. This evaluates the training pairs themselves, complementing the judge-agreement study.

Reproducible downstream utility of the improved questioner. In the separate downstream study, both question collections are paired with answers from the same GPT-5.4 answer generator. The downstream QA improvement from using improved questions reproduces across three training seeds, increasing the five-benchmark average from 61.58 ± 0.24 to 62.89 ± 0.10, with positive gains in all three seed pairs.

Together, the main self-evolution results, human review, and downstream study demonstrate useful self-generated supervision and reproducible gains from improved questions. We will incorporate both the human verification and reproducibility results into the revised manuscript.

W3. Question-length confound

The QG gains persist in length-matched comparisons. We conducted score-blind, within-image word-length matching and recomputed the question-generation metrics on the matched subsets.

ComparisonImagesMatched pairsQG difference95% CI
Base → R160149+0.0918[0.0650, 0.1182]
Base → R267186+0.0924[0.0637, 0.1207]
Base → R366193+0.0929[0.0644, 0.1206]

All three improvements remain positive after length matching, with paired-image confidence intervals excluding zero. The conclusion also holds under a stricter ±2-word matching caliper.

These results indicate that the improvements on the matched subsets cannot be explained solely by increased question length. We will report matching coverage and clarify that this supplementary analysis uses nine questions per image, compared with five in the main evaluation protocol.

W4. Spatial scope of training and evaluation

Our evaluation covers visual perception beyond spatial reasoning, including visual search, counting, and evidence-based reasoning. Our framework targets these capabilities by generating questions that demand closer visual inspection.

We assess these capabilities through both question-generation evaluation and downstream QA.

Question-generation evaluation. The rubric covers visual search, evidence coverage, contextual reasoning, and spatial reasoning. Improvements in the four-dimension mean are also observed under an independent-family judge.

Downstream QA evaluation. We evaluate CV-Bench-2D, CV-Bench-3D, SparBench, VStarBench, and RealWorldQA separately. The three-seed replication shows a substantial improvement on CV-Bench-3D (+6.53 points), smaller improvements on three other benchmarks, and a decline on RealWorldQA (−1.09 points).

These results demonstrate benefits across the evaluated visual-perception tasks. The full downstream per-benchmark results, including gains and declines, are reported in Table 7. Results for the main self-evolution experiments and ablations are reported separately in Appendix F (Table 11). We will make this task scope explicit in the Introduction.

Q2. Is Figure 2 the prompt used in the experiments?

Yes. Figure 2 shows the text prompt used in our experiments. The image is supplied separately as a multimodal input alongside the prompt, rather than being explicitly referenced in its opening text.

The topic and subtopic instructions encourage proposal diversity, while subsequent constraints require questions to depend on the visual input, such that their answers would change or become indeterminable without the image.

We will clarify the multimodal input format and the relationship between these instructions in the figure caption.

The additional controls and experiments strengthen the main findings: the QG gains persist across length and judge checks, and the generated questions yield reproducible downstream benefits. We will include the human review and protocol clarifications in the revised manuscript.

As the discussion period is coming to a close, we wanted to briefly follow up on our rebuttal. If any of your concerns remain, we’d be happy to clarify. If our response addresses your main concerns, we’d appreciate it if you would consider revisiting your rating. Thank you!

I appreciate the authors' efforts to address my concerns and the additional experiments. While the length-controlled evaluation and independent judging substantially alleviate some concerns, I will maintain my original score.

The additional analyses strengthen the evidence for improved question generation, but they do not fully distinguish genuine questioning ability from alignment with the evaluation rubric. Furthermore, the human verification reveals remaining noise in the self-generated QA pairs, while their reliability across successive evolution rounds remains unclear. The generalizability beyond the evaluated visual-perception tasks also remains limited.

I appreciate the authors' revisions and would be happy to reconsider if I have missed or misunderstood any changes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.