acceptodds
Under review as a conference paper at ICLR 2027

VisualReasoningGym: Adaptive and Verifiable Datasets for Compositional Reasoning Across Scientific and Natural Images

Abstract

We experimentally identify a structural limitation of static visual datasets for training multimodal reasoning models for compositional reasoning with RLVR. Static datasets bind images and annotations to fixed questions whose complexity and learning value varies across model families and can decline during training. Yet the same images and annotations can support new questions that recover learning signal and increase the complexity of the visual reasoning task. We therefore argue that visual reasoning datasets should be more effectively represented as a generative process over the images, annotations, and reasoning operations, with questions generated at training time to match the policy, and at evaluation time at increasing difficulty until the model breaks. Building on these findings, we introduce VisualReasoningGym, a framework that repurposes existing static visual datasets and its metadata into compositional visual reasoning programs that factorizes a question into image dependent grounded atomic skills and a verifiable reasoning program. VisualReasoningGym samples multiple atomic skills per image as individual questions and composes their answers using parametric reasoning programs, ranging from aggregation and affine expressions to shortest-path and combinatorial-optimization problems. This decomposition provides exact final-answer verification and exposes intermediate outcomes. Rollout feedback adjusts compositional difficulty to the policy’s evolving capabilities. We evaluate VisualReasoningGym as (1) a source of on-policy RL training data and (2) as benchmark for measuring compositional proficiency: the mean number of consecutive difficulty levels passed by a model. Continuing RLVR from Qwen3-VL-8B raises held-out pass@5 compositional proficiency from 4.55 to 6.19 of 8 levels and accuracy by +22.3 points, approaching the compositional proficiency of frontier models evaluated zero-shot on the same in-distribution benchmark. Training also notably improves average performance on external visual reasoning benchmarks across Qwen3-VL and Nemotron-Omni, demonstrating notable positive transfer.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reasoning, Vision Environments, VLM

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_k7NsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces VisualReasoningGym, an adaptive and verifiable data generation framework for multimodal compositional reasoning. The authors argue that traditional visual question-answering datasets rely on fixed image-question-answer triplets, whose effectiveness in providing meaningful reinforcement learning signals may diminish as model capabilities evolve.

To address this issue, VisualReasoningGym transforms existing images and annotations into environments that dynamically generate new questions. Specifically, the framework constructs visually grounded atomic questions from images and their metadata, and then combines these questions into more complex reasoning tasks using executable symbolic programs. Furthermore, the framework dynamically adjusts task difficulty based on the model's rollout performance to maintain effective training signals for reinforcement learning with verifiable rewards (RLVR).

The authors also introduce VisualGym-Ladder, a benchmark that evaluates compositional reasoning capabilities across eight progressively increasing difficulty levels. Experimental results demonstrate substantial improvements in the compositional reasoning performance of Qwen3-VL-8B-Thinking, along with positive transfer to several external visual reasoning benchmarks.

Critical strengths
  • The paper identifies an important issue in RLVR training, where static visual question-answering datasets may gradually lose their effectiveness in providing meaningful learning signals. The authors further investigate this phenomenon across different models and training stages, making the research motivation practically relevant.

  • By decomposing existing visual data into atomic questions and recombining them, the proposed framework can generate reasoning tasks at different difficulty levels without requiring additional image generation. This improves the utilization of existing visual datasets and annotations.

  • The proposed method achieves substantial improvements on VisualGym-Ladder and demonstrates positive transfer to several external visual reasoning benchmarks. The ablation studies and experiments across different models further support the effectiveness of the approach.

Critical weaknesses
  • Insufficient discussion of related work. The paper lacks a comprehensive discussion of closely related studies, including COGS [1], Composition-RL [2], SynthRL [3], and DIVA-GRPO [4]. These works are closely related to the proposed approach in terms of compositional question generation, verifiable data synthesis, or adaptive reinforcement learning.

  • Unclear novelty relative to existing methods. Long Grounded Thoughts [5] has already explored generating verifiable compositional visual reasoning problems from existing images. Although this work is cited, the paper lacks an in-depth comparison with it. The authors should more clearly distinguish their technical contributions from existing approaches and consider discussing Bee [6] and other relevant studies on multimodal data construction.

  • Insufficient validation of automatically generated data quality. The atomic questions rely on VLM-generated metadata and automatic annotations, which may introduce incorrect supervision signals. In particular, converted questions are retained if at least one out of five verification attempts succeeds. The authors are encouraged to provide human evaluation of annotation quality and ablation studies using stricter filtering criteria.

  • Limited evaluation of compositional generalization. The training and evaluation stages use the same families of symbolic reasoning programs, making it unclear whether the model can generalize to unseen compositional patterns. Furthermore, the current experiments do not sufficiently disentangle improvements in atomic visual perception from improvements in symbolic reasoning capabilities.

[1] Gu et al. Composition-Grounded Data Synthesis for Visual Reasoning. ICLR 2026. https://proceedings.iclr.cc/paper_files/paper/2026/hash/4733cee6af505d5e2782a1dc499c0d7e-Abstract-Conference.html

[2] Xu et al. Composition-RL: Compose Your Verifiable Prompts for Reinforcement Learning of Large Language Models. arXiv:2602.12036, 2026. https://arxiv.org/abs/2602.12036

[3] Wu et al. SynthRL: Scaling Visual Reasoning with Verifiable Data Synthesis. Findings of ACL 2026. https://aclanthology.org/2026.findings-acl.2107/

[4] Gao et al. DIVA-GRPO: Enhancing Multimodal Reasoning through Difficulty-Adaptive Variant Advantage. ICLR 2026. https://proceedings.iclr.cc/paper_files/paper/2026/hash/3adaa3ccd68cf934455f871f95cc95de-Abstract-Conference.html

[5] Acuna et al. Long Grounded Thoughts: Synthesizing Visual Problems and Reasoning Chains at Scale. ICML 2026. https://proceedings.mlr.press/v306/acuna26a.html

[6] Zhang et al. Bee: A High-Quality Corpus and Full-Stack Suite to Unlock Advanced Fully Open MLLMs. ICLR 2026. https://proceedings.iclr.cc/paper_files/paper/2026/hash/171291d8fed723c6dfc76330aa827ff8-Abstract-Conference.html

Questions for authors

same with weaknesses.

3 comments

Thank you for the supportive review and for the references; we will add them to the related work.

Related work and contribution

  • Long Grounded Thoughts (Acuna et al., 2026) also uses grounded image metadata, then asks an LLM to merge multiple-choice questions and generate composed questions and answers. We represent atomic answers as integers, compute composed targets with executable programs, and adjust composition size from rollout feedback. Further, LGT does not dynamically adapt the question complexity to the learner during training.
  • COGS (Gu et al., 2026) recombines perception and reasoning factors and studies intermediate rewards. Our framework combines executable program composition, final-answer supervision, and rollout-based control of composition size.
  • Composition-RL shares our use of verifiable prompt composition and a curriculum. Our setting differs in that the atomic questions are grounded in an image and generated from its stored metadata.
  • SynthRL builds harder variants that preserve seed answers. We generate composed targets by executing programs over atomic answers.
  • DIVA-GRPO adapts question variants and modifies advantage estimation. We share its adaptive generation goal, while controlling composition size through the number of atomic inputs. Our contribution is in data generation: we retain the existing GRPO objective and advantage estimator.
  • Bee studies multimodal corpus construction and curation. Our work is complementary: we reuse existing images and metadata to generate verifiable RL tasks at different composition sizes. Bee could also provide source material for our metadata extraction and task generation.

Our contribution is the finding that fixed visual questions can lose training value, together with a reusable generator that adapts to the base model and to the same policy as it improves. The adaptation takes place in the generated tasks, with the RL update rule unchanged. We evaluate this design through adaptive training, composition ladders, and transfer to external benchmarks. We will expand the related-work discussion to make these connections and distinctions explicit.

Data quality

Following your suggestion, we ran a human audit; the results are in the general response above. Removing the five affected image ladders produces negligible changes in pass@5 proficiency and preserves the ordering of the five reported models. We will filter the final release using human annotations. The one-of-five conversion rule is a screening threshold. Our filtering check tests evaluation robustness; we have not retrained with a stricter conversion filter but will add this to the final version.

Generalization and the source of improvement

We evaluate transfer beyond the generated tasks: performance improves in 11 of 14 external settings, with the mean rising from 52.69% to 56.34%. Within composed D1–D7 responses, atomic accuracy on matched extracted answers rises from 78.6% to 88.6%. These results support better intermediate answers and transfer to external benchmark tasks, which use their own questions and answer formats. The original 100-image VisualGym-Ladder holds out images within known program families; we did not hold out program families, so the external benchmarks are our evidence for transfer beyond them.

Thanks for the rebuttal. My main remaining question is whether the gains extend to program families unseen during training. The improvement on atomic questions is helpful to see, but it is still unclear how much of the gain comes from better visual perception and how much comes from better composition. The human audit addresses the quality of the evaluation labels, while the effect of stricter filtering on training remains open.

Thank you for the follow-up. All three points are fair. We address each one below in details.

1. Unseen program families. This is a good idea. We have not tested this in-domain. Training and the ladder use the same program families, and the ladder holds out images, not families. We will add this experiment to the final version. That said, the external benchmarks are popular multimodal benchmarks not generated by our programs, and 11 of 14 improve, including CharXiv (+3.24), ChartQAPro (+2.23) and BabyVision (+4.82).

2. Perception versus composition. Our evidence points to both, but it does not separate them exactly. We split the saved VisualGym-Ladder responses (Table 1 runs, D1–D7, no new inference) by program. Accuracy in %:

ProgramLaddersBase+ Text-only+ D0-only+ OursOurs − Base (95% CI)
Sum, min, max1269.574.876.090.2+20.7 (6.4 to 37.1)
All other programs8842.847.946.467.0+24.2 (19.3 to 29.3)
  • Perception: single questions (D0) improve from 77.4% to 89.4%, and atomic answers inside composed responses improve from 78.6% to 88.6%.
  • Composition: on the other programs Base scores 42.8% and our model reaches 67.0%, while training on symbolic questions without images (Text-only) or on single visual questions (D0-only) reaches only 47.9% and 46.4%.

An exact split needs an evaluation with the correct sub-answers given as text. We will add it to the final version.

3. Stricter filtering for training. This is still open. The training labels come from the same pipeline as the audited labels (2.17% marked incorrect). As stated in the paper (Appendix B.2), converted source questions (except DeltaPrompts) are kept only if an independent model answers them correctly in at least one of five trials. We will retrain with a three-of-five screen and report the result.

Reviewer_MDZ5OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VisualReasoningGym, which converts existing natural and scientific images and their annotations into dynamically generated visual reasoning tasks. It composes atomic visual questions, including counting, spatial relations, and OCR, through executable programs and adjusts difficulty according to model performance. The authors also introduce VisualGym-Ladder to measure consecutive success across difficulty levels. Experiments show improvements on the proposed benchmark and most external evaluation settings. However, the evidence does not sufficiently distinguish contributions from visual understanding, symbolic computation, and curriculum adaptation.

Critical strengths
  1. The motivation is clear and empirically supported. Cross-model comparisons, training-time tracking, and question regeneration show that the usefulness of fixed questions depends on the current policy and can decline during training. These analyses provide a concrete motivation for dynamic generation beyond simply increasing dataset size.
  2. The framework offers a practical way to reuse existing visual resources. Separating atomic visual tasks from symbolic programs allows one image to support multiple compositions with automatically computed targets. This design supports data reuse, controlled task construction, and systematic investigation of compositional training.
  3. The main model achieves sizeable gains, with evaluation extending beyond one model and benchmark. Qwen accuracy increases from 49.95% to 72.25% on VisualGym-Ladder, while the mean across 14 external settings rises from 52.69% to 56.34%. Comparisons include D0-only training, source-task RLVR, and a manual curriculum, alongside an extension to Nemotron.
Critical weaknesses
  1. Final-answer supervision can conceal visual errors, leaving the source of improvement unclear. The authors need to establish that gains reflect more accurate visual understanding and information integration, rather than error cancellation, guessing, or improved computation alone, and compare fairly against direct visual supervision. Counting errors can cancel under summation, while modulo operations can map different atomic answers to the same output. Correct final answers therefore do not establish correct visual subtask execution. A matched comparison should evaluate atomic-answer rewards, final-answer rewards, and joint rewards, and report how often correct final answers contain incorrect atomic answers.
  2. Exact verification remains conditional on insufficiently validated visual labels. Executable programs can compute targets exactly from supplied atomic answers, but cannot establish whether those answers match the image. Count agreement between a VLM and SAM, together with retaining converted questions after one successful verification out of five trials, does not establish annotation reliability across task types. The authors should conduct stratified human audits, report atomic and final-label error rates, and verify the main findings on a corrected test set.
  3. The controls do not fully isolate the contribution of policy adaptation. D0-only training changes both composition and difficulty, while the manual curriculum represents only one predefined schedule. Matching training steps and hyperparameters also does not ensure equal generated-token costs. Using the same generator, the authors should compare fixed high difficulty, uniform difficulty mixtures, and a tuned manual curriculum under both equal updates and equal total generated-token budgets.
  4. The validity of the proposed proficiency measure requires further examination. Increasing atomic-question count does not necessarily increase reasoning depth: some operations can ignore inputs or compress the answer space, while longer tasks can increase output truncation. Figure 5 explicitly includes token-limit failures. Consecutive-level proficiency therefore combines visual recognition, computation, output-budget effects, and sensitivity to early failures. The authors should report per-level results, computation performance with oracle atomic values, and token-budget sensitivity. With only 100 test images, confidence intervals should also resample complete image ladders.
Questions for authors
  1. Does final-answer supervision reliably reflect improvements in visual understanding?
  2. How reliable are the automatically generated visual labels?
  3. How much of the improvement comes from the adaptive curriculum itself?
  4. Does the proficiency metric validly measure compositional reasoning ability?
5 comments

Visual understanding

“Does final-answer supervision reliably reflect improvements in visual understanding?”

We thank the reviewer for this great question. To answer directly, we ran a new experiment described below based on your feedback. The experiment shows clear evidence that training with final-answer supervision helps with the visual atomic questions. When comparing the exact same questions across both models, atomic accuracy (which requires visual understanding) increased from 78.6% to 88.6%. This points to better visual understanding overall, even if getting the final answer right doesn’t always guarantee the intermediate steps are correct.

Experimental Setting: We directly examined intermediate answers in 7,000 saved D1–D7 responses from the base and our models, covering 100 images and five responses per level per model. GPT-5.4-mini extracted stated atomic answers without access to gold labels, with supporting quotes required. We then scored them against the saved atomic labels.

Metric (%)BaseAdaptive
Atomic accuracy on matched extracted answers78.688.6
Atomic answers extracted (%)92.495.6
Correct final responses with an observed atomic mistake (at least)42.528.1

Across 18,609 matched atomic answers extracted for both models, accuracy improves by 10.0 percentage points (paired image-bootstrap 95% CI: 8.1 to 11.9). The last row counts 685/1,611 and 687/2,443 correct final responses; these fractions are lower bounds because some atomic answers remain unknown. We therefore observe better intermediate answers under final-answer supervision, while confirming that correct final answers can conceal atomic mistakes. Some atomic tasks also involve arithmetic. This analysis measures the learned answers; a matched comparison of atomic-only, final-only, and joint rewards remains unreported.

Annotation quality

“How reliable are the automatically generated visual labels?”

To answer this question, we conducted a human-audit. We report the human-audit results and the filtered-ladder comparison in the general response above. We use the same exclusion rule and the same retained images for every model. Notably, only 2.17% of reviewed questions were marked incorrect by the annotators, showcasing the quality of our data.

The human audit supplements the VLM/SAM agreement and one-of-five conversion screens. It checks atomic labels and supports the comparison on the filtered set; per-task and separate final-label error rates are not measured here. Executable programs compute targets exactly from the supplied atomic labels.

Adaptive curriculum

“How much of the improvement comes from the adaptive curriculum itself?”

Both runs use VisualReasoningGym tasks. Table 1 reports:

ScheduleAccuracy (%)Pass@5 proficiency
Manual65.655.74
Adaptive72.256.19

The adaptive configuration gains 6.60 percentage points in accuracy and 0.45 levels in proficiency over the manual configuration. This supports the tested adaptive training strategy.

Proficiency and per-level results

“Does the proficiency metric validly measure compositional reasoning ability?”

Pass@5 proficiency counts consecutive levels passed from D0, with a level passed when at least one of five responses is correct. D0 contains one atomic question; D1–D7 compose three to nine. We use this metric to summarize sustained success along the ladder under fixed decoding settings. The token-limit failures in Figure 5 illustrate why proficiency depends on the decoding budget. It combines visual answers, computation, and response completion, and stops at the first failed level. An early failure can therefore lower proficiency even when later levels are solved. Composition size need not imply strictly increasing program depth or difficulty. We will clarify this definition in Section 3.2.1 and report per-level accuracy alongside proficiency.

Below, we additionally evaluate accuracy at every level. Notably, accuracy improves at every tested level:

LevelBase accuracy (%)Adaptive accuracy (%)
D077.489.4
D162.679.0
D251.072.2
D346.470.0
D443.473.2
D541.269.2
D639.663.8
D738.061.2

New Larger Evaluation Set

To address your concern about the 100-image evaluation, we also report results on VisualGym-Ladder Large (1,000 images) in the general response. On this set, adaptive training improves accuracy from 53.11% to 70.97% and pass@5 proficiency from 4.68 to 6.07. For the original 100-image VisualGym-Ladder, we resampled whole image ladders, pairing the models and retaining all eight levels and five responses per level. Across 10,000 bootstrap samples, the accuracy gain is 22.30 percentage points (95% CI: 18.05 to 26.75), and the pass@5 proficiency gain is 1.64 levels (95% CI: 1.02 to 2.28). These intervals measure image-sampling uncertainty for the saved runs. Missing final answers fall from 23.73% to 3.55%, showing improved response completion alongside the accuracy gain; this count does not identify every case as token-limit truncation.

Hi thank you for your time and thoughtful feedback. We have submitted our rebuttal addressing your comments and would appreciate it if you could review our responses when you have a chance. Thank you again for your consideration.

Reviewer_24oyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces VisualReasoningGym for generating adaptive visual reasoning tasks from existing images and annotations. It combines atomic visual questions through verifiable symbolic programs. Task difficulty increases based on the model's performance during RL training. The authors also introduce VisualGym-Ladder to evaluate compositional proficiency.

Critical strengths

The paper provides empirical evidence that static visual questions lose their learning value across models and during RL training. Which is an interesting observation.

Critical weaknesses
  1. Compositional visual reasoning? The method combines answers from independent, sometimes unrelated visual questions through symbolic operations. Increasing the number of questions may simply accumulate perceptual errors rather than require deeper visual reasoning. I think the authors should really understand what does compositional visual reasoning stands for, first.

  2. Unreliable automatically generated annotations. Object counts are accepted when GPT-5.5 predictions agree with SAM 3 masks, but agreement between automated models does not guarantee correctness. Authors should provide systematic human validation.

  3. Limited generalization beyond predefined task templates.

  4. Insufficiently controlled baselines. Compositional training exposes the model to multiple atomic visual questions per prompt, whereas D0-only training uses one. Matching training steps does not necessarily match the amount of visual supervision or computation. Difficult to isolate the benefit of composition.

  5. Potential confounding from format learning. Figure 5 shows baseline responses that fail to produce final answers before reaching the token limit. Some improvements may come from more efficient response generation rather than stronger visual reasoning. given the question are just symbolic operation of several answers of different questions, i don’t see this efficient response useful to other question domain.

Questions for authors

Can the authors provide some references to prior work that defines compositional visual reasoning similarly, particularly as combining answers to independent visual subquestions?

10 comments

Definition of composition and reference to previous work

“Compositional visual reasoning?”

We refer to compositional visual reasoning, as explicitly defined in Section 3 (pp. 3–4).

problems whose answer requires solving several grounded visual sub-questions on a single image and combining their answers through a reasoning operation to get to the final answer. The sub-questions may be independent given the image. The composition lies in the reasoning operation over their answers, which the model must carry out without access to the intermediate values.

The definition states that the sub-questions may be independent. The composition is the reasoning operation over their answers. For example, counts of three object categories become inputs to a sum.

On prior work: Long Grounded Thoughts (Acuna et al., 2026) and COGS (Gu et al., 2026) provide closely related precedents. Additionally, CLEVR (Johnson et al., 2017), a diagnostic dataset for "compositional language and elementary visual reasoning", includes questions that compare the counts of two separately specified object sets. GQA (Hudson and Manning, 2019), a dataset for "compositional question answering", joins separate visual checks with AND/OR. NS-VQA (Yi et al., 2018) and NS-CL (Mao et al., 2019) execute compositions of visual primitives. Our setting explicitly presents atomic questions and combines their answers through a program.

On error accumulation: the composed levels are not a longer list of perception questions. The model must also carry out the reasoning operation over values it is never shown, and these operations include sorting, recurrences, graph queries and constrained selection (Section 3.1). Table 1 separates the two ingredients. Training on the same images and atomic questions without composition reaches 53.58% (D0-only), training on symbolic questions without images reaches 54.50% (RLVE, text-only), and training on their composition reaches 72.25%.

We will clarify that composition size controls the number of atomic inputs, without assuming that every added input increases reasoning depth. Our new intermediate-answer analysis directly measures the atomic errors that can accumulate in these tasks (please see our response to Reviewer MDZ5). Please let us know if there is a particular distinction we should clarify further.

Annotation quality and human audit

Following your feedback, we ran a human audit of the VisualGym-Ladder atomic questions, reported in the general response. Our analysis shows that only 2.17% of the atomic questions were marked incorrect by the annotators. For completeness, we went further and ran another experiment removing every ladder that contains such atomic questions. This changes pass@5 proficiency by at most 0.11 levels and leaves the ordering of the models unchanged. We will add the audit to the manuscript.

Generalization beyond the task templates.

We evaluate the trained model on 14 external benchmark settings, each with its own questions and answer formats (Tables 2 and 6). Accuracy improves on 11 of the 14, the other three change by less than one point, and the mean rises from 52.69% to 56.34%. From the DeltaPrompt initialization, 13 of the 14 settings improve and the mean rises from 54.76% to 57.47%.

The framework is also not tied to the current templates. The atomic task families, the program families and the source datasets are extensible (Section 3.1).

Visual supervision and training controls

Table 1 reports the following results on the original 100-image VisualGym-Ladder:

Training settingLadder accuracy (%)
Text-only RLVR54.50
Direct source-task RLVR55.48
D0-only training53.58
Our manual curriculum65.65
Our adaptive curriculum72.25

Composed prompts contain more atomic questions than D0-only prompts, and equal training steps can use different numbers of generated tokens. The adaptive configuration outperforms the D0-only, direct source-task, text-only, and manual-curriculum baselines in Table 1. These results support the full training configuration. They do not tell us how much of the gain comes from composition or scheduling alone, because the total numbers of atomic questions and generated tokens were not matched. We will state this scope explicitly.

Potential confounding from format learning. Figure 5, token limits, and transfer

Figure 5 marks responses that reach the output-token limit as “none”. These are examples of individual responses. Finishing within the budget can contribute to the measured gains, so the figure alone does not establish stronger reasoning.

Output-budget sweep. Following your suggestions, we measure the effect of the output limit directly, we re-evaluated Qwen3-VL-8B-Thinking (Base) on the same VisualGym-Ladder questions with the output limit raised from 8K to 16K, 32K and 64K tokens. Prompts, sampling parameters and scoring are unchanged.

ModelOutput limitAccuracy (%)Pass@5 proficiency
Qwen3-VL-8B-Thinking (Base)8K49.954.55
Qwen3-VL-8B-Thinking (Base)16K59.155.27
Qwen3-VL-8B-Thinking (Base)32K59.504.95
Qwen3-VL-8B-Thinking (Base)64K59.905.22
+ Ours (adaptive schedule)8K72.256.19

The advantage persists when Base is given eight times the output budget. Base rises from 49.95% at 8K to 59.15% at 16K and gains less than one point beyond that, so further budget does not close the gap. At every budget Base remains significantly below + Ours (adaptive schedule) evaluated at 8K.

On transfer and usefulness to other question domains: the 14 external benchmark settings use their own questions and answer formats and none of our composition programs. Mean accuracy rises from 52.69% to 56.34%, with gains in 11 settings (Tables 2 and 6).

The gains therefore do not reduce to the 8K output limit or to missing final answers.

Hi thank you for your time and thoughtful feedback. We have submitted our rebuttal addressing your comments and would appreciate it if you could review our responses when you have a chance. Thank you again for your consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their careful reading and constructive feedback. We appreciate their recognition of our empirical finding that the learning value of fixed visual questions varies across models and can decline during training [24oy, MDZ5, k7Ns], and of its practical motivation for dynamic question generation [MDZ5, k7Ns]. They also highlighted the value of reusing existing images and annotations [MDZ5, k7Ns]. In particular, separating atomic visual tasks from symbolic programs enables multiple compositions per image, automatically computed targets, and controlled task construction [MDZ5], while varying task difficulty without generating new images makes better use of existing data [k7Ns]. We appreciate their recognition of the substantial gains on VisualGym-Ladder and positive transfer to external benchmarks [MDZ5, k7Ns], the breadth of baseline and cross-model evaluations [MDZ5], and the supporting evidence from ablations and experiments across models [k7Ns]. We address the questions below.

Human audit and filtering

We audited atomic question–answer labels across source datasets. We selected 50 images from the original 100-image VisualGym-Ladder in proportion to their source datasets. Annotators checked each image and its nine saved atomic question–answer pairs, marking each Correct, Incorrect, or Unsure, with optional corrections. Final symbolic answers were excluded.

Human auditResult
Annotators7
Reviewed atomic questions marked incorrect by the annotators2.17%

Notably, only 2.17% of reviewed questions were marked incorrect by the annotators who rated them showcasing the quality of our synthesis pipeline. That said, to further ensure correctness, we removed any image ladder containing a question unanimously marked incorrect, including all D0–D7 levels and all five responses per level. This removes five images, leaving the same 95 images for every model. We reused the saved outputs and scoring. Arrows show before → after filtering.

ModelPass@5 proficiency
Qwen3-VL-8B base4.55 → 4.66
Qwen + D0-only4.83 → 4.93
Qwen + adaptive6.19 → 6.18
GPT-6 Astra6.59 → 6.54
Gemini 3.8 Flash6.70 → 6.65

We find negligible changes in pass@5 proficiency (largest change: 0.11 levels), with no change in the ordering of these five models. We will report this check and filter the final released data using human annotations.

VisualGym-Ladder Large evaluation

To address Reviewer MDZ5’s concern about the 100-image evaluation, we report results on VisualGym-Ladder Large, containing 1,000 images. We will add this larger evaluation to the manuscript. The table compares the same Qwen3-VL-8B model variants on the original 100-image VisualGym-Ladder and VisualGym-Ladder Large. Proficiency is pass@5.

Qwen3-VL-8BOriginal accuracy (%)Original proficiencyLadder Large accuracy (%)Ladder Large proficiency
Base49.954.5553.114.68
D0-only53.584.8355.984.94
Manual65.655.7465.255.55
Adaptive72.256.1970.976.07

The ordering of these four models is unchanged on the larger set. On VisualGym-Ladder Large, adaptive training improves over the base model by 17.85 percentage points in accuracy and 1.39 proficiency levels, and over the manual curriculum by 5.72 points and 0.52 levels.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.