acceptodds
Withdrawn from ICLR 2027

FactorGraph-RLVR: Diagnosing Reward Informativeness in Factorized Visual Reasoning Curricula

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has improved multimodal reasoning, but gains attributed to curriculum design may instead reflect how often training produces informative group-relative updates. We introduce \ours, a controlled visual-graph testbed that crosses visual masking and reasoning load in a matched factorial design, together with a parser-gated dense verifier. To separate curriculum composition from optimization opportunity, we compare reasoning-heavy and uniform curricula under a fixed optimizer-step budget, match checkpoints by cumulative informative updates, and ablate dense supervision with exact-only rewards. A preregistered evaluation covers 28 frozen model checkpoints, three final evaluation splits, and four decoding conditions. Under greedy decoding, reasoning-heavy dense RLVR improves exact success over the unadapted model on all three splits. The gains are 4.1, 4.2, and 4.7 percentage points on in-distribution, style-shift, and length-shift data, respectively. All 36 seed–split–decoding contrasts are positive, and every 95% bootstrap confidence interval computed from paired source groups excludes zero. Uniform training yields smaller gains, informative-update matching retains most of the reasoning-heavy effect, and exact-only training remains near the base model. These results provide controlled evidence that curriculum composition influences visual RLVR beyond optimizer-step budget and measured informative-update opportunity.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning with verifiable rewards, multimodal reasoning, visual reasoning, curriculum learning, reward informativeness, group relative policy optimization, controlled evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_oxqZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims at curriculum composition in reinforcement learning with verifiable rewards for visual reasoning, with the goal of disentangling two confounded effects. The authors introduce FACTORGRAPH-RLVR, a synthetic visual graph reasoning testbed built around a 3×3 factorial design that independently varies visual masking and reasoning load. Using this testbed, they compare four fixed curriculum distributions trained via GRPO with LoRA on Qwen3-VL-8B. The paper concludes that curriculum composition influences visual RLVR performance beyond what can be explained by optimizer-step budget or measured reward informativeness.

Critical strengths
  1. The paper addresses a recognized problem, where performance gains from curriculum design are often attributed to better task selection, but they may instead simply reflect a higher rate of informative reward signals. Attempting to separate these two factors is a methodologically relevant goal.

  2. The 3×3 crossed design of visual stress and reasoning stress is a useful diagnostic construct. By keeping the underlying relation tables identical across all nine cells, the testbed enables paired analysis of how performance gains vary across visual occlusion and reasoning depth, providing more granular insight than single-scalar benchmark scores.

  3. The authors follow a frozen, preregistered evaluation pipeline with 28 model checkpoints, three test splits, four decoding conditions, and paired bootstrap confidence intervals. They explicitly separate development and confirmatory results, avoid outcome-based checkpoint selection, and report all planned comparisons including null and negative results.

  4. The idea of using logged within-group reward dispersion to count informative training steps, and then matching checkpoints on this metric, is a creative attempt to control for differences in update signal frequency across curricula.

Critical weaknesses
  1. The entire study is conducted on a single, highly stylized synthetic task: following numeric transition tables on rendered graphs with partial digit masking. This is essentially a symbolic formatting and arithmetic task with trivial visual components, and it bears little resemblance to real-world visual reasoning, multimodal understanding, or standard visual QA benchmarks.

  2. The task primarily tests structured execution of explicit instructions, not abstract reasoning, visual comprehension, or generalization to natural scenes. All conclusions are therefore tightly bounded to this one specific task family; there is no evidence that any of the findings transfer to other VLMs, other reasoning tasks, or practical RLVR applications. For a conference paper, this extremely limited scope severely reduces the potential impact of the work.

  3. The headline effect is surprisingly small in magnitude and critically reverses sign in one of three training seeds. The paired bootstrap interval for seed 49 lies entirely below zero, meaning that for that run, uniform training is reliably better than reasoning-heavy training. The authors’ preregistered criterion sets a very low bar; a result that flips direction across random seeds is not a reliable or generalizable finding.

  4. The operation of informative training steps using a simple threshold on within-group reward standard deviation (σ_t > 10⁻⁸) is methodologically weak. This metric only measures whether rewards vary across rollouts, not whether the variation produces a meaningful gradient, a correct parameter update, or any improvement in policy.

  5. Several key controls and comparisons are missing, limiting the interpretability of the results: The dense vs. exact-only reward ablation is only performed under the mixed-control curriculum, not under reasoning-heavy or visual-heavy curricula. Thus it is impossible to tell whether the benefit of dense reward interacts with curriculum composition, which is directly relevant to the paper’s core question.

  6. The paper does not compare against any existing adaptive curriculum methods (e.g., difficulty filtering, variance-based sampling, exploration-aware curricula). It only compares a handful of fixed distributions, so it offers little practical guidance for RLVR curriculum design.

  7. Synthetic visual reasoning benchmarks with programmatic generation and verifiable rewards already exist in many forms (CLEVR, PuzzleVQA, PuzzleCraft, VisualSphinx, etc.). FACTORGRAPH-RLVR adds a factorial masking/length manipulation, but this is an incremental variation, not a fundamentally new benchmark paradigm.

Questions for authors

Refer to Weaknesses part.

Reviewer_VabPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces FactorGraph-RLVR, a controlled visual reasoning testbed for studying how curriculum composition interacts with reward informativeness in reinforcement learning with verifiable rewards. It independently varies visual masking and reasoning depth in a matched factorial design and provides a parser-gated verifier with intermediate-step rewards. Using Qwen3-VL-8B-Instruct, the authors compare fixed curriculum distributions under equal optimizer-step budgets and at checkpoints matched by cumulative informative updates. They report that a reasoning-heavy curriculum improves exact-trace success on in-distribution, visual-style-shift, and reasoning-length-shift evaluations, and argue that curriculum composition matters beyond the measured availability of informative updates.

Critical strengths
  1. Matched task construction and training-budget controls support a careful comparison of curriculum compositions.
  2. The novel diagnostic setting independently varies visual difficulty and reasoning depth while providing verifiable intermediate traces.
Critical weaknesses
  1. The overly compressed presentation makes the task construction, reward design, and curriculum comparisons difficult to follow.
  2. The contribution is limited, as findings from a narrow synthetic task family do not establish robust, generalizable conclusions.
  3. The analysis offers insufficient insight into why particular curricula help beyond the expected observation that training-data composition affects performance.
Questions for authors
  1. What generalizable insight do the results establish beyond the expected effect of training-data composition, and which evidence supports that insight?
Reviewer_jFk6OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether curriculum gains in visual RLVR persist after controlling the frequency of reward-ranking signals. It introduces synthetic graph-following tasks with different masking levels and path lengths. Experiments with Qwen3-VL-8B-Instruct show that reasoning-heavy training performs best on average, including after approximately matching informative-step counts.

Critical strengths
  1. The testbed varies visual masking and path length over the same underlying graphs, providing a useful setup for studying curriculum effects in a controllable way.

  2. The paper transparently reports variation across training seeds, including results that reverse the average curriculum ranking.

Critical weaknesses
  1. The evidence is too narrow to support a broader conclusion about curriculum effects in visual RLVR. All experiments use one model and one synthetic task family constructed by the authors. The style and length shifts remain within the same setup, so it is unclear whether the conclusions generalize to more realistic visual reasoning tasks or training settings.

  2. The informative-step control leaves alternative explanations unresolved. The paper counts a step as informative when logged reward dispersion exceeds 10−810^{-8}, but provides limited analysis of this choice or how alternative definitions would affect the conclusions.

Questions for authors
  1. Could you validate the findings on more realistic visual reasoning tasks?
  2. Could you provide additional experiments or justification for the choice of the informative-step measure?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.