acceptodds
Withdrawn from ICLR 2027

Rewarding Non-Verifiable Intermediate Steps for Photorealistic Multimodal Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning for reasoning has shown strong results in mathematics and code generation, where intermediate steps can often be checked using deterministic rules or exact outcomes. In photorealistic multimodal reasoning, structured intermediate steps may also be needed, but they often lack a single exact answer and must instead be judged from visual and linguistic context. This makes it difficult to design reliable step-wise rewards for photorealistic multimodal reasoning. We propose a staged GRPO framework that first optimizes format and global accuracy rewards, then introduces step-wise rewards for intermediate reasoning. We consider two ways to specify the reasoning structure: one guided by human strategies and one proposed by a model from task examples. We introduce MOSAIC (Multimodal Object-Specific Ambiguity Inference from Context), a multilingual visual word inference testbed over photorealistic images that requires integrating visual and linguistic information and naturally elicits a structured reasoning process, which we decompose into four human-inspired steps. Experiments across 12 languages show improvements over prompt-based, supervised fine-tuning, and vanilla GRPO baselines, with the largest gains in lower-resource languages. Gains extend across Qwen and InternVL model families, and the larger models improve on three languages unseen during RL training. We further ask whether the reasoning structure must be manually specified. On SpatiaLQA, a distinct spatial-reasoning and action-planning task, we use model-proposed reasoning steps and evaluation criteria, yielding a 4.50-point improvement in F1-score over the baseline.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal reasoning, reinforcement learning, process rewards, vision-language models, multilingual learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_gNLeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates a staged GRPO algorithm for real-photograph scenarios where intermediate steps are difficult to verify precisely. The first stage rewards output structure and final answer quality, while the second replaces the answer reward with rubric-based scores for selected reasoning steps. The authors introduce MOSAIC, a visual missing-word inference task covering 12 training languages and three unseen languages, with a four-step decomposition derived from human strategies. Experiments on Qwen and InternVL demonstrate the method's effectiveness (e.g., increasing Qwen2.5-VL-7B's average MOSAIC accuracy from 81.1% to 85.3%). The paper also validates the model-proposed steps on SpatiaLQA.

Critical strengths
  1. The authors support the method's effectiveness and robustness with sufficient comparative experiments. MOSAIC covers multiple languages and two model families, while SpatiaLQA evaluates a different task with a model-proposed reasoning structure.
Critical weaknesses
  1. There is a fundamental gap between the paper's core claim and its evaluation. The paper claims to study multimodal reasoning in real-photograph scenarios, focusing on the difficulty of rewarding intermediate steps. However, i) the evaluation primarily uses a multilingual task whose characteristics are difficult to connect directly to this motivation; ii) the focus on word filling is too narrow, and small models already achieve very high scores on MOSAIC. As a result, the task cannot comprehensively assess multimodal reasoning ability.
  2. The method lacks sufficient core novelty. Its central ideas, including GRPO, staged reinforcement learning, and dense rewards, have already been validated and published.
  3. The authors manually specify four reasoning steps—Sentence Analysis, Image Analysis, Object Exclusion, and Grammar and Context—and require the model to follow them. This may limit, to some extent, the trained model's generalization to other tasks.
Questions for authors
  1. How were the weights for the individual components of the final reward determined in each of the two GRPO training stages?
  2. Could the authors evaluate the method on more representative benchmarks, including OOD benchmarks such as MM-star, MathVista, and MMMU, to support the stated goal of studying multimodal reasoning in real photographs?
  3. Could the authors report the performance of other representative models, such as Qwen3.5-9B, on MOSAIC? This would help establish the task's level of difficulty.
2 comments

We thank the reviewer for the detailed feedback. Below, we clarify our design choices and present additional experimental results addressing the main concerns.

W1: A gap between the motivation and evaluation; MOSAIC is narrow and already easy for small models

A: MOSAIC is a multimodal question-answering task grounded in photorealistic images, formulated as missing-word inference from an image and an incomplete caption. Multilinguality concerns the language of the input caption; the underlying task remains reasoning jointly over visual and linguistic evidence. We deliberately vary the input language while keeping intermediate reasoning in English to test whether semantic step rewards support reasoning across languages, rather than improvements restricted to English-specific input patterns. Evaluation on languages unseen during RL training further tests transfer beyond the training languages. The multilingual design therefore directly serves our investigation of image-grounded reasoning.

The characterization that small models already achieve very high scores overlooks the reported results. Average accuracy across the six evaluated models ranges from 67.4% to 82.3%. Qwen2.5-VL-3B achieves 71.7% overall and only 34.5% on Hindi, leaving substantial room for improvement. Moreover, the evaluation extends beyond missing-word inference to SpatiaLQA, a challenging public benchmark requiring spatial reasoning and action-plan generation with a different, model-proposed decomposition.

W2: The method lacks novelty because GRPO, staged training, and dense rewards already exist

A: Our contribution is the design and empirical study of task-specific semantic rewards for photorealistic reasoning whose intermediate steps lack uniquely verifiable targets. We define reasoning steps with explicit roles and corresponding evaluation criteria, allowing a frozen, off-the-shelf judge to supervise them without reasoning-demonstration SFT or dedicated reward-model training.

Our experiments establish how this supervision can be used: introducing global and step-wise rewards jointly from the initial model performs poorly, whereas step-wise training after global-reward initialization improves final-answer performance and outperforms continued outcome-based training. SpatiaLQA further demonstrates the framework with a different, model-proposed decomposition. The prior existence of GRPO, staged training, and process rewards does not by itself account for these task-specific reward designs and empirical findings.

W3: The manually prescribed four steps may limit generalization to other tasks

A: The four steps are specific to MOSAIC; the framework does not require other tasks to reuse them. SpatiaLQA directly demonstrates adaptation to a different task: we use five model-proposed reasoning steps and corresponding evaluation criteria for spatial action-plan generation. The framework improves F1 from 40.20 to 44.70, with Stage 2 contributing a further 2.06 points. This provides evidence that the method extends beyond MOSAIC’s manually specified decomposition through task-adapted steps and rubrics.

Q1: How were the reward-component weights determined?

A: We selected the reward weights through grid search on 100 validation examples using Qwen2.5-VL-3B and used the same weights across model sizes, without size-specific tuning.

Q2: Could the method be evaluated on additional OOD benchmarks such as MMStar, MathVista, and MMMU?

A: SpatiaLQA already evaluates our framework on a challenging public spatial-reasoning benchmark, using model-proposed steps for action-plan generation. This directly tests adaptation to a different task beyond MOSAIC. Extending the framework to further tasks requires task-specific steps and rubrics followed by two-stage training, which carries substantial computational cost.

Q3: Could more recent models, such as Qwen3.5-9B, be evaluated on MOSAIC?

A: We additionally evaluated Qwen3.5-9B on MOSAIC using the zero-shot task instruction and GPT-based semantic scoring.

ModelSettingAverage accuracy across 12 seen languages
Qwen3.5-9BZero-shot85.86%
Qwen2.5-VL-7BOur staged GRPO85.91%

The newer and larger Qwen3.5-9B achieves accuracy comparable to our staged Qwen2.5-VL-7B model. This shows that our training brings the smaller, earlier model to a performance level comparable to the newer model on MOSAIC. Moreover, Qwen3.5-9B still makes errors on approximately 14% of examples, indicating that the task remains challenging even for a more recent model.

Thank you to the authors for their detailed response. However, I still find that the objective of "testing whether semantic step rewards support reasoning across languages" is only weakly connected to image-grounded reasoning, and I remain concerned about the method's generalizability to other domains. Furthermore, Qwen3.5-9B already achieves a relatively high accuracy of 85.86% on MOSAIC, raising further concerns about the benchmark's difficulty. Therefore, I have decided to maintain my current score.

Reviewer_fPYjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies process rewards for photorealistic multimodal reasoning, where intermediate steps cannot generally be checked against a unique correct answer. It proposes a two-stage GRPO framework: Stage 1 optimizes structured output and final-answer rewards, while Stage 2 continues training with semantic rewards for selected reasoning steps. The authors introduce MOSAIC, a multilingual visual word-inference task, and prescribe four reasoning steps based on human-inspired strategies. On Qwen2.5-VL-7B, average accuracy over 12 seen languages rises from 81.1% for the zero-shot baseline to 85.3% after both stages. The framework is also applied to SpatiaLQA using a separately proposed five-step decomposition, improving Content F1 from 40.20 to 44.70.

Critical strengths
  1. The paper proposes a concrete way to reward intermediate reasoning steps whose correctness is difficult to verify automatically. For MOSAIC, the authors define task-specific steps and scoring criteria, and use human analysis to identify object exclusion and grammar inference as the steps that need the most improvement.
  2. The experiments cover several useful comparisons: the method is tested against multiple baselines, compared with continued first-stage training to account for extra training time, and evaluated on SpatiaLQA, a spatial reasoning and action-planning benchmark. These results support the method’s effectiveness beyond the main MOSAIC experiment.
Critical weaknesses
  1. The effect of removing the final-answer reward in the second stage has not been isolated. The proposed second stage adds step rewards and removes answer rewards at the same time. The existing controls support the combined design, but do not show whether removing answer rewards is necessary. A comparison that retains both types of reward after the same first-stage checkpoint would clarify this point.
  2. The reasoning steps and scoring criteria are task-specific, and the paper offers limited evidence about how reliably they can be chosen for a new task. The four steps used for word inference are specified in the prompt; spatial planning requires a different set. Performance also drops when the word-inference reasoning prompt is applied to GQA. That result does not isolate the effect of the steps themselves, but it does limit claims about the general applicability of the four-step format.
Questions for authors

How does second-stage training perform when the final-answer reward is retained alongside step rewards, starting from the same first-stage checkpoint and using the same number of updates? How sensitive are the results to the choice of reasoning steps and scoring criteria, and how should these be selected for a new task?

1 comment

We thank the reviewer for the detailed feedback. Below, we clarify our design choices and present additional experimental results addressing the main concerns.

W1: The effect of removing the final-answer reward in Stage 2 has not been isolated

A: Removing final-answer rewards in Stage 2 is a deliberate experimental choice: we ask whether rewarding intermediate reasoning can improve final-answer performance without continuing to reward the answer itself. Keeping outcome rewards would instead test their combined optimization. Stage 2 improves final-answer performance and outperforms continued Stage 1, supporting the usefulness of step-wise supervision after global-reward initialization.

We additionally conducted the requested reward-composition experiment, retaining both the binary and MetaCLIP final-answer rewards alongside step-wise rewards. This run starts from the same Stage 1 checkpoint and follows the same sampling settings and 1,020-update Stage 2 schedule.

MethodAdditional updates after Stage 1Seen Avg. (12 languages)Unseen Avg. (3 languages)
Stage 1—84.3%60.3%
Stage 21,02085.3%61.8%
Stage 2 with answer rewards102085.95%59.00%

One possible explanation for the lower unseen-language accuracy is that retaining final-answer rewards shifts learning toward answer correctness rather than the targeted intermediate reasoning behaviors, which may contribute to weaker transfer to unseen languages. This remains a hypothesis rather than a demonstrated mechanism.

Our design does not require claiming that combined rewards cannot work: removing final-answer rewards allows us to directly study step-wise supervision without continued outcome rewards.

W2: Task-specific reasoning steps and limited evidence about their selection for new tasks

A: The framework transfers through task-specific reasoning steps and evaluation criteria; it does not require reusing MOSAIC’s four-step format across tasks. MOSAIC’s decomposition follows human-inspired ambiguity-resolution strategies. For SpatiaLQA, we provide the task definition and ten representative examples to GPT-5.5 to propose a different five-step decomposition and corresponding rubrics, with only minor wording refinements. This provides a concrete procedure for adapting the framework without deriving the steps from a human reasoning study. On this challenging public spatial-reasoning task, Stage 2 adds 2.06 F1 points over Stage 1.

Reviewer_tnfLOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies rubric-based process rewards for multimodal reasoning tasks whose intermediate steps lack exact verification. The authors propose a two-stage GRPO framework: Stage 1 optimizes a prescribed reasoning format and final-answer quality, while Stage 2 replaces outcome rewards with model-judged rewards for intermediate reasoning steps. The authors introduce MOSAIC, a multilingual visual word-completion benchmark requiring models to infer a missing object noun from an image and an incomplete caption. Its reasoning structure comprises sentence analysis, image analysis, object exclusion, and grammar inference, motivated by human ambiguity-resolution strategies. The framework is also evaluated on SpatiaLQA using a model-proposed reasoning decomposition. Experiments with specific Qwen and InternVL models report task-performance improvements, accompanied by reward ablations, unseen-language evaluations, and analyses of automated judging and computational cost.

Critical strengths

S1: A useful control for the second training stage.

The authors compare Stage 2 against continued Stage-1 training with the same number of additional updates (Appendix M, Table 21). On Qwen2.5-VL-7B, Stage 2 improves average accuracy by 1.13 percentage points on seen languages and 3.19 points on unseen languages over this control. This provides evidence that the second-stage reward change contributes beyond simply extending training.

S2: Helpful visual presentation.

The paper provides clear illustrations of the task setup and the two-stage training pipeline, making the proposed procedure easier to follow.

Critical weaknesses

W1: Insufficient justification for the prescribed reasoning decomposition and limited evidence for generalization across models and tasks.

The paper does not sufficiently establish why a fixed decomposition with named reasoning steps is preferable to more flexible process supervision. The Stage-1 comparison with vanilla GRPO simultaneously introduces the prescribed structure and an additional MetaCLIP-based answer reward, leaving their contributions confounded. Furthermore, the limited human-agreement study does not establish that the frozen rubric judge remains reliable as the policy is optimized against it. Although the authors acknowledge judge limitations and report computational costs, they do not adequately investigate the risks highlighted by DeepSeek-R1’s discussion of process rewards, particularly reward hacking and limited benefits relative to overhead. The existing ablations therefore leave important questions about both the contribution of the prescribed structure and the reliability of its reward signal unresolved.

The training experiments cover Qwen2.5-VL and InternVL3.5-8B, but the latter uses Qwen3-8B as its language backbone. Consequently, the reported cross-family gains do not establish generalization beyond the Qwen language-model lineage, which is particularly relevant given MOSAIC’s dependence on multilingual understanding and the requirement for English intermediate reasoning. Transfer to languages excluded from RL training is also inconsistent: Table 4 reports average changes of −9.0, +6.9, and +1.5 percentage points for the three trained models. Beyond the proposed MOSAIC benchmark, SpatiaLQA provides one additional task instantiation, while GQA primarily serves as a capability-retention check. These results offer limited support for the broader applicability of the framework across independent language backbones and reasoning tasks.

W2: Limited methodological novelty and generalizable insight.

Process supervision, rubric-based RL beyond verifiable domains, and structured multimodal reasoning already have substantial precedents. For example, CoS explicitly structures steps using Name, Thought, and Reflection components and evaluates intermediate reasoning. The present contribution appears primarily to be a task-specific combination of prescribed semantic categories, frozen-judge rubric scoring, and staged GRPO training. Although extending these ideas to photorealistic tasks is potentially useful, the paper does not sufficiently establish what new technical challenge requires this particular design or what reusable insight emerges from it. Table 5 provides evidence for the training schedule, but does not explain which properties of the decomposition make it effective or when similar constraints should be preferred. Even accepting the reported improvements, the experiments mainly demonstrate that a particular recipe works on selected configurations, offering limited insight into the conclusion’s broader question of how MLLMs benefit from structured multimodal reasoning.

Questions for authors

Q1: What justifies the prescribed reasoning structure and process rewards over outcome-based training?

DeepSeek-R1 (§2.2.3 and §4.2) deliberately avoids prescribing specific problem-solving strategies and reports difficulties with process rewards, including step definition, unreliable judgments, reward hacking, and limited benefits relative to their computational overhead. Difficulty in verifying intermediate steps therefore does not, by itself, justify introducing more elaborate rubric-based rewards. In the comparison on page 7, Stage 1 introduces both the four-step structure and an additional MetaCLIP-based answer reward, leaving their contributions confounded. Could the authors compare against free-form reasoning with identical answer-reward components and a matched training budget? Furthermore, although the continued-Stage-1 control is useful, why should Stage 2 remove final-answer rewards rather than combine them with process rewards? These comparisons, accounting for reward-evaluation costs, would clarify the value of prescribing the structure and replacing the reward separately.

Q2: Do the rubrics measure useful reasoning or conformity to a prescribed procedure?

Inspiration from human reasoning does not establish that a particular decomposition, step ordering, or reasoning language is appropriate across models and examples. Rewarding Progress (ICLR 2025) (ICLR 2025) motivates process rewards based on a step’s contribution to eventual success, rather than assuming that local correctness provides the most effective training signal. How do the authors establish that their rubrics reward reasoning that improves task outcomes, rather than compliance with a preferred procedure and presentation style?

This concern also affects generalization across model families. Although Qwen2.5-VL and InternVL3.5 are distinct multimodal systems, the evaluated InternVL3.5-8B uses Qwen3-8B as its language backbone. The experiments therefore do not establish generalization beyond the Qwen language-model lineage. Given MOSAIC’s dependence on multilingual lexical and grammatical understanding, its partially translation-generated captions, and the requirement for English intermediate reasoning, could the gains depend on these models’ language capabilities and compatibility with the prescribed procedure?

Could the authors test alternative valid decompositions and compare English, target-language, and freely chosen reasoning languages under matched conditions, including on a comparably sized VLM with a non-Qwen backbone, such as Gemma 3 4B and Ministral 3 8B? Table 1 only reports the zero-shot evaluation scores of Gemma 3 4B on MOSAIC, but no results of the two-stage training. Independent human assessment could additionally examine whether valid solutions using different step orderings or reasoning languages receive unjustifiably lower rewards. These analyses would clarify when the prescribed structure helps and whether its benefits extend beyond the selected models and language settings.

2 comments

We thank the reviewer for the detailed feedback. Below, we clarify our design choices and present additional experimental results addressing the main concerns.

W1.1: Insufficient justification for the prescribed reasoning decomposition

A: Stage 1 combines a prescribed reasoning structure with task-appropriate global accuracy rewards. Because MOSAIC admits semantically valid answers beyond exact matches, MetaCLIP similarity provides graded feedback on final-answer correctness, complementing the binary accuracy reward. These components are designed to work together, and the original vanilla-GRPO comparison evaluates their combined benefit. Our reward ablation additionally compares the individual global rewards and their combination.

To further examine the contribution of the prescribed structure, we trained a free-form reasoning baseline with the same binary and MetaCLIP final-answer rewards and matched training settings. Both settings retain the reasoning/answer output format; the free-form baseline does not require the four named reasoning steps.

Reasoning structureFinal-answer rewardsSeen Avg. (12 languages)Unseen Avg. (3 languages)
Free-form reasoning (new experiment)Binary + MetaCLIP81.72%56.83%
Prescribed four-step reasoning (Stage 1, reported in the paper)Binary + MetaCLIP84.3%60.3%

The free-form baseline remains below our Stage 1 performance on both seen and unseen languages despite receiving both final-answer rewards. This additional result supports the value of the prescribed reasoning structure.

W1.2: Limited evidence for generalization across models and tasks

A: The framework does not require other tasks to reuse MOSAIC’s human-designed reasoning steps. On SpatiaLQA, a challenging public benchmark for spatial reasoning, models must generate action plans involving spatial relations and dependencies. Note that SpatiaLQA is a substantially different task from MOSAIC’s missing-word inference. We use five model-proposed reasoning steps rather than reusing MOSAIC’s four human-inspired steps. F1 improves from 40.20 to 44.70, with Stage 2 contributing a further 2.06 percentage points. This experiment directly demonstrates that the framework works with a different task, output structure, and model-proposed decomposition.

Across models, our staged framework improves performance on Qwen2.5-VL-3B, Qwen2.5-VL-7B, and InternVL3.5-8B. Although InternVL3.5-8B uses a Qwen3 language backbone, it provides a distinct multimodal system on which Stage 2 yields an additional 2.48 percentage-point improvement over Stage 1. These results demonstrate that the gains extend beyond a single model size or multimodal implementation.

On languages unseen during RL training, the two larger models improve average accuracy by 6.9 and 1.5 percentage points, respectively. The differing result for the 3B model identifies a boundary of the observed transfer, rather than negating the positive evidence from both larger models.

W2: Limited methodological novelty and generalizable insight

A: CoS and our framework share the broad idea of process supervision, but differ substantially in their reasoning structure, supervision requirements, and training procedure:

CoS structures generated steps using a shared Name–Thought–Reflection template. Our framework assigns task-specific reasoning steps explicitly defined roles and corresponding evaluation criteria. We demonstrate different decompositions on two distinct tasks: four human-inspired steps for MOSAIC’s missing-word inference and five model-proposed steps for SpatiaLQA’s spatial action-plan generation. CoS relies on generated reasoning demonstrations to train the policy and annotated reasoning steps to train a dedicated PRM.

Our framework requires neither: it directly trains the policy using global and step-wise rewards, with a frozen, off-the-shelf judge applying explicit step-specific rubrics. Our reasoning roles are shared across examples within each task rather than being example-specific, enabling us to identify recurring weaknesses, such as difficulties with Object Exclusion and Grammar Inference, and target them through the corresponding training rubrics.

The central technical challenge is to supervise image-grounded intermediate reasoning that lacks uniquely verifiable targets. Our experiments support introducing this semantic supervision after global-reward training: Stage 2 improves over continued Stage 1, and the strategy also produces additional gains with model-proposed steps on SpatiaLQA. This provides evidence for a transferable training strategy with task-adapted reasoning steps and rubrics.

Q1: What justifies the prescribed reasoning structure and process rewards over outcome-based training?

A: DeepSeek-R1 explicitly states that its unsuccessful attempts do not imply that these approaches cannot produce effective reasoning models. Its findings therefore do not establish that process rewards are unsuitable for our setting.

Removing final-answer rewards in Stage 2 is a deliberate experimental choice: we ask whether rewarding intermediate reasoning can improve final-answer performance without continuing to reward the answer itself. Keeping outcome rewards would answer a different question about jointly optimizing both objectives.

The two-stage schedule is also experimentally motivated: introducing global and step-wise rewards together from the initial model performs substantially worse in Table 5. Thus, both the initialization stage and the subsequent removal of outcome rewards have explicit motivations; the latter is central to the research question being tested.

We additionally trained Stage 2 with both step-wise and final-answer rewards from the same Stage 1 checkpoint. With answer rewards retained, seen-language accuracy is comparable to the paper-reported Stage 2 result, while unseen-language accuracy is lower. One possible explanation is that retaining final-answer rewards shifts learning toward answer correctness rather than the targeted intermediate reasoning behaviors, which may contribute to weaker transfer to unseen languages.

MethodAdditional updates after Stage 1Seen Avg. (12 languages)Unseen Avg. (3 languages)
Stage 1—84.3%60.3%
Stage 21,02085.3%61.8%
Stage 2 with answer rewards102085.95%59.00%

Q2: Do the rubrics measure useful reasoning or conformity to a prescribed procedure?

A: The rubrics evaluate substantive reasoning content. Object Exclusion assesses whether exclusions are justified, plausible candidates are preserved, and the reasoning is consistent. Grammar Inference assesses the correct application of grammatical cues and penalizes invented rules. Although explanations are written in English, the grammar criteria explicitly concern the original caption language.

Both Stage 1 and Stage 2 already use the prescribed reasoning structure. The comparison with continued Stage 1 therefore evaluates the benefit of changing the supervision while retaining that structure. We assess this benefit through final-answer performance under a separate evaluator, rather than treating higher training-judge scores as evidence of success.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.