acceptodds
Under review as a conference paper at ICLR 2027

Learning to Reason with Procedural Guidance in MLLMs

Abstract

Reinforcement learning with verifiable rewards (RLVR) can improve reasoning in large language models (LLMs). However, when all sampled responses receive the same reward, the group-relative advantage in methods such as Group Relative Policy Optimization (GRPO) is zero, so the group provides no reward-driven learning signal. Hint-based RL helps the model avoid all-incorrect rollout groups by supplying part of a reasoning trace, usually as free-form text written by an oracle model. Instead of relying on an oracle model, we introduce procedural guidance, in which the program that generates problems and computes their answers also provides correct, controllable guidance. We instantiate this concept in a procedural data generation framework called Vision Reasoning Gym (VRGym), specifically designed for multimodal reasoning problems. The derivations generated by VRGym contain maskable values that can be replaced with an anonymous "?" mark while preserving their reasoning structure. Using these derivations, we design On-Policy Procedurally Graduated Guidance (OPGG), which maintains a scalar guidance-withdrawal budget for each task across training steps and updates it based on each rollout group's accuracy. As the budget increases, OPGG first masks selected values and then truncates the remaining derivation, allowing the model to learn to solve problems without guidance. With this guidance schedule, OPGG achieves the highest seven-dataset macro-average accuracy among the evaluated hint-based methods on both backbones.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning with Verifiable Rewards, Multimodal Large Language Models Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_miiwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper targets multimodal reasoning with reinforcement learning and introduces VRGym, which generates problems with verifiable answers and structured guidance, together with OPGG, which progressively masks and removes that guidance based on rollout accuracy. Across two Qwen-VL backbones, OPGG achieves the highest average accuracy among the evaluated methods on VRGym and six external benchmarks.

Critical strengths
  1. VRGym provides a useful way to generate multimodal reasoning problems with correct, controllable guidance.
  2. The method also improves performance on six external benchmarks without using guidance at test time.
Critical weaknesses
  1. The comparison does not isolate the benefit of the OPGG schedule. OPGG uses procedural derivations, while the hint-based baselines use oracle-generated traces. The ablations in §4.4 test components of OPGG, but the comparison with other schedules still mixes differences in guidance and scheduling. I would like to see the schedules compared using the same procedural derivations, including a simple predetermined withdrawal schedule.

  2. Permanent guidance withdrawal is not directly tested. Algorithm 1 freezes a task’s budget once it reaches 2, so later failures cannot restore guidance. The ablations do not test this decision. Please report each task’s unguided accuracy before and after graduation, and test whether allowing guidance to return improves performance.

Questions for authors
  1. How does OPGG compare with a predetermined withdrawal schedule when both use the same procedural guidance?
  2. How does each task’s unguided accuracy change before and after graduation?
  3. What happens if guidance can be restored when a graduated task’s performance declines? 4.How does providing Scaf-GRPO with the procedural solution rather than only the final answer affect its performance?
3 comments

We thank the reviewer for taking the time to review our paper. Based on the feedback from all three reviewers, we realized that some parts of our paper were not clearly written, which may have led to misunderstandings. To address this, we have added a comment titled "Reiterating the Contributions" to clarify and restate the key contributions of our work.

Weaknesses

  1. We agree with the reviewer. As clarified in our "Reiterating the Contributions" comment, our paper focuses on procedural guidance rather than OPGG. Therefore, we have added new experiments in Section 4.4, where we replace the guidance used by existing hint-based RL baselines with procedural guidance while keeping their original scheduling mechanisms unchanged. This allows us to evaluate the effect of procedural guidance independently of OPGG's scheduling mechanism. The results show that procedural guidance further improves the performance of existing on-policy hint-based methods. In particular, Scaf-GRPO improves across all seven benchmarks, with the largest gain on MathVision (+4.93 percentage points) and an average improvement of +1.78 percentage points with procedural guidance.

  2. As explained in our "Reiterating the Contributions" comment, OPGG is a simple and intuitive approach designed to demonstrate the feasibility of procedural guidance. We do not expect small changes to its design to substantially affect performance, as also shown in our other ablation studies. Nevertheless, we have conducted additional experiments to address the reviewer's concerns.

    (1) Performance on graduated and ungraduated tasks

    Below, we report the avg@8 results for graduated and ungraduated tasks, both overall and across the task categories included in training. We observe that graduated tasks are generally easier by design and benefit more from guidance. In contrast, ungraduated tasks tend to be more difficult, making it harder for the model to fully benefit from guidance. This is consistent with Limitation 2, where we acknowledge that our task design may not be perfectly balanced and that some tasks may still be too difficult for the model.

    Model / Task GroupOverallMathLogicAlgoChart QA
    Qwen2.5-VL-7B-Instruct / Graduated26.941.021.632.423.1
    Qwen2.5-VL-7B-Instruct + OPGG / Graduated50.274.231.952.346.5
    Δavg@8 / Graduated+23.4+33.1+10.3+19.9+23.4
    Qwen2.5-VL-7B-Instruct / Ungraduated5.09.45.01.03.1
    Qwen2.5-VL-7B-Instruct + OPGG / Ungraduated10.812.111.76.218.1
    Δavg@8 / Ungraduated+5.8+2.6+6.8+5.1+15.0

    We also report the proportion of graduated and ungraduated tasks in each category below.

    Task GroupOverallMathLogicAlgoChart QA
    Graduated31.4% (16/51)25.0% (3/12)11.8% (2/17)10.0% (1/10)83.3% (10/12)
    Ungraduated68.6% (35/51)75.0% (9/12)88.2% (15/17)90.0% (9/10)16.7% (2/12)

    (2) Removing the permanent guidance withdrawal constraint

    Following the reviewer's suggestion, we also evaluate OPGG without the permanent guidance withdrawal constraint. The results are shown below.

    MethodVRGymMathVistaMathVisionMathVerseDynaMathMMMUMMMU-ProAvg.
    OPGG (default)23.6671.1427.9649.2852.0554.3538.0545.21
    Without permanent guidance withdrawal22.3671.1128.2549.8452.0955.4037.9045.28

    The results show that removing this constraint has little effect on the overall performance. Therefore, this alternative design is also a reasonable choice. As emphasized earlier, OPGG is only one possible way to use procedural guidance, and we chose a simple and intuitive design to demonstrate its feasibility. Our main focus remains on evaluating the benefits of procedural guidance rather than optimizing OPGG's scheduling mechanism.

Questions

  1. We are not entirely sure what the reviewer means by a "predetermined withdrawal schedule." If this refers to withdrawing guidance based only on training steps, we believe such a design is less suitable. It would require testing many scheduling strategies to determine how much guidance should be withdrawn at each step. More importantly, it would not indicate whether the model has actually learned to solve problems without relying on guidance. In contrast, OPGG adjusts guidance based on group accuracy. Similar accuracy-based strategies are also used in previous hint-based methods. For example, Scaf-GRPO [1] increases the number and strength of hints when the model fails to produce a correct answer. Therefore, we believe that using group accuracy to schedule guidance is a simple and reasonable choice.

    Nevertheless, following the reviewer's suggestion, we also conducted an experiment with a linear schedule. Specifically, we linearly increase the budget from 0 at the beginning of training to 2 at the end of training. The results are shown in the table below.

    MethodVRGymMathVistaMathVisionMathVerseDynaMathMMMUMMMU-ProAvg.
    OPGG (default)23.6671.1427.9649.2852.0554.3538.0545.21
    linear22.3170.6426.5248.5650.1154.3838.0344.36
  2. We have provided additional results addressing this concern in our response to Weakness 2.

  3. As shown in the additional experiments in our response to Weakness 2, allowing guidance to be restored does not substantially affect the final performance. We also do not observe a large difference in training speed. When guidance restoration is allowed, ρ fluctuates between 1.95 and 2 for graduated tasks. We report the actual training time below. Allowing guidance restoration increases the average training time per step by approximately 10%.

    OPGG (default)With guidance restoration
    Relative training time per step1.000x1.105x
  4. First, we would like to clarify that Scaf-GRPO [1] uses oracle-generated reasoning and adds hints when the model fails to produce a correct answer. Therefore, the difference is not simply whether a solution is provided. We acknowledge that our original description may have been unclear, and we have revised the related work section accordingly. Furthermore, as discussed in our response to Weakness 1, we have added experiments in Section 4.4 where three existing hint-based RL baselines are trained using our procedural guidance. These results provide additional evidence for the effectiveness of procedural guidance independently of OPGG.

[1] Zhang, Xichen, et al. "Scaf-grpo: Scaffolded group relative policy optimization for enhancing llm reasoning." International Conference on Learning Representations. 2026.

About Presentation

In addition, we learn the reviewer's concern about the clarity of our writing, and we apologize if some parts of the paper were difficult to follow. Since the reviewer raised issues with the presentation, we would appreciate it if you could point out specific sections or sentences that were unclear or ambiguous. This would help us better understand the issues and make targeted improvements to the paper. We sincerely appreciate your feedback and would like to improve the clarity of our work as much as possible.

AuthorsOpenReview ↗

Reminder: Response to Our Rebuttal

Dear Reviewer miiw,

We apologize for our late response, since we needed additional time to conduct supplementary experiments and provide additional experimental results and details.

We have also added an "Additional Summary of Paper" in the general comment section to further clarify the key contributions of our paper.

As the discussion period is approaching its deadline, we would greatly appreciate it if you could take a moment to review our rebuttal and share your feedback.

Thank you very much for your time and consideration.

Best regards, The Authors

Reviewer_Jat5OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper improves upon hint-based RL techniques by having the provided step-by-step reasoning trace and masking come from the generator, VRGym, that generates the question rather than an oracle LLM that could have buggy reasoning traces. Building ontop of VRGym, the authors introduce OPGG, a scheduling mechanism that exploits the controllability of VRGym's procedural guidance.

Critical strengths
  1. The framework introduces an intuitive and elegant approach to RLVR by leveraging problem generators to simultaneously synthesize problems and structured, step-by-step reasoning derivations.
  2. Decent improvement over baseline methods except compared to Scaf-GRPO; but, OPGG is faster, which makes up for it.
Critical weaknesses

[1] Wang et .al, Perception Aware Policy Optimization (PAPO) for Multimodal Reasoning, ICLR 2026 (https://arxiv.org/pdf/2507.06448). [2] Liu et al., NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation, NeurIPS 2025, https://openreview.net/forum?id=9zD2i7YRot. Below I enumerate the concerns that motivated my reject. I am open to discussion if I have any misunderstanding.

  1. The evaluation lacks comparisons against state-of-the-art dynamic sampling methods (such as DAPO) or other visual grounding and policy optimization approaches (e.g., PAPO [1] or NoisyRollout [2])).
  2. The ablation study (Table 4) demonstrates that simplified variants (such as mask-only or removal-only) drop performance by less than 1 percentage point. This suggests that a complex, two-phase guidance-withdrawal schedule (ρt\rho_t) may be an over-engineered solution.
  3. While the paper emphasizes a significant increase in non-zero advantage groups (+2,570 groups, Table 3), it remains unclear whether these extra gradient signals translate to genuine, generalizable reasoning capabilities rather than simply overfitting to synthetic procedural distributions. It would also be nice to see (1) An analysis or breakdown of potential failure modes or noisy outputs from the VRGym procedural generator (e.g., edge cases, malformed questions, or unnatural phrasing; (2) A comparative error analysis contrasting rollouts from OPGG and the baselines would help clarify how guided training specifically alters the model's intermediate reasoning chains or fixes visual-grounding mistakes.
  4. Although the model trains on 51 procedural tasks within VRGym, its evaluation on external benchmarks (such as MathVista, MathVision, and MMMU) yields exceptionally tight performance deltas with other baselines. This could mean that procedural generation could suffer from a domain gap when transitioning from synthetic templates to complex, real-world visual data.
  5. The authors explicitly acknowledge in Section 5 that "uneven task calibration" causes guidance budgets to oscillate on overly difficult tasks or become trivial on easy ones. This needs to be discussed in more detail as it could directly hinder the model's generalization to novel domains remain severely constrained if it requires meticulous meticulous, task-specific tuning of difficulty distributions.
Questions for authors

[1] Wang et .al, Perception Aware Policy Optimization (PAPO) for Multimodal Reasoning, ICLR 2026 (https://arxiv.org/pdf/2507.06448). [2] Liu et al., NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation, NeurIPS 2025, https://openreview.net/forum?id=9zD2i7YRot.

These are a reiteration of the above points in question format:

  1. Do results generalize to DAPO? How does OPGG compare to other visual grounding benchmarks like PAPO [1] and NoisyRollout [2]?
  2. Does the gradient signals from the +2,570 translate to genuine, generalizable reasoning capabilities rather than simply overfitting to synthetic procedural distributions? The deltas on MathVista, MathVision, MMMU, MMLU-Pro are extremely tight. why is this the case?
  3. Are there any potential failure modes or noisy outputs from the VRGym procedural generator (e.g., edge cases, malformed questions, or unnatural phrasing? How does your algorithm take that into account? Some sort of filtering? And how do you the rollouts from baselines compare to OPGG? This would help clarify how guided training specifically alters the model's intermediate reasoning chains or fixes visual-grounding mistakes.
  4. Could you provide on how meticulous tuning is required for different tasks? I worry if it is a lot this will hinder OPGG's out-of-the-box applicability and generalization to novel domains remain severely constrained.
12 comments

We thank the reviewer for taking the time to review our paper. Based on the feedback from all three reviewers, we realized that some parts of our paper were not clearly written, which may have led to misunderstandings. To address this, we have added a comment titled "Reiterating the Contributions" to clarify and restate the key contributions of our work.

Weaknesses

  1. First, the main reason we did not include DAPO [1] as a baseline is that all our compared methods are based on GRPO. Regarding the reviewer's suggestion to use dynamic sampling, we consider its goal different from that of hint-based RL. Hint-based RL aims to help models solve difficult problems that they would otherwise fail to solve, while dynamic sampling resamples problems that are too easy or too difficult. Therefore, we did not include it in our original comparisons. However, dynamic sampling can be directly applied to OPGG and some existing methods. Despite the limited time available for additional training, we conducted experiments with GRPO and OPGG using dynamic sampling. The results are shown below, and we add this result to Appendix E.1 as well.

    Qwen2.5-VL-7B-InstructVRGymMathVistaMathVisionMathVerseDynaMathMMMUMMMU-ProAvg.
    GRPO with Dynamic Sampling26.7570.7426.6447.8153.2754.1737.4645.26
    Δ vs. GRPO+5.52+0.57+1.93-0.25+2.40+0.23+1.96+1.76
    OPGG with Dynamic Sampling27.4771.2528.0848.8254.2955.4938.6646.29
    Δ vs. OPGG+3.81+0.11+0.12-0.46+2.24+1.14+0.61+1.08

    Regarding PAPO [2] and NoisyRollout [3], both methods have been evaluated on visual reasoning tasks, but their main focus differs from that of hint-based RL. Both methods aim to improve the model's visual perception ability, while hint-based RL mainly uses text-based augmentation to improve reasoning and address advantage collapse. It does not directly focus on improving visual understanding. For this reason, we did not include PAPO and NoisyRollout in our comparisons.

    However, our procedural guidance also includes reasoning steps based on visual information from images. We thank the reviewer for pointing out this connection, as exploring how procedural guidance can further improve visual perception is an interesting direction for future work.

  2. We agree that the changes in our ablation studies do not substantially affect OPGG's performance. As clarified in our "Reiterating the Contributions" comment, OPGG is a simple approach we developed to demonstrate the feasibility of procedural guidance. The purpose of our ablation studies is to explain our parameter choices and evaluate whether the design is reasonable.

    The two-phase guidance-withdrawal schedule was designed to make use of the properties of procedural guidance. However, we acknowledge the reviewer's concern that this design may be over-engineered. Our paper focuses on demonstrating the value of procedural guidance, while OPGG's scheduling mechanism is only one way to use it for training and is not a main contribution of this work.

    To better support our main contribution, we have added new experiments in Section 4.4, where we apply procedural guidance to other hint-based RL methods without using OPGG. This allows us to evaluate the effect of procedural guidance independently of OPGG's scheduling mechanism. In particular, Scaf-GRPO improves across all seven benchmarks, with the largest gain on MathVision (+4.93 percentage points) and an average improvement of +1.78 percentage points with procedural guidance.

Weaknesses

  1. We thank the reviewer for raising this important concern. We agree that an increase in non-zero advantage does not necessarily imply that the additional gradient signals are useful. Our original objective in Table 3 was to show that our method increases the proportion of non-zero advantages. Together with the improvements on external benchmarks, this suggests that the additional signals may be helpful. However, we acknowledge that this evidence is indirect. Therefore, we have moved this analysis to the appendix and no longer present it as a major benefit of our method.

    (1) Correctness and unnatural phrasing: Our tasks are generated using strict programmatic rules, and all generated tasks have been manually checked. Therefore, we can ensure the correctness of the generated tasks and their solutions. Regarding unnatural phrasing, our guidance is generated programmatically rather than by an LLM. As a result, the text may be less natural, as it mainly connects reasoning steps and values using predefined phrases to form natural-language descriptions, as shown in Figure 4. We agree that this may be considered unnatural phrasing to some extent. However, the key advantage is that the correctness of the reasoning steps is guaranteed by the program.

    (2) Model's intermediate reasoning chains: To further illustrate this, we have added an inference example from a Chart QA task in Appendix H. In this example, LUFFY [4] produces incorrect reasoning, Hint-GRPO [5] misinterprets the question, and Scaf-GRPO [6] makes a perception error by reading an incorrect value from the chart. In contrast, only the model trained with OPGG and procedural guidance produces a correct reasoning chain. This example suggests that the complete reasoning steps provided by procedural guidance during training can also help reduce certain visual perception errors.

    (3) Visual grounding: Our work does not specifically target visual grounding during training. However, as the reviewer correctly observed, our generated guidance does contain visual grounding information. We treat this information as part of the reasoning process, while the training signal is still based on whether the final answer is correct. Therefore, we do not explicitly optimize visual grounding in this paper. As we mentioned in Weakness 1, although visual grounding is not the primary focus of this work, we believe it presents a promising direction for future research.

  2. We fully agree with the reviewer and have also acknowledged this limitation in our paper. Since our work focuses on mathematics, logical reasoning, and chart understanding, its applicability to real-world scenarios is limited. In particular, extending procedural guidance to tasks involving realistic images would be challenging. However, this is a common limitation of procedural environments, as seen in Reasoning Gym [7] and KORGym [8].

    On the other hand, we believe procedural environments and procedural guidance have strong potential for agentic tasks, such as computer use and web browsing [9], where tasks and interactions can be generated programmatically at scale. By incorporating procedural guidance into these environments, we could provide additional training signals to support agentic RL.

  3. We use the same hyperparameters across all 51 tasks in VRGym, without any task-specific settings. All tasks follow the same rules, so no additional task-specific tuning is required. Although further tuning may improve performance, we prioritize keeping the method simple and clear.

    We also report the budget changes for each task in Appendix G to show how individual tasks improve during training. In addition, we provide the actual graduation results for each task in our response to Reviewer pc9X's Question 4. These results show that improvements are indeed more limited on harder tasks, while tasks that successfully graduate achieve significant improvements.

    As we mentioned in Weakness 4, such as agentic tasks [9], where tasks can also be controlled programmatically. In such settings, OPGG may not be the best approach. OPGG is a demonstrates of how procedural guidance can be used.

Questions

  1. We have addressed this concern in our response to Weakness 1.

  2. We have addressed the gradient signal concern in our response to Weakness 3. Regarding the reviewer's concern about whether the improvements over other baselines are simply due to overfitting to synthetic procedural distributions, we would like to clarify that all baselines are trained on the same synthetic procedural data generated by VRGym. Therefore, the main differences between these methods are the source of guidance and the scheduling mechanism.

    To evaluate whether training on VRGym causes overfitting, we believe it is more appropriate to compare the trained model with the original base model on external benchmarks that were not used for training.

    ModelMathVistaMathVisionMathVerseDynaMathMMMUMMMU-ProAvg.
    Qwen2.5-VL-7B-Instruct68.4424.8847.3249.7453.0436.3846.63
    OPGG71.1427.9649.2852.0554.3538.0548.81

    As shown in Table 1, when excluding the VRGym test split, OPGG still achieves an average improvement of +2.18 points over the original model across six external benchmarks. This is a meaningful improvement for multimodal reasoning using only RLVR training, suggesting that the benefits of our method are not limited to the synthetic procedural training distribution.

  3. As explained in our response to Weakness 3 (1), our reasoning chains are generated using strict programmatic rules and validated for correctness. Therefore, we can ensure that the generated reasoning chains are correct. The main limitation is the unnatural phrasing mentioned by the reviewer. Since procedural guidance is constructed using conditional statements and loops, its language can be less natural than model-generated text. While this is a limitation, it also reflects a key advantage of our approach: full control over the generated content. Regarding the comparison between baseline rollouts and OPGG, we have addressed this in our response to Weakness 3 (2) and provided a concrete example in the Appendix H.

  4. We have addressed this concern in our response to Weakness 5.

[1] Yu, Qiying, et al. "Dapo: An open-source llm reinforcement learning system at scale." Advances in Neural Information Processing Systems 38 (2026): 113222-113244.

[2] Wang, Zhenhailong, et al. "Perception-aware policy optimization for multimodal reasoning." International Conference on Learning Representations. Vol. 2026. 2026.

[3] Liu, Xiangyan, et al. "Noisyrollout: Reinforcing visual reasoning with data augmentation." Advances in Neural Information Processing Systems 38 (2026): 2923-2957.

[4] Yan, Jianhao, et al. "Learning to reason under off-policy guidance." Advances in Neural Information Processing Systems 38 (2026): 117157-117186.

[5] Huang, Qihan, et al. "Boosting mllm reasoning with text-debiased hint-grpo." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.

[6] Zhang, Xichen, et al. "Scaf-grpo: Scaffolded group relative policy optimization for enhancing llm reasoning." International Conference on Learning Representations. Vol. 2026. 2026.

[7] Stojanovski, Zafir, et al. "Reasoning gym: Reasoning environments for reinforcement learning with verifiable rewards." Advances in Neural Information Processing Systems 38 (2026).

[8] Shi, Jiajun, et al. "Korgym: A dynamic game platform for llm reasoning evaluation." Advances in Neural Information Processing Systems 38 (2026): 161286-161314.

[9] Lai, Hanyu, et al. "Computerrl: Scaling end-to-end online reinforcement learning for computer use agents." International Conference on Learning Representations. Vol. 2026. 2026.

Reviewer_pc9XOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the lack of a reward-derived learning signal when all rollouts in a GRPO group are incorrect. The authors introduce VRGym, which generates visual reasoning problems together with answers and derivations containing maskable values. Their method, OPGG, places part of a derivation in the training prompt and maintains a guidance-withdrawal budget for each task. Based on rollout-group accuracy, it first masks values and then removes derivation text. Training still uses a binary final-answer reward and GRPO; evaluation prompts contain no guidance.

Critical strengths
  1. A clear way to provide controllable guidance. VRGym supplies derivations with maskable fields, allowing guidance to be reduced during training while keeping evaluation unguided.
  2. Consistent reported improvements. OPGG has the highest seven-dataset macro-average among the evaluated methods on both backbones. The increase in groups with non-zero reward-derived advantage also speaks directly to the paper’s motivation.
Critical weaknesses
  1. The procedural-versus-oracle comparison does not adequately control the information in the hints. Section 4.5 holds the training setup fixed and disables the schedule, but compares an initial procedural derivation with a free-form Gemini-generated hint. The procedural hint supplies values read from the image and a task-specific reasoning structure while masking computed intermediate results and the final answer. The oracle hint may differ in how much of this information it contains. Thus, the 0.76-point difference does not isolate the benefit of the procedural hint’s reliability; hint content, structure, or task fit could also explain it. A comparison that characterizes and, where feasible, matches these properties would strengthen the claim.
  2. The value of accuracy-driven withdrawal over a simple schedule remains unclear. Section 4.4 compares masking-only and removal-only variants and changes two update-rule settings, but does not compare OPGG with linear or exponential withdrawal using the same procedural hints and training budget. The results establish that the complete method works, but do not show whether feedback from per-task group accuracy is more useful than a preset schedule. Moreover, the non-zero-advantage analysis compares OPGG only with unguided GRPO, leaving its behavior relative to other hint-based methods unclear.
Questions for authors
  1. For Table 5, how do the two hint sources compare in length and in their coverage of image-read values, operations, and reasoning steps? Can the comparison be repeated with these properties matched as closely as feasible?
  2. Can you report the share of groups with non-zero reward-derived advantage for at least one in-prompt hint baseline, counting only the first eight rollouts per prompt for a consistent comparison? For Scaf-GRPO, please report retries and their sampling cost separately.
  3. How were α=0.8\alpha=0.8, δ+=0.05\delta^+=0.05, and δ−=0.01\delta^-=0.01 selected? In addition to the reported α=1.0\alpha=1.0 and equal-step-size ablations, could you examine α=0.4\alpha=0.4 and 0.60.6, vary the withdrawal and restoration step sizes separately, and report task-level budget trajectories?
  4. By the end of the 500-step run, how many tasks have reached the no-guidance state? How does unguided evaluation performance differ between tasks that have and have not reached it?
4 comments

We thank the reviewer for the positive feedback. Based on the comments from all three reviewers, we realized that some parts of our paper were not clearly written, which may have led to misunderstandings. To address this, we have added a comment titled "Reiterating the Contributions" to clarify and restate the key contributions of our work.

Weaknesses

  1. First, we acknowledge that the oracle model may fail to capture all the information in an image. However, this is a limitation of relying on an oracle model, as its accuracy and consistency cannot be guaranteed. Our proposed procedural guidance is designed to address this issue by ensuring that all relevant information in each image is accurately described. This difference in the information provided by the guidance is exactly what we aim to evaluate.

    Additionally, similar concerns about this experiment were raised by another reviewer. We agree that the original experiment was not sufficiently rigorous, so we have moved it to the appendix and added a new experiment in Section 4.4. In this experiment, we replace the guidance used by other baseline methods with our proposed procedural guidance while keeping their scheduling mechanisms unchanged. This allows us to isolate the effect of the guidance itself. The results show that procedural guidance further improves the performance of existing on-policy hint-based methods. In particular, Scaf-GRPO improves across all seven benchmarks, with the largest gain on MathVision (+4.93 percentage points) and an average improvement of +1.78 percentage points with procedural guidance.

  2. We consider OPGG to be a simple scheduling method as well. If guidance is reduced based only on training steps rather than group accuracy, it is difficult to determine whether the model has learned to solve problems without relying on guidance. It is also unclear how much guidance should be withdrawn at each step. Using model performance to adjust guidance is also common in previous hint-based methods. For example, Scaf-GRPO [1] increases the number and strength of hints when the model fails to produce a correct answer. Therefore, we believe that using group accuracy to schedule guidance is a simple and reasonable choice.

    Nevertheless, following the reviewer's suggestion, we also conducted an experiment with a linear schedule. Specifically, we linearly increase the budget from 0 at the beginning of training to 2 at the end of training. The results are shown in the table below.

    MethodVRGymMathVistaMathVisionMathVerseDynaMathMMMUMMMU-ProAvg.
    OPGG (default)23.6671.1427.9649.2852.0554.3538.0545.21
    linear22.3170.6426.5248.5650.1154.3838.0344.36

    Regarding the non-zero advantage analysis, we did not report results for other methods because differences in their mechanisms would make a direct comparison unfair. Another reviewer also raised a related concern. After further examination, we agree that an increase in non-zero advantage does not necessarily mean that the additional gradient signals are useful. Our original goal in Table 3 was to show that our method increases the proportion of non-zero advantages. Together with the improvements on external benchmarks, this suggests that the additional signals are helpful. However, we acknowledge that this evidence is indirect. Therefore, we have moved this analysis to the appendix and no longer present it as a major benefit of our method.

[1] Zhang, Xichen, et al. "Scaf-grpo: Scaffolded group relative policy optimization for enhancing llm reasoning." International Conference on Learning Representations. 2026.

Questions

  1. We can provide the average hint length for both types of guidance. However, comparing the coverage of image-read values, operations, and reasoning steps is less straightforward. Our procedural guidance defines a step-by-step solution for each problem, while oracle-generated guidance is harder to control. When prompting an oracle model to generate a solution, we cannot guarantee that it will refer to specific visual features or include a certain number of reasoning steps. This is one of the main advantages of our procedural guidance, as each step can be explicitly controlled. An example is shown in Figure 4.

    The table below reports the average hint length in tokens for each method. Note that these methods generate guidance in different ways. For example, Scaf-GRPO [3] prompts the oracle model to generate different types of content, which are then combined to form guidance of varying strengths. In contrast, LUFFY [1] simply prompts the oracle model to generate a reasoning process.

    Procedural guidance (VRGym)Oracle guidance (LUFFY [1])Oracle guidance (Hint-GRPO [2])Oracle guidance (Scaf-GRPO [3])
    token length415.887.6184.7263.9

    As discussed in our response to Weakness 2, we agree with the reviewer's concern that this experiment was not sufficiently rigorous. Therefore, we have moved it to the appendix and no longer present it as a major benefit of our method.

    Regarding Scaf-GRPO [3], we found that our original description was inaccurate, which may have led to a misunderstanding. We have corrected the paper and would like to clarify the mechanism here. For groups in which all responses are incorrect, Scaf-GRPO [3] generates 12 additional rollouts with different levels of hints. These rollouts are still generated in parallel, using vLLM through the verl framework to handle inference.

  2. Regarding sampling cost, the average training time is reported in Table 2. In terms of the number of sampled rollouts, Scaf-GRPO [3] requires 1.81× as many rollouts as the original setting on average. Under our training setup, each step originally contains 256 rollouts, while Scaf-GRPO generates approximately 207 additional rollouts per step on average.

  3. As clarified in our initial contribution reiteration, the hyperparameters of OPGG were not specifically tuned. Our ablation studies are intended to show that these parameter choices are reasonable. OPGG is one way to apply procedural guidance, and our main goal is to show that the fine-grained control provided by procedural guidance is more reliable than the guidance used in previous hint-based RL methods. As shown in our ablation results, the performance is not highly sensitive to these hyperparameter choices. Nevertheless, we have tried to address the reviewer's requested ablations as much as possible.

    • Effect of α: Due to limited experimental time, we provide an additional result with α = 0.4.
    MethodVRGymMathVistaMathVisionMathVerseDynaMathMMMUMMMU-ProAvg.
    OPGG (default)23.6671.1427.9649.2852.0554.3538.0545.21
    α = 0.422.7171.1126.6948.6152.3354.1537.9244.79
    • Withdrawal and restoration: We have already provided results comparing symmetric and asymmetric withdrawal and restoration settings. It is difficult to analyze these two mechanisms independently because our goal is to gradually remove guidance so that the model learns to solve problems without relying on it. One possible additional experiment would be to set the restoration parameter δ⁻ to 0, allowing only withdrawal. However, we do not consider this experiment necessary for supporting our main conclusions.
    • Task-level budget changes: Since our training covers many tasks, we have added the training budget changes for each of the 51 tasks in Appendix G.

Questions

  1. Below, we report the avg@8 results for graduated and ungraduated tasks, both overall and across the task categories included in training. We observe that graduated tasks are generally easier by design and benefit more from guidance. In contrast, ungraduated tasks tend to be more difficult, making it harder for the model to fully benefit from guidance. This is consistent with Limitation 2, where we acknowledge that our task design may not be perfectly balanced and that some tasks may still be too difficult for the model.

    Model / Task GroupOverallMathLogicAlgoChart QA
    Qwen2.5-VL-7B-Instruct / Graduated26.941.021.632.423.1
    Qwen2.5-VL-7B-Instruct + OPGG / Graduated50.274.231.952.346.5
    Δavg@8 / Graduated+23.4+33.1+10.3+19.9+23.4
    Qwen2.5-VL-7B-Instruct / Ungraduated5.09.45.01.03.1
    Qwen2.5-VL-7B-Instruct + OPGG / Ungraduated10.812.111.76.218.1
    Δavg@8 / Ungraduated+5.8+2.6+6.8+5.1+15.0

    We also report the proportion of graduated and ungraduated tasks in each category below.

    Task GroupOverallMathLogicAlgoChart QA
    Graduated31.4% (16/51)25.0% (3/12)11.8% (2/17)10.0% (1/10)83.3% (10/12)
    Ungraduated68.6% (35/51)75.0% (9/12)88.2% (15/17)90.0% (9/10)16.7% (2/12)

[1] Yan, Jianhao, et al. "Learning to reason under off-policy guidance." Advances in Neural Information Processing Systems 38 (2026): 117157-117186.

[2] Huang,Qihan, et al. "Boosting mllm reasoning with text-debiased hint-grpo." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.

[3] Zhang, Xichen, et al. "Scaf-grpo: Scaffolded group relative policy optimization for enhancing llm reasoning." International Conference on Learning Representations. Vol. 2026. 2026.

AuthorsOpenReview ↗

Reminder: Response to Our Rebuttal

Dear Reviewer pc9X,

We apologize for our late response, since we needed additional time to conduct supplementary experiments and provide additional experimental results and details.

We have also added an "Additional Summary of Paper" in the general comment section to further clarify the key contributions of our paper.

As the discussion period is approaching its deadline, we would greatly appreciate it if you could take a moment to review our rebuttal and share your feedback.

Thank you very much for your time and consideration.

Best regards, The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Reiterating the Contributions

Our writing may not have been clear enough and may have unintentionally misled the reviewers. To address this, we would like to clarify the main contributions of our paper.

  • Our goal is to introduce a new training paradigm, with OPGG serving as an example. The core insight of this work is not OPGG itself, but the introduction of procedural guidance and empirical evidence of its effectiveness. Our ablation studies are intended only to show that our design and parameter choices are reasonable. Based on our limited experiments, small changes in these parameters do not substantially affect the final results. Our main contribution is to provide empirical evidence that accurate and complete procedural guidance can outperform existing hint-based RL methods.

  • Based on the clarifications above, we have revised the main paper, moved some analyses to the appendix, and added new experiments in Section 4.4 to more directly evaluate the effectiveness of procedural guidance. These changes aim to provide stronger evidence for procedural guidance rather than focusing on OPGG.

    In the new experiments, we replace oracle guidance in existing baseline methods with VRGym's procedural guidance, while keeping their original guidance schedules and other training settings unchanged. The results show that procedural guidance further improves the performance of existing on-policy hint-based methods, such as Scaf-GRPO [1]. We also provide relevant portion of the updated Table 4 below for reference.

    Qwen2.5-VL-7B-InstructVRGymMathVistaMathVisionMathVerseDynaMathMMMUMMMU-ProAvg.
    Scaf-GRPO [1]24.61 (+1.56)72.38 (+1.22)29.56 (+4.93)50.44 (+1.46)51.58 (+1.85)54.10 (+0.37)37.64 (+1.08)45.76 (+1.78)

    As shown in Table 4, procedural guidance improves performance across all seven benchmarks, with the largest gain on MathVision (+4.93 points). On average, it achieves an improvement of +1.78 percentage points, further supporting the effectiveness of procedural guidance beyond OPGG.

  • All newly added content and material moved to the appendix are marked in blue.

[1] Zhang, Xichen, et al. "Scaf-grpo: Scaffolded group relative policy optimization for enhancing llm reasoning." International Conference on Learning Representations. Vol. 2026. 2026.

AuthorsOpenReview ↗

Additional Summary of Paper

To further clarify the key contributions of our paper and address the remaining questions raised by the reviewers, we would like to provide a brief summary.

Our paper introduces three components: VRGym (dataset), Procedural Guidance (guidance), and OPGG (guidance scheduling mechanism).

We consider VRGym and Procedural Guidance to be our main contributions, especially Procedural Guidance, as highlighted in our paper title. OPGG is a minor contribution that provides a simple heuristic scheduling mechanism for applying Procedural Guidance to visual reasoning tasks.

We believe these three components should be evaluated separately, using different experimental results.

1. Is the VRGym task design effective?

The most direct way to evaluate VRGym is to compare the original models with models trained using standard GRPO. The results are shown below.

ModelVRGymMathVistaMathVisionMathVerseDynaMathMMMUMMMU-ProAvg.Avg. w/o VRGym
Qwen2.5-VL-7B-Instruct14.1568.4424.8847.3249.7453.0436.3841.9946.63
+GRPO21.2370.1724.7148.0650.8753.9435.5043.5047.21
Δ (GRPO − Base)+7.08+1.73−0.17+0.74+1.13+0.90−0.88+1.50+0.57
Qwen3.5-4B-Base29.6974.6932.4064.4071.8666.5150.5655.7360.07
+GRPO41.7176.6538.0368.7474.3468.0753.7260.1863.26
Δ (GRPO − Base)+12.02+1.96+5.63+4.34+2.48+1.56+3.16+4.45+3.19

The improvement on the Instruct model is relatively limited. We believe this is expected because applying RL to models that have already undergone extensive post-training can be less effective, as these models typically have lower entropy and limited exploration ability [1]. As a result, further training with standard GRPO often brings only small improvements, mainly on in-domain tasks (VRGym).

In contrast, when training a base model, we observe clear improvements across external benchmarks. The average score excluding VRGym increases by 3.19 points, compared to only 0.57 points for the Instruct model.

These results suggest that VRGym provides useful training tasks that can improve generalization beyond its own domain.

Note: Since Qwen2.5-VL and Qwen3-VL do not release base models, we use Qwen3.5-4B-Base for this experiment.

2. Is Procedural Guidance better than Oracle Guidance?

Our new experiments in Sec. 4.4 show that Procedural Guidance can improve on-policy hint-based RL.

Qwen2.5-VL-7B-InstructVRGymMathVistaMathVisionMathVerseDynaMathMMMUMMMU-ProAvg.Avg. w/o VRGym
Scaf-GRPO + Oracle Guidance (Gemini 3.7 Flash)23.0571.1624.6348.9849.7353.7336.5643.9847.47
Scaf-GRPO + Procedural Guidance24.6172.3829.5650.4451.5854.1037.6445.7649.28
Δ (Procedural − Oracle)+1.56+1.22+4.93+1.46+1.85+0.37+1.08+1.78+1.82

By changing only the guidance source, Procedural Guidance achieves better results across all benchmarks, including an average improvement of 1.82 points on external benchmarks.

These results show that Procedural Guidance can improve on-policy hint-based RL by providing reliable guidance with correct answers and reasoning steps, leading to better performance even on external mathematical reasoning tasks.

3. Is the OPGG mechanism reasonable?

Through our ablation studies, we show that the default parameter choices and design of OPGG are reasonable.

However, we still consider OPGG a minor contribution, as it is simply a heuristic scheduling mechanism.

As shown in the experiments above, Procedural Guidance can also be applied to other on-policy hint-based RL methods. OPGG is just one simple approach we designed to demonstrate how Procedural Guidance can be used during RL training.

Conclusion

We believe Procedural Guidance has strong potential for future research. For any procedural environment, we can make small changes to the code so that it generates not only questions but also detailed reasoning steps with guaranteed correctness. We refer to these reasoning steps as “Procedural Guidance”.

Therefore, the concept of Procedural Guidance is not limited to VRGym. It can be applied to other procedural environments and integrated into existing training methods that use procedurally generated tasks.

We believe this is the key value of Procedural Guidance and an important direction for future research.

[1] Yue, Yang, et al. "Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?." Advances in Neural Information Processing Systems 38 (2026): 57654-57689.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.