acceptodds
Under review as a conference paper at ICLR 2027

Edit, Don't Imitate: Minimal-Edit On-Policy Distillation for Multimodal Reasoning

Abstract

Visual mathematical reasoning can fail before reasoning begins: when a vision-language model (VLM) misreads a diagram, subsequent reasoning proceeds from false premises. This perceptual bottleneck is especially pronounced in smaller models, while accurate textual descriptions can substantially narrow the gap to larger models. Because obtaining such descriptions at inference requires additional annotation or model calls, we propose minimal-edit corrective on-policy distillation. Using reference figure descriptions and verified reference solutions as training-only information, a teacher scores incorrect student rollouts under a minimal-edit objective, and only teacher-disfavored tokens contribute to the corrective loss. Correct rollouts receive task-reward reinforcement learning. This preserves on-policy supervision while avoiding full-response imitation. Across model scales, the method requires no descriptions at test time and meets or exceeds the corresponding base model supplied with a reference figure description at inference. It also improves perception-oriented performance, particularly low-level geometric perception, and transfers to external mathematics and perception benchmarks. Because the minimal-edit teacher evaluates corrections rather than generating its own solution, the method remains effective with a frozen same-size teacher, although its gains are smaller than with a larger teacher on most metrics.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision Language Models, Multimodal Mathematics Reasoning, On-Policy Distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ffoDOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3(+1)
Soundness
2(+1)
Read review
Paper summary

The paper introduces minimal-edit corrective on-policy distillation, a post-training method that aims to improve the perceptual capabilities of pretrained VLMs. The method modifies the standard GRPO algorithm by scoring the incorrect rollouts using a teacher that has access to a reference solution and a figure description. The teacher is instructed to minimally edit the incorrect rollout, and its logprobs of the existing incorrect rollouts are used as targets in a reverse KL objective. Only the token positions that the teacher disagrees with are used for the optimization. Correct rollouts are reinforced with the standard GRPO algorithm. The authors validate the method using two small (2B, 8B) models across four datasets against multiple baselines.

Critical strengths
  • The idea is simple, intuitive, and does not introduce a lot of overhead as it does not need to generate from the teacher.
Critical weaknesses
  • As part of the ablation study, Table 5 sweeps various configurations and shows that the method is extremely sensitive to the hyperparameter lambda. The method’s best performance reported in this table matches the one in the main table, which reports test set performance. This is model selection based on the test set, and it undermines confidence in the reported results. This is my largest concern.
  • Some seemingly important choices were not ablated (see questions).
  • The method’s experimental breadth is lacking. It uses two small, single-family (Qwen) models and evaluates on 4 datasets.

Minor: The methodology’s structure is a bit confusing. It starts with an experiment and discusses its results. The methodology is also too long (almost 3 pages) for an empirical paper.

Questions for authors
  • How did you do model selection? Did you have a separate validation set?
  • xeditx_{\text{edit}} is defined as xedit:=(d,A,y)x_{\text{edit}}:=(d, A, y). Is there a missing xx and a missing instruction here? According to section 3.2 and Figure 3 the instruction is consequential. Can you provide an ablation to this with other instructions, ideally ones that do not mention minimal editing?
  • Can you provide the experimental setup and the final performance metrics (i.e., augment Table 4) for the component configuration experiments?
  • I am surprised by the fact that performing GRPO + token-level OPD at a random 2% of the tokens ends in divergence with training curves similar to the unmasked variants. How did you tune the models for these ablations?
  • Why did you pick these datasets for evaluation? The paper’s experimental breadth could be improved by evaluating on MathVista, MM-Math, DynaMath, LogicVista and by adding at least one non-qwen-family model.
Reviewer_QHDuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to transfer training-only diagram descriptions and verified solutions into vision-language models for visual mathematical reasoning. It proposes minimal-edit corrective on-policy distillation, which applies teacher-based token losses only to teacher-disfavored tokens of incorrect rollouts while applying GRPO to correct rollouts.

Critical strengths

A simple but useful corrective-distillation mechanism. Combining rollout routing, minimal-edit teacher conditioning, and a token-level sign mask together produces great results. The ablations in Figure 4 are helpful: neither unmasked minimal-edit scoring nor a random mask with the same roughly 2% keep rate remains stable. This suggests that the gains come from which tokens are selected, rather than sparsity alone.

Consistent gains across the evaluated tasks. Tables 1 and 2 show improvements on Geometry3K, MathVerse, and Geoperception, with particularly large gains in Geometry3K pass@1 and Geoperception for the 2B model. Table 3 also provides a useful comparison with existing Qwen2.5-VL-7B checkpoints on the same backbone. The method achieves the best Geometry3K and MathVerse results among the compared models, despite using a relatively small training set.

Critical weaknesses
  1. The base models seem somewhat outdated. Does the benefit of privileged information still hold for newer, more thoroughly trained models? It would be useful to see whether the same gains can be observed with stronger starting points.

  2. Figures 1 and 2 show noticeable signs of AI-generated illustrations, which makes the presentation look less polished.

  3. An obvious missing citation is MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Engine, which is closely related to the paper's topic.

Questions for authors
  1. The main experiments use Qwen3-VL-2B/8B and Qwen2.5-VL-7B. Given the rapid progress in VLMs, does minimal-edit distillation still provide substantial gains when applied to newer, more thoroughly trained models? In particular, does the benefit persist when the base model already has strong visual perception and mathematical reasoning capabilities?

  2. The training pipeline retains candidate solutions whose final answers pass a rule-based verifier. However, a correct final answer does not guarantee that the intermediate reasoning is correct. How often do the retained reference solutions contain flawed or invalid reasoning steps, and how sensitive is the method to the quality of these references?

Reviewer_tbK1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies visual mathematical reasoning in small VLMs. It first shows that adding a reference textual figure description at inference substantially improves Qwen3-VL models on Geometry3K, and argues that perception is a major bottleneck. To internalize this signal without overhead from test-time descriptions, it proposes "minimal-edit corrective on-policy distillation". Student rollouts are verified by a rule-based checker. Correct rollouts are trained with a GRPO loss. Incorrect rollouts are scored by a frozen teacher conditioned on the figure description, a verified reference solution, the student's own rollout, and an instruction to minimally edit it. Only tokens with positive log-ratio (about 2% of tokens) contribute to a corrective loss. Experiments on Qwen3-VL-2B/8B and Qwen2.5-VL-7B, trained on ~1.8K Geometry3K problems, report gains on Geometry3K, MathVerse, figure-description recall, and Geoperception, including with a frozen same-size self-teacher.

Critical strengths
  1. The motivating measurement is clean. It quantifies how much a faithful textual reading of the diagram helps at each model scale.

  2. Scoring the student's own rollout under a minimal-edit prompt is a cheap way to localize errors. It needs one teacher forward pass and no generation or dedicated rewriter model.

  3. The method trains on a small dataset (~1.8K problems) and reports competitive numbers against published Qwen2.5-VL-7B checkpoints trained on much more data.

Critical weaknesses

W1. The diagnosis does not connect to the method, which is my major concern.

The paper is motivated by a perception bottleneck, but nothing in the method is specific to perception:

W1.1 The mechanism is error-type agnostic. The teacher scores tokens and the loss lowers the probability of tokens the teacher disfavors. This applies equally to perceptual, reasoning, and arithmetic errors. One of the paper's own qualitative examples (Figure 8) is an arithmetic error. The same pipeline would run unchanged on text-only math.

W1.2. The diagnostic and the method test different things. Figure 2 measures giving the description to the student at inference. In the method the description appears only in the teacher's context. The paper never ablates the description in the teacher (it acknowledges this in Section 4.2), so it is unknown how much of the gain comes from the description at all.

W1.3. The reference solution, not the description, appears to drive the gains. Removing the reference solution drops Geometry3K pass@1 from 0.554 to 0.471 (Table 5). A full verified solution carries reasoning, not only perceptual information.

W1.4. Perception metrics do not support "internalized perception". On 2B, minimal-edit figure-description recall (0.633) is below privileged OPD (0.742) and below several SFT variants (Table 1). Evidence that selected tokens target perceptual errors consists of three hand-picked examples. There is no statistic on what fraction of selected tokens correspond to perceptual errors.

The perception story is therefore an introductory framing that the method and experiments do not validate.

W2. The "GRPO" branch is basically on-policy rejection-sampling fine-tuning

Correct rollouts are the only inputs to the task loss. With binary rewards and group normalization, every correct rollout in a group receives the same positive advantage. Incorrect rollouts never receive a negative advantage. If, as in the GRPO baselines (Appendix D), each rollout batch receives a single on-policy update, the importance ratio is 1 and clipping is inactive. The branch then reduces to difficulty-weighted maximum likelihood on the model's own correct samples. In other words, it is online rejection-sampling fine-tuning. Calling it GRPO is therefore misleading.

W3. The "OPD" framing does not hold

The minimal-edit teacher's context contains the very rollout it scores, with an instruction to copy correct parts. The teacher distribution therefore depends on the sample and assigns high probability to most of whatever the student wrote. This is not a fixed distillation target. Without the mask, the teacher cross-entropy term is nearly constant, and the reverse-KL objective degenerates toward entropy maximization. This matches the paper's own ablation, where the unmasked variant has a strongly negative loss and exploding entropy (Figure 4). Framing the method as OPD is therefore misleading.

Moreover, in Algorithm 1 (line 24) the teacher’s log-probability is a constant, so the gradient comes only from the student term. The corrective loss therefore just lowers the probability of the selected tokens. The teacher only decides which tokens to penalize. This is unlikelihood training, not distillation.

W4. Narrow experimental scope

--Training uses a single dataset, Geometry3K.

--The evaluation covers only geometry diagrams; other visual reasoning settings are not tested, despite the general title ("Multimodal Reasoning").

--It is unspecified which model generated the reference solutions in the self-teacher setting. If they come from the larger teacher, the "same-size teacher" claim is confounded.

Questions for authors
  1. Is the corrective term in the implementation differentiated directly, as Algorithm 1 suggests, or is the per-token log-ratio used as an advantage in a policy-gradient update?

  2. In the self-teacher experiments, which model generated the reference solutions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.