- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3(+1)
- Soundness
- 2(+1)
Read review
Paper summary
The paper introduces minimal-edit corrective on-policy distillation, a post-training method that aims to improve the perceptual capabilities of pretrained VLMs. The method modifies the standard GRPO algorithm by scoring the incorrect rollouts using a teacher that has access to a reference solution and a figure description. The teacher is instructed to minimally edit the incorrect rollout, and its logprobs of the existing incorrect rollouts are used as targets in a reverse KL objective. Only the token positions that the teacher disagrees with are used for the optimization. Correct rollouts are reinforced with the standard GRPO algorithm. The authors validate the method using two small (2B, 8B) models across four datasets against multiple baselines.
Critical strengths
- The idea is simple, intuitive, and does not introduce a lot of overhead as it does not need to generate from the teacher.
Critical weaknesses
- As part of the ablation study, Table 5 sweeps various configurations and shows that the method is extremely sensitive to the hyperparameter lambda. The method’s best performance reported in this table matches the one in the main table, which reports test set performance. This is model selection based on the test set, and it undermines confidence in the reported results. This is my largest concern.
- Some seemingly important choices were not ablated (see questions).
- The method’s experimental breadth is lacking. It uses two small, single-family (Qwen) models and evaluates on 4 datasets.
Minor: The methodology’s structure is a bit confusing. It starts with an experiment and discusses its results. The methodology is also too long (almost 3 pages) for an empirical paper.
Questions for authors
- How did you do model selection? Did you have a separate validation set?
- is defined as . Is there a missing and a missing instruction here? According to section 3.2 and Figure 3 the instruction is consequential. Can you provide an ablation to this with other instructions, ideally ones that do not mention minimal editing?
- Can you provide the experimental setup and the final performance metrics (i.e., augment Table 4) for the component configuration experiments?
- I am surprised by the fact that performing GRPO + token-level OPD at a random 2% of the tokens ends in divergence with training curves similar to the unmasked variants. How did you tune the models for these ablations?
- Why did you pick these datasets for evaluation? The paper’s experimental breadth could be improved by evaluating on MathVista, MM-Math, DynaMath, LogicVista and by adding at least one non-qwen-family model.