- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Mode-wise Likelihood Optimization (MILO), a post-training method intended to increase the diversity of correct reasoning strategies produced across multiple samples while preserving repeated-sampling correctness. MILO assigns each observed correct mode equal total weight in a mode-wise policy-gradient term and combines it with a pooled-correctness objective. The paper provides an analysis relating the update to expected CorrectModes@k and evaluates it on several mathematical reasoning datasets. Overall, I find the problem interesting and the proposed objective promising, but the motivation for prioritizing mode coverage and several aspects of the method's presentation need clarification. The empirical evaluation covers a reasonable set of datasets for one paper, though the number and diversity of evaluated base models are limited.
Critical strengths
-
MILO proposes a meaningful objective beyond pooled correctness to distinguish a pool with repeated versions of one approach from a pool with several correct strategies.
-
The weight gives a practical control over the emphasis on broader mode coverage.
-
The experiments include GSM8K and additional arithmetic/reasoning datasets, as well as multiple mode labelers and labeler-free diversity measures.
Critical weaknesses
-
The practical value of mode coverage needs sharper motivation. The paper argues that broader coverage may help downstream selection, verification, or self-training, but it is not clear that more modes help when the task only requires finding one correct answer and no verifier or downstream process benefits from alternative strategies. Since the reported pass@k is already near its ceiling in key experiments, the paper should better distinguish settings where mode coverage provides measurable utility from settings where it is simply an additional diversity objective.
-
Equal weighting of observed modes may be counterproductive. The mode-wise term gives a rare observed mode the same total weight as a frequent mode. This is intuitive if the goal is broad coverage, but the paper could better explain why equal weighting is desirable, rather than, for example, applying a softer frequency correction or weighting modes by downstream value. Correctness alone does not indicate that all strategies are equally useful. A clearer discussion and ablations over alternative mode weights would help establish when this bias improves outcomes and when it may overemphasize noisy or low-value modes. The role of is helpful, but does not by itself answer why the target distribution should be uniform over modes.
-
Several central concepts and theoretical claims need more intuition. The abstract uses CorrectModes@k before defining it; a brief definition would make the contribution easier to grasp. The introduction also invokes a “distributional objective” and a “local correctness optimum” without enough plain-language explanation of what readers should take away. Theorem 2 and Eq. 3 would also benefit from a compact worked example showing how the pooled and mode-wise terms combine for correct and incorrect rollouts. Finally, the contrast with UCPO would be clearer with a small example: several distinct responses can express the same strategy, so uniformity over responses need not mean uniformity over reasoning modes.
-
The labeling pipeline and model generality need clearer treatment. The paper uses auxiliary models to construct mode labels: Qwen2.5-0.5B-Instruct for training strategy descriptions and Qwen2.5-1.5B-Instruct for evaluation, with Qwen3-generated responses used to construct the respective cluster banks. These are not correctness verifiers; correctness is checked against the answer key. Still, the multi-model labeling pipeline adds inference and implementation complexity, and the paper should make the roles, costs, and reproducibility implications especially clear. Separately, the main post-training experiments center on Qwen3-4B-Base, with a Qwen3-8B result in the appendix. The dataset coverage is reasonable, but evaluation on more base-model families and scales is needed to establish that the gains are not specific to Qwen3.
Questions for authors
-
In what realistic settings does increased correct-mode coverage improve end-to-end task success beyond pass@k?
-
What motivates equal total weight for each observed correct mode? How does MILO compare with softer frequency corrections or mode weights based on downstream utility, and how sensitive are results to noisy or unstable mode assignments?
-
Could the paper add a simple worked example for Eq. 3 and for the distinction between UCPO's response-level uniformity and MILO's mode-level weighting? Could it also define CorrectModes@k in the abstract and explain the implications of Theorem 2 more directly?
-
Could the authors evaluate additional base-model families and clarify the compute and data flow for the auxiliary mode-labeling models, separate from the ground-truth correctness verifier?