acceptodds
Withdrawn from ICLR 2027

MILO: Mode-wise Likelihood Optimization for Post-Training Diversity

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

People often ask language models for multiple candidate solutions, but sampling more responses does not necessarily produce more ways of solving a problem: many samples may repeat the same underlying strategy. We introduce Mode-wise Likelihood Optimization (MILO), a post-training method for increasing the diversity of correct solution strategies while preserving accuracy. MILO groups verified-correct responses that use the same strategy into correct modes, gives each mode its own log-likelihood term, and combines the resulting mode-wise objective with pooled correctness using a weight . Its mode-wise update gives each observed mode equal total weight, rather than favoring modes simply because they appear more often. We show that this update directly targets correct-mode coverage. For finite rollout groups, its expected mode-wise update is the gradient of a -weighted sum of expected CorrectModes@, where CorrectModes@ counts the distinct correct modes among samples. At a local correctness optimum, a small MILO step with improves this coverage objective to first order when its gradient is nonzero, while correctness changes only at second order. On Qwen3-4B-Base GSM8K, improves both single-sample accuracy and CorrectModes@, raising coverage from to . Increasing to raises coverage to while keeping single-sample accuracy comparable to the base model and pass@ nearly unchanged. The gains extend across datasets, sampling budgets, and independent mode labelers, and are also reflected by labeler-free diversity metrics. A controlled verifier-blind-spot experiment further illustrates how broader correct-mode coverage can help candidate pools retain usable correct answers when a verifier fails on common strategies.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Post-Training, Diverse Reasoning, Mode Coverage

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_jAGzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Mode-wise Likelihood Optimization (MILO), a post-training method intended to increase the diversity of correct reasoning strategies produced across multiple samples while preserving repeated-sampling correctness. MILO assigns each observed correct mode equal total weight in a mode-wise policy-gradient term and combines it with a pooled-correctness objective. The paper provides an analysis relating the update to expected CorrectModes@k and evaluates it on several mathematical reasoning datasets. Overall, I find the problem interesting and the proposed objective promising, but the motivation for prioritizing mode coverage and several aspects of the method's presentation need clarification. The empirical evaluation covers a reasonable set of datasets for one paper, though the number and diversity of evaluated base models are limited.

Critical strengths
  1. MILO proposes a meaningful objective beyond pooled correctness to distinguish a pool with repeated versions of one approach from a pool with several correct strategies.

  2. The weight λ\lambda gives a practical control over the emphasis on broader mode coverage.

  3. The experiments include GSM8K and additional arithmetic/reasoning datasets, as well as multiple mode labelers and labeler-free diversity measures.

Critical weaknesses
  1. The practical value of mode coverage needs sharper motivation. The paper argues that broader coverage may help downstream selection, verification, or self-training, but it is not clear that more modes help when the task only requires finding one correct answer and no verifier or downstream process benefits from alternative strategies. Since the reported pass@k is already near its ceiling in key experiments, the paper should better distinguish settings where mode coverage provides measurable utility from settings where it is simply an additional diversity objective.

  2. Equal weighting of observed modes may be counterproductive. The mode-wise term gives a rare observed mode the same total weight as a frequent mode. This is intuitive if the goal is broad coverage, but the paper could better explain why equal weighting is desirable, rather than, for example, applying a softer frequency correction or weighting modes by downstream value. Correctness alone does not indicate that all strategies are equally useful. A clearer discussion and ablations over alternative mode weights would help establish when this bias improves outcomes and when it may overemphasize noisy or low-value modes. The role of λ\lambda is helpful, but does not by itself answer why the target distribution should be uniform over modes.

  3. Several central concepts and theoretical claims need more intuition. The abstract uses CorrectModes@k before defining it; a brief definition would make the contribution easier to grasp. The introduction also invokes a “distributional objective” and a “local correctness optimum” without enough plain-language explanation of what readers should take away. Theorem 2 and Eq. 3 would also benefit from a compact worked example showing how the pooled and mode-wise terms combine for correct and incorrect rollouts. Finally, the contrast with UCPO would be clearer with a small example: several distinct responses can express the same strategy, so uniformity over responses need not mean uniformity over reasoning modes.

  4. The labeling pipeline and model generality need clearer treatment. The paper uses auxiliary models to construct mode labels: Qwen2.5-0.5B-Instruct for training strategy descriptions and Qwen2.5-1.5B-Instruct for evaluation, with Qwen3-generated responses used to construct the respective cluster banks. These are not correctness verifiers; correctness is checked against the answer key. Still, the multi-model labeling pipeline adds inference and implementation complexity, and the paper should make the roles, costs, and reproducibility implications especially clear. Separately, the main post-training experiments center on Qwen3-4B-Base, with a Qwen3-8B result in the appendix. The dataset coverage is reasonable, but evaluation on more base-model families and scales is needed to establish that the gains are not specific to Qwen3.

Questions for authors
  1. In what realistic settings does increased correct-mode coverage improve end-to-end task success beyond pass@k?

  2. What motivates equal total weight for each observed correct mode? How does MILO compare with softer frequency corrections or mode weights based on downstream utility, and how sensitive are results to noisy or unstable mode assignments?

  3. Could the paper add a simple worked example for Eq. 3 and for the distinction between UCPO's response-level uniformity and MILO's mode-level weighting? Could it also define CorrectModes@k in the abstract and explain the implications of Theorem 2 more directly?

  4. Could the authors evaluate additional base-model families and clarify the compute and data flow for the auxiliary mode-labeling models, separate from the ground-truth correctness verifier?

Reviewer_oSWXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Mode-wise Likelihood Optimization (MILO), a post-training method designed to improve the diversity of correct reasoning strategies while maintaining solution accuracy. MILO assigns equal aggregate weight to each observed correct mode and combines mode-wise likelihood optimization with a pooled correctness objective. The paper provides theoretical connections between the proposed update and expected mode coverage, and demonstrates improvements across several mathematical reasoning benchmarks.

Critical strengths
  • The paper evaluates several representative baselines and includes experiments across different datasets, model sizes, and sampling budgets. The additional evaluations using independent mode labelers and labeler-free diversity metrics further strengthen the empirical findings.
Critical weaknesses
  • The abstract is unnecessarily divided into multiple paragraphs, which is unconventional for an ICLR submission and affects the overall readability and presentation. More broadly, the manuscript in its current form falls short of the clarity and polish expected of a top conference paper and requires substantial revision.
  • Although MILO maintains high pass@64, it substantially reduces pass@1 compared with MaxRL (0.753 to 0.639 on GSM8K at λ = 0.1). Since pass@64 is already close to saturation for most baselines, the practical significance of preserving this metric is somewhat limited.
  • The proposed method relies on automatically generated strategy descriptions and clustering to identify reasoning modes, but lacks systematic human validation of these labels.
  • The verifier-blind-spot experiment artificially rejects each policy's most frequent correct modes, which naturally favors a more uniform mode distribution.
  • MILO uses additional strategy labels generated by an auxiliary language model, whereas the compared baselines do not have access to the same information.
  • The local correctness-preservation guarantee assumes a stationary correctness optimum and sufficiently small optimization steps, which may not hold during full post-training.
Questions for authors

Please see the weaknesses.

Reviewer_95XvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that the diversity of correct trajectory modes should be considered in reinforcement learning. The authors focus on the different ways of reaching the correct solution, which is relatively less explored compared with other directions to improve diversity. They propose a new objective that increases the weight of less sampled correct solution mode trajectory. Evaluated on GSM8K with Qwen3 model, the proposed method is able to cover significantly more correct ways of solving problems during sampling at the cost of non-ignorable pass@1 performance drop.

Critical strengths

While promoting the diversity of completions for LLM during RL is a well studied directions, the specific focus on improving coverage of sampled completion for the correct subset is less studied, which means the paper has potential to offer new perspectives into the field. The way the paper classifying mode is also reasonable on the setup of small model on a relatively easy deterministic math task which have a limited set of possible strategies to solve the question.

Critical weaknesses

The major weakness of the paper, as expected, is the performance. While the proposed improve sampling mode coverage, the pass@1 witnesses a meaningful drop as a cost. However, contemporary work like G²RPO: Geometric GRPO Escaping LLMs’ Reasoning Ruts to Break the Accuracy–Entropy Trade-off don't have need to have the sacrifices. As the method's target derived from trying to improve the metrics of coverage of mode, there is actually no surprise that it outperforms alternatives when comparing under the mode coverage metrics, making it appear somewhat circular. Another concern is the generalizability and scalability of the method. The paper defines mode of answer as different strategies to correctly solving the problem. While such definition is relatively easy to quantify on short math questions, the boundary of different strategies seems less clear when it comes to more competitive math, where each solution is usually hard to classify. This concern also affects adoption of the method to a wider RLVR setting like end-to-end code writing. Additionally, I fail to locate a paragraph where the author clarified the scope of the proposed method.

Questions for authors
  1. Do you consider the lower pass@1 accuracy a significant weakness of the paper? If the aim of the method is to obtain different solutions to the same task, wouldn't prompting the model to generate multiple solutions or sequentially prompting it to try different methods be an easier and more reliable way to achieve this? What is the expected use case for the proposed method if it comes at the cost of accuracy?
  2. How well do you think the method and the definition of a correct mode would generalize to other domains, such as coding? In long-running tasks, where each completion is usually significantly different from the others, how can we define different strategies for solving the problem in a meaningful way, and how would these definitions affect the quality of training?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.