MILO: Mode-wise Likelihood Optimization for Post-Training Diversity

People often ask language models for multiple candidate solutions, but sampling more responses does not necessarily produce more ways of solving a problem: many samples may repeat the same underlying strategy. We introduce Mode-wise Likelihood Optimization (MILO), a post-training method for increasing the diversity of correct solution strategies while preserving accuracy. MILO groups verified-correct responses that use the same strategy into correct modes, gives each mode its own log-likelihood term, and combines the resulting mode-wise objective with pooled correctness using a weight $\lambda$. Its mode-wise update gives each observed mode equal total weight, rather than favoring modes simply because they appear more often. We show that this update directly targets correct-mode coverage. For finite rollout groups, its expected mode-wise update is the gradient of a $1/k$-weighted sum of expected CorrectModes@$k$, where CorrectModes@$k$ counts the distinct correct modes among $k$ samples. At a local correctness optimum, a small MILO step with $\lambda >0$ improves this coverage objective to first order when its gradient is nonzero, while correctness changes only at second order. On Qwen3-4B-Base GSM8K, $\lambda =0.01$ improves both single-sample accuracy and CorrectModes@$64$, raising coverage from $8.4$ to $14.6$. Increasing $\lambda$ to $0.1$ raises coverage to $22.6$ while keeping single-sample accuracy comparable to the base model and pass@$64$ nearly unchanged. The gains extend across datasets, sampling budgets, and independent mode labelers, and are also reflected by labeler-free diversity metrics. A controlled verifier-blind-spot experiment further illustrates how broader correct-mode coverage can help candidate pools retain usable correct answers when a verifier fails on common strategies.