acceptodds
Under review as a conference paper at ICLR 2027

MIPS: Mitigating Mode Collapse in Reinforcement Learning for Large Outcome Spaces

Abstract

Expected-reward maximization in reinforcement learning concentrates probability mass on a few outcomes, even in tasks that require many distinct, high-quality solutions, a failure known as mode collapse. Inverse Probability Scaling (IPS) counters this in group-based policy gradients such as GRPO by dividing each reward by the outcome's probability, estimated from its frequency within the sampled group. When the outcome space is large relative to the group, outcomes rarely repeat and these estimates become uninformative. Since an outcome can be constructed along many trajectories, its probability is a sum over all of them, and computing it directly is intractable. We propose Marginal IPS (MIPS), which performs marginal evaluation by combining the sampled trajectory with a learned model of the trajectories that lead to each outcome. On Hypergrids, where the exact sampling distribution is computable, MIPS stays near the reward-distribution target as the outcome space grows relative to the rollout group, while frequency-based IPS degrades. In molecular synthesis and phylogenetic inference, MIPS sustains diverse, high-quality sampling where GRPO and frequency-based IPS collapse, showing that MIPS serves as a reliable solution to mode collapse in RL.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, mode collapse, inverse probability scaling, reward-proportional sampling, diverse generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Mp8JOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MIPS (Marginal Inverse Probability Scaling), a method that replaces within-group outcome frequencies in Inverse Probability Scaling (IPS) with the ratio of a forward trajectory probability to a learned backward (trajectory-conditional) probability, and integrates these scaled rewards into Group Relative Policy Optimization (GRPO). This addresses a core limitation of frequency-based IPS in large outcome spaces by avoiding the need for repeated observations of identical outcomes within a group. The authors validate the method on controlled Hypergrid environments (measuring exact terminal probabilities against the target distribution), molecular generation (comparing discovered high-reward modes and scaffolds against RGFN), and phylogenetic tree inference (evaluating alignment and likelihood metrics against PhyloGFN and GRPO variants).

Critical strengths

Strong controlled validation and promising molecular generation gains: Evaluating exact terminal probabilities on Hypergrids is substantially more informative than relying on sample-diversity statistics alone; at the largest grid with group size 16, MIPS achieves a JSD of 0.0428 compared to 0.5437 for IPS-GRPO. Furthermore, the molecular generation experiments show substantial improvements over RGFN across reported seeds (70 vs. 18 high-reward modes and 149 vs. 25 high-reward scaffolds) while appropriately evaluating retained diversity on samples drawn from the final policy rather than cumulative training discoveries.

Clear motivation and practical problem formulation: The paper clearly distinguishes reward-proportional sampling from simple action-entropy maximization or expected-reward maximization, and proposes a clean mechanism to bypass the sample-collision bottleneck of frequency-based IPS in large outcome spaces.

Critical weaknesses

Unacknowledged theoretical connection to importance-sampling and forward-KL GFlowNet training, alongside incomplete ablations: For any normalized conditional model whose support covers the policy's supported trajectories, the expected scaled reward conditioned on an outcome is already an unbiased estimator of the exact IPS reward (R(o) / P_theta(o)), meaning accuracy of the conditional model reduces within-outcome variance rather than enabling unbiasedness. Moreover, in the idealized whole-trajectory score-function setting, the expected gradient is proportional to the negative gradient of the forward KL divergence from a backward-policy-defined target distribution to the forward policy, closely connecting MIPS to existing divergence-based GFlowNet and Entropic PPO formulations. Because the ablation sets the conditional factor to 1 (which fails to preserve a normalized distribution over multiple construction paths), the experiments do not isolate the benefit of learning the conditional model versus simply using a fixed, normalized backward policy, nor do they isolate GRPO's specific contribution against learned-backward GFlowNet or Entropic PPO baselines.

Flawed and internally inconsistent phylogenetic evaluation: First, evaluating reward-proportional sampling via Pearson correlation between log-probability and log-reward is insufficient because any power-scaled distribution R(o)^alpha yields a perfect correlation of 1.0 despite having arbitrarily different concentration; furthermore, evaluating MIPS terminal probabilities using the same learned conditional model used during training risks measuring internal ratio consistency rather than true sampling accuracy. Second, Table 3 ("MLL") and Appendix Table 10 ("Marginal log-lik." vs. "MLL") are internally inconsistent: Table 3 reports −4366.5 for PhyloGFN and −4372.9 for MIPS-GRPO (matching the "Marginal log-lik." column of Table 10), whereas the "MLL" column of Table 10 averages to −4672.4 for PhyloGFN and −4664.0 for MIPS-GRPO, which reverses the quality ordering between the two methods.

Questions for authors

Reconciling Phylogenetic Tables (Table 3 vs. Table 10): Can you define the exact distinction between "MLL" and "Marginal log-lik." in Table 10, explain why Table 3 reports the values from the "Marginal log-lik." column under the label "MLL", and clarify how the reversed ordering in the other column affects your conclusions regarding the quality-coverage trade-off?

Isolating Learned Backward Models and Algorithmic Contributions: How does MIPS perform on Hypergrids when using a fixed, uniform normalized backward policy or an oracle conditional model (rather than setting the conditional factor to an unnormalized 1)? Additionally, how does MIPS compare against a GFlowNet baseline equipped with a learned backward policy (or Entropic PPO) on Hypergrids and phylogenetic inference?

Phylogenetic Calibration and Independent Probability Validation: What are the log-probability calibration slopes and residuals (beyond Pearson correlation) on the phylogenetic task, and can you validate terminal probabilities on a tractable subset using exact summation or converged importance sampling rather than relying solely on the learned training conditional? Please also specify the exact positive reward definition (including temperature and priors) and clarify why Figure 5 states that absolute log-probabilities of the same discrete outcomes are "not comparable."

Implementation and Estimator Mechanics: How do you handle zero-variance groups and extreme log-weights during updates? Are off-policy exploration probabilities in phylogenetic inference included in the trajectory likelihoods? How are valid backward transitions enumerated and normalized for canonical molecules reached via different construction histories? Finally, does "trained until convergence" on Hypergrids simply mean the fixed 10,000 updates in Table 6, and how does the nonzero entropy bonus affect stationary distribution matching?

1 comment

When is the learned conditional model necessary?

In our earlier reply, we agreed that a uniform backward model suffices on the standard Hypergrid. We ran an experiment that pinpoints exactly when this holds and when it fails. It directly addresses the reviewer's hypothesis that the learned model "mainly reduces variance."

Why uniform cc works on the open grid and when it stops working. Let T(o)T(o) be the set of trajectories that can actually reach outcome oo. For any normalized cc, let Fc(o)=∑τ∈T(o)c(τ∣o)F_c(o) = \sum_{\tau \in T(o)} c(\tau \mid o) be the fraction of cc's mass on those feasible trajectories. Since Pθ(τ)=Pθ(o)Pθ(τ∣o)P_\theta(\tau) = P_\theta(o) P_\theta(\tau \mid o),

Eτ∼Pθ(⋅∣o)[R(o)c(τ∣o)Pθ(τ)]=R(o)Fc(o)Pθ(o)\mathbb{E}_{\tau \sim P_\theta(\cdot \mid o)}\left[\frac{R(o)c(\tau \mid o)}{P_\theta(\tau)}\right] = \frac{R(o)F_c(o)}{P_\theta(o)}

MIPS converges to Pθ∝R⋅FcP_\theta \propto R \cdot F_c. On the fully open Hypergrid, every reverse path is feasible, so Fc≡1F_c \equiv 1 for uniform cc, which explains why uniform and learned cc matched in Table R1. When some construction paths are infeasible, a feasibility-unaware uniform cc assigns mass to reverse paths that the policy can never generate. Then Fc(o)<1F_c(o) < 1 varies across outcomes, and the fixed point is biased by a factor that no amount of sampling can remove. Real-world tasks have this structure: not every route to a molecule is synthetically valid. A cc trained by maximum likelihood on on-policy trajectories is supported only on feasible paths, so Fc≡1F_c \equiv 1 again.

Experiment. We modified our 2D Hypergrid experiment by randomly designating 20% of non-origin cells as absorbing blocks. Stepping onto a block ends the episode, so the blocks interrupt paths and make some cells unreachable, effectively rendering some trajectories invalid. Everything else matches the paper (G=64G = 64, 10k updates). We compare the exact distributions against P⋆∝RP^{\star} \propto R over reachable outcomes and against R⋅FcR \cdot F_c, the biased fixed point predicted for uniform cc. We modified the environment so that the contrast between both method is clear and we can distinguish the effects more clearly.

HHReachableJSD to P⋆P^{\star} (unif / learned)TV to P⋆P^{\star} (unif / learned)JSD to R⋅FcR \cdot F_c (unif / learned)R2 modes hit (unif / learned)ESS (unif / learned)
327140.388 / 0.0060.74 / 0.050.0003 / 0.3761/3 / 3/30.99 / 0.90
642,2230.558 / 0.1220.91 / 0.310.0001 / 0.5261/3 / 3/30.99 / 0.85
12811,6960.586 / 0.3570.93 / 0.720.0003 / 0.4671/3 / 1/30.97 / 0.87
25645,1450.673 / 0.3860.99 / 0.750.0003 / 0.6380/2 / 1/20.98 / 0.70
512153,4300.683 / 0.5011.00 / 0.870.0001 / 0.6400/2 / 1/20.98 / 0.70

What this shows.

  1. Uniform cc converges to the predicted biased target, not to P⋆P^{\star}. Its JSD to R⋅FcR \cdot F_c is at most 3×10−43 \times 10^{-4} at every scale, while its TV distance to P⋆P^{\star} ranges from 0.74 to 1.00: nearly all of its mass is misplaced. It also neglects the highest-reward modes; at H=64H = 64, it assigns a probability mass of 0.009 to R2 cells, compared to a target of 0.21.
  2. Learned cc escapes the biased target at every scale. It reduces JSD to P⋆P^{\star} by a factor of 60 at H=32H = 32 and 4.6 at H=64H = 64, and recovers all reachable R2 modes in both settings.
  3. The failure of uniform cc is due to bias, not variance. Uniform cc has higher ESS (0.97 to 0.99 vs. 0.70 to 0.90) and lower within-outcome variance of log⁡w\log w (about 5×10−45 \times 10^{-4} vs. 0.02 to 0.10) than learned cc, yet converges to the wrong distribution. The learned model performs better despite noisier weights, so its role here is to correct the outcome marginal, not to reduce variance.

At H≥128H \geq 128, learned cc does not fully reach P⋆P^{\star}; this is a harder optimization problem, and we will report it as such. The qualitative separation holds throughout: uniform cc is provably locked to the wrong fixed point, and learned cc is not.

Thus, a normalized uniform cc suffices when all construction paths are feasible, and when feasibility is path-dependent, learning cc is what makes the estimator target P∝RP \propto R. We will add this analysis and experiment to the paper.

Reviewer_Wsg2OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper tackles mode collapse in GRPO style RL where the frequency based inverse probability scaling fails once outcomes rarely repeat in a group and authors proposes MIPS which estimates each outcome's probability from one trajectory using a learned backward model and plugs the scaled reward into GRPO. On Hypergrids MIPS stays close to the reward proportional target as the outcome space grows and on molecular synthesis and phylogenetics it covers far more high reward outcomes than GRPO and compares well with task specific GFlowNets.

Critical strengths
  1. The paper explains clearly why frequency based IPS breaks down when the group is small relative to the outcome space. The fix is to estimate the outcome probability from one sampled trajectory and a learned trajectory conditional model and the scaled reward goes into the standard GRPO policy update without changing it.
  2. The evaluation is thorough where on Hypergrids the paper computes the exact outcome distribution of each trained policy over 64 grid and group size settings and on molecules MIPS finds about four times as many high reward modes as RGFN without lowering their quality and on trees it matches PhyloGFN's coverage with the same number of training trajectories and better reward alignment.
Critical weaknesses
  1. The proposed method is very close to GFlowNets trained by policy gradients. The scaled reward is the trajectory balance ratio and the target PP proportional to RR is the GFlowNet target which the paper acknowledges and similar methods such as Entropic PPO which can learn a backward policy by likelihood also exist. The paper argues that MIPS differs because the learned trajectory model cc only estimates the current marginal inside GRPO. GFlowNets are compared on molecules and trees and there is no GFlowNet on Hypergrid, the one setting where the exact distribution can be checked. So it is hard for me to tell what MIPS adds beyond a policy gradient GFlowNet with a learned backward model.
  2. The ablation that removes the learned model sets c=1c = 1 which by my understanding is not MIPS without learning but a different target that favours outcomes reachable by many paths. A uniform backward model would be the better choice and if it works as well then the learned model mainly reduces variance rather than being important for estimating the marginal as the paper concludes.
Questions for authors
  1. Why is there no GFlowNet in Figure 1 given that it targets the same reward proportional distribution?
  2. Is the entropy regularized RL panel GRPO with the entropy coefficient of 0.1 from Table 6 and with which H and G?
  3. What are the wall clock and memory costs of training learned trajectory model cc compared with GRPO and RGFN?
  4. On molecules only G=100G = 100 is used. Does IPS-GRPO recover with a larger group as it does on small Hypergrids or does it stay collapsed as on trees with G=4096G = 4096?
Reviewer_YonnOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Marginal Inverse Probability Scaling (MIPS), a variant of GRPO that learns a stochastic policy that samples proportionally to a reward function defined over the outcome space, i.e., the same neural sampling problem that GFlowNet papers aim to solve. By observing the limitation of IPS, which is its frequency-based estimation of the marginal likelihood of an outcome, MIPS introduces an alternative way to estimate the marginal by learning a model. They combine MIPS with GRPO-style training and evaluate it on standard GFlowNets sampling benchmarks: Hypergrid, biological sequence design (sEH), and phylogenetic tree generation.

Critical strengths

None

Critical weaknesses

1. I think the proposed method is a combination of existing findings in the GFlowNet literature. If not, please clarify.

The trajectory-conditional model can be seen as a GFlowNet backward policy, and its maximum-likelihood objective (Eq. 2) is considered in [1], which is not cited. The scaled reward R(o)cϕ(τ∣o)/Pθ(τ)R(o)c_{\phi}(\tau \mid o) / P_{\theta}(\tau) is the exponentiated negative trajectory-balance residual, so, up to GRPO’s length normalisation and entropy terms, the policy update is the importance-weighted forward-KL gradient [2], which is in turn equivalent (up to scale) to the Linex(1) loss in [3] with a per-batch normaliser; both works are not cited. The current manuscript lacks clarification on the correspondence and what is new in this work. Last but not least, these works should also be considered as baselines in the experiments.

2. Lack of justification that the proposed algorithm achieves the goal of reward-proportional sampling.

First, for those who are not familiar with IPS could struggle to understand the method. I had to read IPS paper to fully understand what the proposed method achieves at the optimum. Step-by-step justification for why the proposed algorithm achieves the goal of reward-proportional sampling should be included in the main section. Regarding this, I think the GRPO-style update makes its expected gradient different from forward KL divergence minimisation, and it is worth investigating, e.g., the fixed point of the algorithm. I believe the entropy regulariser changes the fixed point and introduces bias (i.e., it doesn't sample from the outcome reward distribution). Other components like length normalisation or clipping don't seem to change the fixed point of the algorithm.

[1] Gritsaev, Timofei, et al. "Optimizing backward policies in GFlownets via trajectory likelihood maximization." ICLR 2025.
[2] da Silva, Tiago, et al. "On divergence measures for training GFlownets." NeurIPS 2024.
[3] Hu, Rui, et al. "Beyond squared error: Exploring loss design for enhanced training of generative flow networks." ICLR 2025.

Questions for authors

None

Comments on the paper

1 comment

Thank you for the response. Unfortunately, my assessment is unchanged.

The authors agreed on the equivalences to previous GFlowNet papers [1-3]. Regarding their new contributions, I don't understand why "Why IPS fails in GRPO" matters when the proposed algorithm in this paper essentially solves the same problem as those GFlowNet works. The second contribution ("A fix that keeps the GRPO update ...") can be misleading, because [2]'s forward-KL also doesn't require flow or log Z. As the authors agreed, MIPS-GRPO is essentially [1]’s learned backward policy + [2]’s forward-KL update + GRPO’s normalisations, clipping, and an entropy bonus (which adds bias).

W2 is only partially resolved. I appreciate the ablations in Tables R2 and R3 and the acknowledgement that the entropy bonus biases the sampler. However, the justification itself is still only promised (the authors said, "We will include the derivation in ...," and there is no manuscript update).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.