- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes MIPS (Marginal Inverse Probability Scaling), a method that replaces within-group outcome frequencies in Inverse Probability Scaling (IPS) with the ratio of a forward trajectory probability to a learned backward (trajectory-conditional) probability, and integrates these scaled rewards into Group Relative Policy Optimization (GRPO). This addresses a core limitation of frequency-based IPS in large outcome spaces by avoiding the need for repeated observations of identical outcomes within a group. The authors validate the method on controlled Hypergrid environments (measuring exact terminal probabilities against the target distribution), molecular generation (comparing discovered high-reward modes and scaffolds against RGFN), and phylogenetic tree inference (evaluating alignment and likelihood metrics against PhyloGFN and GRPO variants).
Critical strengths
Strong controlled validation and promising molecular generation gains: Evaluating exact terminal probabilities on Hypergrids is substantially more informative than relying on sample-diversity statistics alone; at the largest grid with group size 16, MIPS achieves a JSD of 0.0428 compared to 0.5437 for IPS-GRPO. Furthermore, the molecular generation experiments show substantial improvements over RGFN across reported seeds (70 vs. 18 high-reward modes and 149 vs. 25 high-reward scaffolds) while appropriately evaluating retained diversity on samples drawn from the final policy rather than cumulative training discoveries.
Clear motivation and practical problem formulation: The paper clearly distinguishes reward-proportional sampling from simple action-entropy maximization or expected-reward maximization, and proposes a clean mechanism to bypass the sample-collision bottleneck of frequency-based IPS in large outcome spaces.
Critical weaknesses
Unacknowledged theoretical connection to importance-sampling and forward-KL GFlowNet training, alongside incomplete ablations: For any normalized conditional model whose support covers the policy's supported trajectories, the expected scaled reward conditioned on an outcome is already an unbiased estimator of the exact IPS reward (R(o) / P_theta(o)), meaning accuracy of the conditional model reduces within-outcome variance rather than enabling unbiasedness. Moreover, in the idealized whole-trajectory score-function setting, the expected gradient is proportional to the negative gradient of the forward KL divergence from a backward-policy-defined target distribution to the forward policy, closely connecting MIPS to existing divergence-based GFlowNet and Entropic PPO formulations. Because the ablation sets the conditional factor to 1 (which fails to preserve a normalized distribution over multiple construction paths), the experiments do not isolate the benefit of learning the conditional model versus simply using a fixed, normalized backward policy, nor do they isolate GRPO's specific contribution against learned-backward GFlowNet or Entropic PPO baselines.
Flawed and internally inconsistent phylogenetic evaluation: First, evaluating reward-proportional sampling via Pearson correlation between log-probability and log-reward is insufficient because any power-scaled distribution R(o)^alpha yields a perfect correlation of 1.0 despite having arbitrarily different concentration; furthermore, evaluating MIPS terminal probabilities using the same learned conditional model used during training risks measuring internal ratio consistency rather than true sampling accuracy. Second, Table 3 ("MLL") and Appendix Table 10 ("Marginal log-lik." vs. "MLL") are internally inconsistent: Table 3 reports −4366.5 for PhyloGFN and −4372.9 for MIPS-GRPO (matching the "Marginal log-lik." column of Table 10), whereas the "MLL" column of Table 10 averages to −4672.4 for PhyloGFN and −4664.0 for MIPS-GRPO, which reverses the quality ordering between the two methods.
Questions for authors
Reconciling Phylogenetic Tables (Table 3 vs. Table 10): Can you define the exact distinction between "MLL" and "Marginal log-lik." in Table 10, explain why Table 3 reports the values from the "Marginal log-lik." column under the label "MLL", and clarify how the reversed ordering in the other column affects your conclusions regarding the quality-coverage trade-off?
Isolating Learned Backward Models and Algorithmic Contributions: How does MIPS perform on Hypergrids when using a fixed, uniform normalized backward policy or an oracle conditional model (rather than setting the conditional factor to an unnormalized 1)? Additionally, how does MIPS compare against a GFlowNet baseline equipped with a learned backward policy (or Entropic PPO) on Hypergrids and phylogenetic inference?
Phylogenetic Calibration and Independent Probability Validation: What are the log-probability calibration slopes and residuals (beyond Pearson correlation) on the phylogenetic task, and can you validate terminal probabilities on a tractable subset using exact summation or converged importance sampling rather than relying solely on the learned training conditional? Please also specify the exact positive reward definition (including temperature and priors) and clarify why Figure 5 states that absolute log-probabilities of the same discrete outcomes are "not comparable."
Implementation and Estimator Mechanics: How do you handle zero-variance groups and extreme log-weights during updates? Are off-policy exploration probabilities in phylogenetic inference included in the trajectory likelihoods? How are valid backward transitions enumerated and normalized for canonical molecules reached via different construction histories? Finally, does "trained until convergence" on Hypergrids simply mean the fixed 10,000 updates in Table 6, and how does the nonzero entropy bonus affect stationary distribution matching?
When is the learned conditional model necessary?
In our earlier reply, we agreed that a uniform backward model suffices on the standard Hypergrid. We ran an experiment that pinpoints exactly when this holds and when it fails. It directly addresses the reviewer's hypothesis that the learned model "mainly reduces variance."
Why uniform c works on the open grid and when it stops working. Let T(o) be the set of trajectories that can actually reach outcome o. For any normalized c, let Fc(o)=∑τ∈T(o)c(τ∣o) be the fraction of c's mass on those feasible trajectories. Since Pθ(τ)=Pθ(o)Pθ(τ∣o),
Eτ∼Pθ(⋅∣o)[Pθ(τ)R(o)c(τ∣o)]=Pθ(o)R(o)Fc(o)
MIPS converges to Pθ∝R⋅Fc. On the fully open Hypergrid, every reverse path is feasible, so Fc≡1 for uniform c, which explains why uniform and learned c matched in Table R1. When some construction paths are infeasible, a feasibility-unaware uniform c assigns mass to reverse paths that the policy can never generate. Then Fc(o)<1 varies across outcomes, and the fixed point is biased by a factor that no amount of sampling can remove. Real-world tasks have this structure: not every route to a molecule is synthetically valid. A c trained by maximum likelihood on on-policy trajectories is supported only on feasible paths, so Fc≡1 again.
Experiment. We modified our 2D Hypergrid experiment by randomly designating 20% of non-origin cells as absorbing blocks. Stepping onto a block ends the episode, so the blocks interrupt paths and make some cells unreachable, effectively rendering some trajectories invalid. Everything else matches the paper (G=64, 10k updates). We compare the exact distributions against P⋆∝R over reachable outcomes and against R⋅Fc, the biased fixed point predicted for uniform c. We modified the environment so that the contrast between both method is clear and we can distinguish the effects more clearly.
What this shows.
At H≥128, learned c does not fully reach P⋆; this is a harder optimization problem, and we will report it as such. The qualitative separation holds throughout: uniform c is provably locked to the wrong fixed point, and learned c is not.
Thus, a normalized uniform c suffices when all construction paths are feasible, and when feasibility is path-dependent, learning c is what makes the estimator target P∝R. We will add this analysis and experiment to the paper.