acceptodds
Under review as a conference paper at ICLR 2027

DFPO:Enhancing Robustness and Generalization for Robust RL through Distributional Value Flow Modeling in LLM Post-Training

Abstract

Training reinforcement learning (RL) systems in real-world environments remains challenging due to noisy supervision and poor out-of-domain (OOD) generalization, especially in LLM post-training. Recent distributional RL methods improve robustness by modeling values with multiple quantile points, but they still learn each quantile independently as a scalar. This results in rough-grained value representations that lack fine-grained conditioning on state information, struggling under complex and OOD conditions. We propose DFPO (Distributional Value Flow Policy Optimization), a robust distributional RL framework that models values as continuous flows across time steps. By learning a value flow field instead of isolated quantile predictions, DFPO captures richer state information for more accurate advantage estimation. To stabilize training under noisy feedback, DFPO further integrates conditional risk control and consistency constraints along value flow trajectories. Experiments on dialogue, math reasoning, and scientific tasks show that DFPO outperforms PPO, FlowRL, and other robust baselines under noisy supervision, achieving improved training stability and generalization.

Primary area:
reinforcement learning
Keywords:
Value distributions, Conditional risk control, Consistency constraints, Robustness and generalization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Ln9SOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes DFPO, a flow-based distributional critic for LLM post-training under noisy rewards. It combines distributional GAE with sensitivity weighting, anchor regularization, geometric consistency, and tail constraints, reporting improved training stability and out-of-domain performance.

Critical strengths

The paper addresses a relevant problem through critic design rather than only modifying policy updates. The reported late-stage stability on dialogue tasks is encouraging.

Critical weaknesses

1. The risk-control interpretation is not sufficiently justified. The actor averages the distributional advantages, which algebraically reduces to standard GAE based on the critic’s mean. Meanwhile, Eq. (4) matches predicted and target tail means rather than imposing a policy-level risk constraint. Distributional training may improve mean estimation through regularization, but this does not by itself establish risk-averse behavior or suppression of catastrophic outcomes.

2. The geometric consistency argument contains a logical gap. Eq. (3) enforces equal terminal projections at tt and 1−t1-t, but this symmetry does not imply that the projection is constant over time, as assumed in Proposition 3. This weakens the stated justification for straight trajectories and accurate single-step inference. Moreover, the sampling-step ablation measures downstream accuracy, not whether single-step sampling preserves the modeled distribution.

3. The experiments do not fully isolate the contribution of flow modeling. The Robust Bellman baseline uses minimum-based scalarization, whereas DFPO uses a mean, introducing a confound beyond critic parameterization. The improvement over DVPO is also relatively small on the reported reasoning comparison (40.25% versus 39.63%). Multiple seeds for DFPO alone do not establish the statistical reliability of this difference. Matched critic comparisons and multi-seed results for the strongest baselines are needed.

Questions for authors
  1. Is the learning target the true return distribution or a regularized surrogate for improving mean-advantage estimation? How should its bias be interpreted?
  2. Can the authors revise Proposition 3 and compare single-step versus multi-step sampling using a frozen critic, reporting distributional and tail-estimation errors?
  3. Do the gains persist against a quantile critic with matched scalarization, regularization, and training budget, and against DVPO across multiple training seeds?
  4. Appendix B.1 analyzes a geometric mixture of return distributions, whereas the algorithm uses rank-wise sums of quantile TD errors, It seems that there is a theory-practice gap.
6 comments
AuthorsOpenReview ↗

De-biased Surrogate Target, Prop. 3 Refinement & Multi-Seed Controls

Thank you for your rigorous critique, which refined our theoretical guarantees and experimental controls. We address all concerns below.


1. Surrogate Target, Bias & Supervision Risk Filtering (W1, Q1)

  • Critic-level filtering vs. actor-level pessimism: As Distributional GAE is linear over quantiles with A^t=Mean(AD(st))\hat{A}_t = Mean(A_D(s_t)), policy updates operate on the critic mean Vˉ(st)\bar{V}(s_t). Under heavy LLM reward noise, late-stage formatting collapse (Table 1) stems from extrinsic supervision risk, false-negative spikes corrupting critic gradients, not intrinsic environment stochasticity. Actor-level pessimism (CVaR PG, Robust Bellman min-scalarization) conflates label noise with true risk, causing underestimation and collapse (36.23% Science). DFPO learns a risk-filtered surrogate distribution to provide a de-biased, noise-resilient mean advantage anchor Vˉ(st)\bar{V}(s_t).
  • De-biasing via non-local flow coupling: The empirical target ZtgtZ^{tgt} is skewed by false-negative spikes; unbiased fitting (MSE, symmetric flow matching) drags Vˉ(st)\bar{V}(s_t) downward. DFPO introduces intentional asymmetric structural bias to de-bias Vˉ(st)\bar{V}(s_t) toward clean underlying returns: (1) LRiskL_{Risk} (Eq. 4) anchors first-order tail locations; LShapeL_{Shape} (Eq. 5) acts as a distributional noise filter: left-tail concavity (∇2≤0\nabla^2 \le 0) contracts spurious negative spikes, right-tail convexity (∇2≥0\nabla^2 \ge 0) preserves sparse high-reward reasoning chains. (2) Our spectral-normalized velocity field vθ(zτ,τ,hs)v_\theta(z_\tau, \tau, h_s) is globally Lipschitz-continuous with shared θ\theta; suppressing lower-tail explosion (LShapeL_{Shape}) and target variance (LBCFML_{BCFM}) prevents corrupted tail gradients from distorting the global field. This shifts Vˉ(st)=(1/K)∑kz^1(k)\bar{V}(s_t) = (1/K)\sum_{k} \hat{z}_1^{(k)} away from negative outliers, cutting CVaR0.10CVaR_{0.10} error by 19.7% vs. Value Flows and eliminating destructive advantage spikes that trigger policy collapse.

2. Proposition 3 & Frozen-Critic Fidelity (W2, Q2)

  • Even-odd synergy (Prop. 3): In Prop. 3 (App. B.3), global time-invariance of g(t)=x^1(zt,t)g(t) = \hat{x}_1(z_t, t) holds under all-pairs sampling (t1,t2∼U[0,1]t_1, t_2 \sim U[0,1]). Conjugate sampling (t′=1−tt'=1-t, Eq. 3) and linear CFM (LUDCFML_{UDCFM}, Eq. 1) form an even-odd decomposition around t=0.5t=0.5: LconsL_{cons} eliminates odd-order anti-symmetric fluctuations (godd(t)=0g_{odd}(t)=0); LUDCFML_{UDCFM} (regressing vθv_\theta to constant velocity x1−x0x_1-x_0) suppresses even-order curvature. Prop. 3 has been updated to formalize this.
  • Frozen-critic step fidelity: Evaluating our frozen DFPO critic across N∈{1,5,10,20}N \in \{1,5,10,20\} Euler steps on 3,200 MC reference samples:
Steps NNW1↓W_1\downarrowKS↓KS\downarrowCVaR0.05↓CVaR_{0.05}\downarrowCVaR0.10↓CVaR_{0.10}\downarrow
1 (Ours)0.67340.46211.59401.3926
50.67340.46221.59431.3928
100.67340.46191.59511.3935
200.67230.46211.60821.4060

Across N=1N=1–2020, W1W_1 varies by <0.0011<0.0011 and KSKS by <0.0003<0.0003, confirming 1-step inference faithfully preserves global and tail distributions.

3. Matched Critics & Multi-Seed Reliability (W3, Q3)

  • Matched mean scalarization: Comparing a Matched Quantile (mean) critic and flow critics (Value Flows, PCBF) with identical regularizers and Science training budget:
Critic (Scalarization)SimpleQAGPQAHLEMATH500AIME24MinervaAMC23AVG
Quantile (min)3.144.173.3887.4045.0027.2183.3336.23
Quantile (mean)3.053.993.8088.2045.0027.5780.0035.94
PCBF (mean)2.844.353.8090.0050.0030.5180.0037.36
Value Flows (mean)2.594.713.4890.2055.0029.0485.8338.69
DFPO (mean)3.175.434.0391.8056.6732.3588.3340.25

Removing scalarization confounds, DFPO outperforms Matched Quantile (mean) by +4.31% (40.25% vs. 35.94%).

  • Multi-seed & noise study: On OOD Math, DVPO (seeds 42/123/2026) achieves 65.98±0.45%65.98\pm0.45\%; DFPO (4 seeds, Table 13) reaches 66.90±0.47%66.90\pm0.47\% (p<0.05p<0.05), with 7-task AVG 39.96±0.25%39.96\pm0.25\% (all seeds exceed DVPO’s 39.63%, Table 14). On Dialogue (1000 steps, Table 15), DFPO leads by +12.34% (87.31% vs. 74.97%). Under 0%→40% label noise, OOD accuracy drops only 1.45% vs. 2.09% for PPO.

4. Theory-Practice Alignment in App. B.1 (Q4)

Eq. (11) mixes measures by averaging CDFs FF, while Sec. 3.3 averages sorted quantiles F−1F^{-1} (comonotonic addition). Since W1(μ,ν)=∫01∣Fμ−1(τ)−Fν−1(τ)∣dτW_1(\mu,\nu) = \int_0^1 |F_\mu^{-1}(\tau) - F_\nu^{-1}(\tau)|d\tau in 1D, rank-wise quantile GAE is an exact W1W_1 contraction with identical modulus Γ=γ(1−λ)1−λγ<1\Gamma = \frac{\gamma(1-\lambda)}{1-\lambda\gamma} < 1 via the L1L_1 triangle inequality. This proof has been added to Sec. 3.3 and App. B.1.


Thank you again for feedback strengthening the paper. Please let us know if you have further questions, and we will respond promptly. If our replies are satisfactory, we kindly ask you to consider updating your score and confidence accordingly.

Thanks for the response and additional experiments. The matched critic comparisons and extra seeds strengthen the empirical results.

My main concerns are not fully resolved. Reshaping noisy targets does not necessarily remove bias. How does the method distinguish incorrect negative feedback from genuinely poor outcomes? For Proposition 3, fitting constant velocities along interpolation paths does not by itself establish straight learned ODE trajectories. The contraction result also needs to clarify what fixed point the implemented surrogate converges to. The frozen critic experiment is helpful, but similar errors against a common reference do not establish that different solvers produce similar distributions. A direct comparison between one step and a sufficiently accurate numerical solve would be more helpful.

Overall, I appreciate the efforts, but still have some concerns about the theoretical justification and denoising explanation.

AuthorsOpenReview ↗

Response part(1/4)

Thank you for your thoughtful follow-up and for acknowledging that our matched critic comparisons and multi-seed experiments strengthen the empirical results. Your four remaining questions touch upon the exact mathematical and numerical foundations of DFPO. Below, we provide rigorous theoretical characterizations and direct frozen-critic empirical verifications to fully resolve each point.


1. How DFPO Distinguishes Incorrect Negative Feedback (False Negatives) from Genuinely Poor Outcomes (True Negatives)

A natural concern is whether left-tail concavity (LShapeL_{Shape}) blindly pushes up all negative values, thereby failing to penalize genuinely poor outcomes. DFPO separates extrinsic false-negative supervision noise from genuinely poor outcomes both mathematically and empirically:

  • Mathematical Mechanism (Translation Invariance of ∇2\nabla^2 vs. Localized Tail Spikes): For a token state hsh_s, the critic models the conditional return distribution Z(s)Z(s) across continuations:
    1. Genuinely poor outcomes (Global Location Shift): When a reasoning prefix ss contains a genuine logical error, continuations consistently fail, shifting the entire target quantile set z(1)tgt,…,z(K)tgtz^{tgt}_{(1)}, \dots, z^{tgt}_{(K)} downward by a negative offset −C-C. Crucially, our curvature operator ∇2z^(k)=z^(k+2)−2z^(k+1)+z^(k)\nabla^2 \hat{z}_{(k)} = \hat{z}_{(k+2)} - 2\hat{z}_{(k+1)} + \hat{z}_{(k)} is strictly translation-invariant: ∇2(z^(k)−C)≡∇2z^(k)\nabla^2(\hat{z}_{(k)} - C) \equiv \nabla^2 \hat{z}_{(k)}. Thus, LShapeL_{Shape} imposes zero penalty on a global downward shift. Guided by first-order flow matching (LUDCFML_{UDCFM}) and tail-mean anchoring (LRiskL_{Risk}), the critic faithfully assigns a low expected state value Vˉ(s)\bar{V}(s) to genuinely poor states.
    2. Incorrect negative feedback (Localized High-Curvature Discontinuity): When a valid reasoning state receives a sporadic false-negative reward due to brittle exact-match rules or noisy RMs, it conflicts with the shared semantic representation hsh_s learned across valid states and manifests as an isolated lower-tail spike (∇2z^(k)≫0\nabla^2 \hat{z}_{(k)} \gg 0). Left-tail concavity (∇2z^(k)≤0\nabla^2 \hat{z}_{(k)} \le 0) specifically contracts this localized second-order jump while LBCFML_{BCFM} anchors the conditional mean, preventing false-negative spikes from dragging Vˉ(s)\bar{V}(s) down.
  • Direct Empirical Verification on True Negatives vs. False Negatives: To test this directly on our frozen critics, we categorized the validation rollouts with completed boxed answers (N=787N=787 out of 3200 rollouts) by comparing the noisy single-trace training label (label ∈{0,1}\in \{0,1\}) against the true correctness label: (A) True Negatives (label=0 and answer is genuinely incorrect, n=148n=148), (B) False Negatives (label=0 due to formatting/rule mismatch, but answer is genuinely correct, n=68n=68), and (C) True Positives (label=1 and answer is genuinely correct, n=569n=569). Note that all reported numbers below represent the critic's predicted state value Vˉ(s)\bar{V}(s) (the mean of K=50K=50 quantile particles at the terminal response token) under paired random seeds:
Frozen CriticA: True Neg Value Vˉ(s)\bar{V}(s)B: False Neg Value Vˉ(s)\bar{V}(s)C: True Pos Value Vˉ(s)\bar{V}(s)Denoising Gap: B$-$ADiscrimination Gap: C-\A
Value Flows0.07840.11330.0794+0.0349+0.0349+0.0011+0.0011
PCBF0.32480.38350.3443+0.0587+0.0587+0.0195+0.0195
DFPO (Ours)0.36450.39100.4000+0.0265+0.0265+0.0355+0.0355

This evaluation establishes three key findings:

  1. Effective Recovery from False-Negative Supervision: On False Negatives (B), where the model produced a genuinely correct answer but received a corrupted training reward of 00, DFPO predicts a substantially higher expected value (Vˉ(B)=0.3910\bar{V}(B) = 0.3910, close to True Positives (C) at 0.40000.4000) than on True Negatives (A) (0.36450.3645), confirming that DFPO does not treat mislabeled correct reasoning as genuinely poor outcomes.
  2. Preserving Ground-Truth Value Ordering (A<B<CA \lt B \lt C): DFPO is the only method that maintains the proper monotonic value ordering Vˉ(True Negative)<Vˉ(False Negative)<Vˉ(True Positive)\bar{V}(\text{True Negative}) \lt \bar{V}(\text{False Negative}) \lt \bar{V}(\text{True Positive}). Both Value Flows and PCBF suffer from value inversion (Vˉ(B)>Vˉ(C)\bar{V}(B) > \bar{V}(C)).
  3. Clear True Outcome Discrimination (C−AC - A): Under noisy supervision, Value Flows loses the ability to distinguish genuinely correct outcomes from genuinely poor ones (Vˉ(C)=0.0794\bar{V}(C) = 0.0794 vs. Vˉ(A)=0.0784\bar{V}(A) = 0.0784, gap +0.0011+0.0011), and PCBF exhibits a much weaker separation (+0.0195+0.0195) distorted by false-negative over-estimation. DFPO achieves the largest and cleanest value separation between True Positives and True Negatives (+0.0355+0.0355), confirming that our regularizers filter out supervision noise while preserving genuine credit assignment.
Reviewer_hLAHOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The authors propose DFPO, an algorithm using a flow-based distributional critic to improve LLM post-training under noisy reward. The critic learns token-level return distributions with additional losses to stabilize the flow and control side. The experiments cover dialogue, math, and scientific QA, with both in-domain and cross-domain tests. The results show better overall performance and less degradation during extended dialogue training.

Critical strengths
  1. The paper addresses an important practical problem: noisy rewards can destabilize LLM post-training and damage existing capabilities. Therefore improving stability in this setting would make RL training more reliable.

  2. The reported results are promising across dialogue, math, and scientific QA and the paper is clearly written and east to follow.

Critical weaknesses
  1. My main concern is that the methodological contribution appears incremental. The authors present distributional flow-based critic learning as a key contribution, but flow-based critics [1,2] and sensitivity-based weighting already exist. Other components, including the flow-matching losses from Value flows [1](e.g., BCFMs, DCFMs), also closely relate to existing methods. The paper omits PCBF [2], which addresses source-boundary consistency in Bellman-coupled flow critics and could therefore provide a strong baseline when adapted to the same LLM training setup. The authors should clarify what is new beyond adapting and combining these ideas for LLM post-training.

  2. The experiments do not sufficiently isolate the benefits of the proposed design. They lack (i) comparisons with existing flow critics [1,2] adapted to the same LLM training setup and (ii) direct evaluations of value or return-distribution accuracy, such as Wasserstein distance to reference return distributions. Experiments on OGBench [3] could provide additional evidence, but are not essential given the paper’s focus; comparisons with existing flow critics in the LLM setting are more important. Without these checks, the claim that DFPO provides more accurate advantage estimates remains insufficiently. I would consider raising my score if the rebuttal provides convincing evidence addressing these two points.

References

[1] Dong et al. “Value Flows.” ICLR, 2026.

[2] Xu et al. “Path-Coupled Bellman Flows for Distributional Reinforcement Learning.” ICML, 2026.

[3] Park, Seohong, et al. "Ogbench: Benchmarking offline goal-conditioned rl." International Conference on Learning Representations. Vol. 2025. 2025.

Questions for authors

Please address above concerns regarding methodological novelty, baseline comparisons, and critic accuracy detailed in the Critical Weaknesses section.

Besides, I have a following question:

  1. The method introduces several loss weights in Eq. (6). Appendix C.1 lists their values, but it is unclear how they were selected and how much tuning was needed. Please clarify the selection process and confirm whether the same values were used across all tasks.
6 comments
AuthorsOpenReview ↗

Methodological Novelty, Flow-Critic Benchmarks & Distribution Fidelity

Thank you for your constructive feedback and guidance on evaluating DFPO. We have completed all requested flow-critic adaptations (Value Flows, PCBF) and distributional accuracy evaluations, detailed below.


1. Methodological Novelty Beyond Value Flows & PCBF (W1) While DFPO builds on flow matching foundations (DCFM, BCFM, sensitivity weighting), direct application of Value Flows and PCBF fails in LLM post-training. Both frame critic learning as exact density matching, assuming clean Bellman targets and minimizing transport error over the full support. Under severe reward corruption, DFPO shifts from passive density matching to active geometric signal filtering via three interlocked, mutually enabling components:

  • Asymmetric geometric filtering (LRisk,LShapeL_{Risk}, L_{Shape}): Value Flows’ sensitivity weighting rescales loss but leaves transport geometry unconstrained, fitting corrupted outliers and causing lower-tail explosion (Fig. 3). PCBF enforces source-boundary consistency across 1-step Bellman backups, locking in noisy pseudo-labels and propagating boundary errors across token horizons. DFPO instead imposes curvature priors on the velocity field: left-tail concavity (∇2≤0\nabla^2 \le 0) suppresses negative noise spikes; right-tail convexity (∇2≥0\nabla^2 \ge 0) preserves sparse high-reward reasoning chains critical for OOD generalization.
  • Structural interlock of rectification (LconsL_{cons}) and tail shaping: This is not a loose combination of losses—each enables the other. Curvature constraints on standard multi-step ODE critics require backpropagation through an unrolled solver, incurring prohibitive memory cost and gradient instability. LconsL_{cons} regularizes trajectories to constant-velocity paths, reducing ODE integration to an exact differentiable 1-step projection (NFE=1, z^1=z0+vθ(z0,0,hs)\hat z_1 = z_0 + v_\theta(z_0,0,h_s)), allowing direct tail-gradient regularization of vθv_\theta without ODE unrolling.
  • Comonotonic multi-step distributional GAE: Unlike prior 1-step Bellman designs, DFPO leverages strict monotonicity of 1D Lipschitz flows (∂z1/∂z0>0\partial z_1 / \partial z_0 > 0) to preserve quantile rank identity across token transitions, yielding a multi-step GAE operator with exact 1D comonotonic optimal transport.

2. Head-to-Head Comparison with Flow Critics (W2.i) We adapted Value Flows, PCBF, and a Matched Quantile (mean) critic to our full LLM pipeline (Qwen3-8B, SuperGPQA; identical actor/optimizer/budget); PCBF will be cited in the revision:

Critic MethodSimpleQAGPQAHLEMATH500AIME24MinervaAMC23ALL AVG
Matched Quantile (mean)3.053.993.8088.2045.0027.5780.0035.94
PCBF2.844.353.8090.0050.0030.5180.0037.36
Value Flows2.594.713.4890.2055.0029.0485.8338.69
DFPO (Ours)3.175.434.0391.8056.6732.3588.3340.25

DFPO outperforms Value Flows (+1.56%) and PCBF (+2.89%), isolating the empirical gains of co-designed tail filtering and trajectory rectification.

3. Distribution Accuracy & OGBench Scope (W2.ii)

  • Distribution fidelity: 3,200 Monte Carlo samples (200 validation prompts × 16 rollouts) evaluate frozen critics on W1W_1, KSKS, and lower-tail CVaR error:
MethodW1↓W_1 \downarrowKS↓KS \downarrowCVaR0.05CVaR_{0.05} Err ↓\downarrowCVaR0.10CVaR_{0.10} Err ↓\downarrow
Value Flows0.68600.48942.03771.7333
PCBF0.69180.45421.95421.6461
DFPO (Ours)0.67340.46211.59401.3926

DFPO achieves the lowest W1W_1 and reduces CVaR0.10CVaR_{0.10} error by 19.7% vs. Value Flows and 15.4% vs. PCBF, confirming superior return and tail-risk estimation.

  • OGBench: It targets low-dimensional offline RL (MuJoCo), mismatched with LLM reasoning. Per your advice, we prioritized LLM flow-critic comparisons and MC distribution evaluations.

4. Hyperparameter Selection & Universal Sharing (Q1) Minimal tuning was required: weights (λreg=0.1,λcons=0.01,λrisk=0.5,λshape=0.5\lambda_{reg}=0.1, \lambda_{cons}=0.01, \lambda_{risk}=0.5, \lambda_{shape}=0.5) were selected from 4 coarse magnitude candidates on 500 validation samples to balance initial gradient norms. Ablations (Tables 9, 11) confirm broad stability, and weights remain strictly identical across Dialogue/Math/Science with no task-specific tuning.


DFPO advances robust LLM post-training via: (1) active asymmetric geometric flow-critic filtering; (2) 1-step trajectory rectification (NFE=1NFE=1) for scalable token-level inference and tail-gradient backpropagation; (3) verified superiority over Value Flows and PCBF in distribution fidelity and downstream accuracy.

Thank you again for your valuable guidance. If the newly added flow-critic comparisons and distributional fidelity evaluations address your concerns, we kindly ask you to consider raising your score to a positive rating (e.g., 3 / Weak Accept). Your recognition would be greatly appreciated.

I thank the authors for providing the direct comparison with flow critics and the evaluation of distributional accuracy. However, after carefully reviewing the revision, my core concern remains: DFPO still appears to be an existing flow-critic framework (A) augmented with a few additional regularizers (B). Furthermore, the newly provided comparison results demonstrate that B yields only a marginal performance benefit. I detail my reasoning below:

  1. Head-to-Head Comparison with Flow Critics (W2.i). By looking at Table 4, I think the most of the improvement comes from prior work. According to the paper: the row “Above+DCFM Loss” corresponds to standard distributional value flow modeling PPO, which is an existing approach. This row already improves the base model by +3.68 of the total +5.61 gain. Consequently, the components presented as your novel contributions (the risk, shape, consistency, and Lipschitz terms) contribute only about +1.9 in total.

  2. Distribution Accuracy & OGBench Scope (W2.ii). I don't think the comparison in Table 17 is fair. All critics are evaluated with a single integration step (N = 1), yet DFPO is designed for one-step inference through its consistency regularization, whereas the flow-critic baselines are multi-step critics. Moreover, the remark at the end of Appendix B.4 states that, for an unregularized CFM critic, a one-step Euler projection collapses to the conditional mean. Unless the baselines were trained with a comparable straightening regularizer, they are therefore evaluated in a regime where they are expected to degenerate. Even so, DFPO reduces W1 by only about 2–3% (0.6734 vs. 0.6860 and 0.6918). These results suggest that DFPO’s learned distribution is possibly not substantially more accurate than a nearly degenerate one.

If DFPO’s advantage holds only at N = 1, it is mainly attributable to the straightening regularizer, which is an established technique applied here to an existing flow critic. In my view, such a gain reflects improved inference efficiency rather than more accurate distributional estimation, and I do not consider this contribution sufficiently novel.

Although I appreciate the additional experiments, they do not convincingly demonstrate the benefit of the proposed components over existing flow critics. Therefore, my main concern remains unresolved, and I am sorry that I will maintain my current evaluation.

AuthorsOpenReview ↗

Response part(1/4)

Thank you for carefully reviewing our revision and articulating your perspective so clearly. Reading your critique helped us understand why our contribution appeared to be "an existing flow critic (A) + minor regularizers (B)". We did not clearly lay out the problem-driven reasoning behind DFPO, including what new optimization problem arises in LLM post-training, why we introduce continuous flows to address it, what inherent dynamical challenge emerges with flows, and how our geometric regularizers resolve it as a single closed-loop system. This gap is compounded by two presentation ambiguities in Table 4's caption and Appendix B.4's Remark.


1. Problem-Driven Philosophy: From Problem Definition to Closed-Loop Flow Control

Step 1: The New Problem in Long-Sequence LLM Post-Training

In embodied and continuous-control RL, return uncertainty reflects genuine physical stochasticity (e.g., actuator noise). Prior flow critics (Value Flows, Dong et al., 2025; PCBF, Xu et al., 2026) are therefore designed to solve a passive density-matching problem. Assuming 1-step Bellman targets ZtgtZ^{tgt} are uncorrupted ground truth, the goal is for a critic to replicate the empirical distribution as faithfully as possible.

In auto-regressive LLM post-training over long token sequences, solving that old problem is actively harmful. Sequence-level rewards rely on brittle rule verifiers or imperfect reward models that frequently assign corrupted 0 labels (false negatives) to valid reasoning chains due to minor formatting mismatches (Fig. 6), while cross-domain (OOD) generalization relies on rare, high-reward exploratory trajectories. Under extrinsic label corruption, faithfully matching the empirical distribution simply memorizes false-negative spikes (Fig. 3 Left), depresses value estimates, and injects chaotic gradient variance into policy updates (Fig. 4 Left).

Thus, our work is driven by a fundamentally different problem: how can a critic dynamically filter out extrinsic false-negative label noise without weakening penalties on genuinely poor actions, preserve rare OOD exploratory signals, and propagate this denoised credit to key reasoning tokens across long sequences?

Reviewer_REgROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DFPO, a distributional value flow policy optimization framework for robust LLM post-training under noisy supervision. Instead of using scalar value prediction or discrete quantile learning, DFPO models the return distribution with a continuous flow-based value head, and further introduces uncertainty-weighted flow matching, consistency regularization, and risk-sensitive distributional constraints. Experiments are conducted on multi-turn dialogue, mathematical reasoning, and scientific QA tasks. The results show that DFPO improves training stability and cross-domain generalization compared with PPO, GRPO, FlowRL, and other baselines.

Critical strengths
  1. This paper targets an important issue in LLM post-training: noisy reward supervision can make value estimation and advantage estimation unstable. Improving critic robustness is a worthwhile direction. 
  2. Combining distributional value modeling with flow matching is an interesting attempt. The consistency constraint and risk/shape constraints are also designed with a clear motivation, namely to stabilize the learned value distribution while keeping useful high-return information.
Critical weaknesses
  1. DFPO models the full return distribution, but if the final policy update still aggregates distributional advantages into a scalar and feeds it into the standard PPO objective, then under the stated linear, quantile-wise Distributional GAE recursion, this is essentially equivalent to applying scalar GAE to the mean of the predicted value distribution. In this case, the full distribution, risk constraint, and tail-shape constraint mainly act on the critic learning objective, rather than directly entering the actor objective. The method then looks more like “PPO with a distributionally regularized critic” than a true distributional policy optimization method. 
  2. The paper models the value distribution as a continuous-time ODE flow and shapes it with consistency, risk, and shape constraints. However, the actual training relies on sorted quantile particles, rank-wise Distributional GAE recursion, and a finite-step Euler solver. Two structural issues arise. Flow sampling produces continuous distribution particles, while GAE couples quantiles by rank across time steps. This requires the quantile identity to remain stable across environment transitions and reordering, but the author does not explain why this rank alignment is structurally compatible with the flow-generated distribution.
Questions for authors

I would greatly appreciate it if the author could clarify or explain the issues I raised in Weakness.

1 comment
AuthorsOpenReview ↗

Expectation-Coupled Flow Critic & Rank-Wise GAE Compatibility

We thank you for your positive assessment (Weak Acceptance) and insightful analysis of our actor-critic optimization mechanism and flow-based Distributional GAE structural compatibility.


1. Mechanism of Expectation-Coupled Distributional Flow Critic (W1)

Your mathematical observation is exact: because Distributional GAE operates linearly over sorted quantiles and the policy update aggregates advantages via A^t=Mean(AD(st))\hat{A}_t = \text{Mean}(A_D(s_t)), the actor consumes the expected advantage derived from the predicted distributional mean Vˉ(st)\bar{V}(s_t). Rather than applying non-linear risk distortions directly to the actor loss, which often causes premature entropy collapse and discards high-value exploratory rollouts in long-horizon LLM generation, coupling a flow- and risk-regularized distributional critic to the actor via its expectation is a deliberate architectural design that fundamentally changes and stabilizes Vˉ(st)\bar{V}(s_t) under heavy reward corruption:

  • Non-Local Gradient Coupling in Lipschitz Velocity Fields: Unlike discrete quantile critics where separate output heads learn isolated scalars, DFPO parameterizes all quantiles through a unified, spectral-normalized neural ODE velocity field vθ(zτ,τ,hs)v_\theta(z_\tau, \tau, h_s) sharing parameters θ\theta and backbone representation hsh_s. Under noisy supervision, standard scalar L2L_2 regression absorbs corrupted negative outliers directly into the conditional expectation, causing severe mean drift. In DFPO, enforcing left-tail concavity (LShapeL_{Shape}) and tail matching (LRiskL_{Risk}) bounds the gradient magnitude of corrupted lower-tail particles while expanding upper-tail exploratory mass. Because vθv_\theta is globally Lipschitz-continuous, shaping the asymmetric tails directly regularizes the global velocity field and shifts the integrated expectation Vˉ(st)=1K∑kz^1(k)\bar{V}(s_t) = \frac{1}{K}\sum_k \hat{z}_1^{(k)}, protecting the actor's advantage signal from downward noise bias while preserving rare reasoning chains.
  • Direct Empirical Validation: Evaluated against 3,200 Monte Carlo reference return samples, DFPO achieves a lower 1-Wasserstein distance (W1=0.6734W_1 = \mathbf{0.6734}) than Value Flows (0.68600.6860) and PCBF (0.69180.6918), while reducing lower-tail CVaR0.10\text{CVaR}_{0.10} error by 19.7% (from 1.73331.7333 to 1.3926\mathbf{1.3926}). Holding mean scalarization and training budgets strictly identical on Science training, DFPO (40.25%) significantly outperforms a Matched Quantile (mean) critic (35.94%), PCBF (37.36%), Value Flows (38.69%), and scalar PPO (37.34%). This confirms that continuous flow and tail regularizations yield a substantially more accurate advantage anchor for policy optimization.

2. Structural Compatibility of Continuous Flows, Rank-Wise GAE, and Euler Solvers (W2)

We address two structural questions on reconciling continuous-time ODE flows with rank-wise GAE coupling across transitions and finite-step Euler integration:

  • Quantile order invariance across transitions: In our 1D scalar value space (z∈Rz \in \mathbb{R}), vθ(zτ,τ,hs)v_\theta(z_\tau, \tau, h_s) is Lipschitz-continuous in zτz_\tau via spectral normalization (Sec. 3.2). By the Picard–Lindelöf uniqueness theorem, 1D ODE trajectories cannot intersect, giving ∂z1/∂z0>0\partial z_1 / \partial z_0 > 0 for any state hsh_s. The flow is thus a strictly monotonic, order-preserving diffeomorphism: ordered initial Gaussian particles z0(1)<⋯<z0(K)z_0^{(1)} < \dots < z_0^{(K)} preserve their exact rank at terminal states z1(1)<⋯<z1(K)z_1^{(1)} < \dots < z_1^{(K)} for both sts_t and st+1s_{t+1} without crossing or rank inversion. The Sort operation acts as a formal safeguard; the kk-th particle deterministically tracks the same cumulative probability level τk=Φ(z0(k))\tau_k = \Phi(z_0^{(k)}) across st→st+1s_t \to s_{t+1} transitions. Rank-wise particle coupling across timesteps is therefore structurally consistent, equivalent to 1D comonotonic coupling—the optimal transport plan minimizing 1-Wasserstein distance.
  • Finite-step Euler solver consistency: While general ODE flows suffer discretization drift under finite-step solvers, DFPO incorporates geometric consistency (LconsL_{cons}) and linear flow targets to rectify trajectories toward constant velocity (dvθ/dτ≈0dv_\theta/d\tau \approx 0), where Euler local truncation error vanishes. Evaluating the frozen DFPO critic across N∈{1,5,10,20}N \in \{1, 5, 10, 20\} Euler steps against the MC reference distribution shows negligible discrepancy: W1W_1 varies by <0.0011<0.0011 (0.67340.6734 at N=1N=1 to 0.67230.6723 at N=20N=20); KSKS varies by <0.0003<0.0003. This confirms 1-step Euler inference faithfully preserves the continuous flow distribution and its rank structure.(See Reviewer Ln9S)

We again thank you for your constructive feedback, which has substantially strengthened the paper. Please let us know if you have further questions, and we will respond promptly. If our replies are satisfactory, we kindly ask you to consider updating your score and confidence accordingly.

Comments on the paper

6 comments
AuthorsOpenReview ↗

summary part3

  • Tail constraints de-bias the critic mean Vˉ(st)\bar{V}(s_t): The actor updates via expected advantage A^t=Mean(AD(st))\hat{A}_t = Mean(A_D(s_t)), an expectation directly shaped by our critic design. Unlike discrete quantile critics with separate heads, DFPO generates all quantiles from a single velocity field vθ(zτ,τ,hs)v_\theta(z_\tau, \tau, h_s) spectrally normalized for Lipschitz continuity. With shared parameters θ\theta across all quantiles, constraining lower-tail downward motion via LShapeL_{Shape} and LRiskL_{Risk} blocks extreme negative gradients from corrupting intermediate quantiles. The integrated mean Vˉ(st)=(1/K)∑kz^1(k)\bar{V}(s_t) = (1/K)\sum_{k} \hat{z}_1^{(k)} is thus shielded from downward bias, avoiding actor-level pessimism (CVaR PG, min-scalarization) that confuses noise with true risk and triggers early entropy collapse.

4. Algorithmic Co-Design: Why 1-Step Trajectory Straightening Enables Token-Level Tail Shaping

DFPO is not a simple sum of independent losses; flow straightening and tail shaping are mutually dependent for LLM sequences:

  • Tractable Tail Gradients with NFE=1NFE=1: Evaluating LRiskL_{Risk} and LShapeL_{Shape} requires differentiating terminal quantile particles z^1(k)\hat{z}_1^{(k)} w.r.t. θ\theta at every token. Standard multi-step ODE critics (Value Flows, PCBF) require unrolling ODE solvers, which creates prohibitive memory cost and gradient instability over long sequences. Combining geometric consistency (LconsL_{cons}) with constant-velocity CFM targets (x1−x0x_1 - x_0), DFPO straightens trajectories to nearly constant velocity across τ∈[0,1]\tau \in [0,1]. This collapses ODE integration into one linear step: z^1=z0+vθ(z0,0,hs)\hat{z}_1 = z_0 + v_\theta(z_0, 0, h_s). Gradients from sorted particles z^(k)\hat{z}_{(k)} flow directly into vθv_\theta with no ODE unrolling.
  • Rank-Coupled Flow Particles for Multi-Step Distributional GAE: For 1D Lipschitz vθv_\theta, Picard–Lindelöf uniqueness guarantees non-crossing trajectories ∂z1/∂z0>0\partial z_1 / \partial z_0 > 0 for ordered initial samples z0(1)<⋯<z0(K)z_0^{(1)} \lt \cdots \lt z_0^{(K)}. The kk-th particle at sts_t and st+1s_{t+1} correspond to the same quantile level τk=Φ(z0(k))\tau_k = \Phi(z_0^{(k)}). Rank-wise TD summation in Distributional GAE is well-defined and equivalent to 1D comonotonic optimal transport. We prove this is an exact W1W_1 contraction with modulus Γ=γ(1−λ)1−λγ<1\Gamma = \frac{\gamma(1-\lambda)}{1-\lambda\gamma} \lt 1 in quantile space (revised Sec. 3.3 & App. B.1).

5. Summary of New Controlled Experiments & Direct Distribution Evaluations

We add controlled evaluations in point-by-point responses and revised Appendix C.6 to validate each design component:

  • Head-to-Head Comparison under Identical Mean Scalarization: Tested on our Qwen3-8B Science pipeline with matched scalarization A^t=Mean(AD(st))\hat{A}_t = Mean(A_D(s_t)), regularizers and compute budget, DFPO (40.25%) outperforms Value Flows (38.69%), PCBF (37.36%), and Matched Quantile (mean) critic (35.94%). Across seeds, DFPO surpasses DVPO on OOD Math (66.90±0.47%66.90 \pm 0.47\% vs 65.98±0.45%65.98 \pm 0.45\%, p<0.05p \lt 0.05) and avoids DVPO’s late-stage collapse on Dialogue at 1000 steps (87.31% vs 74.97%).
  • Return-Distribution Fidelity & Frozen Solver Check: Evaluated against 3200 Monte Carlo reference samples, DFPO achieves lower W1=0.6734W_1 = 0.6734 than Value Flows (0.6860) and PCBF (0.6918), and reduces lower-tail CVaR0.10CVaR_{0.10} error by 19.7% vs Value Flows and 15.4% vs PCBF. Sweeping Euler steps N∈{1,5,10,20}N \in \{1,5,10,20\} on the frozen critic yields negligible distribution drift (W1W_1 varies <0.0011\lt0.0011, KSKS varies <0.0003\lt0.0003), confirming 1-step projection preserves global and tail distributions.
  • Task-Agnostic Hyperparameters: Loss weights (λreg=0.1\lambda_{reg}=0.1, λcons=0.01\lambda_{cons}=0.01, λrisk=0.5\lambda_{risk}=0.5, λshape=0.5\lambda_{shape}=0.5) are selected from a coarse 4-candidate sweep to balance gradient norms and kept identical across Dialogue, Math and Science with no task-specific tuning.

Please refer to our individual responses below for detailed mathematical derivations and full experimental tables.

AuthorsOpenReview ↗

summary part2

2. Core Philosophy: Balancing Robustness and Generalization via Asymmetric Tail Control (Not Unbiased Estimation)

Instead of treating the critic as a passive unbiased estimator of raw empirical returns, we address a core question for real-world LLM post-training: how can a distributional critic simultaneously maximize optimization robustness under noisy supervision and preserve OOD generalization during dynamic policy updates?

In noisy LLM post-training, unbiased fitting of empirical targets is actively harmful, as it conflates two opposite extremes in the tails of the return distribution:

  • Why the lower tail governs robustness: Extrinsic reward corruption, such as false-negative 0 labels on valid reasoning chains from minor formatting mismatches (Fig. 6), appears as extreme spikes in the lower tail. Unbiased estimation memorizes these spikes, depresses value estimates, and injects chaotic advantage variance that causes late-stage policy collapse (PPO/GRPO collapse in multi-turn dialogue, Table 1). Naive pessimism such as CVaR or min-scalarization over-penalizes the lower tail, confusing label noise with true risk and triggering premature entropy collapse.
  • Why the upper tail governs generalization: Cross-domain reasoning breakthroughs rely on sparse, high-reward exploratory trajectories in the upper tail. Standard mean regression or symmetric smoothing treats these rare breakthroughs as outliers and smooths them away, weakening OOD generalization.
  • Quasi-normal signal sculpting for asymmetric tail control: Discrete quantile heads (QR-DQN, DVPO) learn isolated points without unified spatial geometry, while scalar critics collapse both tails into a single mean E[Y∣X]\mathbb{E}[Y|X]. Evolving a continuous velocity field vθ(zτ,τ,hs)v_\theta(z_\tau, \tau, h_s) from a Gaussian prior, DFPO uses the flow as a smooth quasi-normal information bottleneck that decouples the two tails. Through asymmetric curvature control (LRiskL_{Risk}, LShapeL_{Shape}), it contracts destructive lower-tail noise dispersion (∇2z^(k)≤0\nabla^2 \hat{z}_{(k)} \le 0 for robustness) while preserving upper-tail convexity (∇2z^(k)≥0\nabla^2 \hat{z}_{(k)} \ge 0 to amplify rare OOD exploratory signals), keeping the central mass smooth and unimodal to anchor de-biased advantage estimation.
  • Sensitivity-guided token-level credit assignment: Effective long-horizon LLM credit assignment requires identifying semantic pivot tokens where future returns bifurcate sharply. Unlike static quantile predictors, continuous ODE flows expose local Jacobian sensitivity ∥J(1)∥2=∥∂z1/∂z0∥2\|J(1)\|^2 = \|\partial z_1 / \partial z_0\|^2 along trajectories. Through wsens(s)=σ(∥J(1)∥2/τtemp)+0.5w_{sens}(s) = \sigma(\|J(1)\|^2 / \tau_{temp}) + 0.5, the critic dynamically focuses representational capacity on high-sensitivity decision tokens such as key logical operators and domain markers (Figs. 2 & 13), rather than applying uniform supervision across sequences.
  • Minimal transport energy for coherent cross-token guidance: Unconstrained paths often cross and oscillate, injecting chaotic gradient variance into policy updates (Fig. 4). Regularizing velocities toward straight-line optimal transport paths via LconsL_{cons} and LBCFML_{BCFM} minimizes Wasserstein transport energy (dvθ/dτ≈0dv_\theta / d\tau \approx 0). This ensures tail-denoised distributional signals propagate coherently across token transitions via rank-coupled Distributional GAE, providing stable high-resolution guidance for real-world capability optimization.

3. Concrete Execution Mechanism: Reward Noise Filtering and Mean Advantage De-biasing

While prior flow critics (Value Flows, PCBF) treat empirical Bellman targets ZtgtZ^{tgt} as clean ground truth and fit them uniformly, LLM rewards suffer severe false-negative spikes. Blind matching pulls lower quantiles toward extreme negatives (Fig. 3), depresses the critic mean, and injects harmful advantage spikes into the actor. DFPO executes energy control through two mathematical mechanisms:

  • Inter-particle spacing regulation (LRiskL_{Risk}, LShapeL_{Shape}): Instead of matching corrupted targets globally, DFPO regularizes spacing of sorted terminal quantile particles z^(1)<⋯<z^(K)\hat{z}_{(1)} \lt \cdots \lt \hat{z}_{(K)} via second-order finite differences ∇2z^(k)=z^(k+2)−2z^(k+1)+z^(k)\nabla^2 \hat{z}_{(k)} = \hat{z}_{(k+2)} - 2\hat{z}_{(k+1)} + \hat{z}_{(k)}. Left-tail concavity (∇2≤0\nabla^2 \le 0) restricts outward dispersion and blocks downward drag from false negatives; right-tail convexity (∇2≥0\nabla^2 \ge 0) lets upper particles spread, preserving rare high-reward trajectories instead of smoothing them as noise.
AuthorsOpenReview ↗

summary part1

We sincerely thank the Area Chair and all reviewers (REgR, hLAH, Ln9S) for their constructive feedback. We are encouraged by the shared recognition that stabilizing LLM post-training under noisy reward supervision is a vital problem, and that DFPO achieves strong training stability and cross-domain generalization across dialogue, math, and scientific reasoning tasks.

At the heart of DFPO is a shift in what a distributional critic should optimize for LLM post-training: rather than pursuing passive, unbiased density estimation of noisy empirical returns, DFPO treats the value flow as a dynamic supervisory energy controller designed to balance training robustness and out-of-domain (OOD) generalization. In long-sequence LLM RL, the two tails of the return distribution govern these two competing objectives—lower-tail proxy noise drives optimization collapse, while sparse upper-tail breakthroughs fuel OOD generalization. By sculpting asymmetric tail curvature over an order-preserving continuous flow manifold, DFPO filters destructive lower-tail false negatives without smoothing away rare upper-tail exploratory trajectories.

To provide a clear executive overview of our work and revisions (highlighted in blue in the updated manuscript), we outline DFPO’s methodological novelty, tail-centric robustness–generalization philosophy, noise-filtering and co-design mechanisms, and new empirical & theoretical results below.


1.Methodological Novelty: Scaling Robust RL via Risk- and Consistency-Controlled Value Flows

DFPO advances robust RL for LLM post-training across three fundamental literature dimensions:

  • Beyond Heuristic Advantage Clipping & Reweighting (BAPO, KTAE, λ\lambda-GRPO): Prior robust RL methods stabilize policy updates by truncating or reweighting noisy scalar advantages. While this reduces variance, it does not explicitly guide out-of-domain (OOD) exploration. Instead, DFPO scales value modeling at the representation level, transforming raw scalar feedback into a structured, asymmetric quasi-normal distributional signal that jointly stabilizes optimization and guides cross-domain generalization.
  • From Isolated Discrete Quantiles to Continuous Value Flow Fields (QR-DQN, DVPO, Robust Bellman): Existing distributional critics predict fixed quantiles via independent scalar heads, resulting in coarse-grained state conditioning and isolated head drift under noisy supervision. DFPO replaces discrete quantile learning with a continuous-time generative velocity field vθ(zτ,τ,hs)v_\theta(z_\tau, \tau, h_s) over a virtual horizon τ∈[0,1]\tau \in [0,1]. Mapping a Gaussian prior through this shared, spectral-normalized flow field captures fine-grained state-conditioned uncertainty, skewness, and multi return dynamics across tokens.
  • From Passive Density Matching to Active Risk & Trajectory Control (FlowRL, Value Flows, PCBF): Unlike policy-side trajectory reward matching (FlowRL) or prior flow critics (Value Flows, PCBF) that passively fit empirical Bellman targets across the full support, DFPO integrates conditional risk/shape control (LRisk,LShapeL_{Risk}, L_{Shape}) and geometric trajectory rectification (LconsL_{cons}) with multi-step comonotonic Distributional GAE. This actively contracts lower-tail supervision noise (∇2z^(k)≤0\nabla^2 \hat{z}_{(k)} \le 0) while preserving upper-tail exploratory reasoning chains (∇2z^(k)≥0\nabla^2 \hat{z}_{(k)} \ge 0) in a single differentiable step (NFE=1NFE=1).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.