- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes DFPO, a flow-based distributional critic for LLM post-training under noisy rewards. It combines distributional GAE with sensitivity weighting, anchor regularization, geometric consistency, and tail constraints, reporting improved training stability and out-of-domain performance.
Critical strengths
The paper addresses a relevant problem through critic design rather than only modifying policy updates. The reported late-stage stability on dialogue tasks is encouraging.
Critical weaknesses
1. The risk-control interpretation is not sufficiently justified. The actor averages the distributional advantages, which algebraically reduces to standard GAE based on the critic’s mean. Meanwhile, Eq. (4) matches predicted and target tail means rather than imposing a policy-level risk constraint. Distributional training may improve mean estimation through regularization, but this does not by itself establish risk-averse behavior or suppression of catastrophic outcomes.
2. The geometric consistency argument contains a logical gap. Eq. (3) enforces equal terminal projections at and , but this symmetry does not imply that the projection is constant over time, as assumed in Proposition 3. This weakens the stated justification for straight trajectories and accurate single-step inference. Moreover, the sampling-step ablation measures downstream accuracy, not whether single-step sampling preserves the modeled distribution.
3. The experiments do not fully isolate the contribution of flow modeling. The Robust Bellman baseline uses minimum-based scalarization, whereas DFPO uses a mean, introducing a confound beyond critic parameterization. The improvement over DVPO is also relatively small on the reported reasoning comparison (40.25% versus 39.63%). Multiple seeds for DFPO alone do not establish the statistical reliability of this difference. Matched critic comparisons and multi-seed results for the strongest baselines are needed.
Questions for authors
- Is the learning target the true return distribution or a regularized surrogate for improving mean-advantage estimation? How should its bias be interpreted?
- Can the authors revise Proposition 3 and compare single-step versus multi-step sampling using a frozen critic, reporting distributional and tail-estimation errors?
- Do the gains persist against a quantile critic with matched scalarization, regularization, and training budget, and against DVPO across multiple training seeds?
- Appendix B.1 analyzes a geometric mixture of return distributions, whereas the algorithm uses rank-wise sums of quantile TD errors, It seems that there is a theory-practice gap.
6 comments
Thanks for the response and additional experiments. The matched critic comparisons and extra seeds strengthen the empirical results.
My main concerns are not fully resolved. Reshaping noisy targets does not necessarily remove bias. How does the method distinguish incorrect negative feedback from genuinely poor outcomes? For Proposition 3, fitting constant velocities along interpolation paths does not by itself establish straight learned ODE trajectories. The contraction result also needs to clarify what fixed point the implemented surrogate converges to. The frozen critic experiment is helpful, but similar errors against a common reference do not establish that different solvers produce similar distributions. A direct comparison between one step and a sufficiently accurate numerical solve would be more helpful.
Overall, I appreciate the efforts, but still have some concerns about the theoretical justification and denoising explanation.
Response part(1/4)
Thank you for your thoughtful follow-up and for acknowledging that our matched critic comparisons and multi-seed experiments strengthen the empirical results. Your four remaining questions touch upon the exact mathematical and numerical foundations of DFPO. Below, we provide rigorous theoretical characterizations and direct frozen-critic empirical verifications to fully resolve each point.
1. How DFPO Distinguishes Incorrect Negative Feedback (False Negatives) from Genuinely Poor Outcomes (True Negatives)
A natural concern is whether left-tail concavity () blindly pushes up all negative values, thereby failing to penalize genuinely poor outcomes. DFPO separates extrinsic false-negative supervision noise from genuinely poor outcomes both mathematically and empirically:
- Mathematical Mechanism (Translation Invariance of vs. Localized Tail Spikes): For a token state , the critic models the conditional return distribution across continuations:
- Genuinely poor outcomes (Global Location Shift): When a reasoning prefix contains a genuine logical error, continuations consistently fail, shifting the entire target quantile set downward by a negative offset . Crucially, our curvature operator is strictly translation-invariant: . Thus, imposes zero penalty on a global downward shift. Guided by first-order flow matching () and tail-mean anchoring (), the critic faithfully assigns a low expected state value to genuinely poor states.
- Incorrect negative feedback (Localized High-Curvature Discontinuity): When a valid reasoning state receives a sporadic false-negative reward due to brittle exact-match rules or noisy RMs, it conflicts with the shared semantic representation learned across valid states and manifests as an isolated lower-tail spike (). Left-tail concavity () specifically contracts this localized second-order jump while anchors the conditional mean, preventing false-negative spikes from dragging down.
- Direct Empirical Verification on True Negatives vs. False Negatives: To test this directly on our frozen critics, we categorized the validation rollouts with completed boxed answers ( out of 3200 rollouts) by comparing the noisy single-trace training label (
label) against the true correctness label: (A) True Negatives (label=0and answer is genuinely incorrect, ), (B) False Negatives (label=0due to formatting/rule mismatch, but answer is genuinely correct, ), and (C) True Positives (label=1and answer is genuinely correct, ). Note that all reported numbers below represent the critic's predicted state value (the mean of quantile particles at the terminal response token) under paired random seeds:
| Frozen Critic | A: True Neg Value | B: False Neg Value | C: True Pos Value | Denoising Gap: B$-$A | Discrimination Gap: C-\A |
|---|---|---|---|---|---|
| Value Flows | 0.0784 | 0.1133 | 0.0794 | ||
| PCBF | 0.3248 | 0.3835 | 0.3443 | ||
| DFPO (Ours) | 0.3645 | 0.3910 | 0.4000 |
This evaluation establishes three key findings:
- Effective Recovery from False-Negative Supervision: On False Negatives (B), where the model produced a genuinely correct answer but received a corrupted training reward of , DFPO predicts a substantially higher expected value (, close to True Positives (C) at ) than on True Negatives (A) (), confirming that DFPO does not treat mislabeled correct reasoning as genuinely poor outcomes.
- Preserving Ground-Truth Value Ordering (): DFPO is the only method that maintains the proper monotonic value ordering . Both Value Flows and PCBF suffer from value inversion ().
- Clear True Outcome Discrimination (): Under noisy supervision, Value Flows loses the ability to distinguish genuinely correct outcomes from genuinely poor ones ( vs. , gap ), and PCBF exhibits a much weaker separation () distorted by false-negative over-estimation. DFPO achieves the largest and cleanest value separation between True Positives and True Negatives (), confirming that our regularizers filter out supervision noise while preserving genuine credit assignment.
De-biased Surrogate Target, Prop. 3 Refinement & Multi-Seed Controls
Thank you for your rigorous critique, which refined our theoretical guarantees and experimental controls. We address all concerns below.
1. Surrogate Target, Bias & Supervision Risk Filtering (W1, Q1)
2. Proposition 3 & Frozen-Critic Fidelity (W2, Q2)
Across N=1–20, W1 varies by <0.0011 and KS by <0.0003, confirming 1-step inference faithfully preserves global and tail distributions.
3. Matched Critics & Multi-Seed Reliability (W3, Q3)
Removing scalarization confounds, DFPO outperforms Matched Quantile (mean) by +4.31% (40.25% vs. 35.94%).
4. Theory-Practice Alignment in App. B.1 (Q4)
Eq. (11) mixes measures by averaging CDFs F, while Sec. 3.3 averages sorted quantiles F−1 (comonotonic addition). Since W1(μ,ν)=∫01∣Fμ−1(τ)−Fν−1(τ)∣dτ in 1D, rank-wise quantile GAE is an exact W1 contraction with identical modulus Γ=1−λγγ(1−λ)<1 via the L1 triangle inequality. This proof has been added to Sec. 3.3 and App. B.1.
Thank you again for feedback strengthening the paper. Please let us know if you have further questions, and we will respond promptly. If our replies are satisfactory, we kindly ask you to consider updating your score and confidence accordingly.