- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Diffusion Bridge Critics (DBC), a distributional critic that maps quantile levels to return quantiles through a deterministic bridge. Training combines quantile regression with an auxiliary anchor loss based on empirical target quantiles. An integral-consistent sampler is introduced to address discretization error. The paper argues that vanilla diffusion critics degrade to a Gaussian under repeated Bellman updates (Theorem 4.1) while DBC does not, and evaluates the method on MuJoCo and DMC benchmarks, including integrations with SAC, TD3, and QVPO.
Critical strengths
1. An interesting critic-side idea that can complement different policy models.
DBC combines explicit quantile conditioning with multi-step bridge computation, so each output estimates a specified return quantile. The integrations with SAC, TD3, and QVPO show that the design can be used with different actor architectures, including a diffusion policy. This makes the proposal potentially useful beyond one particular actor-critic combination.
2. Promising results in the learning curves.
DBC has the highest plotted endpoint in all nine panels of Fig. 3, with particularly visible gaps on Walker2d-v5, Ant-v5, Hopper-v5 and Humanoid-Run. This is encouraging empirical evidence, although the magnitude of the gains needs clarification (Weakness 2). The ablations also examine the main design choices: the number of bridge steps, the anchor-loss weight, and the interpolation schedule.
Critical weaknesses
1. The method's rationale is difficult to follow, and its central theoretical claim is not established.
My background with RL and control, including diffusion/flow policies, is more on the actor side, with less direct experience in distributional critics. My main difficulty is understanding why DBC should work. Section 3.2 moves quickly to UniDB's optimal-control formulation without adequately explaining the bridge's role in critic learning. Section 4 then introduces targets, losses, and discretization without making the connection to Gaussian degradation clear.
An easier-to-follow progression would be: first define the vanilla diffusion critic and its failure mode; then explain why the quantile-conditioned bridge should address it; next introduce the training objective and state what is proven about it, under which assumptions; and finally describe how the critic is trained and sampled.
For the anti-degradation claim, I would expect a theorem or proposition explaining, under stated assumptions, why repeated approximate Bellman updates with DBC avoid spuriously collapsing a non-Gaussian return distribution to a Gaussian. Theorem 4.2 establishes endpoint consistency, while the anchor-loss results concern fitting error and fixed-point deviation; their connection to this stronger claim is unclear. If the claim is instead supported mainly by experiments, its scope should be made explicit.
There is also a concrete gap in Theorem 4.1. In Appendix B.1, independence and zero mean do not suffice to make the discounted error sum Gaussian. Take a deterministic true return , , and independent errors with . Then
which is not Gaussian. The proof also takes , which is different from increasing the number of Bellman updates at a fixed discount. Theorem 4.1 is therefore not established as stated. The proof also does not explain which assumption or step distinguishes vanilla diffusion critics from DBC.
2. The table summaries and learning curves are difficult to reconcile.
The relationship between tasks, tables, and curves is difficult to follow, especially for similarly named MuJoCo and DMC tasks (e.g., Humanoid-v5 vs. Humanoid-Run). More importantly, several Table 2 values lie well above both the endpoints and peaks of the corresponding curves in Figure 3. Representative examples are below.
| Task / method | Table 2 | Fig. 3 endpoint | Fig. 3 peak |
|---|---|---|---|
| Hopper-v5 / DSAC | 3508.8 ± 22.3 | ≈ 1.4k | ≈ 2.0k |
| Hopper-v5 / SAC | 3648.7 ± 49.0 | ≈ 2.2k | ≈ 3.3k |
| Walker2d-v5 / DSAC | 3320.5 ± 1198.4 | ≈ 1.3k | ≈ 1.3k |
| Humanoid-v5 / VD | 4872.2 ± 233.4 | ≈ 2.3k | ≈ 2.3k |
| Humanoid-v5 / VF | 5004.8 ± 58.5 | ≈ 3.2k | ≈ 3.5k |
These differences need to be explained; otherwise, the efficacy of the proposed DBC method is questionable
Questions for authors
1. What does the bridge add? (Clarification.)
Why use a multi-step bridge rather than a direct quantile predictor, and what do the additional steps contribute? A short conceptual explanation connecting the bridge and the training objectives before the implementation details would help.
2. Why does DBC avoid degradation? (Clarification/correction.)
Which critics does Theorem 4.1 cover, and which step of its argument fails for DBC? Can the authors address the counterexample and limiting argument in Weakness 1 and distinguish proven guarantees from empirical observations?
3. What do the tables report? (Clarification.)
Do Tables 2–3 report the final checkpoint, each seed's best checkpoint, or another statistic? How does this account for the examples above? Do the tables and curves use the same runs and settings? What do the ± values and shaded regions denote, and what training budget is used for each DMC task?
I am open to raising my score if the method's rationale and theoretical claims are clarified or corrected, and the experimental reporting is convincingly reconciled.
4 comments
Weakness2&Q3(1): What do the tables report? (Clarification.)
Do Tables 2–3 report the final checkpoint, each seed's best checkpoint, or another statistic? What do the ± values and shaded regions denote,
AW2&A3(1): Thank you for your question. Tables 2–3 report the mean of each seed's best return, with indicating the standard deviation across seeds. The shaded regions also represent the standard deviation across seeds.
Q3(2) How does this account for the examples above? Do the tables and curves use the same runs and settings?
A3(2): Thank you for pointing this out. The tables and curves use the same runs and settings but report different statistics. The tables average each seed's best return, whereas the curves show smoothed mean returns across seeds. The learning curves are mainly intended to illustrate the overall reward trends. Therefore, the curves may not directly match the tables.
Q3(3) what training budget is used for each DMC task?
A3(3): Thank you for your question. The training budgets are listed below. We use smaller budgets for Cartpole-Balance and Reacher-Easy due to their relative simplicity.
| Task | Training Budget |
|---|---|
| Cartpole-Balance-v0 | 100k |
| Reacher-Easy-v0 | 100k |
| Walker-Run-v0 | 500k |
| Humanoid-Stand-v0 | 1000k |
| Humanoid-Run-v0 | 1000k |
Thank you for the responses. Please update the statistics you report in the tables and figures, maybe in the captions. They can be misleading now. Other than that, I don't have any further questions, and I've raised my score.
Thank you for your careful reading and support. We have revised the captions of Tables 2 and 3 to clarify how we summarize each seed’s best return, and the caption of Figure 3 to explain what the curves and shaded bands represent.
AW1(a): Thank you for your valuable feedback. We agree that this ordering is more reader-friendly. We have revised the paper accordingly: first, we added a brief introduction to the Diffusion Critic in Section 3.1. We then added the motivation for introducing the Diffusion Bridge in Section 3.2, and in Section 4 we briefly explain why the Diffusion Bridge can avoid the problems of the Diffusion Critic. Additionally, we added a new experiment in Appendix B.6 showing that the Diffusion Critic fits the mean and variance quickly, but fits shape information more slowly. The corresponding training objectives are then presented in the subsequent sections.
AW1(b)&A2: We thank the reviewer for their careful reading. Our revised theorem no longer relies on assumptions about γ→1. We have revised the proof of Gaussian degradation in Appendix B.1. At a fixed discount, each shift of the Bellman target moves the shape component of a vanilla diffusion critic’s score field, so the critic must refit it. Under the conditions in Appendix B.1, partial fitting leaves Gaussian source noise at each update and drives the return distribution toward a Gaussian. In DBC, the same shift preserves the shape velocity at each quantile level, so the critic can continue fitting the same shape across updates.
A1: We thank the reviewer for this question. Although IQN does not suffer from Gaussian degradation, its single-step MLP may be less effective at modeling complex return distributions. In contrast, Diffusion Bridge provides greater modeling flexibility through multi-step refinement, enabling a gradual distribution-to-distribution transport. As proven in Appendix B.5 (Theorem B.6), quantile coupling is optimal for scalar distributions. The benefits are further supported by the example in Appendix B.6 (Figure 5) and our ablation in Table 5, where five-step DBC consistently outperforms its single-step counterpart.