acceptodds
Under review as a conference paper at ICLR 2027

Distributional Reinforcement Learning with Diffusion Bridge Critics

Abstract

Recent advances in diffusion-based reinforcement learning (RL) methods have demonstrated promising results in a wide range of continuous control tasks. However, existing works in this field focus on the application of diffusion policies while leaving the diffusion critics unexplored. Actually, accurate value estimation is an important component for efficient RL, as policies are optimized by the values or gradients estimated by the critic model. Furthermore, given the stochasticity of most decision-making tasks, it has been confirmed that the critic is more appropriately depicted with a distributional model. Motivated by these points, we propose a novel distributional RL method with Diffusion Bridge Critics (DBC). DBC directly models the inverse cumulative distribution function of return. This allows us to accurately capture the value distribution and prevents it from collapsing into a trivial Gaussian distribution owing to the strong distribution-matching capability of the diffusion bridge. Moreover, we further derive an analytic integral formula to address discretization errors in DBC, which is essential in value estimation. To our knowledge, DBC is the first work to employ the diffusion bridge model as the critic. Notably, DBC is also a plug-and-play component and can be integrated into most existing RL frameworks. Experimental results on MuJoCo robot control benchmarks and DMC benchmarks, demonstrating the superiority of DBC compared with previous distributional critic models.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Diffusion Bridge, Quantile Critic

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DZCsOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Diffusion Bridge Critics (DBC), a distributional critic that maps quantile levels to return quantiles through a deterministic bridge. Training combines quantile regression with an auxiliary anchor loss based on empirical target quantiles. An integral-consistent sampler is introduced to address discretization error. The paper argues that vanilla diffusion critics degrade to a Gaussian under repeated Bellman updates (Theorem 4.1) while DBC does not, and evaluates the method on MuJoCo and DMC benchmarks, including integrations with SAC, TD3, and QVPO.

Critical strengths

1. An interesting critic-side idea that can complement different policy models.

DBC combines explicit quantile conditioning with multi-step bridge computation, so each output estimates a specified return quantile. The integrations with SAC, TD3, and QVPO show that the design can be used with different actor architectures, including a diffusion policy. This makes the proposal potentially useful beyond one particular actor-critic combination.

2. Promising results in the learning curves.

DBC has the highest plotted endpoint in all nine panels of Fig. 3, with particularly visible gaps on Walker2d-v5, Ant-v5, Hopper-v5 and Humanoid-Run. This is encouraging empirical evidence, although the magnitude of the gains needs clarification (Weakness 2). The ablations also examine the main design choices: the number of bridge steps, the anchor-loss weight, and the interpolation schedule.

Critical weaknesses

1. The method's rationale is difficult to follow, and its central theoretical claim is not established.

My background with RL and control, including diffusion/flow policies, is more on the actor side, with less direct experience in distributional critics. My main difficulty is understanding why DBC should work. Section 3.2 moves quickly to UniDB's optimal-control formulation without adequately explaining the bridge's role in critic learning. Section 4 then introduces targets, losses, and discretization without making the connection to Gaussian degradation clear.

An easier-to-follow progression would be: first define the vanilla diffusion critic and its failure mode; then explain why the quantile-conditioned bridge should address it; next introduce the training objective and state what is proven about it, under which assumptions; and finally describe how the critic is trained and sampled.

For the anti-degradation claim, I would expect a theorem or proposition explaining, under stated assumptions, why repeated approximate Bellman updates with DBC avoid spuriously collapsing a non-Gaussian return distribution to a Gaussian. Theorem 4.2 establishes endpoint consistency, while the anchor-loss results concern fitting error and fixed-point deviation; their connection to this stronger claim is unclear. If the claim is instead supported mainly by experiments, its scope should be made explicit.

There is also a concrete gap in Theorem 4.1. In Appendix B.1, independence and zero mean do not suffice to make the discounted error sum Gaussian. Take a deterministic true return ZZ, γ=1/2\gamma = 1/2, and independent errors with P(ϵk=1)=P(ϵk=−1)=1/2\mathbb{P}(\epsilon_k = 1) = \mathbb{P}(\epsilon_k = -1) = 1/2. Then

Z^−Z=∑k=0∞2−kϵk=dUniform[−2,2],\widehat Z - Z = \sum_{k=0}^{\infty} 2^{-k} \epsilon_k \overset{d}{=} \mathrm{Uniform}[-2, 2],

which is not Gaussian. The proof also takes γ→1\gamma \to 1, which is different from increasing the number of Bellman updates at a fixed discount. Theorem 4.1 is therefore not established as stated. The proof also does not explain which assumption or step distinguishes vanilla diffusion critics from DBC.

2. The table summaries and learning curves are difficult to reconcile.

The relationship between tasks, tables, and curves is difficult to follow, especially for similarly named MuJoCo and DMC tasks (e.g., Humanoid-v5 vs. Humanoid-Run). More importantly, several Table 2 values lie well above both the endpoints and peaks of the corresponding curves in Figure 3. Representative examples are below.

Task / methodTable 2Fig. 3 endpointFig. 3 peak
Hopper-v5 / DSAC3508.8 ± 22.3≈ 1.4k≈ 2.0k
Hopper-v5 / SAC3648.7 ± 49.0≈ 2.2k≈ 3.3k
Walker2d-v5 / DSAC3320.5 ± 1198.4≈ 1.3k≈ 1.3k
Humanoid-v5 / VD4872.2 ± 233.4≈ 2.3k≈ 2.3k
Humanoid-v5 / VF5004.8 ± 58.5≈ 3.2k≈ 3.5k

These differences need to be explained; otherwise, the efficacy of the proposed DBC method is questionable

Questions for authors

1. What does the bridge add? (Clarification.)

Why use a multi-step bridge rather than a direct quantile predictor, and what do the additional steps contribute? A short conceptual explanation connecting the bridge and the training objectives before the implementation details would help.

2. Why does DBC avoid degradation? (Clarification/correction.)

Which critics does Theorem 4.1 cover, and which step of its argument fails for DBC? Can the authors address the counterexample and limiting argument in Weakness 1 and distinguish proven guarantees from empirical observations?

3. What do the tables report? (Clarification.)

Do Tables 2–3 report the final checkpoint, each seed's best checkpoint, or another statistic? How does this account for the examples above? Do the tables and curves use the same runs and settings? What do the ± values and shaded regions denote, and what training budget is used for each DMC task?

I am open to raising my score if the method's rationale and theoretical claims are clarified or corrected, and the experimental reporting is convincingly reconciled.

4 comments

Weakness 1: The method's rationale is difficult to follow, and its central theoretical claim is not established.

1(a): My background with RL and control, including diffusion/flow policies, is more on the actor side, with less direct experience in distributional critics. My main difficulty is understanding why DBC should work. Section 3.2 moves quickly to UniDB's optimal-control formulation without adequately explaining the bridge's role in critic learning. Section 4 then introduces targets, losses, and discretization without making the connection to Gaussian degradation clear.

An easier-to-follow progression would be: first define the vanilla diffusion critic and its failure mode; then explain why the quantile-conditioned bridge should address it; next introduce the training objective and state what is proven about it, under which assumptions; and finally describe how the critic is trained and sampled.

AW1(a): Thank you for your valuable feedback. We agree that this ordering is more reader-friendly. We have revised the paper accordingly: first, we added a brief introduction to the Diffusion Critic in Section 3.1. We then added the motivation for introducing the Diffusion Bridge in Section 3.2, and in Section 4 we briefly explain why the Diffusion Bridge can avoid the problems of the Diffusion Critic. Additionally, we added a new experiment in Appendix B.6 showing that the Diffusion Critic fits the mean and variance quickly, but fits shape information more slowly. The corresponding training objectives are then presented in the subsequent sections.

Weakness1(b) For the anti-degradation claim, I would expect a theorem or proposition explaining, under stated assumptions, why repeated approximate Bellman updates with DBC avoid spuriously collapsing a non-Gaussian return distribution to a Gaussian. Theorem 4.2 establishes endpoint consistency, while the anchor-loss result concerns fitting error and fixed-point deviation; their connection to this stronger claim is unclear. If the claim is instead supported mainly by experiments, its scope should be made explicit.

There is also a concrete gap in Theorem 4.1. In Appendix B.1, independence and zero mean do not suffice to make the discounted error sum Gaussian. Take a deterministic true return Z, γ = 1/2, and independent errors with P(ε_k = 1) = P(ε_k = −1) = 1/2. Then

Z−Z~=∑k=1∞2−kεk=dUniform[−2,2]Z − \tilde{Z} = \sum_{k=1}^{\infty} 2^{−k} \varepsilon_k \overset{d}{=} \mathrm{Uniform}[−2, 2],

which is not Gaussian. The proof also takes γ → 1, which is different from increasing the number of Bellman updates at a fixed discount. Theorem 4.1 is therefore not established as stated. The proof also does not explain what assumption or step distinguishes vanilla diffusion critics from DBC.

Q2: Why does DBC avoid degradation? (Clarification/correction.)

Which critics does Theorem 4.1 cover, and which step of its argument fails for DBC? Can the authors address the counterexample and limiting argument in Weakness 1 and distinguish proven guarantees from empirical observations?

AW1(b)&A2: We thank the reviewer for their careful reading. Our revised theorem no longer relies on assumptions about γ→1\gamma\to 1. We have revised the proof of Gaussian degradation in Appendix B.1. At a fixed discount, each shift of the Bellman target moves the shape component of a vanilla diffusion critic’s score field, so the critic must refit it. Under the conditions in Appendix B.1, partial fitting leaves Gaussian source noise at each update and drives the return distribution toward a Gaussian. In DBC, the same shift preserves the shape velocity at each quantile level, so the critic can continue fitting the same shape across updates.

Q1: What does the bridge add? (Clarification.)

Why use a multi-step bridge rather than a direct quantile predictor, and what do the additional steps contribute? A short conceptual explanation connecting the bridge and the training objectives before the implementation details would help.

A1: We thank the reviewer for this question. Although IQN does not suffer from Gaussian degradation, its single-step MLP may be less effective at modeling complex return distributions. In contrast, Diffusion Bridge provides greater modeling flexibility through multi-step refinement, enabling a gradual distribution-to-distribution transport. As proven in Appendix B.5 (Theorem B.6), quantile coupling is optimal for scalar distributions. The benefits are further supported by the example in Appendix B.6 (Figure 5) and our ablation in Table 5, where five-step DBC consistently outperforms its single-step counterpart.

Diffusion StepsAnt-v5Humanoid-v5Walker2d-v5
15150.45524.75809.8
56481.95877.26162.8

Weakness2&Q3(1): What do the tables report? (Clarification.)

Do Tables 2–3 report the final checkpoint, each seed's best checkpoint, or another statistic? What do the ± values and shaded regions denote,

AW2&A3(1): Thank you for your question. Tables 2–3 report the mean of each seed's best return, with ±\pm indicating the standard deviation across seeds. The shaded regions also represent the standard deviation across seeds.

Q3(2) How does this account for the examples above? Do the tables and curves use the same runs and settings?

A3(2): Thank you for pointing this out. The tables and curves use the same runs and settings but report different statistics. The tables average each seed's best return, whereas the curves show smoothed mean returns across seeds. The learning curves are mainly intended to illustrate the overall reward trends. Therefore, the curves may not directly match the tables.

Q3(3) what training budget is used for each DMC task?

A3(3): Thank you for your question. The training budgets are listed below. We use smaller budgets for Cartpole-Balance and Reacher-Easy due to their relative simplicity.

TaskTraining Budget
Cartpole-Balance-v0100k
Reacher-Easy-v0100k
Walker-Run-v0500k
Humanoid-Stand-v01000k
Humanoid-Run-v01000k

Thank you for the responses. Please update the statistics you report in the tables and figures, maybe in the captions. They can be misleading now. Other than that, I don't have any further questions, and I've raised my score.

Thank you for your careful reading and support. We have revised the captions of Tables 2 and 3 to clarify how we summarize each seed’s best return, and the caption of Figure 3 to explain what the curves and shaded bands represent.

Reviewer_6c6sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Diffusion Bridge Critics (DBC), a distributional RL method that uses diffusion bridges to model return quantiles. The approach is motivated by the limited expressiveness of some existing critics and the distributional degradation that the authors observe with vanilla diffusion critics under repeated Bellman updates. DBC combines a quantile-conditioned critic with an auxiliary Anchor Loss and an Integral-Consistent Discretization scheme. The authors evaluate the method on MuJoCo and DMC benchmarks.

Critical strengths
  1. The identification of Gaussian degradation is well-motivated, with an intuitive illustration of Figure 2.
  2. A theoretical result (Theorem 4.1) is provided to justify replacing vanilla value diffusion with a diffusion-bridge critic that models the quantile function.
  3. Good experimental results are obtained.
Critical weaknesses
  1. The theoretical justification of Gaussian degradation is not fully convincing. Theorem 4.1 assumes independent, zero-mean approximation errors without justifying these assumptions in bootstrapped learning. Figure 2 illustrates the phenomenon in a specific setting but cannot establish the general claim.
  2. The comparison against Value Diffusion (VD) is not sufficient to support the paper's central claim.
Questions for authors
  1. Why are those assumptions on the approximation errors (independence, zero mean, bounded variance, etc.) made in the proof of Theorem 4.1? Under bootstrapped Bellman updates, consecutive targets may be highly correlated—how are these assumptions justified?
  2. The comparison against Value Diffusion (VD) is insufficient. The comparison is confined to the MuJoCo results in Table 2 and is absent from the DMC evaluation (Table 3) and the plug-and-play experiments (Table 4). Excluding VD from these benchmarks weakens the evidential basis for the paper's central claim.
  3. Since Appendix F states that VD has no official implementation and was reproduced from scratch, can the authors provide evidence that this re-implementation reproduces VD's reported performance on at least one task?
2 comments

Weakness 1: The theoretical justification of Gaussian degradation is not fully convincing. Theorem 4.1 assumes independent, zero-mean approximation errors without justifying these assumptions in bootstrapped learning. Figure 2 illustrates the phenomenon in a specific setting but cannot establish the general claim.

Q1: Why are those assumptions on the approximation errors (independence, zero mean, bounded variance, etc.) made in the proof of Theorem 4.1? Under bootstrapped Bellman updates, consecutive targets may be highly correlated—how are these assumptions justified?

AW1&A1: Thank you for your question. We have revised the proof of Gauss Degradation in Appendix B.1. In the new proof, we start from the observation that in Diffusion/Flow Matching, after each translation of the target distribution, the shape-related component of the velocity field must be refitted, and we then prove that under insufficient fitting, the distribution degrades to a Gaussian distribution.

Weakness 2: The comparison against Value Diffusion (VD) is not sufficient to support the paper's central claim.

Q2: The comparison against Value Diffusion (VD) is insufficient. The comparison is confined to the MuJoCo results in Table 2 and is absent from the DMC evaluation (Table 3) and the plug-and-play experiments (Table 4). Excluding VD from these benchmarks weakens the evidential basis for the paper's central claim.

AW2 & Q2: We thank the reviewer for this comment. We have added the experimental results for Value Diffusion on DMC, as shown in the table below.

MethodCartpole-BalanceReacher-EasyHumanoid-Run
Value Diffusion999.38 ± 0.46178.62 ± 153.0931.59 ± 67.12
SAC999.0 ± 0.9871.3 ± 95.977.4 ± 65.4
Ours999.3 ± 0.2972.6 ± 8.0138.9 ± 23.6

Q3: Since Appendix F states that VD has no official implementation and was reproduced from scratch, can the authors provide evidence that this re-implementation reproduces VD's reported performance on at least one task?

A3: We thank the reviewer. Our from-scratch VD implementation achieves approximately similar performance on Humanoid-v5 comparable to that reported in the original paper.

our implementationorigin paperDBC
Humanoid-v54872.2 ± 233.45497.15 ± 48.585877.2 ± 188.7

Dear Reviewer 6c6s,

With the discussion period closing tomorrow, we would like to confirm whether our response has resolved your concerns. If anything is still unclear, we are happy to explain further. Thank you again for your thoughtful comments.

Reviewer_B1V6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper claims that diffusion critics suffer from Gaussian degradation. Because small error could accumulate across Bellman updates and the learnt return distribution might blur into Gaussian. They use theorem and toy example to show the question exist. Then the paper proposes Diffusion Bridge Critic (DBC) to solve this question. Instead of generating samples from noise, DBC uses a deterministic diffusion bridge, which maps quantile level to corresponding return quantile. The critic trained with not just quantile loss, but also an anchor loss to help training. For experiments, the paper includes five MuJoCo and five DMC tasks, to report DBC can outperform other baselines.

Critical strengths
  1. The paper identifies a new failure mode for diffusion critics, and proposes a new critic design DBC.
  2. The paper shows DBC achieve good performance with several different actor designs.
Critical weaknesses
  1. The experiments are not very reliable, because plain SAC is almost the best baseline. If the proposed method is mainly built on previous distributional critics, the chosen tasks should be ones where modeling the return distribution is useful. If the standard SAC critic is the best baseline here, it could be that the tasks are not well chosen, or the other baselines are not well trained. Either way, it's hard to say DBC is better because it models the distribution better.
  2. The paper's problem is about diffusion critics, but its solution mainly comes from the quantile framework that IQN already have. The paper also shows IQN is free of degradation in Table 1. The new part is diffusion bridge, but it's not clear why this part is necessary.
Questions for authors
  1. Why is the standard SAC critic almost the best baseline?
  2. Can you show something similar to fig 2 on real tasks, to show Gaussian degradation actually happen while DBC can avoid it?
  3. Why is the setup for each model not unified, such as in Table 10-12?
  4. Why does Table 3 only include three baselines?
4 comments

Weakness 1: The experiments are not very reliable, because plain SAC is almost the best baseline. If the proposed method is mainly built on previous distributional critics, the chosen tasks should be ones where modeling the return distribution is useful. If the standard SAC critic is the best baseline here, it could be that the tasks are not well chosen, or the other baselines are not well trained. Either way, it's hard to say DBC is better because it models the distribution better.

Q1: Why is the standard SAC critic almost the best baseline?

AW1 & A1: We thank the reviewer for raising this concern.We thank the reviewer for raising this concern. One possible reason is that we used an update-to-data (UTD) ratio of 1 for fairness following convention in the RL literature, whereas the official DSAC-v1 implementation uses a UTD ratio of 0.05 (one update every 20 environment steps), which may have affected its performance. For ValueFlows, its original focus on offline RL may explain its weaker performance in our online setting. We otherwise followed the official implementations and their recommended configurations.

Weakness 2: The paper's problem is about diffusion critics, but its solution mainly comes from the quantile framework that IQN already have. The paper also shows IQN is free of degradation in Table 1. The new part is diffusion bridge, but it's not clear why this part is necessary.

AW2: We thank the reviewer for raising this question. Although IQN does not suffer from Gaussian degradation, its single-step MLP may still be less effective at capturing complex return distributions than a multi-step Diffusion Bridge. Through iterative refinement, DBC provides greater flexibility in distribution modeling. Moreover, Diffusion Bridge offers a principled framework for distribution-to-distribution transport, where quantile coupling is optimal for scalar distributions under the squared Wasserstein-2 metric, as proven in our newly added Appendix B.5 (Theorem B.6). This advantage is illustrated by the toy example in Appendix B.6 (Figure 5) and further supported by our ablation in Table 5, where five-step DBC consistently outperforms its single-step counterpart.

Diffusion StepsAnt-v5Humanoid-v5Walker2d-v5
15150.45524.75809.8
56481.95877.26162.8

[1]: Xiao, Zhisheng, et al. "Tackling the Generative Learning Trilemma with Denoising Diffusion GANs." International Conference on Learning Representations, 2022.

[2]: Gao, Weiguo, and Ming Li. "Toward Theoretical Insights into Diffusion Trajectory Distillation via Operator Merging." Neural Networks, vol. 202, 2026, article 109023.

Q2: Can you show something similar to fig 2 on real tasks, to show Gaussian degradation actually happen while DBC can avoid it?

A2: Thank you for your comments. We have added to Appendix B.2 the Critic-modeled distributions for Value Diffusion and DBC on HalfCheetah-v5, Walker2d-v5, and Hopper-v5, with standard normalization applied to highlight the shape of the reward distribution. The figures show that Value Diffusion still tends to form a Gaussian-like distribution, whereas on Walker2d-v5 DBC more clearly recovers the shape of the true distribution.

Q3: Why is the setup for each model not unified, such as in Table 10-12?

A3: Thank you for raising this point. The hyperparameters in Tables 10 and 11 are primarily those used in the original paper, while Table 12 lists the hyperparameters used for the Drop-Top aggregation in DBC. We describe this mechanism in Appendix F. Regarding the choice of the Drop-Top rate in DBC, due to time constraints we were unable to conduct more extensive ablation studies. On Gym-MuJoCo tasks, however, a Drop-Top rate of about 10% already yields good results. Only for unstable tasks such as Hopper-v5 does the rate need to be increased (to 25%) to maintain stable training. We additionally tested the drop-rate ablation on Humanoid-v5. Except for drop rate = 0, the other settings are relatively stable.

Drop rateReturn
0%3720.52 ± 868.63
3%5922.13 ± 1099.36
6%5965.76 ± 344.34
9%6187.82 ± 379.87
12%5617.54 ± 221.10

Q4: Why does Table 3 only include three baselines?

A4: We thank the reviewer for this comment. We have added the experimental results for Value Diffusion on DMC, as shown in the table below.

MethodCartpole-BalanceReacher-EasyHumanoid-Run
VD999.38 ± 0.46178.62 ± 153.0931.59 ± 67.12
VF996.50 ± 2.02467.54 ± 285.85113.29 ± 5.97
SAC999.0 ± 0.9871.3 ± 95.977.4 ± 65.4
Ours999.3 ± 0.2972.6 ± 8.0138.9 ± 23.6

Dear Reviewer B1V6,

As the discussion period closes tomorrow, may we kindly check whether our response has addressed your concerns? We would be glad to clarify any remaining points. Thank you for your valuable time.

Thanks for the rebuttal. I appreciate the new fig4 and fig5. But I'm still concerned about the baseline results. In Table 2, most distributional critics underperform standard SAC, sometimes by large margins. Also the new DropTop results make it more difficult to isolate the diffusion bridge's contribution. So I'll maintain the score at 2.

Thank you for your feedback. The baseline experimental setup follows the official implementation. Regarding Drop-Top, we clarify that the 1-step and 5-step DBC ablation in Table 5 uses same settings except for the diffusion steps. In the diffusion-step ablation, the 1-step setting corresponds to IQN with Drop-Top. We believe that the consistent improvements therefore support the contribution of diffusion bridge.

Diffusion StepsAnt-v5Humanoid-v5Walker2d-v5
DBC (1 STEP IQN)5150.45524.75809.8
DBC (5 STEP)6481.95877.26162.8

Comments on the paper

3 comments
~Haque_M_Ishfaq1OpenReview ↗

Comparison with Langevin Soft Actor Critic (LSAC)

Hi authors,

We had a highly related work from ICLR 2025 : Ishfaq, H., Wang, G., Islam, S., & Precup, D. (2025, May). Langevin soft actor-critic: Efficient exploration through uncertainty-driven critic learning. In International Conference on Learning Representations.

In this paper, we consider Langevin Monte Carlo based Thompson Sampling for uncertainty estimation in critic in SAC and DSAC for sample efficient RL with improved exploration. Many of the motivations in your paper aligns with ours. So, we would appreciate if you could consider citing our work as well as use LSAC as a baseline for your experiments.

We open sourced our code as well: https://github.com/hmishfaq/LSAC

Thank you for recommending your work to us. Due to limited resources, we were unable to complete the experiments. However, we have added a citation to LSAC in the revised manuscript and will consider including the experiments in the camera-ready version if feasible.

We sincerely thank all three reviewers for their feedback. B1V6 recognized the identification of a new failure mode for diffusion critics, while 6c6s appreciated the motivation behind it. B1V6 and DZCs also noted DBC's good performance with different actor designs, and the reviewers gave positive feedback on the experimental results. We especially thank DZCs for raising their score and confirming that they had no further questions.

During the rebuttal, we made the following revisions:

  1. Gaussian Degradation and Diffusion Bridge. We have revised the analysis of Gaussian degradation(Section 4.1, Appendix B.1). The new analysis explains both why diffusion critics suffer from Gaussian degradation and why DBC avoids it. We also provided a theoretical foundation (Appendix B.5) for the quantile coupling of the diffusion bridge from an optimal transport perspective.

  2. Additional Experiments. Figure 4 reports return distributions on three MuJoCo tasks: Value Diffusion yields approximately Gaussian return distributions, whereas DBC also captures the non-Gaussian component. On multimodal targets (Figure 5), DBC attains lower fitting error than both IQN and Flow Matching. We also report Value Diffusion results on DMC. We thank B1V6 for the positive comments on these figures.

  3. Paper Revisions. Following DZC's suggestion, we reorganized Sections 3–4: the motivation now precedes the training procedure, and the diffusion bridge construction is presented before the algorithmic details. We updated the captions of Tables 2–3 and Figure 3 to explain the reported statistics.

The manuscript changes are highlighted in blue. We thank the AC and reviewers again for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.