acceptodds
Under review as a conference paper at ICLR 2027

SDRB: A Synthetic Benchmark for Distributional Reinforcement Learning

Abstract

Distributional reinforcement learning (DRL) models complete return distributions, yet standard benchmarks primarily evaluate control performance rather than the fidelity of the learned distribution. We introduce the Synthetic Distributional Reinforcement Learning Benchmark (SDRB), a controlled testbed for directly evaluating return-distribution reconstruction against exact or high-precision ground truth. SDRB isolates six statistical regimes: Gaussian, mixture, heavy-tailed, sparse, distribution drift, and zero-shot out-of-distribution evaluation. We demonstrate the benchmark through a 540-run factorial study crossing three quantile representations (QR-DQN, IQN, and FQF) with three learning objectives (Quantile Huber, MMD, and Sinkhorn divergence). The experiments reveal substantial representation–objective interactions, reconstruction errors in tail and point-mass structure that are not fully captured by global metrics, failure to recover under persistent stale replay after distribution drift, and mean-preserving OOD shifts that substantially increase distributional error while leaving expected-value error essentially unchanged. These results demonstrate how controlled reconstruction evaluation can expose behavior obscured by aggregate control or expected-value metrics. SDRB complements complex control benchmarks by treating return-distribution reconstruction as a distinct evaluation axis.

Primary area:
datasets and benchmarks
Keywords:
distributional reinforcement learning, distribution reconstruction, reinforcement learning, benchmarking, return distributions

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_xzthOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a synthetic benchmark for evaluating return-distribution estimation in DRL. It constructs simple finite-horizon MDPs across six statistical regimes, including Gaussian, mixture, heavy-tail, sparse, distribution drift, and OOD settings. They compare three distributional representations(QR-DQN, IQN, FQF) with three learning objectives(Quantile Huber, MMD and Sinkhorn). It demonstates the interactions between representation and objection.

Critical strengths

This paper addresses a meaningful problem in distributional RL evaluation focusing on the accuracy of learned distributions. The experiments are relatively systematic. The inclusion of regime-specific diagnostics provides additional insights beyond global metrics.

Critical weaknesses
  1. Prior work, such as Théate et al. (2023) and Prasad (2026) have already examined the accuracy of distributional reconstruction and assessed learned return distributions against ground truth. In addition, some of the observations made in this paper—such as the interactions between representation and objective and the shortcomings of expected-value metrics—are already established. The authors should therefore clarify what fundamentally new diagnostic capabilities SDRB offers that go beyond these prior studies.
  2. SDRB is restricted to deterministic chain MDPs with a fixed horizon of 10 and a fixed evaluation policy. In this setting, actions influence reward distributions but not state transitions. Consequently, the benchmark leaves out stochastic transitions, complex state spaces, and policy-dependent dynamics—all of which are important challenges in distributional RL.
  3. The experiments largely rely on fixed reward distributions with predefined parameters and do not systematically examine how reconstruction performance varies with horizon length, mixture separation, rare-event probability, or drift magnitude.
  4. In the Drift setting, persistent stale replay prevents all methods from recovering, making their adaptation capabilities hard to distinguish. Likewise, the OOD experiment applies a mean-preserving variance shift with frozen predictors, so the unchanged expected-value error follows directly from the design. These experiments therefore do not sufficiently show that the benchmark can reveal new or unexpected algorithmic behaviors.
Questions for authors

See the weeknesses.

2 comments

We thank the reviewer for identifying limitations in SDRB's novelty, experimental coverage, and interpretation. We agree that the manuscript should more clearly distinguish its diagnostic contribution from prior reconstruction studies.

Novelty. We do not claim that reference-based reconstruction evaluation, representation–objective interactions, or the limitations of expected-value metrics are individually new. SDRB's intended contribution is a common controlled protocol combining known return distributions, a representation × objective factorial study, regime-specific structural diagnostics, and targeted probability-location controls.

The distinction we aim to emphasize is diagnostic actionability. For example, FQF–MMD exhibits poor reconstruction with adaptive fractions, and fixing fractions improves Gaussian Wasserstein-1 from 1.883 to 0.294 and Mixture from 2.128 to 0.380. This connects an observed failure to a specific optimization-component intervention. We will sharpen the comparison with Théate et al., Prasad, and the other cited reconstruction studies rather than claim novelty for established phenomena.

Environment coverage. We agree that a deterministic fixed-policy chain with H = 10 does not capture stochastic transition branching, complex states, policy-dependent dynamics, or long-horizon error propagation. The present design deliberately isolates reward-distribution reconstruction, but this should be presented as a limited diagnostic setting, not a comprehensive distributional RL benchmark.

We also agree that the existing experiments do not establish robustness across horizon length, mixture separation, rare-event probability, quantile resolution, or drift magnitude. We will explicitly identify these as missing evaluations and avoid generalizing the observed rankings beyond the tested configurations.

Sensitivity and methodological interpretation. We conducted an additional Gaussian Huber sensitivity study using κ ∈ {0.1, 0.5, 1, 2} for QR-DQN and IQN over five matched seeds. Reducing κ from 1 to 0.1 improves mean Wasserstein-1 from 0.808 to 0.360 for QR-DQN and from 0.778 to 0.296 for IQN. This substantially changes the interpretation of the original underdispersion and demonstrates the importance of objective calibration. It does not substitute for the requested horizon or reward-parameter sweeps.

Drift and OOD. We agree that persistent stale replay makes adaptation difficult by construction, while a frozen predictor cannot respond to an unobserved distribution shift. The unchanged expected-value error under the mean-preserving OOD shift is a designed property, not a newly discovered generalization phenomenon.

We will therefore describe these experiments as illustrative diagnostic stress tests. Fresh-replay comparisons, observable or task-conditioned shifts, and variation in shift magnitude would be required to evaluate adaptation mechanisms.

Overall, we will narrow SDRB's claims to controlled reconstruction diagnosis and within-benchmark intervention evidence, while clarifying the additional experimental coverage necessary to establish broader significance.

Thank you for the response. I will maintain my score.

Reviewer_efiLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

SDRB uses a short deterministic chain with controlled reward distributions and reference return laws to evaluate distributional representations and training losses across stationary, sparse-reward, drift, and out-of-distribution settings.

Critical strengths

The use of reference return distributions. It makes it possible to inspect location, dispersion, and structural errors directly, including failures that would be easy to miss if one only looked at aggregate return.

Critical weaknesses
  1. The benchmark’s additional diagnostic value is not sufficiently established. Several findings concern familiar phenomena, such as Huber-related underdispersion and representation–loss interactions. I am still unsure which previously unresolved failure SDRB identifies and what we learn from that diagnosis. Showing differences between methods is a useful starting point, but I would expect a benchmark paper to explain how those differences inform our understanding or method design.

  2. The experiments need a clearer organizing argument. I found it difficult to extract a consistent message from the experiments because they mix effects of representation, optimization, replay, and available information. The probability-location controls are useful, but frozen out-of-distribution predictions and replay-induced lag under drift complicate the interpretation. I would like the paper to make clearer which explanation each experiment is testing and which controls rule out the alternatives.

Questions for authors
  1. What new and useful diagnosis does SDRB enable beyond known representation–loss interactions, and what methodological conclusion does it support?
  2. Which controls separate distributional-learning effects from the frozen predictor, unchanged inputs, and replay mechanism in the out-of-distribution and drift results?
  3. Does any existing result show that an SDRB diagnosis guides a corrective intervention or predicts behavior beyond the tested representation–loss combinations? Such evidence would help clarify the benchmark’s practical value.
2 comments

We thank the reviewer for emphasizing that a diagnostic benchmark should provide actionable explanations rather than simply rank existing methods. We agree that the manuscript does not currently make this distinction sufficiently clear.

Q1. What new diagnostic value does SDRB provide? We do not claim that underdispersion or representation–objective interactions are newly discovered phenomena. SDRB's intended contribution is a controlled protocol that uses reference return distributions and structural diagnostics to identify configuration-dependent reconstruction failures and evaluate targeted corrections.

The clearest example is FQF with permutation-invariant objectives. Its canonical fraction-update rule assumes ordered quantile values, while MMD does not enforce this ordering. SDRB exposes poor reconstruction in this configuration and permits an intervention that changes the fraction-update mechanism without changing the underlying reward regime.

A second example comes from our new Huber sensitivity study. Reducing κ from 1 to 0.1 improves Gaussian Wasserstein-1 from 0.808 to 0.360 for QR-DQN and from 0.778 to 0.296 for IQN, while substantially improving dispersion calibration. This demonstrates that much of the original underdispersion was associated with the chosen smoothing threshold rather than being an intrinsic limitation of the representations.

Q2. Which explanations are isolated? We agree that the current presentation combines mechanisms that should be distinguished. We will reorganize the interpretation into:

  • Stationary reconstruction: performance of complete representation–objective configurations, without claiming that representation and statistical-objective effects are independently identified.
  • Probability-location controls: targeted interventions on FQF's fraction-update mechanism.
  • Huber sensitivity: an intervention on loss calibration within the Gaussian configurations.
  • Drift: reconstruction under deliberately persistent obsolete replay data, not a general test of adaptation.
  • OOD: frozen-predictor sensitivity to a mean-preserving reward-law shift, not adaptation or generalization to an observed change.

In particular, unchanged expected-value error in the OOD experiment follows from the mean-preserving construction, and the drift protocol does not control for fresh or recency-weighted replay. We will present these as illustrative stress tests rather than discoveries of unexpected adaptation failures.

Q3. Does a diagnosis guide an intervention? The fixed-fraction FQF control provides direct within-benchmark intervention evidence. For FQF–MMD, Gaussian Wasserstein-1 improves from 1.883 to 0.294 and Mixture from 2.128 to 0.380 when fraction adaptation is disabled. This supports the practical conclusion that greater representational flexibility can be counterproductive when its optimization components are incompatible.

The Huber study provides an independent calibration intervention. At κ = 0.1, IQN's mean Gaussian Wasserstein-1 is 0.296, compared with 0.345 for MMD on matched seeds, although the difference is small relative to variability. Thus, an apparent objective advantage under the original configuration can change after targeted calibration.

These interventions establish actionable findings within the tested settings, but they do not demonstrate transfer to new algorithms, stochastic transitions, or longer horizons. We will narrow the contribution claim accordingly and make the diagnostic-to-intervention connection central to the revised manuscript.

Thank you for the response. I will maintain my score. I found it hard to justify this work significantly push forward our understanding to the design of RL.

Reviewer_8UWCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SDRB, a synthetic benchmark for evaluating return-distribution reconstruction under a fixed policy in a deterministic finite-horizon chain. A reported 540-run study crosses QR-DQN, IQN, and FQF representations with Quantile Huber, MMD, and Sinkhorn objectives. Using analytical or Monte Carlo reference distributions, the paper examines reconstruction accuracy, tail and point-mass errors, persistent stale replay, and frozen-predictor distribution shifts.

Critical strengths

Evaluating distributional fidelity separately from control performance is a worthwhile objective. The controlled reward laws and independently generated reference distributions make the benchmark interpretable, while ten seeds and a shared training protocol improve comparability. Weighted objectives for FQF, reported gradient-isolation checks, and fixed-fraction controls demonstrate attention to implementation details. The manuscript also appropriately acknowledges that its OOD experiment does not test adaptation and that its drift results concern a deliberately restrictive replay protocol.

Critical weaknesses
  1. The structural metrics are insufficiently specified for unordered outputs. MMD and Sinkhorn do not preserve quantile-level ordering, so tail and zero-plateau errors must use the predicted measure’s inverse CDF rather than raw output indices. Table 8’s QR-DQN–MMD values—W1=0.316W_1=0.316 and zero-plateau MAE 3.6803.680—appear inconsistent under the stated 51-atom interpretation; this requires reconciliation before interpreting the structural findings.

  2. The FQF experiments combine permutation-invariant value losses with an order-dependent fraction-update objective. The authors acknowledge this mismatch, and fixing fractions substantially improves MMD performance. However, the controls do not test a compatible adaptive formulation, such as a monotone value network. Consequently, some reported interactions may reflect this particular hybrid’s incompatibility rather than limitations of adaptive quantile representations.

  3. Objective calibration is asymmetric. MMD and Sinkhorn receive diagnostic tuning, whereas Quantile Huber remains fixed at κ=1\kappa=1. Reproducible underdispersion across seeds does not rule out systematic smoothing bias. A threshold sweep, an unsmoothed quantile-loss control, and comparable validation budgets are needed to strengthen the objective comparisons.

  4. The OOD and drift protocols have limited diagnostic discrimination. A frozen predictor receives no information identifying the changed reward law, while drift retains obsolete samples throughout training. These are useful illustrations, but informative controls—such as task-conditioned prediction and fresh or recency-weighted replay—are needed to distinguish avoidable algorithmic failures from limitations imposed by the protocols.

  5. Coverage remains narrow even for a diagnostic benchmark: one horizon, one quantile budget, and a fixed policy on a non-branching chain. Moreover, the “Heavy-Tail” regime is a finite Gaussian mixture and therefore light-tailed in the usual mathematical sense. Horizon and resolution sweeps, genuine heavy-tailed rewards, and an empirical-return baseline would better separate statistical estimation from bootstrapping and representation errors.

  6. Relevant quantile-based work [1–3] is missing. References [1,2] directly address non-crossing representations, particularly relevant to the ordering issues identified here; [3] studies complete return-distribution estimation for policy evaluation. The authors should cite and discuss these connections and include a representative monotone-quantile baseline, or explain its omission. 15270_SDRB_A_Synthetic_Benchma NeurIPS Proceedings

Relevant references [1] Zhou, F., Wang, J., and Feng, X. (2020). Non-Crossing Quantile Regression for Distributional Reinforcement Learning. Advances in Neural Information Processing Systems, 33. NeurIPS Proceedings [2] Zhou, F., Zhu, Z., Kuang, Q., and Zhang, L. (2021). Non-decreasing Quantile Function Network with Efficient Exploration for Distributional Reinforcement Learning. Proceedings of IJCAI-21, 3455–3461. doi:10.24963/ijcai.2021/476. IJCAI [3] Kuang, Q., Wang, C., Jiao, Y., and Zhou, F. (2026). Distributional Off-Policy Evaluation with Deep Quantile Process Regression. Journal of the American Statistical Association. doi:10.1080/01621459.2026.2671449.

Questions for authors
  1. For QR-DQN–MMD, the zero-return mass is approximately 0.810=0.10740.8^{10}=0.1074. Under a sorted, equally weighted 51-atom interpretation, the five complete bins inside this plateau would contribute approximately (5/51)×3.680=0.361(5/51)\times3.680=0.361 to W1W_1, already exceeding the reported 0.3160.316. How are these two metrics computed, and can you provide the underlying outputs and metric routines?
  2. How does the common 51-point evaluation interface preserve FQF’s learned probability masses? Are all distributional diagnostics invariant to jointly permuting atoms and their weights?
  3. Do the underdispersion and objective rankings persist after varying the Huber threshold and comparing against unsmoothed quantile regression under matched tuning budgets?
  4. Does enforcing monotonicity in the FQF value network resolve the poor MMD results while retaining adaptive fractions?
  5. Can you add fresh-replay controls and compare bootstrapped learning with direct empirical-return estimation across horizons, including H=1H=1, to identify which failures specifically arise from temporal bootstrapping?
1 comment

We thank the reviewer for identifying a genuine inconsistency in the Sparse structural diagnostic. We have corrected the implementation, independently verified the resulting metric, and conducted additional sensitivity analyses.

Q1. Sparse zero-plateau discrepancy. The original zero-plateau MAE compared predicted atoms in raw output order against ordered reference quantiles. This was inappropriate for permutation-invariant objectives such as MMD and Sinkhorn. Wasserstein-1 was computed from the predicted distribution, whereas the plateau diagnostic implicitly assumed that output indices represented ordered quantile positions.

We corrected the diagnostic to sort the predicted equal-weight atoms before comparison and reran all 90 Sparse configurations (three representations × three objectives × ten seeds). We exported the 51 evaluation atoms and independently recomputed the corrected diagnostic across all 900 final-episode state evaluations; the recorded values matched.

For QR-DQN–MMD, the ordering-sensitive plateau MAE in the rerun is 3.672 ± 1.041, while the corrected value is 0.317 ± 0.244. The corresponding rerun Wasserstein-1 is 0.311 ± 0.089. This resolves the inconsistency identified by the reviewer. IQN–MMD's corrected plateau MAE is 0.122 ± 0.100, with Wasserstein-1 of 0.366 ± 0.096.

Q2. FQF native weights and evaluation. The audit now exports FQF's native atom locations, fraction locations, and probability weights separately from its common 51-point evaluation outputs. All 300 final-episode FQF state evaluations had positive native weights summing to approximately one.

The corrected common-grid plateau metric is invariant to permutations of its equally weighted atoms. However, this verification does not establish equivalence between native-weighted and common-grid diagnostics. We will clarify the distinction and avoid claiming that all native FQF structural diagnostics have been independently validated.

Some Sparse reruns also differ in global Wasserstein-1 from the submitted results, particularly FQF–MMD (0.541 originally versus 0.782 in the rerun). The plateau correction itself cannot explain those differences. We investigated the added evaluation code and found no changes to tracked model or PyTorch RNG state during the tested FQF–MMD evaluation. A targeted seed-126 rerun reproduced the original training trajectory and all shared numerical metrics exactly. We will disclose the revised rerun values and distinguish the verified metric correction from the unresolved differences between some archived training trajectories.

Q3. Huber calibration. We conducted a five-seed Gaussian sensitivity study with κ ∈ {0.1, 0.5, 1, 2}. Reducing κ from 1 to 0.1 improves QR-DQN Wasserstein-1 from 0.808 to 0.360 and IQN from 0.778 to 0.296. Their dispersion ratios improve from 0.411 to 0.896 and from 0.416 to 0.926.

Against matched five-seed MMD baselines (0.250 for QR-DQN; 0.345 for IQN), the mean objective ranking reverses for IQN, although the difference is small relative to seed variability. We agree that the original κ = 1 comparison is not calibration-independent. This targeted study does not replace unsmoothed quantile regression or an exhaustive matched-budget hyperparameter search.

Q4. Monotone adaptive FQF. The existing fixed-fraction intervention improves FQF–MMD Gaussian Wasserstein-1 from 1.883 to 0.294 and Mixture from 2.128 to 0.380. It identifies a consequential interaction with the implemented fraction-update mechanism but does not establish that a compatible monotone adaptive formulation would fail. We will discuss the suggested non-crossing and non-decreasing quantile-network work and explicitly identify the missing baseline.

Q5. Replay, direct-return estimation, and coverage. We agree that fresh-replay controls, direct empirical-return fitting, longer horizons, and varying quantile budgets are needed to separate replay, temporal bootstrapping, and representation effects. These controls have not been conducted, so we will restrict the corresponding conclusions.

We also agree that our finite Gaussian-contamination mixture is light-tailed in the strict asymptotic sense. We will replace the term “Heavy-Tail” with terminology such as “Gaussian Contamination” or “Outlier Contamination” and discuss the additional cited literature.

Reviewer_9FQ9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces the Synthetic Distributional Reinforcement Learning Benchmark (SDRB), a controlled environment for evaluating the accuracy of learned return distributions rather than policy performance. The benchmark uses a deterministic finite-horizon chain with stochastic rewards and considers six regimes: Gaussian, Gaussian mixture, heavy-tail contamination, sparse rewards, distribution drift, and out-of-distribution evaluation.

The authors conduct a factorial study comparing three distributional representations (QR-DQN, IQN, and FQF) with three learning objectives (Quantile Huber, MMD, and Sinkhorn divergence), across six regimes and ten seeds, totaling 540 runs. Reconstruction is evaluated against analytical or Monte Carlo reference distributions using Wasserstein-1 distance and additional structural diagnostics.

The results suggest that reconstruction quality depends on the representation–objective combination. MMD generally performs well with QR-DQN, while Quantile Huber exhibits underdispersion in several configurations. The authors also report difficulties in reconstructing rare-event and point-mass structure, failure to recover from distribution drift under persistent stale replay, and substantial distributional error under mean-preserving OOD shifts.

Critical strengths
  1. A relevant motivation for evaluating distributional reconstruction independently of policy performance.

The paper addresses a meaningful limitation of conventional RL evaluation: good expected returns do not necessarily imply accurate estimates of the full return distribution. The use of controlled reward distributions with analytical or high-precision reference targets enables direct examination of reconstruction errors, including dispersion, tails, and discrete probability mass. This provides a potentially useful diagnostic complement to performance-oriented RL benchmarks.

  1. A systematic comparison of established representations and learning objectives.

The factorial design provides a common experimental protocol for comparing QR-DQN, IQN, and FQF under different learning objectives. The additional structural diagnostics and probability-location controls are useful, particularly for identifying potential incompatibilities between FQF's adaptive fraction updates and permutation-invariant objectives. The authors also clearly acknowledge several limitations of the benchmark's scope.

Critical weaknesses
  1. Limited novelty and insufficient coverage of important distributional RL challenges.

The contribution beyond existing distributional RL diagnostic studies is insufficiently established. Bellemare et al. (ICML 2017) and Rowland et al. (ICML 2019) already investigate distributional reconstruction and Bellman approximation errors, while Nguyen-Tang et al. (AAAI 2021) evaluate MMD-based distributional learning on stochastic chains with varying lengths and delayed rewards. Théate et al. (2023) and Sun et al. (NeurIPS 2024) also examine interactions between distributional representations and probability metrics.

More importantly, the proposed benchmark uses only a deterministic chain with H=10H=10, where actions affect reward distributions but not state transitions, and evaluation follows a fixed policy. This creates several important limitations:

  • No stochastic transition branching: All uncertainty comes from random rewards along a predetermined trajectory. The benchmark does not examine multimodal returns arising from different successor states, stochastic paths, or uncertain goal-reaching outcomes, which are central to distributional RL.
  • Very short horizon: With only ten Bellman steps, the benchmark cannot adequately examine long-horizon distributional error propagation or delayed reward learning. The restriction to H=10H=10 is not sufficiently justified, particularly since Gaussian return distributions remain analytically tractable for arbitrary horizons.
  • Limited sparse-reward setting: The Sparse regime generates an immediate nonzero reward independently at each step with probability 0.2. This is substantially different from long-horizon goal-conditioned tasks, where rewards occur only after reaching a distant goal and successful trajectories may be extremely rare. Such problems can also be constructed using stochastic chains with exactly computable hitting-time and return distributions.

Therefore, the benchmark could cover substantially more challenging and representative distributional RL problems without sacrificing its main advantage of known ground truth. Given that related stochastic-chain and reconstruction experiments have already appeared in prior literature, the scientific novelty and significance of SDRB remain insufficiently demonstrated.

  1. Limited scientific insight from the OOD and distribution-drift experiments.

Several highlighted findings largely follow from the experimental construction. In the OOD experiment, the predictor is frozen while the reference distribution changes. Under a mean-preserving variance shift, the unchanged expected-value error is guaranteed by construction, rather than demonstrating a new generalization phenomenon. Similarly, the drift experiment deliberately retains all obsolete pre-drift transitions in the replay buffer, making poor adaptation largely expected. Although the authors acknowledge these limitations, the experiments provide limited new understanding of distributional generalization or adaptation. More informative comparisons would involve observable distribution shifts, adaptation mechanisms, or alternative replay strategies.

  1. Methodological concerns about the validity of representation–objective comparisons.

The compared learning objectives do not necessarily estimate the same population return distribution under sampled Bellman updates. In particular, minimizing the expected Sinkhorn discrepancy against individual sampled Bellman-target distributions generally differs from minimizing the discrepancy against the full marginal Bellman target distribution. The analogous sampled Wasserstein-1 objective can even favor a collapsed distribution rather than the true return distribution, whereas squared MMD with a fixed kernel has a different consistency property. Consequently, the reported advantage of MMD may partly reflect intrinsic statistical objective bias rather than the claimed representation–objective interactions.

Furthermore, FQF retains its canonical order-dependent fraction update when combined with permutation-invariant MMD and Sinkhorn objectives. Appendix Table 11 shows that fixing FQF fractions reduces Gaussian MMD reconstruction error from 1.883 to 0.294, with similar improvements in other regimes. This suggests that a substantial part of the reported interaction may arise from incompatible optimization components. Quantile Huber also uses a fixed smoothing parameter without sufficient sensitivity analysis despite known Huber-related distributional bias. Comparisons between oracle distribution fitting and sampled Bellman learning would help distinguish these effects.

Questions for authors

Q1. Benchmark novelty, scope, and missing distributional RL challenges.

The paper motivates SDRB as a benchmark for diagnosing return-distribution reconstruction, but several closely related settings have already been studied. For example, Bellemare et al. (2017) evaluate distributional reconstruction against Monte Carlo ground truth, Rowland et al. (2019) examine statistical approximation and distributional Bellman updates, and Nguyen-Tang et al. (2021) study stochastic chains with varying lengths K=1,…,15K=1,\ldots,15 and delayed rewards.

Could the authors clarify what new diagnostic capabilities SDRB provides beyond these existing studies? In particular:

  • Why are all primary experiments restricted to H=10H=10? Since Gaussian return distributions remain analytically tractable for arbitrary horizons, the benchmark could examine how reconstruction error changes with Bellman depth without sacrificing exact ground truth. Are the observed objective rankings stable for substantially longer horizons?
  • The current chain has deterministic transitions, so all return uncertainty originates from stochastic rewards. How would the benchmark diagnose errors arising from stochastic successor states, alternative future trajectories, or multimodal hitting-time distributions?
  • The Sparse regime generates an independent nonzero reward with probability 0.2 at each step, which differs substantially from delayed goal-reaching tasks where successful trajectories may be extremely rare. Stochastic goal-reaching chains can also have analytically computable hitting-time and return distributions. Why are such settings excluded?

These questions are important because the proposed benchmark should offer meaningful diagnostic coverage beyond existing distribution-fitting experiments. Clarification of the benchmark's distinctive capabilities, and any available evidence about horizon sensitivity, would materially affect my assessment of its novelty and significance.

Q2. Statistical consistency of learning objectives under sampled Bellman targets.

A central concern is whether the compared objectives have the same population-optimal solution under the sampled Bellman training protocol. The benchmark aims to recover the marginal return distribution, but Sinkhorn is applied to individual sampled Bellman-target distributions. In general, these define different population objectives, where QR,S′Q_{R,S'} denotes the conditional Bellman-target distribution associated with a sampled transition.

This distinction can introduce intrinsic reconstruction bias even with infinite data and perfect optimization. For example, with terminal rewards R∼0.8δ0+0.2δ1R\sim 0.8\delta_0+0.2\delta_1, minimizing the expected sampled Wasserstein-1 loss favors the collapsed prediction δ0\delta_0 rather than the true return distribution. In contrast, squared MMD with a fixed kernel has a population consistency property because of the linearity of kernel mean embeddings, although this property requires additional care when the kernel is adapted to individual targets.

Could the authors clarify:

  • Whether the compared objectives share the true return distribution as their population optimum under the implemented sampled Bellman updates, particularly for Sinkhorn?
  • Whether the reported superiority of MMD could partly reflect differences in intrinsic objective bias rather than distributional representation quality?
  • Whether an oracle-fitting control, comparing direct fitting of the known return distribution with sampled Bellman learning, could separate objective bias from representation and temporal bootstrapping errors?

This distinction seems particularly important because the paper's main conclusion concerns representation–objective interactions, yet the experiment does not clearly separate differences in statistical targets from differences in optimization or approximation.

Q3. Interpretation of representation–objective interactions and baseline calibration.

The reported interaction between representations and objectives may be substantially influenced by the compatibility of their optimization components. In particular, FQF retains its canonical order-dependent fraction-update rule when its value network is trained using permutation-invariant MMD or Sinkhorn objectives. These objectives do not necessarily preserve the ordered quantile interpretation assumed by the fraction update.

Appendix Table 11 provides notable evidence: fixing FQF fractions reduces MMD reconstruction error from 1.883 to 0.294 on Gaussian and from 2.128 to 0.380 on Mixture. The fixed-fraction results also become close to QR-DQN-MMD. This raises the question of whether the reported ranking reversals primarily reflect incompatibility between the fraction-update rule and value objective, rather than a fundamental interaction between representation capacity and probability metrics.

Could the authors clarify how much of the central representation–objective interaction remains after controlling for adaptive fraction updates? Additionally, FQF is trained using its learned probability masses but evaluated through a common uniform quantile grid. Could this mismatch between the native representation and evaluation interface contribute to the observed errors?

Finally, Quantile Huber uses a fixed κ=1\kappa=1, while the Gaussian reward standard deviation is 0.5. Given known Huber-related distributional bias, how sensitive are the reported reconstruction errors and objective rankings to κ\kappa? A small sensitivity analysis or comparison with unsmoothed quantile regression would help establish whether the observed underdispersion is intrinsic to the representation or primarily caused by the selected loss configuration.

1 comment

We thank the reviewer for the detailed assessment. We agree that SDRB's contribution should be framed as a controlled diagnostic protocol, rather than as a new distributional RL algorithm or a comprehensive evaluation of distributional RL.

Q1. Novelty and scope. SDRB combines known reference return distributions, a representation × objective factorial design, structural reconstruction diagnostics, and probability-location interventions within a common experimental protocol. Its intended value is to connect reconstruction failures to specific properties of the learning configuration, rather than merely rank methods by aggregate error.

We acknowledge that deterministic transitions, a fixed policy, and horizon H = 10 limit the scope of the findings. In particular, the Sparse regime models independent rare rewards rather than delayed goal-reaching events. We will clarify these distinctions relative to Bellemare et al., Rowland et al., Nguyen-Tang et al., Théate et al., and Sun et al. We have not established stability across longer horizons or stochastic branching and will not claim such generality.

Q2. Population-objective consistency. We agree that minimizing expected discrepancies against conditional sampled Bellman targets need not recover the marginal return distribution. For fixed-kernel squared MMD, the sampled and marginal population objectives differ by a prediction-independent constant. This guarantee does not automatically extend to our minibatch-adaptive MMD bandwidth, and no analogous general equivalence is guaranteed for Sinkhorn divergence.

We conducted a deterministic one-step audit using our implemented objective routines. For equally likely terminal rewards of −1 and +1, the fixed-regularization Sinkhorn loss assigns approximately equal expected sampled-target discrepancies (0.99896) to the correct distribution and a prediction with half its spread, although their marginal reconstruction discrepancies are 0 and 0.5. This illustrates weak discrimination under the sampled objective, not proof that the partially dispersed prediction is a global optimum.

We will therefore distinguish statistical-objective alignment from representation and optimization effects. The factorial results characterize complete learning configurations; they do not isolate representation capacity or establish a calibration-independent superiority of MMD. An oracle-fitting versus sampled-Bellman control remains necessary to quantify the contribution of objective bias.

Q3. FQF compatibility and Quantile Huber calibration. We agree that FQF's order-dependent fraction update may be incompatible with permutation-invariant value objectives. Our fixed-fraction intervention improves FQF–MMD Gaussian Wasserstein-1 from 1.883 to 0.294 and Mixture from 2.128 to 0.380. This demonstrates a consequential optimization-component interaction, not an intrinsic limitation of adaptive fractions. We will distinguish native FQF probability masses from the common-grid evaluation interface and acknowledge the absence of a compatible monotone adaptive-fraction baseline.

We also conducted a Gaussian Huber sensitivity study over κ ∈ {0.1, 0.5, 1, 2}, using five matched seeds for QR-DQN and IQN. Reducing κ from 1 to 0.1 improves mean Wasserstein-1 from 0.808 to 0.360 for QR-DQN and from 0.778 to 0.296 for IQN. Predicted-to-reference standard-deviation ratios improve from 0.411 to 0.896 and from 0.416 to 0.926.

On the same five seeds, MMD achieves Wasserstein-1 of 0.250 for QR-DQN and 0.345 for IQN. Thus, calibrated Huber has a slightly lower mean error than MMD for IQN, although the difference is small relative to seed variability. We will revise our conclusions to make clear that the original κ = 1 underdispersion and objective rankings depend substantially on loss calibration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.