Q1. Benchmark novelty, scope, and missing distributional RL challenges.
The paper motivates SDRB as a benchmark for diagnosing return-distribution reconstruction, but several closely related settings have already been studied. For example, Bellemare et al. (2017) evaluate distributional reconstruction against Monte Carlo ground truth, Rowland et al. (2019) examine statistical approximation and distributional Bellman updates, and Nguyen-Tang et al. (2021) study stochastic chains with varying lengths K=1,…,15 and delayed rewards.
Could the authors clarify what new diagnostic capabilities SDRB provides beyond these existing studies? In particular:
- Why are all primary experiments restricted to H=10? Since Gaussian return distributions remain analytically tractable for arbitrary horizons, the benchmark could examine how reconstruction error changes with Bellman depth without sacrificing exact ground truth. Are the observed objective rankings stable for substantially longer horizons?
- The current chain has deterministic transitions, so all return uncertainty originates from stochastic rewards. How would the benchmark diagnose errors arising from stochastic successor states, alternative future trajectories, or multimodal hitting-time distributions?
- The Sparse regime generates an independent nonzero reward with probability 0.2 at each step, which differs substantially from delayed goal-reaching tasks where successful trajectories may be extremely rare. Stochastic goal-reaching chains can also have analytically computable hitting-time and return distributions. Why are such settings excluded?
These questions are important because the proposed benchmark should offer meaningful diagnostic coverage beyond existing distribution-fitting experiments. Clarification of the benchmark's distinctive capabilities, and any available evidence about horizon sensitivity, would materially affect my assessment of its novelty and significance.
Q2. Statistical consistency of learning objectives under sampled Bellman targets.
A central concern is whether the compared objectives have the same population-optimal solution under the sampled Bellman training protocol. The benchmark aims to recover the marginal return distribution, but Sinkhorn is applied to individual sampled Bellman-target distributions. In general, these define different population objectives, where QR,S′ denotes the conditional Bellman-target distribution associated with a sampled transition.
This distinction can introduce intrinsic reconstruction bias even with infinite data and perfect optimization. For example, with terminal rewards R∼0.8δ0+0.2δ1, minimizing the expected sampled Wasserstein-1 loss favors the collapsed prediction δ0 rather than the true return distribution. In contrast, squared MMD with a fixed kernel has a population consistency property because of the linearity of kernel mean embeddings, although this property requires additional care when the kernel is adapted to individual targets.
Could the authors clarify:
- Whether the compared objectives share the true return distribution as their population optimum under the implemented sampled Bellman updates, particularly for Sinkhorn?
- Whether the reported superiority of MMD could partly reflect differences in intrinsic objective bias rather than distributional representation quality?
- Whether an oracle-fitting control, comparing direct fitting of the known return distribution with sampled Bellman learning, could separate objective bias from representation and temporal bootstrapping errors?
This distinction seems particularly important because the paper's main conclusion concerns representation–objective interactions, yet the experiment does not clearly separate differences in statistical targets from differences in optimization or approximation.
Q3. Interpretation of representation–objective interactions and baseline calibration.
The reported interaction between representations and objectives may be substantially influenced by the compatibility of their optimization components. In particular, FQF retains its canonical order-dependent fraction-update rule when its value network is trained using permutation-invariant MMD or Sinkhorn objectives. These objectives do not necessarily preserve the ordered quantile interpretation assumed by the fraction update.
Appendix Table 11 provides notable evidence: fixing FQF fractions reduces MMD reconstruction error from 1.883 to 0.294 on Gaussian and from 2.128 to 0.380 on Mixture. The fixed-fraction results also become close to QR-DQN-MMD. This raises the question of whether the reported ranking reversals primarily reflect incompatibility between the fraction-update rule and value objective, rather than a fundamental interaction between representation capacity and probability metrics.
Could the authors clarify how much of the central representation–objective interaction remains after controlling for adaptive fraction updates? Additionally, FQF is trained using its learned probability masses but evaluated through a common uniform quantile grid. Could this mismatch between the native representation and evaluation interface contribute to the observed errors?
Finally, Quantile Huber uses a fixed κ=1, while the Gaussian reward standard deviation is 0.5. Given known Huber-related distributional bias, how sensitive are the reported reconstruction errors and objective rankings to κ? A small sensitivity analysis or comparison with unsmoothed quantile regression would help establish whether the observed underdispersion is intrinsic to the representation or primarily caused by the selected loss configuration.
We thank the reviewer for identifying limitations in SDRB's novelty, experimental coverage, and interpretation. We agree that the manuscript should more clearly distinguish its diagnostic contribution from prior reconstruction studies.
Novelty. We do not claim that reference-based reconstruction evaluation, representation–objective interactions, or the limitations of expected-value metrics are individually new. SDRB's intended contribution is a common controlled protocol combining known return distributions, a representation × objective factorial study, regime-specific structural diagnostics, and targeted probability-location controls.
The distinction we aim to emphasize is diagnostic actionability. For example, FQF–MMD exhibits poor reconstruction with adaptive fractions, and fixing fractions improves Gaussian Wasserstein-1 from 1.883 to 0.294 and Mixture from 2.128 to 0.380. This connects an observed failure to a specific optimization-component intervention. We will sharpen the comparison with Théate et al., Prasad, and the other cited reconstruction studies rather than claim novelty for established phenomena.
Environment coverage. We agree that a deterministic fixed-policy chain with H = 10 does not capture stochastic transition branching, complex states, policy-dependent dynamics, or long-horizon error propagation. The present design deliberately isolates reward-distribution reconstruction, but this should be presented as a limited diagnostic setting, not a comprehensive distributional RL benchmark.
We also agree that the existing experiments do not establish robustness across horizon length, mixture separation, rare-event probability, quantile resolution, or drift magnitude. We will explicitly identify these as missing evaluations and avoid generalizing the observed rankings beyond the tested configurations.
Sensitivity and methodological interpretation. We conducted an additional Gaussian Huber sensitivity study using κ ∈ {0.1, 0.5, 1, 2} for QR-DQN and IQN over five matched seeds. Reducing κ from 1 to 0.1 improves mean Wasserstein-1 from 0.808 to 0.360 for QR-DQN and from 0.778 to 0.296 for IQN. This substantially changes the interpretation of the original underdispersion and demonstrates the importance of objective calibration. It does not substitute for the requested horizon or reward-parameter sweeps.
Drift and OOD. We agree that persistent stale replay makes adaptation difficult by construction, while a frozen predictor cannot respond to an unobserved distribution shift. The unchanged expected-value error under the mean-preserving OOD shift is a designed property, not a newly discovered generalization phenomenon.
We will therefore describe these experiments as illustrative diagnostic stress tests. Fresh-replay comparisons, observable or task-conditioned shifts, and variation in shift magnitude would be required to evaluate adaptation mechanisms.
Overall, we will narrow SDRB's claims to controlled reconstruction diagnosis and within-benchmark intervention evidence, while clarifying the additional experimental coverage necessary to establish broader significance.