acceptodds
Withdrawn from ICLR 2027

Offline-to-Online Distributional Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Distributional reinforcement learning (DRL) has demonstrated strong performance and improved training stability by modeling the full distribution of returns. Despite this progress, we observe that in the offline-to-online setting, DRL suffers severe performance degradation at the onset of online finetuning. We trace this instability to quantile penalization during offline training, which may violate the intrinsic monotonic ordering of quantile values and induce substantially biased value estimates. To address this issue, we propose a monotonic quantile positional embedding and a monotonic quantile value network, which provably preserve quantile ordering under offline penalization. By enforcing this structural property, our method mitigates quantile estimation bias and stabilizes policy transfer from offline pretraining to online fine-tuning. We further introduce a quantile-guided exploration strategy that leverages the learned return distribution as an uncertainty signal to facilitate efficient online exploration. Experiments on the D4RL benchmark demonstrate that our method substantially reduces performance degradation at the onset of online interaction and consistently achieves superior final performance compared with existing offline-to-online reinforcement learning methods.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, offline-to-online, distributional, quantile, exploration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 5 reviews, above 27% of the others.
Reviewer_H78hOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies early fine-tuning degradation in O2O RL from the perspective of Distributional RL, arguing that offline conservative quantile penalization can disrupt quantile ordering and introduce value bias. The authors propose COOL-DRL, which enforces monotonicity of the quantile function through Monotonic Quantile Positional Embedding (MQPE) and Monotonic Quantile Value Network (MQVN), and further uses quantile dispersion to construct an online exploration policy. Experiments cover D4RL MuJoCo, AntMaze, and several Adroit tasks.

Critical strengths
  • Studying O2O RL stability from the perspective of distributional value structure has some novelty. Compared with prior work that mainly focuses on scalar Q-value bias, this paper examines how conservative penalties affect quantile crossing, and the problem is clearly defined.

  • The experimental evaluation is relatively comprehensive and includes key ablations.

Critical weaknesses
  • Interpreting quantile dispersion directly as "uncertainty" and using it for UCB exploration is conceptually questionable. In Distributional RL, quantile spread primarily reflects the randomness of the return distribution itself, i.e., aleatoric variability, and is not equivalent to epistemic uncertainty about Q(s,a)Q(s,a).

  • Theorem 3.1 explicitly establishes ∂Fθ/∂τ≥0\partial F_\theta/\partial\tau \ge 0, while the network uses clipped ReLU, whose derivative can be zero. Therefore, the result only guarantees a nondecreasing function, not a strictly increasing one. However, the paper repeatedly claims in the experiments and conclusion that COOL-DRL provides "theoretically guaranteed strict monotonicity."

  • The main experiments compare only against IQL, CalQL, SO2, and WSRL. However, the paper itself discusses SUF, PEX, 2025 online pre-training methods, and diffusion-generation O2O methods, but does not include them in the experiments. In particular, SUF also focuses on early fine-tuning stability and is therefore a direct baseline.

  • There are still many basic editing issues in the main text and references. For example, the Figure 8 caption contains "confirming the its crucial role," and the main text contains "resepectively." Capitalization is also inconsistent in the references, including "Suf," "Cal-ql," "offline rl," "MIT press," and "Cambridge university press." These are not core technical issues, but the manuscript requires systematic proofreading for an ICLR submission.

Questions for authors

See Weaknesses.

Reviewer_b2gfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies performance drops when distributional RL agents move from offline training to online fine-tuning. The authors investigate this instability to conservative penalties that disrupt quantile ordering and distort value estimates. They propose COOL-DRL, which uses a monotonic quantile embedding and network to preserve this ordering, together with a quantile-based exploration strategy for online learning. Experiments on D4RL tasks show improved transfer stability and competitive final performance. Ablation studies examine the contributions of the monotonic architecture and exploration strategy.

Critical strengths
  1. The paper identifies a practical problem: distributional RL agents can suffer a sharp performance drop when moving from offline training to online fine-tuning. The proposed architecture guarantees nondecreasing quantile estimates, and the ablations support its role in improving transfer stability. This is useful when online adaptation must preserve the performance of an existing policy.

  2. The empirical results are promising, particularly on MuJoCo tasks. The paper includes component ablations, comparison with a non-crossing quantile method, and an equal-computation-budget evaluation. These experiments provide meaningful evidence that the proposed approach can improve stability and final performance.

Critical weaknesses

My main concern is that the paper does not sufficiently establish why enforcing quantile monotonicity reduces value bias and stabilizes fine-tuning. Monotonicity constrains the ordering of quantiles over τ\tau, but not their magnitude. A monotone curve can still severely underestimate returns, and it is unclear why the penalty in Eq. (2) would cause less underestimation with a monotone architecture.

The current results (Fig. 2, Table 6, and Sec. 4.6.1) rely on learned reference critics, which may also be biased and may evaluate different policies. A more direct test would compare predicted Q-values with Monte Carlo returns obtained by taking the same initial action at the same state and then following the corresponding offline-trained policy. Reporting the 1-Wasserstein distance between the predicted distribution and empirical rollout returns would better support the claimed bias reduction.

Questions for authors

Please address above concern in the Critical Weaknesses section.

Besides, I have following questions:

  1. Why should enforcing quantile monotonicity reduce underestimation under Eq. (2)? Can authors provide direct evidence by measuring mean estimation error and 1-Wasserstein distance against Monte Carlo returns under the corresponding offline-trained policy?

  2. Are the penalty settings, entropy backup, and learning rate described in Appendix D.2 held fixed across the ablations? Please clarify whether these comparisons isolate the effect of replacing the standard network with MQVN.

  3. How does Theorem 3.3 account for the approximation bias of the fixed quantile formulas? For example, Eq. (11) gives approximately 1.0224 for exact standard normal quantiles, although the true standard deviation is 1. How is this consistent with the claimed �0��0� error?

Reviewer_P2GROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies offline-to-online distributional RL and proposes COOL-DRL to improve training stability during online fine-tuning. The method uses a monotonic quantile network to prevent quantile crossing and a quantile-based exploration strategy. Experiments on D4RL show improved stability and competitive performance over existing methods.

Critical strengths
  1. The paper identifies quantile crossing in offline distributional RL and shows its potential impact on online fine-tuning.
  2. The experimental results are generally strong. The ablation studies also show that the monotonic network helps stabilize training and the exploration strategy improves final performance.
Critical weaknesses
  1. Why distributional RL is particularly needed for offline-to-online learning, as well as what advantages it offers over standard Q-value-based methods, are not well explained and motivated. In addition, monotonic quantile networks have been studied previously, and the differences from existing approaches need to be better explained.
  2. The variance of the return distribution does not reflect uncertainty due to limited data. In addition, Theorem 3.3 does not account for the approximation bias introduced by estimating moments from a fixed set of quantiles.
Questions for authors
  1. Why is distributional RL necessary for offline-to-online learning? What makes it more suitable than conventional Q-value learning?
  2. What are the main technical differences between the proposed monotonic quantile network and existing monotonic quantile methods?
  3. How does return variance reflect the uncertainty needed for exploration? How is the approximation bias of the quantile-based estimators handled in Theorem 3.3?
Reviewer_HYrfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies instability during the transition from offline training to online fine-tuning in distributional reinforcement learning. The authors observe that CODAC, an offline method using conservative quantile penalties, can suffer substantial performance degradation when online fine-tuning begins, and associate this phenomenon with quantile crossing and biased value estimates. They propose COOL-DRL, which combines a monotonic quantile positional embedding (MQPE) with a monotonic quantile value network (MQVN) to structurally guarantee that quantile estimates are nondecreasing with respect to the quantile fraction. The method also uses predicted quantiles to estimate the mean and dispersion of the return distribution and guide online exploration.

The evaluation covers MuJoCo and AntMaze tasks from D4RL, with additional experiments on expert datasets and Adroit tasks. The results show improved transfer stability and final performance on multiple tasks. The paper also includes ablations of the monotonic network, positional embedding, and exploration strategy, as well as a comparison with CQR. However, the causal relationship between improved monotonicity, reduced value bias, and better online performance, along with the specific advantages of the proposed architecture over existing monotonic quantile methods, requires further clarification.

Critical strengths

S1. The paper addresses a practically relevant problem. It investigates performance degradation when distributional RL transitions from offline pretraining to online fine-tuning and seeks to explain this issue through structural properties of the quantile representation. MQPE and MQVN provide an analytically supported monotonicity guarantee, enforcing quantile ordering through the network architecture. This offers a useful perspective on the relationship between offline regularization and online transfer.

S2. The evaluation is relatively comprehensive and provides initial evidence separating the roles of different components. Beyond multiple MuJoCo and AntMaze tasks, the paper includes evaluations on additional datasets, computation-budget comparisons, and hyperparameter sensitivity analyses. The ablations in Fig. 8, Fig. 9, and Appendix I.1 support the distinct contributions of the monotonic network to transfer stability and the exploration strategy to final performance. These results demonstrate practical value, although the underlying mechanism and advantages over existing architectures require further validation.

Critical weaknesses

W1. The central causal explanation is insufficiently validated, and the conservative penalty settings introduce a potential confound. The paper attributes early online performance degradation to quantile crossing and value estimation bias induced by conservative penalties. Although the ablations in Fig. 8 and Appendix I.1 support the contribution of the monotonic network to transfer stability, they do not fully establish the causal chain from quantile crossing to value bias and subsequent performance degradation. Moreover, Appendix D.2 states that COOL-DRL penalizes only quantiles with tau > 0.5 on some tasks, so its improvement over CODAC may partly result from this change in the penalty mechanism. I recommend a controlled comparison crossing standard versus monotonic networks with full versus selective quantile penalization under otherwise identical training settings, reporting both value estimation error and transfer performance to isolate their contributions.

W2. The technical contribution and architectural advantages over existing monotonic quantile networks remain unclear. Existing approaches such as UMNN and Monotonic Splines can represent continuous monotonic quantile functions, and the main components of MQVN draw on SMNN. Although Appendix J compares against CQR and Appendix I.1 ablates MQPE against the IQN embedding, the former does not cover other continuous, globally monotonic representations, while the latter also changes whether the complete network retains its monotonicity guarantee. Consequently, these results do not clearly establish the advantages of the specific MQPE/MQVN design over other order-preserving architectures. I recommend clarifying the technical contributions beyond prior work and comparing against an existing globally monotonic network and a simple order-preserving architecture that takes tau directly as input, under the same training protocol.

Questions for authors
  1. Do the standard-network ablations in Fig. 8 and Table 9 use exactly the same selective quantile penalization, learning rates, and entropy backup settings as the full model? In a controlled comparison crossing network monotonicity with selective penalization, how much does each factor contribute to transfer stability?

2.Can the authors evaluate value estimation error using independent Monte Carlo returns or empirical quantiles under a fixed policy and on the same state-action pairs, while tracking quantile crossing and online performance degradation, to strengthen the evidence for the proposed causal mechanism?

  1. Under the same training protocol, what specific advantages do MQPE/MQVN offer over a monotonic network taking tau directly as input, or continuous monotonic quantile representations based on UMNN or monotonic splines? Which advantages arise from the proposed design beyond monotonicity itself?
Reviewer_sr9HOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes COOL-DRL to improve the stability of distributional RL during offline-to-online transfer. It combines a monotonic quantile positional embedding, a monotonic critic architecture, and exploration based on quantile-derived return moments. The authors argue that conservative offline penalties disrupt quantile ordering and destabilize subsequent finetuning. Experiments on D4RL tasks report improved transfer stability and competitive final performance, supported by architectural ablations and additional evaluations in the appendix.

Critical strengths

The paper addresses a practically relevant failure mode in offline-to-online learning and provides an explicit architectural guarantee of nondecreasing quantile predictions. The empirical evaluation is substantially developed: the main MuJoCo comparison reports five seeds, and the appendix includes component ablations, a CQR comparison, additional AntMaze and Adroit tasks, sensitivity analyses, and an equal-computation-budget comparison. These experiments provide encouraging evidence that the overall method can improve transfer stability and performance, even though the proposed explanations require stronger validation.

Critical weaknesses
  1. Theorem 3.3 is false as stated. Substituting exact standard-normal quantiles into Eq. (11) gives approximately 1.02238, not the true standard deviation of 1. Thus, the estimator retains nonvanishing approximation bias even with infinitely many observations. Appendix C assumes exact moment recovery without establishing it for the stated distribution families; root-NN quantile estimation alone does not imply root-NN moment consistency.

  2. Theorem 3.2 does not establish calibration of the learned critic. It bounds differences between empirical and population Bellman targets under a realizability assumption, without accounting for conservative penalization, approximation error, or optimization error. Monotonicity alone cannot prevent a systematically shifted quantile function. The claimed theoretical connection from the architecture to reduced estimation bias and stable transfer therefore remains incomplete.

  3. The exploration bonus measures return dispersion, not uncertainty about the estimated mean. This dispersion can remain large even when the return distribution is perfectly known, so μ^+σ^\widehat\mu+\widehat\sigma is not a statistical upper confidence bound. Figure 4 likewise displays return-spread intervals rather than confidence intervals. Matched-candidate controls using mean-only selection and unweighted Gaussian perturbations would clarify the bonus’s actual contribution.

  4. The CODAC comparison changes more than architecture and exploration. Appendix D.2 additionally removes entropy backup and restricts offline penalties to upper-half quantiles on selected tasks, both of which directly alter pessimism. Although architectural ablations are provided, the manuscript does not clearly isolate these changes. The attribution of improved transfer specifically to eliminating quantile crossing remains confounded.

  5. The value-bias diagnostics lack independent, policy-matched ground truth. Figures 2 and 11 use an online SAC critic or the best-performing finetuned critic as reference, although these may evaluate different continuation policies. Moreover, rank correlation cannot establish value or distributional calibration. Rollout-based estimates under explicitly matched policies are needed to substantiate the claimed bias reduction.

  6. Novelty positioning remains incomplete. Reference [1] is cited, but Appendix J does not establish its claimed query-batch inconsistency for the original fixed-head formulation. Missing reference [2] already constructs a continuous monotone quantile function and uses distribution-based exploration, making it an essential comparator. Reference [3] is also relevant to continuous quantile estimation, although its primary objective is OPE rather than offline-to-online control.

Relevant references

[1] Zhou, F., Wang, J., and Feng, X. (2020). Non-Crossing Quantile Regression for Distributional Reinforcement Learning. Advances in Neural Information Processing Systems, 33, 15909–15919. NeurIPS Proceedings

[2] Zhou, F., Zhu, Z., Kuang, Q., and Zhang, L. (2021). Non-decreasing Quantile Function Network with Efficient Exploration for Distributional Reinforcement Learning. Proceedings of IJCAI-21, 3455–3461. doi:10.24963/ijcai.2021/476. IJCAI

[3] Kuang, Q., Wang, C., Jiao, Y., and Zhou, F. (2026). Distributional Off-Policy Evaluation with Deep Quantile Process Regression. Journal of the American Statistical Association. doi:10.1080/01621459.2026.2671449.

Questions for authors

Questions for Authors

  1. How do the authors reconcile Theorem 3.3 with the standard-normal counterexample? Can the result separate fixed approximation bias, empirical-quantile sampling error, and neural-critic estimation error?
  2. Can the authors isolate architecture, upper-half penalization, and entropy-backup changes in a matched experiment? Which settings are shared by CODAC, the MLP ablation, and COOL-DRL?
  3. Does the dispersion bonus outperform mean-only candidate selection and Gaussian exploration using identical candidate actions and computation? Does it distinguish reducible estimation uncertainty from irreducible return variability?
  4. Can the authors report policy-matched Monte Carlo value and quantile errors, rather than relying on another learned critic and rank correlation?
  5. How does MQVN compare with NDQFN [2] under the same offline-to-online protocol? What evidence supports Appendix J’s batch-dependence criticism of the fixed-head method in [1]?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.