- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies early fine-tuning degradation in O2O RL from the perspective of Distributional RL, arguing that offline conservative quantile penalization can disrupt quantile ordering and introduce value bias. The authors propose COOL-DRL, which enforces monotonicity of the quantile function through Monotonic Quantile Positional Embedding (MQPE) and Monotonic Quantile Value Network (MQVN), and further uses quantile dispersion to construct an online exploration policy. Experiments cover D4RL MuJoCo, AntMaze, and several Adroit tasks.
Critical strengths
-
Studying O2O RL stability from the perspective of distributional value structure has some novelty. Compared with prior work that mainly focuses on scalar Q-value bias, this paper examines how conservative penalties affect quantile crossing, and the problem is clearly defined.
-
The experimental evaluation is relatively comprehensive and includes key ablations.
Critical weaknesses
-
Interpreting quantile dispersion directly as "uncertainty" and using it for UCB exploration is conceptually questionable. In Distributional RL, quantile spread primarily reflects the randomness of the return distribution itself, i.e., aleatoric variability, and is not equivalent to epistemic uncertainty about .
-
Theorem 3.1 explicitly establishes , while the network uses clipped ReLU, whose derivative can be zero. Therefore, the result only guarantees a nondecreasing function, not a strictly increasing one. However, the paper repeatedly claims in the experiments and conclusion that COOL-DRL provides "theoretically guaranteed strict monotonicity."
-
The main experiments compare only against IQL, CalQL, SO2, and WSRL. However, the paper itself discusses SUF, PEX, 2025 online pre-training methods, and diffusion-generation O2O methods, but does not include them in the experiments. In particular, SUF also focuses on early fine-tuning stability and is therefore a direct baseline.
-
There are still many basic editing issues in the main text and references. For example, the Figure 8 caption contains "confirming the its crucial role," and the main text contains "resepectively." Capitalization is also inconsistent in the references, including "Suf," "Cal-ql," "offline rl," "MIT press," and "Cambridge university press." These are not core technical issues, but the manuscript requires systematic proofreading for an ICLR submission.
Questions for authors
See Weaknesses.