- Rating
- 1(−1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper replaces the teacher-student setting proposed in FQL by a single one-step actor. For each state-action pair , the approach consist of sampling actions (using latents samples) with a one-step actor and use only the action that is closest ( distance) to to update the policy using a regularized actor-critic loss. The author evaluate the approach on OGbench benchmark along with some ablations studies and theoretical justification.
Critical strengths
The main idea and the motivation of the paper is easy to understand and well presented. Empirical results are competitive with the baselines specially when the hyperparameter (number of action candidat is tuned per task).
Critical weaknesses
-
Except the illustrative examples in Figure 1 and Figure 2 (Figure 2 is not well explain and is difficult to interpreted, how was it produced, the abbreviations etc), there is no a clear theoretical justification why the proposed framework mitigate the tension between the two objectives (maximize the Q-function and remain near the dataset support) better than previous previous approach.
-
It is not clear how the proposed approach capture a multi-modal distribution
-
In term of mitigating the tension the tension between maximizing the Q-function and remain near the dataset support (which is the main problem the paper attend to address), recent methods like GFP [1] have being proposed. How does DROL compare to these methods.
-
Even though the main idea is easy to understand the paper remain difficult to follow, the writing is not precise e.g in row 39 and 40 what do you clearly mean by 'the loss resolves the conflict by compromising on that sample', such kind of claim can found throughout the paper.
-
Limited related work
-
The theory provide in the paper are general facts and does not justify the proposed method.
- How is section 3.3 relevent?
- Proposition 4.1 is just a general fact.
- Except the change in term of the gradient of the objectif, Proposition 4.2 does not establish and improvement in term of the resulting policy.
- Proposition 4.3 is general fact with strong unrealistic assumption e.g supposing that a value function is m-strongly convexe even locally. The framing of the proposition itself is questionable e.g what do you mean by "Let be a closed convex local analysis neighborhood" ? etc
-
As the author point out in section 5.3 (Conflict in a pointwise update .....) the experiment does not show DROL reduces the conflict or than the policy performance is related to the conflict.
-
Can the authors also clarify how they defines and mesure the change of responsibility in their experimental setup (section 5.4)
-
In term of experiment, how does DROL perform on noisy dataset from OGBench which can be seen as a good proxy multi-modal dataset? Or can the illustrate the approach on a true multi-modal example?
[1] [Guided Flow Policy: Learning from High-Value Actions in Offline Reinforcement Learning](https://arxiv.org/abs/2512.03973
Questions for authors
see above...