- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper argues that using a single policy for both action execution and bootstrapping is problematic and proposes RAPO, a method that separates the policy's roles and trains them accordingly. In TD3+BC, the method splits the actor into an execution actor and a critic target actor, and instead of fixing the coefficient alpha, which is usually treated as a hyperparameter, it learns and adjusts alpha during training. The method can also be extended to IQL, and the paper reports results on D4RL.
Critical strengths
Given that the regularization parameter is usually fixed, the paper takes an interesting approach by adapting alpha during training.
Critical weaknesses
The paper does not sufficiently justify its design choices. There is neither a theoretical argument nor an experiment showing that using a single policy for two roles is actually problematic, which makes the motivation hard to accept. In addition, the method consists of many components that are presented without specific justification, so it is unclear why each part of the method is necessary.
The experiments are insufficient. Major baselines such as ReBRAC are missing, and most of the reported results do not appear to differ from the baselines once the standard deviations are taken into account.
Questions for authors
- Is there any theoretical or empirical evidence that using a single policy for both roles in TD3+BC is actually a problem? Since this is the main motivation of the paper, I think it is important to justify it.
- Compared with the numbers reported in the ReBRAC paper, there appears to be no performance difference. Does this not suggest that fixed coefficients can perform just as well? Given that the authors cite ReBRAC, a direct comparison seems important.
- Taking the standard deviations in Table 1 into account, the results do not seem to demonstrate the superiority of RAPO. I am curious how the method performs on other benchmarks such as Adroit or OGBench.
- The trends in Figure 2 look quite monotonic. Could a simple hand-designed schedule achieve the same effect?
- In Section 4.3, why does the Q variant perform worse? I would like to know the authors' explanation for the ablation results in general.
- In the ablations of Sections 4.3 and 4.4, if the coefficient learning rate were also tuned for each control condition, would they not reach similar performance? Table 7 suggests that the results vary considerably with the learning rate, so an experiment that rules out this effect seems necessary.
- Unlike the baselines, the proposed method uses two policies and a meta-learning procedure. How much additional computational cost does this incur?
Thanks for the additional experiments. Two of my main concerns remain.
Motivation and design. The paper never shows why using one policy for both execution and bootstrapping is a problem, or why each component is needed. The response gives a final-score comparison, but that doesn't show the problem exists. Is there any direct evidence that the two roles actually need different regularization? The same applies to the other components, which are still justified only by ablation scores rather than by an explanation of what goes wrong without them.
Experiments. The response does not address my question about other benchmarks such as Adroit or OGBench. With only 15 D4RL tasks, where many results are within one std of the strongest baselines, it is hard to tell whether RAPO is actually better. The added CIs don't help here, since they are only against TD3+BC and IQL.