- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Summary
The paper introduces DARE, a two-stage method that perturbs exploration directions by comparing rollouts to encourage diverse responses. DARE is supposidly algorithm agnostic and can encourage task-progression states to be visited more often.
Critical strengths
Strengths
- I think the motivation is clear, well-motivated, and well-written. I found it easy to follow the paper. The presentation is good.
- The idea, at the risk of minimising it temporarily, is sensible and elegant. Action-behaviour diversity is a real issue and measuring responses instead of assuming them feels like a fresh angle.
- The plug-and-play nature is even better. And if I am correct, if , it reverts back to standard isotropic OU noise, which is elegant also.
- The TransportBox task is particularly motivating (if not an outlier) for the method.
Critical weaknesses
Weaknesses
- The results are interesting since for the majority of experiments, DARE is quite close/ties to the comparable methods. This is less convincing.
- Why do the different experiments use different amount of random seeds? For example, Tab 1 uses five seeds, and Tab 2 uses 3? Regardless, both are quite small and 10 would be preferable.
- Is there a potential confound between the DARE and OU comparison? Why does OU use a fixed of 8 for each task but DARE has task specific of varying values. Perhaps the gap in results are due to this mismatch. If the paper's claim was the response-shaped covariance beats isotropic covariance then I feel like this mismatch wouldn't exist to empirically back this up.
- Section 4.1, what is DAB? Similarly, with Appendix B, what is RW-DAB?
Minor issues
- Presentation of figures is not the best, preferabbly I would like a clearly labelled x and y axis.
- Figure 2, the result subplots don't line up.
Questions for authors
Questions
- Is there any intuition why SAC with no noise beats DARE on draw_triangle L_1 and L_2?
- Can you report final-policy success, or at least returns, to show better exploration leads to better learning?
- Why do OU and DARE have different values and how is this considered fair?
2 comments
Thank you for this detailed discussion back, and thank you for your hard work during this rebuttal period. I wanted to check all of the additional content and the other reviewers' concerns carefully. Given the discussion around this paper, I maintain my score.
Author Response
Reviewer 3
We thank the reviewer for the helpful and constructive review.
[Matched OU Correlation Horizons]
All OU–DARE comparisons were rerun with matched HOU. On TransportBox-L0, DARE obtains 345.2 vs. 73.0 mean reaches for OU at HOU=8, including discovery (Table 14). With matched features, horizon and nominal energy, Spectral focused obtains 630.2 vs. 77.6 for Uniform focused (Table 2). These gains cannot stem from horizon mismatch; spectral allocation does not consistently win on DrawTriangle.
[Seed Counts and Variability]
Main SAC/TD3/PPO comparisons and three-task ablations use five matched seeds; Appendix C.4 reports individual seed results, with aggregate means and sample SD reported in the corresponding tables.
[SAC-Only and DARE on DrawTriangle]
DARE’s mean reaches are 3629.4 vs. 3716.0 at L1 and 2581.2 vs. 2604.0 at L2 (2.3%/0.9% lower), but 27.0 vs. 4.0 at L3 (Table 14). SAC-only is already stochastic; external perturbations need not improve every stage. This does not establish a causal trade-off between early and deeper reaches.
[Exploration and Policy Performance]
Milestones measure concrete task progress under matched budgets and learner settings, revealing differences obscured by rare terminal success. Better exploration does not automatically produce a better final policy: value estimation, credit assignment, use of rare successes, and policy-optimization objectives and stability also matter. Reaching counts therefore complement, rather than substitute for, final-policy evaluation.
[Task-Dependent Exploration Gains]
Table 2 shows task-dependent configuration rankings. DARE allocates more energy to stronger responses, not higher task rewards; its benefit depends on whether those responses support task progress. It therefore need not outperform every exploration mechanism on every task. Substantial gains on some tasks and comparable performance on others support the value of response-guided exploration without implying a universally preferable configuration.
[Terminology]
DAB/RW-DAB were working names; we now consistently use DARE (method) and ARE (action-response eigenbasis).