acceptodds
Under review as a conference paper at ICLR 2027

Response-Aware Exploration for Continuous Control Reinforcement Learning

Abstract

In reinforcement learning, exploration with sparse rewards and continuous action spaces on long-horizon tasks remains difficult, and a common strategy is to perturb the policy's actions during exploration. However, diverse action perturbations do not necessarily lead to diverse behavioral responses. Existing methods may encourage behavioral diversity indirectly, but few derive the perturbation directions themselves from the behavioral responses they actually produce. In this paper, we propose Discovered Action-Response Eigenbasis (DARE), a method that derives exploration directions by comparing perturbed and unperturbed rollouts from the same initial state and applies temporally correlated perturbations along those directions. DARE is reward-free, and is designed as an agile method that can be integrated into existing RL pipelines with minimal changes. Experimental results show that DARE outperforms representative action-space perturbation and temporally structured exploration baselines on sparse-reward tasks in Maniskill. DARE enables exploration to reach task-progress states more often, achieving up to 504% as many reaches as the strongest evaluated baseline, while remaining effective across different RL algorithms.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Sparse Reward, Continuous Control, Long Horizon, Exploration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VP5vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary

The paper introduces DARE, a two-stage method that perturbs exploration directions by comparing rollouts to encourage diverse responses. DARE is supposidly algorithm agnostic and can encourage task-progression states to be visited more often.

Critical strengths

Strengths

  • I think the motivation is clear, well-motivated, and well-written. I found it easy to follow the paper. The presentation is good.
  • The idea, at the risk of minimising it temporarily, is sensible and elegant. Action-behaviour diversity is a real issue and measuring responses instead of assuming them feels like a fresh angle.
  • The plug-and-play nature is even better. And if I am correct, if α=0\alpha=0, it reverts back to standard isotropic OU noise, which is elegant also.
  • The TransportBox task is particularly motivating (if not an outlier) for the method.
Critical weaknesses

Weaknesses

  • The results are interesting since for the majority of experiments, DARE is quite close/ties to the comparable methods. This is less convincing.
  • Why do the different experiments use different amount of random seeds? For example, Tab 1 uses five seeds, and Tab 2 uses 3? Regardless, both are quite small and 10 would be preferable.
  • Is there a potential confound between the DARE and OU comparison? Why does OU use a fixed HOUH_{OU} of 8 for each task but DARE has task specific HOUH_{OU} of varying values. Perhaps the gap in results are due to this mismatch. If the paper's claim was the response-shaped covariance beats isotropic covariance then I feel like this mismatch wouldn't exist to empirically back this up.
  • Section 4.1, what is DAB? Similarly, with Appendix B, what is RW-DAB?

Minor issues

  • Presentation of figures is not the best, preferabbly I would like a clearly labelled x and y axis.
  • Figure 2, the result subplots don't line up.
Questions for authors

Questions

  • Is there any intuition why SAC with no noise beats DARE on draw_triangle L_1 and L_2?
  • Can you report final-policy success, or at least returns, to show better exploration leads to better learning?
  • Why do OU and DARE have different HOUH_{OU} values and how is this considered fair?
2 comments
AuthorsOpenReview ↗

Author Response

Reviewer 3

We thank the reviewer for the helpful and constructive review.

[Matched OU Correlation Horizons]

All OU–DARE comparisons were rerun with matched HOUH_{\mathrm{OU}}. On TransportBox-L0, DARE obtains 345.2 vs. 73.0 mean reaches for OU at HOU=8H_{\mathrm{OU}}=8, including discovery (Table 14). With matched features, horizon and nominal energy, Spectral focused obtains 630.2 vs. 77.6 for Uniform focused (Table 2). These gains cannot stem from horizon mismatch; spectral allocation does not consistently win on DrawTriangle.

[Seed Counts and Variability]

Main SAC/TD3/PPO comparisons and three-task ablations use five matched seeds; Appendix C.4 reports individual seed results, with aggregate means and sample SD reported in the corresponding tables.

[SAC-Only and DARE on DrawTriangle]

DARE’s mean reaches are 3629.4 vs. 3716.0 at L1 and 2581.2 vs. 2604.0 at L2 (2.3%/0.9% lower), but 27.0 vs. 4.0 at L3 (Table 14). SAC-only is already stochastic; external perturbations need not improve every stage. This does not establish a causal trade-off between early and deeper reaches.

[Exploration and Policy Performance]

Milestones measure concrete task progress under matched budgets and learner settings, revealing differences obscured by rare terminal success. Better exploration does not automatically produce a better final policy: value estimation, credit assignment, use of rare successes, and policy-optimization objectives and stability also matter. Reaching counts therefore complement, rather than substitute for, final-policy evaluation.

[Task-Dependent Exploration Gains]

Table 2 shows task-dependent configuration rankings. DARE allocates more energy to stronger responses, not higher task rewards; its benefit depends on whether those responses support task progress. It therefore need not outperform every exploration mechanism on every task. Substantial gains on some tasks and comparable performance on others support the value of response-guided exploration without implying a universally preferable configuration.

[Terminology]

DAB/RW-DAB were working names; we now consistently use DARE (method) and ARE (action-response eigenbasis).

Thank you for this detailed discussion back, and thank you for your hard work during this rebuttal period. I wanted to check all of the additional content and the other reviewers' concerns carefully. Given the discussion around this paper, I maintain my score.

Reviewer_J24KOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DARE, an exploration-enhancement method for reinforcement learning. It uses paired trajectories to estimate local mappings from actions to behavioral shifts, applies feature decomposition to find exploration directions, and injects time-correlated OU noise. It doesn't require modifying the underlying SAC/TD3 architecture and can be plugged into different algorithms via action perturbation.

Critical strengths
  1. Exploration is a persistent bottleneck in RL, making this a relevant problem to study.
  2. The method is simple and plug-and-play, making it easy to adapt to different baseline algorithms.
Critical weaknesses
  1. In Figure 2, DARE doesn't stand out as the top performer, showing strong results on only 3 tasks.
  2. The evaluation metrics focus on "exploration behavior" rather than actual policy performance. Success rate is crucial here, since exploring well doesn't mean the task is solved.
  3. For levels L3 to L5 across most tasks, performance is essentially zero. This raises concerns about whether the method can actually handle higher-complexity settings.
  4. The Allegro environment only appears in the ablation study, but its main results are missing from Figure 2.
Questions for authors
  1. In Table 2, why does ObjectPose-7D outperform General-73D? What is the intuition behind this?
  2. The ablation studies are only conducted on the Allegro In-Hand Rotation task. Isn't this too narrow, risking conclusions that don't generalize well to other tasks?
1 comment
AuthorsOpenReview ↗

Author Response

We thank the reviewer for the helpful and constructive review.

[Role of Allegro and Cross-Task Ablations]

We added DrawTriangle and TransportBox ablations, with five matched seeds (Table 2); configuration rankings vary across tasks. Allegro provides measurable reaches through L4, supporting diagnosis at early and deeper stages. Its results appear in Table 2 and Appendix C.2.3, separately from Figure 2’s main benchmark; we do not treat its ranking as universal.

[Response Representation and Milestone Reaches]

ObjectPose-7D has higher means than General-73D at Allegro-L3 (189.6 vs. 160.4) and L4 (54.4 vs. 34.8); General leads at L0–L2 (Table 2). Policy inputs and OU horizon are unchanged. Additional joint, fingertip, and velocity responses may shape exploration without advancing rotation. This is a possible explanation, not a demonstrated cause or universal advantage for focused features.

[Milestone Reachability and Policy Performance]

Gains extend beyond early stages: DrawTriangle-L3 reaches average 27.0 versus 16.6 for OU (Table 14); Allegro-L4 averages 54.4 for Spectral focused versus 43.8 for Uniform focused, with matched features and OU horizon (Table 2). Allegro-L4 is native task success; later stages remain rarely reached elsewhere.

Milestones measure concrete task progress under matched budgets and learner settings, revealing differences obscured by rare terminal success. Better exploration does not automatically produce a better final policy: value estimation, credit assignment, use of rare successes, and policy-optimization objectives and stability also matter. Reaching counts therefore complement, rather than substitute for, final-policy evaluation.

[Task-Dependent Exploration Gains]

Table 2 shows task-dependent configuration rankings. DARE allocates more energy to stronger responses, not higher task rewards; its benefit depends on whether those responses support task progress. It therefore need not outperform every exploration mechanism on every task. Substantial gains on some tasks and comparable performance on others support the value of response-guided exploration without implying a universally preferable configuration.

Reviewer_kWdbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper deals with the problem of exploration in continuous control, and proposes an exploration technique that adds correlated action noise along trajectories. Crucially, the noise is sampled along directions that are estimates to maximally induce changes in the state space. The key idea is to collect paired rollouts (with and without action perturbation) from the same starting state and random seed. These are projected to a handcrafted feature space, and the function from action perturbation to change in trajectory features is approximately linearized. Correlated noise is then applied along its eigenvectors. The resulting method, named DARE, is applied on top of SAC (and, at a smaller scale, on top of TD3 and PPO), and compared with an array of action-space exploration techniques. While in most ManiSkill tasks it does not outperform the better baselines, it induces significant improvements in several tasks. The authors further ablate the components of the method, and its synergy with exploration techniques acting at a different level.

Critical strengths
  • The paper is clear and well written. In particular, the taxonomy of exploration methods in the introduction on two levels positions the paper cleanly.
  • The method is, to the best of my understanding, well-motivated, elegant and sound.
Critical weaknesses

The main limitation of the method is that, unlike the chosen baselines, it relies on a restrictive preliminary phase with somewhat stringent requirements:

  • The method needs to be able to control both state resets and random seeds. This is an exceptionally strong assumption in the standard episodic RL setting, and restricts the method to be used within a simulator.
  • The method relies on handcrafted features, which are task-dependent and may require significant engineering efforts.
  • This phase requires additional interactions which are not counted in the exploration budget (about 1000 rollouts judging from Table 9). If counted, the method would underperform baselines in most Maniskill tasks.

These three constraints are the main reason for my current recommendation, and I will reconsider my score if they can be lifted.

Questions for authors

Main questions/comments:

  • Performance improvement is localized in a handful of tasks, which may not warrant the privileged estimation phase mentioned above.
  • How does the method perform if the feature mapping is fixed to be an identity in all tasks?
  • Section 5.4 does not clearly describe the variants of the method in Table 2.
  • How many seeds were used for experiments? Table 6 refers to 3 or 5, but it is not clear in which setting. More generally, Table 6 and A.3 appear to be written with heavy LLM assistance, resulting in very technical and hard-to-parse text, e.g., "Segment-wise IID, OU, Pink, gSDE-style, and DARE use nominal pre-clipping external-bias energy".
  • The abstract claims "up to 504% as many reaches as the strongest evaluated baseline", which is somewhat cherry-picked. The actual improvements are much milder, as mentioned further down in the text: "DARE remains competitive across tasks, with its clearest gains appearing on milestones that require sustained progress". I would tone down the claims accordingly.

Minor questions:

  • How could the proposed method scale to visual domains?
  • Is the injection of action-level noise ever problematic for on-policy methods like PPO?
  • Why are per-seed results reported only for the PPO/TD3 ablation?

Minor comments:

  • The template is wrong (especially evident in the font).
  • I would recommend aligning the notation between Eq. 1 and Eq. 19.
  • The DAB acronym on line 167 is not introduced.
  • "behavioral-response vectors" are not introduced clearly on line 183.
  • Bolding in tables is skipped when the method does not win (e.g. Table 3).
  • SDAR and VIME are not introduced nor cited in Section 5.5.
  • The theory section in the Appendix feel rather detached from the main paper. I would suggest either referring to them, or removing if they do not support any section in particular.
2 comments
AuthorsOpenReview ↗

Author Response

Reviewer 1

We thank the reviewer for the helpful and constructive review.

[Discovery Cost and Task-Dependent Benefits]

The original presentation reported discovery separately. For directly comparable total costs, we recomputed existing-run curves and counts across the main benchmark, cross-algorithm evaluation, and ablations. Discovery budget D is included within the common total budget B, leaving B−DB-D for DARE’s online training and B for all other baselines.

Each discovery rollout lasts eight steps. Discovery uses 8,192–26,624 transitions per seed (0.5%–5.2% of the budget). After charging this cost, TransportBox-L0 retains 345.2 mean reaches versus 73.0 for OU, the strongest baseline: 4.73 times as many (Table 14).

[Response Features and Identity Mapping]

We reran all 11 tasks with identity mappings under matched budgets and seeds (Appendix D.1, Table 22). Larger absolute reductions concentrate in PlaceAppleInBowl, PlugCharger, and TransportBox; the other eight show smaller absolute differences, though some low-count milestones show appreciable relative changes. Identity provides a starting point without task-specific feature selection, with focused features introduced where useful rather than designed for every task in advance.

[Seed Counts and Reporting Clarity]

Main SAC, TD3/PPO, and three-task ablations all use five matched seeds; Appendix C.4 reports individual seed results, while aggregate means and sample SD are provided in the corresponding result tables. Table 6 and Appendix A.3 are simplified.

[Discovery Requirements]

Paired rollouts do require the perturbed and unperturbed trajectories to share the same starting conditions and random seed. However, this requirement applies only to the initial basis discovery. Once the basis has been obtained, DARE can be used for online exploration without controlling resets or random seeds. We have clarified this distinction in the revised manuscript.

[PPO Likelihood under Action Perturbations]

PPO stores the original latent sample and collection-time log probability, computing its ratio on that sample—not the perturbed executed action (Table 10). ARE and perturbation parameters remain fixed across collection and updates. This preserves sampling–likelihood consistency without claiming that noise cannot affect learning.

[Ablation Variant Definitions]

Section 5.4 and Table 11 clarify the variants across three tasks. All OU variants use HOU=8H_{\mathrm{OU}}=8; response features are defined below.

VariantFeaturesCoefficients / allocation
Raw focusedFocusedDiscrete hold-8; identically distributed, energy-normalized
Uniform focusedFocusedOU; equal variance
Spectral generalGeneralOU; eigenvalue-weighted variance
Spectral focusedFocusedOU; eigenvalue-weighted variance
TaskFocused featuresGeneral features
AllegroObject position (3) + quaternion (4): 7DJoint positions/velocities (16+16), fingertip poses (28), object pose (7), object linear/angular velocities (6): 73D
TransportBoxBox pose (7), box position − right TCP position (3), box position − left TCP position (3): 13DJoint positions/velocities (25+25), TCP poses (14), box pose (7), box linear/angular velocities (6): 77D
DrawTriangleTCP position (3): 3DJoint positions/velocities (7+7), TCP pose (7): 21D

Listed dimensions are per step; features are concatenated across eight steps. Policy inputs remain unchanged: Allegro 102D, TransportBox 77D, DrawTriangle 49D.

[Scope of Improvement]

Highlighting “up to 504%” could imply benchmark-wide gains, although it described a task-specific maximum. We replaced it with a clearer summary: DARE remains competitive across tasks, with its clearest gains where competing exploration makes limited progress.

[Extension to Visual Observations]

A possible, untested extension would use tracked robot/object keypoints, pose estimates, or regional visual features for paired response fitting and ARE construction. Uncancelled background, lighting, or tracking changes could distort allocation, making tracking and normalization important. Compact features would reduce fitting cost relative to raw pixels; eigendecomposition would remain in action space. Matched resets and random streams would still be required.

Thank you for the response. I am struggling to understand this sentence:

[Discovery Requirements] Matched resets and random streams are required only for one-time paired-rollout discovery. Fixed-basis online exploration requires neither. We have clarified this condition in the manuscript.

Could you please rewrite in a way that's easier to interpret, without asking an LLM to do it? I can understand the rest of the response, although it would be much easier to read if it was human-written.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.