acceptodds
Under review as a conference paper at ICLR 2027

Evolution Only When Needed: Triggered Evolutionary Data Injection for Off-Policy Reinforcement Learning

Abstract

Evolutionary reinforcement learning (ERL) can expand exploration for off-policy learners, but maintaining a population throughout training requires repeated auxiliary rollouts even when the learner is still improving. Moreover, indiscriminately inserting population-generated trajectories into replay can shift the replay distribution away from the current learner and degrade off-policy updates. We propose Triggered Evolutionary Data Injection (TEDI), which treats evolutionary assistance as an on-demand replay-data intervention rather than a persistent policy-optimization process. TEDI jointly decides when to invoke evolutionary exploration, which generated experience should enter replay, and how much influence each intervention may have. When persistent learning stagnation is detected, TEDI creates a temporary population of parameter-perturbed policy probes, collects their trajectories outside the main replay buffer, filters candidate experience at both transition and trajectory levels, and injects only a bounded, balanced subset before discarding the population. Thus, evolutionary search affects learning only through selectively admitted replay data. Across four MuJoCo continuous-control tasks, TEDI achieves the highest mean peak return in comparisons with six baselines. Ablation studies show that triggering, filtering, and bounded injection each contribute to performance, while runtime comparisons show that TEDI has the lowest average training time among the compared ERL methods.

Primary area:
reinforcement learning
Keywords:
evolutionary reinforcement learning, off-policy reinforcement learning, experience replay, exploration

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ASGsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

TEDI is a conditional parameter-space exploration and selective replay-admission mechanism built on top of TD3. When sustained stagnation is detected, TEDI generates temporary parameter-perturbed policies, collects their experience outside the main replay buffer, and admits only a filtered, balanced, and capped subset of transitions into replay without replacing the main learner. The authors validate the method across four continuous-control environments (Ant, HalfCheetah, Hopper, and Swimmer), reporting peak-return improvements of approximately 4.2%, 9.8%, 5.6%, and 1.0% over the strongest baselines, alongside ablation studies and auxiliary interaction/runtime accounting.

Critical strengths

Coherent and practical decoupling of exploration and replay insertion: Separating candidate experience acquisition from replay buffer admission allows the system to screen and reject poor trajectories before exposing the off-policy learner to them, while temporary probes avoid the overhead of maintaining a persistent population. This is a clean, practical design choice that directly addresses replay pollution in hybrid methods.

Promising empirical gains and transparent limitation reporting: The reported peak-return improvements over strong baselines (particularly on Ant and HalfCheetah) show real potential, and the authors strengthen their presentation's credibility by transparently accounting for rejected shadow interactions, acknowledging TD3's speed advantage, and reporting negative ablation findings (such as Full TEDI's lower Hopper AUC).

Critical weaknesses

Lack of causal isolation due to an unmatched TD3 backbone and unaligned evaluation protocols: The underlying TEDI learner relies on extensive task-specific modifications (LayerNorm, EMA evaluation, custom learning rates, replay sizes, discount factors, and gradient clipping) that are not clearly matched in the "Reference TD3" baseline. Most tellingly, three of five Full TEDI runs on Hopper trigger zero interventions yet still contribute to the reported gains, meaning performance there is driven entirely by the configured backbone and evaluation setup. Furthermore, relying on single-episode maximum observed returns (in Ant and Swimmer), task-varying evaluation frequencies, and undocumented baseline evaluation protocols confounds whether improvements stem from TEDI's interventions or from backbone and evaluation differences.

Extensive per-task rule customization and missing matched-schedule controls: Across just four tasks, the method changes not only scalar hyperparameters but also discrete algorithmic rules—stagnation detectors, admission predicates, critic-scoring rules, mutation masks, and stopping conditions—several of which (like the exact stagnation predicates in Equation 3 and Table 9) are not fully specified mathematically in the text. In addition, the paper omits the closest conceptual baselines (such as E-ERL, conventional parameter-space noise, and periodic or random probes matched for auxiliary interaction budget), making it impossible to separate the benefit of stagnation-based scheduling and replay filtering from the sheer volume of auxiliary interactions and task-specific engineering.

Questions for authors

Backbone-matched control: Does TEDI's performance gain persist when compared against a TD3 baseline that uses the exact same learner architecture, hyperparameters, exploration schedule, EMA evaluation actor, and evaluation frequency, with only TEDI interventions disabled? Specifically, how does performance compare on the three zero-event Hopper seeds?

Matched-budget scheduling: How does stagnation-triggered scheduling perform against a periodic or randomly scheduled temporary probe baseline matched for total auxiliary environment interactions? In the "without trigger" ablation, what was the exact invocation schedule and realized interaction cost?

Evaluation protocol alignment and alternative metrics: Were baseline evaluation actors (e.g., EMA), evaluation frequencies, and episode counts per checkpoint identical to TEDI's across all tasks? Do the method rankings hold when comparing final-checkpoint returns, independently re-evaluated checkpoints, and main-baseline AUCs?

Task-specific customization and admission diagnostics: How were the task-specific algorithmic rules (stagnation predicates, admission filters, critic scoring) selected, and what are the exact mathematical definitions for the "legacy" and "normalized windows" rules in Table 9? What are the actual admitted transition counts and acceptance rates per seed, and can a single shared configuration work across tasks?

1 comment
AuthorsOpenReview ↗

Thank You and a Brief Follow-up

Dear Reviewer ASGs, Thank you again for your time and thoughtful review. As the discussion period is nearing its end, we would appreciate your feedback on our point-by-point responses and revised manuscript. We have added matched-learner controls (Appendix D.2), auxiliary-budget-matched periodic/random probe comparisons (Appendix D.6), and further evaluation and implementation details. Please let us know if any concerns remain so that we can clarify them before the discussion closes. If our responses and the additional evidence have addressed your concerns, we kindly ask you to consider revising your rating upward.

Reviewer_kYAuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TEDI augments TD3 with temporary evolutionary exploration when learning appears to stagnate. Perturbed policies collect data in separate buffers, candidate transitions are screened, and a bounded amount of accepted data is added to the learner's replay buffer. The method reports improved peak returns on four MuJoCo tasks over five seeds, together with ablations and computational-cost measurements.

Critical strengths
  • The method makes a clear distinction between deciding when to explore, selecting useful experience, and limiting its influence on replay. Temporary probes are discarded after each intervention, which keeps the design simpler than maintaining a persistent population. This is a reasonable way to combine evolutionary exploration with an off-policy learner.

  • The reported peak-return gains on Ant and HalfCheetah are encouraging. For example, on HalfCheetah TEDI reaches 13,079.09, compared with 11,907.69 for the strongest baseline (CEM-RL).

Critical weaknesses
  • The comparison does not sufficiently isolate the contribution of evolutionary injection. TEDI's learner adds LayerNorm and gradient clipping on all tasks, EMA-actor evaluation on three tasks, and task-specific learning rates, discount factors, and exploration schedules, while Table 7 gives only a broad description of the baseline configurations. This matters particularly on Hopper, where three of five TEDI runs trigger no interventions. That observation does not establish the cause of the aggregate gain, but it makes an otherwise identical learner with injection disabled an important control.
Questions for authors
  • How does the same TEDI learner perform with interventions disabled, keeping the architecture, hyperparameters, and evaluation actor unchanged? Please also clarify which of these settings are shared with the TD3 baseline.
Reviewer_jpZKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Triggered Evolutionary Data Injection (TEDI). When learning stops improving for a sustained period, TEDI creates temporary policies by adding noise to policy parameters. These policies collect data in separate buffers. The data pass through transition-level and trajectory-level filters before a limited, balanced subset enters the main replay buffer. The temporary policies are then discarded. Using TD3 on four MuJoCo tasks, the paper reports the highest mean peak returns among the compared methods and shorter training time than the compared evolutionary RL methods.

Critical strengths
  • The method clearly separates temporary exploration from the main learning process. Candidate data remain outside the replay buffer until they pass the filters, and temporary policies do not directly replace the learner (§3.3–3.5).

  • The paper reports results over five seeds, training time, and actual intervention counts. It also separates the number of transitions added to replay from the extra environment interactions used to collect them, helping readers assess both benefits and costs (Tables 1–4 and Appendix B.3).

Critical weaknesses
  • Outcome-based filtering may bias value estimates; Equations (8)–(14) select data based on rewards, next-state scores, and rollout returns. In stochastic environments, this may change the outcome distribution for a given state-action pair, causing standard TD3 updates to learn values from the filtered distribution. The paper does not explain how it handles this bias or whether it assumes deterministic environments.

  • The main stagnation rule is not fully defined. For example, Equation (3) uses (D_{\mathrm{stag}}) without giving its full decision rule. Table 9 lists “legacy” and “normalized windows” but does not provide the complete conditions, window size, or thresholds. Thus, even with the full return history, we cannot determine the trigger decisions from the paper alone, making the core algorithm difficult to check.

  • The effect of negative Q values on the filtering thresholds is not explained. Hopper’s Explorer uses (q>0.8m_t^q) in Table 5. When the median is −100, its threshold is −80, which is stricter than the Conservative threshold of −100. The order reverses when the median is positive. The paper does not explain whether this matches the intended roles.

Questions for authors
  1. How does outcome-based filtering affect the conditional outcome distribution and Bellman targets when rewards or transitions are stochastic? Does the method rely on a deterministic-environment assumption?

  2. What are the complete decision rules, window sizes, and thresholds for the two versions of (D_{\mathrm{stag}}), so that a given return history leads to clearly defined trigger decisions?

  3. Is it intended that the Explorer’s Q-value filter becomes stricter than the Conservative filter when the median Q value is negative? What is the reason for this choice?

3 comments

Thank you for the responses and revisions; they address most of my concerns. I have one remaining question: when a successful rollout passes the trajectory-level gate, could transition-level filtering discard low-reward steps needed to reach high-return states, and how is this trade-off handled? I will take your clarifications into account before finalizing my rating.

AuthorsOpenReview ↗

Clarification on Low-Reward Transitions

Thank you for this thoughtful follow-up. Low immediate reward does not automatically imply rejection. A transition that satisfies the role-specific local conditions remains eligible for replay if its rollout also passes the trajectory-level gate (Eq. (14)). However, rollout acceptance cannot override a failed local predicate, so a useful preparatory step can still be excluded. This screening applies only to auxiliary probe experience. All main-environment transitions enter replay without auxiliary screening (Algorithm 1), so the learner continues to receive low-reward preparatory experience when it encounters it and can learn its longer-term value through TD3's bootstrapped updates.
To account for long-term usefulness, the roles with quality conditions consider critic estimates of future return alongside immediate-reward thresholds (Table 6). For illustration, HalfCheetah's quality score includes a bootstrapped next-state value (Eq. (6)). Its Explorer also permits rewards down to 0.75 times a positive active-population mean when the quality and nontermination conditions hold. These rules allow some low-immediate-reward transitions to be retained. The selected transitions come from temporary parameter-perturbed probes that supplement the learner's ordinary experience.
In the original five HalfCheetah runs, 11,642 of 71,000 steps (16.40%) from trajectory-qualified rollouts were rejected locally. In a separate HalfCheetah training diagnostic, a Conservative transition with immediate reward -1.009 and estimated quality score 375.909 passed both gates and entered replay. Table 4 shows higher mean peak returns for Full TEDI than for w/o Filtering on all four evaluated tasks, supporting the combined admission design in these tasks. We have added an explanation in Appendix D.7 (p. 28) and a cross-reference following Eq. (14). The added discussion explains how future-value estimates, reward requirements, and unfiltered ordinary experience interact in this trade-off.

AuthorsOpenReview ↗

Brief Follow-up on Low-Reward Transitions

Dear Reviewer jpZK,
Thank you again for the time and care you put into reviewing our work. Your question about low-reward preparatory steps helped us improve the manuscript by explaining the filtering trade-off more clearly. We added this explanation in Appendix D.7 (p. 28), with a cross-reference after Eq. (14).
If any questions remain, we’d be happy to address them before the discussion closes. If our clarification addresses your remaining concern, we would be grateful if you could take it into account when finalizing your overall rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Area Chair,
As the discussion period draws to a close, we would like to summarize our work's contribution and the revisions made in response to the three reviewers.
Our research goal is to improve the learning performance of an existing off-policy DRL learner through evolutionary assistance while preserving its configured architecture and gradient-update rules. We designed TEDI as a lightweight framework around this goal. It jointly controls when temporary policy probes are activated, which probe-generated transitions enter replay, and how much is admitted at each intervention. The probes collect candidate data outside the main replay buffer and are discarded after use. Evolutionary assistance reaches the learner through selected experience, without replacing the learner's policy or maintaining a persistent population. Our experiments examine both performance benefits and sampling and computational costs. TEDI achieves the lowest measured training time among the compared evolutionary RL implementations on each of the four tasks (Table 1).
We have uploaded the revised manuscript with changes marked in blue. The main additions address the reviewers' concerns as follows.
Reviewer jpZK: filtering effects and implementation clarity. We added a conditional-outcome analysis and a frozen-model reward diagnostic to explain how outcome-dependent admission can affect training targets under stochastic outcomes (Appendix D.5). We also fully specified the stagnation rules and clarified the sign dependence of the Q thresholds. Reconstruction reproduces all 194 recorded invocations across the 20 original Full TEDI runs (Appendix D.1). The reviewer stated that our responses and revisions addressed most of their concerns. We subsequently answered the remaining question about low-reward preparatory steps and added the explanation to Appendix D.7, with a cross-reference after Eq. (14). This explains how value estimates, local reward conditions, and unfiltered ordinary learner experience interact in the filtering trade-off.
Reviewer kYAu: isolating the intervention contribution. We added a matched-learner control across all four tasks, preserving the learner configuration and evaluation protocol while disabling TEDI's interventions. Full TEDI achieves higher mean peak returns on Ant, HalfCheetah, and Hopper, while Swimmer's peaks are closely clustered (Table 12). On Hopper, the three zero-intervention seeds have identical recorded evaluation trajectories in Full and matched. Both intervention seeds achieve higher independently evaluated final returns with TEDI. Across all five seeds, mean final return improves by 5.1%, and episode completion increases from 47% to 81%, with auxiliary interactions amounting to 0.22% of the main training budget (Table 2). Reviewer kYAu confirmed that the matched comparison addressed their main concern.
Reviewer ASGs: attribution, auxiliary budgets, evaluation, and task configurations. In addition to the matched learner, we added auxiliary-budget-matched periodic and random probe controls on Ant and HalfCheetah. Keeping the learner, probe generation, and admission mechanisms fixed, Full TEDI achieves higher mean peak return, independently evaluated final return, and raw-return AUC than both controls at equal or nearly equal auxiliary budgets (Appendix D.6, Tables 19-20). These results support the contribution of conditional intervention beyond the amount of auxiliary experience collected. We also documented evaluation protocols, examined additional performance metrics, specified the task configurations and invocation rules, and provided actual replay-admission counts and acceptance rates for all 20 Full TEDI runs.
These additions strengthen the evidence for TEDI as an effective lightweight framework and clarify where its interventions contribute. The supplementary experimental code is available in our anonymous rebuttal repository.
We hope this summary helps you assess our contribution and the additional evidence. We would appreciate your considering the revised manuscript and the full discussion alongside the original reviews when preparing your meta-review.
Thank you for your time and for coordinating the review process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.