- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
TEDI is a conditional parameter-space exploration and selective replay-admission mechanism built on top of TD3. When sustained stagnation is detected, TEDI generates temporary parameter-perturbed policies, collects their experience outside the main replay buffer, and admits only a filtered, balanced, and capped subset of transitions into replay without replacing the main learner. The authors validate the method across four continuous-control environments (Ant, HalfCheetah, Hopper, and Swimmer), reporting peak-return improvements of approximately 4.2%, 9.8%, 5.6%, and 1.0% over the strongest baselines, alongside ablation studies and auxiliary interaction/runtime accounting.
Critical strengths
Coherent and practical decoupling of exploration and replay insertion: Separating candidate experience acquisition from replay buffer admission allows the system to screen and reject poor trajectories before exposing the off-policy learner to them, while temporary probes avoid the overhead of maintaining a persistent population. This is a clean, practical design choice that directly addresses replay pollution in hybrid methods.
Promising empirical gains and transparent limitation reporting: The reported peak-return improvements over strong baselines (particularly on Ant and HalfCheetah) show real potential, and the authors strengthen their presentation's credibility by transparently accounting for rejected shadow interactions, acknowledging TD3's speed advantage, and reporting negative ablation findings (such as Full TEDI's lower Hopper AUC).
Critical weaknesses
Lack of causal isolation due to an unmatched TD3 backbone and unaligned evaluation protocols: The underlying TEDI learner relies on extensive task-specific modifications (LayerNorm, EMA evaluation, custom learning rates, replay sizes, discount factors, and gradient clipping) that are not clearly matched in the "Reference TD3" baseline. Most tellingly, three of five Full TEDI runs on Hopper trigger zero interventions yet still contribute to the reported gains, meaning performance there is driven entirely by the configured backbone and evaluation setup. Furthermore, relying on single-episode maximum observed returns (in Ant and Swimmer), task-varying evaluation frequencies, and undocumented baseline evaluation protocols confounds whether improvements stem from TEDI's interventions or from backbone and evaluation differences.
Extensive per-task rule customization and missing matched-schedule controls: Across just four tasks, the method changes not only scalar hyperparameters but also discrete algorithmic rules—stagnation detectors, admission predicates, critic-scoring rules, mutation masks, and stopping conditions—several of which (like the exact stagnation predicates in Equation 3 and Table 9) are not fully specified mathematically in the text. In addition, the paper omits the closest conceptual baselines (such as E-ERL, conventional parameter-space noise, and periodic or random probes matched for auxiliary interaction budget), making it impossible to separate the benefit of stagnation-based scheduling and replay filtering from the sheer volume of auxiliary interactions and task-specific engineering.
Questions for authors
Backbone-matched control: Does TEDI's performance gain persist when compared against a TD3 baseline that uses the exact same learner architecture, hyperparameters, exploration schedule, EMA evaluation actor, and evaluation frequency, with only TEDI interventions disabled? Specifically, how does performance compare on the three zero-event Hopper seeds?
Matched-budget scheduling: How does stagnation-triggered scheduling perform against a periodic or randomly scheduled temporary probe baseline matched for total auxiliary environment interactions? In the "without trigger" ablation, what was the exact invocation schedule and realized interaction cost?
Evaluation protocol alignment and alternative metrics: Were baseline evaluation actors (e.g., EMA), evaluation frequencies, and episode counts per checkpoint identical to TEDI's across all tasks? Do the method rankings hold when comparing final-checkpoint returns, independently re-evaluated checkpoints, and main-baseline AUCs?
Task-specific customization and admission diagnostics: How were the task-specific algorithmic rules (stagnation predicates, admission filters, critic scoring) selected, and what are the exact mathematical definitions for the "legacy" and "normalized windows" rules in Table 9? What are the actual admitted transition counts and acceptance rates per seed, and can a single shared configuration work across tasks?
Thank You and a Brief Follow-up
Dear Reviewer ASGs, Thank you again for your time and thoughtful review. As the discussion period is nearing its end, we would appreciate your feedback on our point-by-point responses and revised manuscript. We have added matched-learner controls (Appendix D.2), auxiliary-budget-matched periodic/random probe comparisons (Appendix D.6), and further evaluation and implementation details. Please let us know if any concerns remain so that we can clarify them before the discussion closes. If our responses and the additional evidence have addressed your concerns, we kindly ask you to consider revising your rating upward.