acceptodds
Under review as a conference paper at ICLR 2027

Can Heads Explore Better Than One in Online Reinforcement Learning?

Abstract

In reinforcement learning, generative policies can represent diverse behaviors, yet effective exploration remains a challenge in online reinforcement learning. Ensembles offer a natural approach, but training policy heads against a shared objective can reduce the diversity needed for exploration. We introduce Ensemble Sampling Actor-Critic (ESAC), a framework that combines generative policy populations with critic guided training and action selection. Circular critic pairing supplies head specific value weightings for diffusion policy regression, while pooled proposals allow the agent to explore actions from the full population. For fixed regression objectives, we prove contraction in output space under a shared objective and characterize when critic dependent targets induce distinct update operators. We further introduce Time Indexed Disagreement Exploration (TIDE), which uses policy disagreement across denoising times to guide action selection and calibrates its bonus through correlation with critic disagreement. Experiments with diffusion and flow matching policies on eight dense reward continuous control benchmarks over five seeds show improved returns over single policy controls on several locomotion tasks. Controlled comparisons and behavioral analyses examine the contribution of the policy population, while ablations establish the task-dependent benefit of calibrated disagreement. These results demonstrate the potential of generative policy ensembles for directed exploration and characterize the trade-off between improved returns and additional computation.

Primary area:
reinforcement learning
Keywords:
online reinforcement learning, policy ensembles, generative policies, directed exploration, continuous control

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gechOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Ensemble Sampling Actor-Critic (ESAC) for online reinforcement learning with generative policies. ESAC maintains multiple policy and critic networks, uses circular critic pairing to provide different value weightings for diffusion-policy regression, and pools action proposals across policy heads. The proposed Time-Indexed Disagreement Exploration (TIDE) scores candidates using policy-predictor disagreement at multiple time labels, with a bonus coefficient gated by its correlation with critic disagreement. The paper analyzes shared-objective regression in output space and evaluates diffusion and flow-matching variants on continuous-control tasks.

Critical strengths
  • The paper identifies a concrete technical issue: independently initialized generative-policy heads can receive training signals that encourage similar predictions. Studying this issue at the level of the regression objective provides a useful starting point for designing policy ensembles.
  • The algorithm is specified through explicit objectives, an action-selection rule, pseudocode, and substantial implementation details. The appendix carefully distinguishes fixed-output-space analysis from neural-network training and acknowledges several limitations of the disagreement interpretation.
  • The evaluation includes single-policy flow controls, update-rate and parameter-scaling comparisons, selection-rule ablations, computational costs, and behavioral diagnostics. Reporting tasks where ensembles provide little benefit or underperform the single-policy baseline helps establish the limitations of the approach.
Critical weaknesses
  • The shared-target flow-matching objective does not implement the proposed diversity-preservation mechanism, leaving the strongest-performing variant insufficiently explained.
  • The theoretical results concern idealized fixed-target regression and do not establish collapse prevention in the implemented neural actor–critic system.
  • TIDE’s deployed score is not directly justified by the trajectory analysis, and its gate ablation confounds adaptive gating with reduced bonus magnitude.
  • The experiments do not sufficiently separate the benefits of policy diversity and exploration from larger candidate pools, additional critics, and greater computation.
  • The excessively tight line spacing makes the manuscript difficult to read.
Questions for authors
  • What mechanism maintains diversity in the flow-matching variant when all heads share the training-action distribution and conditional regression target? Can the shared- versus separate-critic diagnostics be reported and interpreted separately for diffusion and flow matching?
  • Which claims about collapse prevention are intended as theoretical guarantees, and which are finite-training observations? How should the stronger statements in the introduction and Section 4 be reconciled with the qualifications in Appendices A.1–A.2?
  • How does TIDE compare with zero-bonus ensemble-mean action selection under otherwise identical training and candidate budgets? How does adaptive gating compare with a tuned fixed bonus coefficient of comparable magnitude?
  • Does the advantage of multiple policy heads persist when the total candidate count and critic configuration are held fixed?
4 comments

We thank the reviewer for the thoughtful evaluation of our paper and specific experimental suggestions, which helped us refine the analysis and claims.

W1. Shared-target flow matching

The gains come from different places, in diffusion, multiple policies help (4043 versus 3180 for 1 policy), while in flow matching the gain comes from more proposals and more critics (details explained in Q1). We will state this split clearly in the paper.

W2. Idealized theory

The theory covers fixed-target regression; network diagnostics are observations. We will align the main text with Appendices A.1–A.2 (more details in Q2).

W3. TIDE's deployed score and gate ablation

TIDE evaluates heads on the same candidate at four time labels, not along their own sampling paths. The trajectory analysis motivates this. Fixed coefficients match mean gates: 1.2 on Ant, 1.1 on Walker2d. This controls bonus magnitude and shows no added adaptive benefit (Q3).

W4. Candidates, critics, and computation

Ant had the largest original flow gain (2555 to 5646). New controls use 160 total proposals, 1M steps, critic UTD 0.2, and 5 seeds. One-policy/shared-critic controls maximize ensemble-mean critic value.

SetupFlow matchingDiffusion
1 policy, 2 critics4425 ± 2461912 ± 380
5 policies, 2 shared critics3710 ± 8152338 ± 656
1 policy, 5 critics5387 ± 1733180 ± 718
5 policies, 5 critics5646 ± 10014043 ± 617

Extra critics explain most of the flow gain, 5 versus 1 policy adds little relative to seed variation. Diffusion retains a higher multi-policy mean with candidates/critics matched. Parameters/time remain unmatched: ESAC takes about 3 times SDAC's Ant training time. We will report this in the paper.

W5. Line spacing

We sincerely apologize for this and will increase the line spacing.

Q1. Flow diversity and separate diagnostics

Circular pairing does not enter the flow actor loss. All heads regress to the same target; the positive multiplier mˉk≈0.8\bar{m}_k\approx0.8 scales updates without changing their common fixed-target minimizer:

v∗(xt,t,s)=E[ϵ−a†∣xt,t,s].v^{\ast}(x_t,t,s)=\mathbb{E}[\epsilon-a^{\dagger}\mid x_t,t,s].

The contraction analysis therefore predicts convergence toward a common solution under its assumptions (Eq. 10, Appendix A.1).

For diffusion, head-specific critic weights give fk∗=∑iwkiuif_k^{\ast}=\sum_i w_{ki}u_i, so:

fk∗−fj∗=∑i(wki−wji)ui.f_k^{\ast}-f_j^{\ast}=\sum_i(w_{ki}-w_{ji})u_i.

Matched Ant controls use 160 proposals, 1M steps, critic UTD 0.2, and 5 seeds:

SetupFlow matchingDiffusion
1 policy, 2 critics4425 ± 2461912 ± 380
1 policy, 5 critics5387 ± 1733180 ± 718
5 policies, 5 critics5646 ± 10014043 ± 617

The gains come from different places, in diffusion, multiple policies help (4043 versus 3180 for 1 policy), while in flow matching the gain comes from more proposals and more critics. We will state this split clearly in the paper. Flow diversity at 256 states, across 5 seeds:

CriticsPrediction disagreement, start → 1MRelative variance, start → 1M
2 shared1.43 → 0.1000.11 → 0.031
5 paired1.43 → 0.0920.11 → 0.024

Disagreement is normalized output distance on identical inputs. Relative variance divides action variance across heads using common noise by within-head variance across noises. Across-head variance is only 2–3% of within-head variance at 1M.

Q2. Guarantees versus observations

Eq. 10/Appendix A.1 gives output-space contraction for shared fixed-target regression under η<1/(max⁡iwi)\eta < 1/(\max_i w_i) and its assumptions.

Eq. 11/Appendix A.2 as explained in Q1. Critic-dependent weights can produce distinct minimizers, but differences can cancel. Prediction distances, Swimmer gait groups, and Q1 are observations. We will change the introduction/conclusion, label the figure's seed/architecture limitations, and restrict the pairing mechanism to diffusion.

Q3. Zero bonus and a magnitude-matched fixed coefficient

Ablation: 5 policies, 5 critics, 160 candidates, 5 Ant seeds, 3 Walker2d seeds:

SelectionAntWalker2d
Zero bonus5175 ± 6111855 ± 1262
Adaptive TIDE4713 ± 6072919 ± 1927
Fixed coefficient5373 ± 4012822 ± 1282

Within this ablation, Ant shows no adaptive benefit. Walker2d has a higher adaptive mean with large variation: 2 zero-bonus seeds fall early; 2 adaptive seeds stay upright throughout. Fixed coefficients match mean gates (1.2/1.1), without a tuning sweep. Fixed has a higher Ant mean and similar Walker2d mean.

Q4. Multiple policies with candidates/critics fixed

This has been explained in detail with experimentations in W4.

We tried to answer all the feedbacks to the best of our ability. To adhere to the character count we kept the answers concise and happy to explain more if needed. We sincerely hope that you would reconsider your rating.

Thank you for the additional experiments. One remaining comparison would materially affect my assessment. With five diffusion policies, five critics, 160 candidates, and identical action-selection rules and training budgets, does circular critic pairing yield greater policy diversity and higher returns than training all policy heads with the same critic pair? Please support the comparison with diversity measurements. This would directly test the contribution of the proposed diversity-preservation mechanism.

Thank you for this suggestion. We ran this control on Ant with 5 seeds: diffusion, 5 policies, 5 critics, 160 candidates, the same action selection, 1M steps and critic UTD 0.2. With circular pairing, policy k uses critics k and k+1; in the control, every policy uses critics 1 and 2. All 5 critics are trained identically and score the candidates in both runs.

Critic pairingReturnPrediction disagreement (start → 1M)Relative ensemble variance (start → 1M)
Same pair for all policies3477 ± 1751.42 → 0.1560.27 → 0.010
Circular (ESAC)3707 ± 1401.41 → 0.2680.27 → 0.040

Return is shown as mean ± standard deviation over 5 seeds. Prediction disagreement is 0 for identical policies and about 1.4 for unrelated networks. Relative ensemble variance is 0 for identical policies and 1 when switching policies changes the action as much as drawing new noise. With one shared pair, the policies end with about 40% less disagreement and a quarter of the relative variance, on every seed. Circular pairing also reaches a higher return on 4 of 5 seeds and learns faster early on (mean return 2614 versus 2112 at 550k steps).

From the results. circular critic pairing keeps the diffusion policies measurably more diverse than a shared critic pair under identical selection and budget, and this comes with higher return.

Please let us know if you have any further questions. We will be happy to answer.

Thank you for running this controlled comparison. It addresses my remaining concern. I am therefore raising my rating from 2 to 3. Please incorporate this comparison into the paper and keep the claims consistent with its scope.

Reviewer_xDq1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ESAC, an actor-critic approach where the actor is an ensemble of generative policies. ESAC addresses the problem of actors mode collapse with a circular critic pairing where no two policies share a training signal (diffusion variant) . The actor ensemble is sampled using TIDE where actions are ranked based on critics’ mean value plus a bonus for disagreement between policies.

Critical strengths
  1. Circular pairing keeps actors from converging to the same behavior. This is a useful, clearly demonstrated observation for ensemble design.
  2. ESAC presents rubost and stable performance on MuJoCo tasks.
Critical weaknesses
  1. On Ant Ma et al. (2025) report 5683 ± 138 for SDAC under the same budget and setup (vs your 798 ± 190). Similar gaps appear on Walker2d, Humanoid and Hopper. Against the Ma et al. numbers, ESAC does not improve on any task. The discrepancy is not discussed.

  2. Diversity collapse in multi-head and ensemble actors trained online is a known problem, addressed by explicit diversity terms and by selection or pruning in actor ensembles. See https://openreview.net/forum?id=I5ymMVdmaR and https://openreview.net/forum?id=qS9pPu8ODt

  3. The anti-collapse mechanism is inactive in the best variant. In ESAC (FM), all policies regress toward the same critic-selected action, and circular pairing does not enter the actor loss. The paper’s own contraction analysis would predict collapse for these policies. The promised explanation of why they remain distinct is not given. The FM results therefore do not support circular pairing as the source of diversity.

  4. Missing baselines for comparison: a. Published SDAC (Ma et al., 2025): either its reported numbers or a reproduction that matches them. The paper’s own run is far below the published results. b. DDiffPG (Li et al., 2024) c. MaxEntDP (Dong et al., 2025) d. AEAP (ZHANG, 2025) e. CrossQ, BRO or SimbaV2

Questions for authors

Can you provide explanations to the weaknesses I pointed?

2 comments

We thank the reviewer's careful assessment of our paper and constructive comments on the comparisons, theoretical interpretation, and related work.

W1. SDAC results versus Ma et al. (2025)

The Ant result 5683 ± 138 is DPMD while SDAC result is 1391 ± 202 in the original paper that introduces both DPMD and SDAC in their latest version on arxiv titled "Efficient Online Reinforcement Learning for Diffusion Policy". ESAC builds on SDAC. The comparison between our reproduction of SDAC and published numbers are below:

TaskOur SDAC reproductionPublished SDAC
Ant1063 ± 1441391 ± 202
Walker2d3819 ± 8903995 ± 498
Hopper3177 ± 2802955 ± 370
HalfCheetah12062 ± 59412210 ± 964
Humanoid6523 ± 3196437 ± 177

Results are closer on four tasks. For completeness:

TaskESAC (FM)ESAC (Diff)Published DPMD
Ant5646 ± 10014043 ± 6175683 ± 138
Walker2d4305 ± 5294172 ± 4834365 ± 266
Hopper2759 ± 4372284 ± 11473275 ± 55
HalfCheetah10328 ± 111911310 ± 72811924 ± 609
Humanoid6773 ± 5106493 ± 6726959 ± 460

FM means exceed published SDAC on Ant, Walker2d, and Humanoid. We will report these results and summary definitions.

W2. Diversity collapse is already known

We will cite and discuss both papers. We agree that collapse is a known problem. Our contribution is to explain why it happens in Q-weighted generative training, heads sharing candidates and weights solve the same regression problem and contract under the assumptions of Eq. 10. For diffusion, critic-dependent weights change head-specific targets (Eq. 11). This differs from both papers mentioned. AEAP uses random actor selection for acting/updating and pruning in a TD3 ensemble. Gamid-PG trains a Gaussian-mixture actor with deterministic policy gradients. ESAC retains all generative policies and changes diffusion targets through critic weights. We will clarify this contribution relative to prior work.

W3. The mechanism is inactive in ESAC-FM

We agree. Circular pairing does not enter the flow actor loss. All heads regress to the same target; the positive multiplier mˉk≈0.8\bar{m}_k\approx0.8 scales updates without changing their common fixed-target minimizer:

v∗(xt,t,s)=E[ϵ−a†∣xt,t,s].v^{\ast}(x_t,t,s)=\mathbb{E}[\epsilon-a^{\dagger}\mid x_t,t,s].

The contraction analysis therefore predicts convergence toward a common solution under its assumptions (Eq. 10, Appendix A.1).

For diffusion, head-specific critic weights give fk∗=∑iwkiuif_k^{\ast}=\sum_i w_{ki}u_i, so:

fk∗−fj∗=∑i(wki−wji)ui.f_k^{\ast}-f_j^{\ast}=\sum_i(w_{ki}-w_{ji})u_i.

Matched Ant controls use 160 proposals, 1M steps, critic UTD 0.2, and 5 seeds:

SetupFlow matchingDiffusion
1 policy, 2 critics4425 ± 2461912 ± 380
1 policy, 5 critics5387 ± 1733180 ± 718
5 policies, 5 critics5646 ± 10014043 ± 617

The gains come from different places, in diffusion, multiple policies help (4043 versus 3180 for 1 policy), while in flow matching the gain comes from more proposals and more critics. We will state this split clearly in the paper. Flow diversity at 256 states, across 5 seeds:

CriticsPrediction disagreement, start → 1MRelative variance, start → 1M
2 shared1.43 → 0.1000.11 → 0.031
5 paired1.43 → 0.0920.11 → 0.024

Disagreement is normalized output distance on identical inputs. Relative variance divides action variance across heads using common noise by within-head variance across noises. Across-head variance is only 2–3% of within-head variance at 1M.

W4. Missing baselines

(a) Published SDAC and DPMD, plus our reproduction, appear in W1.

(b) DDiffPG: initial MetaWorld run after porting the original codebase to our setup (one seed, 20 evaluation episodes) gave 0% success on door-open at 200k steps and peg-insert-side at 500k. After correcting exploration noise, critic range, and batch size, reach achieved 100% at 200k. The full comparison is running and we wil report it as it finishes.

(c–e) MaxEntDP, AEAP, CrossQ, BRO, and SimbaV2 are relevant. MaxEntDP addresses diffusion exploration, AEAP actor ensembles, and the others sample efficiency with different critic/training configurations. Our generative comparisons fix critic UTD at 0.2, so update schedules and computation need explicit accounting. We will expand related work and are porting these baselines with matched update schedules.

Q1. Explanations of the weaknesses

W1–W4 explain the reporting discrepancy, prior work, mechanism mismatch, and baseline status.

We tried to answer all your weaknesses with thorough and detailed experimentations, and we therefore hope that you would reconsider your rating.

Reviewer_xDq1OpenReview ↗

Thank you for the detailed response and the substantial new experiments.

W1. Thank you for the clarification. The arXiv version I consulted (2502.00361v4) labels 5683 ± 138 on Ant as “SDAC (ours)” and contains no DPMD row. I accept that in the version you cite this result belongs to DPMD. My concern therefore becomes the following. Against the strongest published single-policy method from Ma et al. (DPMD), ESAC (FM) has a lower mean on all five tasks, clearly so on Hopper and HalfCheetah, at roughly three times the training time. Two clarifications would help. (a) Your rebuttal SDAC reproduction differs from Table 1 (e.g., Ant 1063 vs. 798, Hopper 3177 vs. 2472). Which summary statistic does each use? Ma et al. report the best mean return over training, and ESAC should be summarized the same way. (b) The TIDE ablation in your reply to Reviewer gech reports 4713 ± 607 on Ant and 2919 ± 1927 on Walker2d for five policies and five critics, below Table 1 for either variant. What explains the difference?

W2. Thank you. The proposed positioning is reasonable.

W3. I appreciate the analysis. It seems to show that for flow matching the heads collapse regardless of pairing, and that the gain (probably) comes from additional critics and proposals rather than multiple policies. For diffusion, the multi-policy gain is on one task with five seeds, and the shared-critic control also changes the number of critics. I agree with Reviewer gech that comparing circular and shared pairing with five diffusion policies and five critics, with diversity measurements, is the key missing test.

W4. Noted. These comparisons are still pending.

Overall. The new results are informative, but they narrow the main claim considerably. For the best variant, multiple policies do not explain the gains. The adaptive TIDE gate shows no benefit over a fixed coefficient or zero bonus on Ant. ESAC does not improve over the strongest published single-policy baseline, at higher cost. The required revisions to the title, claims, mechanism and main comparisons are substantial, so I maintain my score.

Reviewer_1HnPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an ensemble of generative policies for online RL namely ESAC, together with TIDE, which selects pooled action proposals using critic value and policy disagreement across generative time indices. For diffusion policies, head-specific critic pairs are intended to prevent different policy heads from collapsing toward the same regression target. Their method improves return on several dense-reward locomotion tasks, most notably Ant, and the paper provides evidence that shared-critic diffusion heads collapse in function space.

Critical strengths

In general, the paper is well-written and appears to be sound. A few things that stand out to me:

  1. The shared-objective collapse phenomenon is clearly motivated and supported specifically that the fixed-regression analysis shows why policies optimized toward the same targets contract in output space, and the Ant experiment shows shared-critic heads becoming nearly functionally identical despite retaining different parameters
  2. Reporting negative results where ESAC doesn’t help rather than just favorable benchmark numbers is not only commendable but could also be useful for future works.
Critical weaknesses
  1. Main weakness: the paper does not isolate “multiple learned heads” from simply searching many more actions. I looked at the K=1K=1 vs K=5K=5 comparison and it confounds policy diversity with a much larger action-search budget; The single-policy flow control generates 32 proposals, while K=5 ESAC pools 160 proposals, in addition to using more critics, parameters, and computation. The paper itself explicitly states that this comparison “does not isolate multiple learned proposal distributions.” Therefore the current experiments establish that the larger ESAC system can outperform the smaller baseline, but not that having K distinct learned policies is the reason. I believe this is a central problem in the current version of the paper because the main empirical claim is not supported which is mainly why I recommend weak rejection although a K=1, N=160 experiment with the same critic/action-selection supporting the main claims cpuld change my assessment.
  2. Second weakness: the paper's anti-collapse mechanism does not directly explain its strongest variant, ESAC-FM. For diffusion policies, different critic pairs produce head-specific weighted regression targets. In contrast, the flow implementation selects one common a†a^\dagger and all flow heads regress toward the corresponding velocity target. Thus the critic-pair mechanism analyzed theoretically does not directly explain why ESAC-FM heads remain diverse. The paper therefore lacks a clear causal connection between its main proposed diversity mechanism and its strongest empirical evidence.
Questions for authors
  1. Can the authors report a K=1,N=160 generative-policy control using the same critic/action-selection budget as K=5 ESAC?
  2. Can the authors report a K=5 shared-objective/shared-critic control with the same 160-proposal budget?
  3. For ESAC-FM, what specifically maintains useful diversity when all heads share the same a†a^\dagger-derived training target? Is diversity still present if the random loss-scaling mechanism is removed?
4 comments

We thank the reviewer for carefully reviewing our paper and suggesting controls that helped clarify the sources of improvement. We address each point below.

W1. Multiple learned heads versus a larger action-search budget

We ran the requested K=1, N=160 control with five critics for both policy families. All results below use Ant, 1M steps, critic UTD 0.2, and five seeds (mean ± standard deviation). Every configuration generates 160 total proposals. One-policy and shared-critic controls select the highest ensemble-mean critic value; with one policy, TIDE reduces to this rule.

SetupFlow matchingDiffusion
1 policy, 2 critics4425 ± 2461912 ± 380
5 policies, 2 shared critics3710 ± 8152338 ± 656
1 policy, 5 critics5387 ± 1733180 ± 718
ESAC: 5 policies, 5 critics5646 ± 10014043 ± 617

For flow matching, with the same 160 proposals, the extra critics explain most of the gain, going from 2 to 5 critics raises the score from 4425 to 5387. With the same 5 critics, five policies are only slightly above one policy (5646 versus 5387), within seed variation. We acknowledge that a single flow policy can be enough on this task. For diffusion, five policies remain clearly above one policy with the same proposals and critics (4043 versus 3180). We will add these controls to the main text and state the source of each gain.

W2. The anti-collapse mechanism and ESAC-FM

We agree that critic pairing changes diffusion targets but does not explain diversity in ESAC-FM. Each flow head minimizes the expected squared distance between its velocity prediction vk(xt,t,s)v_k(x_t,t,s) and the common target ϵ−a†\epsilon-a^{\dagger}, multiplied by a positive factor mˉk≈0.8\bar{m}_k\approx0.8. Under the fixed-target formulation, every head therefore has the same minimizer:

v∗(xt,t,s)=E[ϵ−a†∣xt,t,s].v^{*}(x_t,t,s)=\mathbb{E}[\epsilon-a^{\dagger}\mid x_t,t,s].

The multiplier changes update scaling, not this minimizer. The contraction analysis (Eq. 10, Appendix A.1) predicts convergence toward a common solution under its assumptions, consistent with Q3. For diffusion, the fixed-target minimizer of head k is the weighted sum of targets u_i with its own critic-derived weights wki:w_ki:

fk∗=∑iwkiui.f_k^{*}=\sum_i w_{ki}u_i.

The difference between two minimizers is:

fk∗−fj∗=∑i(wki−wji)ui.f_k^{\ast}-f_j^{\ast}=\sum_{i}(w_{ki}-w_{ji})u_i.

Different critic weights can produce distinct targets, but their effects can cancel; this is not a guarantee of persistent separation in neural actor-critic training (Eq. 11, Appendix A.2). This is why the gains come from different places, in diffusion, multiple policies help (4043 versus 3180 for one policy, W1), while in flow matching the gain comes from more proposals and more critics (W1). We will state this split clearly in the paper.

Q1. K=1, N=160 with the same critic/action-selection budget

Yes. W1 reports both controls with five critics: flow 5387 ± 173 versus 5646 ± 1001; diffusion 3180 ± 718 versus 4043 ± 617 (five seeds).

Q2. K=5 with a shared objective/shared critics and 160 proposals

Yes; W1 includes five-policy controls with two shared critics. Flow scores 3710 ± 815 versus 4425 ± 246 for one policy. Diffusion scores 2338 ± 656 versus 1912 ± 380.

Q3. What maintains flow diversity, including without loss scaling?

The gains in flow matching coming from more critic and more proposals as explained in W2. We evaluated 256 states at initialization and 1M steps on Ant, across five seeds. Prediction disagreement is normalized output distance on identical inputs, zero for identical outputs, maximum two, and approximately 1.4 at initialization. Relative ensemble variance (Pathak, Gandhi and Gupta, "Self-Supervised Exploration via Disagreement", ICML 2019, Sekar, Rybkin, Daniilidis, Abbeel, Hafner and Pathak, "Planning to Explore via Self-Supervised World Models" (Plan2Explore), ICML 2020) is action variance across heads sharing the same noise, divided by within head action variance across different noises; zero means identical actions and one means equal variance.

Flow configurationPrediction disagreement, start → 1MRelative variance, start → 1M
2 shared critics1.43 → 0.1000.11 → 0.031
5 paired critics1.43 → 0.0920.11 → 0.024
5 paired, no loss scaling1.41 → 0.0870.11 → 0.021

Disagreement falls sharply in all cases. Across-head variance is only about 2–3% of within-head variance at 1M steps, including without loss scaling. These measurements do not support pairing or random loss scaling as effective diversity-preservation mechanisms for ESAC-FM. We will revise the explanation accordingly.

We again thank you for your insightful review and feedback on the paper. We tried to address all the weaknesses and question to the best of our ability. We sincerely hope that you would positively reconsider your rating.

This is a nice and detailed response, thank you authors! I see and recognize that you ran the controls I requested, in particular the new flow-matching experiments seem to confirm my concern that multiple heads do not provide much additional benefit once the proposal and critic budgets are matched, and that the proposed pairing mechanism doesn't preserve substantial diversity.

Your diffusion results are a good improvement, but I am still unsure whether this improvement can be attributed specifically to multiple learned proposal distributions because per my read the full ESAC configuration also uses TIDE while the single policy control uses critic only selection. In general, while the paper is interesting and the rebuttal to the reviews is substantial, I think the evidence supports a smaller more diffusion-specific picture than the paper claims.

Thank you for the follow up and for reading the new controls closely. We ran the diffusion comparison with the same selection rule for every row, so TIDE is not part of it. All rows use diffusion on Ant, 5 critics, 160 candidates, 1M steps, critic UTD 0.2 and 5 seeds, and all pick the candidate with the highest mean critic value, as the one policy control does.

Diffusion setup (same selection, no TIDE)ReturnPrediction disagreement at 1MRelative ensemble variance at 1M
1 policy3180 ± 718--
5 policies, same critic pair for all3477 ± 1750.1560.010
5 policies, circular pairing (ESAC)3707 ± 1400.2680.040

Return is mean ± standard deviation over 5 seeds. Prediction disagreement is 0 for identical policies and about 1.4 for unrelated networks (all runs start near 1.41). Relative ensemble variance is 0 for identical policies and 1 when switching policies changes the action as much as drawing new noise. With the selection rule held fixed, five diffusion policies still score above one policy (3707 versus 3180) are higher on 4 of 5 seeds, and vary much less across seeds. Giving every policy the same critic pair lowers both diversity and return, which ties the gain to the learned policies themselves and to circular pairing.

We agree that the evidence supports a smaller, diffusion specific claim. We will revise the paper to say that for flow matching the gain comes from more critics and proposals, and that for diffusion the gain comes from multiple policies kept distinct by circular pairing, with TIDE reported as a separate component.

Please let us know if you have any further questions. We will be happy to answer. We sincerely hope that you would positively reconsider your rating.

Thank you for the quick reply; I appreciate that the new controls address my original methodological concerns. That said, we both agreed that the current evidence makes the contribution specialized because the strongest evidence is now for circular critic pairing in diffusion policies on Ant, while the flow-matching and adaptive TIDE results do not establish the original mechanisms. I do like the results here but I still think they do not support acceptance. I would encourage you to try to investigate these mechanisms more in detail and aim for future venues rather than rush the current form to ICLR.

Comments on the paper

1 comment

We thank all the reviewers for their suggestions to focus on diffusion based policies. We agree that limiting the paper to diffusion policies makes the paper and its contributions clearer. This narrower scope also matches current work on online generative policies, which builds on diffusion policies [1, 2].

Most papers study only one type of generative policy, either diffusion or flow matching, and we thought studying both would be interesting. However, we agree that including the flow matching study in the main text makes the paper harder to follow. We have already revised some sections of paper including abstract, introduction and conclusion and uploaded here and started revising the rest of draft to focus on diffusion policies. We are happy to move the flow matching study to the appendix and investigate those results further as future work.

We believe this sharper focus highlights the benefit of our method in the more standard diffusion based setting.

[1] Ma et al., Efficient Online Reinforcement Learning for Diffusion Policy (SDAC/DPMD), 2025 [2] Wang et al., Diffusion Actor-Critic with Entropy Regulator (DACER), 2024

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.