- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Ensemble Sampling Actor-Critic (ESAC) for online reinforcement learning with generative policies. ESAC maintains multiple policy and critic networks, uses circular critic pairing to provide different value weightings for diffusion-policy regression, and pools action proposals across policy heads. The proposed Time-Indexed Disagreement Exploration (TIDE) scores candidates using policy-predictor disagreement at multiple time labels, with a bonus coefficient gated by its correlation with critic disagreement. The paper analyzes shared-objective regression in output space and evaluates diffusion and flow-matching variants on continuous-control tasks.
Critical strengths
- The paper identifies a concrete technical issue: independently initialized generative-policy heads can receive training signals that encourage similar predictions. Studying this issue at the level of the regression objective provides a useful starting point for designing policy ensembles.
- The algorithm is specified through explicit objectives, an action-selection rule, pseudocode, and substantial implementation details. The appendix carefully distinguishes fixed-output-space analysis from neural-network training and acknowledges several limitations of the disagreement interpretation.
- The evaluation includes single-policy flow controls, update-rate and parameter-scaling comparisons, selection-rule ablations, computational costs, and behavioral diagnostics. Reporting tasks where ensembles provide little benefit or underperform the single-policy baseline helps establish the limitations of the approach.
Critical weaknesses
- The shared-target flow-matching objective does not implement the proposed diversity-preservation mechanism, leaving the strongest-performing variant insufficiently explained.
- The theoretical results concern idealized fixed-target regression and do not establish collapse prevention in the implemented neural actor–critic system.
- TIDE’s deployed score is not directly justified by the trajectory analysis, and its gate ablation confounds adaptive gating with reduced bonus magnitude.
- The experiments do not sufficiently separate the benefits of policy diversity and exploration from larger candidate pools, additional critics, and greater computation.
- The excessively tight line spacing makes the manuscript difficult to read.
Questions for authors
- What mechanism maintains diversity in the flow-matching variant when all heads share the training-action distribution and conditional regression target? Can the shared- versus separate-critic diagnostics be reported and interpreted separately for diffusion and flow matching?
- Which claims about collapse prevention are intended as theoretical guarantees, and which are finite-training observations? How should the stronger statements in the introduction and Section 4 be reconciled with the qualifications in Appendices A.1–A.2?
- How does TIDE compare with zero-bonus ensemble-mean action selection under otherwise identical training and candidate budgets? How does adaptive gating compare with a tuned fixed bonus coefficient of comparable magnitude?
- Does the advantage of multiple policy heads persist when the total candidate count and critic configuration are held fixed?
4 comments
Thank you for the additional experiments. One remaining comparison would materially affect my assessment. With five diffusion policies, five critics, 160 candidates, and identical action-selection rules and training budgets, does circular critic pairing yield greater policy diversity and higher returns than training all policy heads with the same critic pair? Please support the comparison with diversity measurements. This would directly test the contribution of the proposed diversity-preservation mechanism.
Thank you for this suggestion. We ran this control on Ant with 5 seeds: diffusion, 5 policies, 5 critics, 160 candidates, the same action selection, 1M steps and critic UTD 0.2. With circular pairing, policy k uses critics k and k+1; in the control, every policy uses critics 1 and 2. All 5 critics are trained identically and score the candidates in both runs.
| Critic pairing | Return | Prediction disagreement (start → 1M) | Relative ensemble variance (start → 1M) |
|---|---|---|---|
| Same pair for all policies | 3477 ± 175 | 1.42 → 0.156 | 0.27 → 0.010 |
| Circular (ESAC) | 3707 ± 140 | 1.41 → 0.268 | 0.27 → 0.040 |
Return is shown as mean ± standard deviation over 5 seeds. Prediction disagreement is 0 for identical policies and about 1.4 for unrelated networks. Relative ensemble variance is 0 for identical policies and 1 when switching policies changes the action as much as drawing new noise. With one shared pair, the policies end with about 40% less disagreement and a quarter of the relative variance, on every seed. Circular pairing also reaches a higher return on 4 of 5 seeds and learns faster early on (mean return 2614 versus 2112 at 550k steps).
From the results. circular critic pairing keeps the diffusion policies measurably more diverse than a shared critic pair under identical selection and budget, and this comes with higher return.
Please let us know if you have any further questions. We will be happy to answer.
Thank you for running this controlled comparison. It addresses my remaining concern. I am therefore raising my rating from 2 to 3. Please incorporate this comparison into the paper and keep the claims consistent with its scope.
We thank the reviewer for the thoughtful evaluation of our paper and specific experimental suggestions, which helped us refine the analysis and claims.
W1. Shared-target flow matching
The gains come from different places, in diffusion, multiple policies help (4043 versus 3180 for 1 policy), while in flow matching the gain comes from more proposals and more critics (details explained in Q1). We will state this split clearly in the paper.
W2. Idealized theory
The theory covers fixed-target regression; network diagnostics are observations. We will align the main text with Appendices A.1–A.2 (more details in Q2).
W3. TIDE's deployed score and gate ablation
TIDE evaluates heads on the same candidate at four time labels, not along their own sampling paths. The trajectory analysis motivates this. Fixed coefficients match mean gates: 1.2 on Ant, 1.1 on Walker2d. This controls bonus magnitude and shows no added adaptive benefit (Q3).
W4. Candidates, critics, and computation
Ant had the largest original flow gain (2555 to 5646). New controls use 160 total proposals, 1M steps, critic UTD 0.2, and 5 seeds. One-policy/shared-critic controls maximize ensemble-mean critic value.
Extra critics explain most of the flow gain, 5 versus 1 policy adds little relative to seed variation. Diffusion retains a higher multi-policy mean with candidates/critics matched. Parameters/time remain unmatched: ESAC takes about 3 times SDAC's Ant training time. We will report this in the paper.
W5. Line spacing
We sincerely apologize for this and will increase the line spacing.
Q1. Flow diversity and separate diagnostics
Circular pairing does not enter the flow actor loss. All heads regress to the same target; the positive multiplier mˉk≈0.8 scales updates without changing their common fixed-target minimizer:
v∗(xt,t,s)=E[ϵ−a†∣xt,t,s].
The contraction analysis therefore predicts convergence toward a common solution under its assumptions (Eq. 10, Appendix A.1).
For diffusion, head-specific critic weights give fk∗=∑iwkiui, so:
fk∗−fj∗=∑i(wki−wji)ui.
Matched Ant controls use 160 proposals, 1M steps, critic UTD 0.2, and 5 seeds:
The gains come from different places, in diffusion, multiple policies help (4043 versus 3180 for 1 policy), while in flow matching the gain comes from more proposals and more critics. We will state this split clearly in the paper. Flow diversity at 256 states, across 5 seeds:
Disagreement is normalized output distance on identical inputs. Relative variance divides action variance across heads using common noise by within-head variance across noises. Across-head variance is only 2–3% of within-head variance at 1M.
Q2. Guarantees versus observations
Eq. 10/Appendix A.1 gives output-space contraction for shared fixed-target regression under η<1/(maxiwi) and its assumptions.
Eq. 11/Appendix A.2 as explained in Q1. Critic-dependent weights can produce distinct minimizers, but differences can cancel. Prediction distances, Swimmer gait groups, and Q1 are observations. We will change the introduction/conclusion, label the figure's seed/architecture limitations, and restrict the pairing mechanism to diffusion.
Q3. Zero bonus and a magnitude-matched fixed coefficient
Ablation: 5 policies, 5 critics, 160 candidates, 5 Ant seeds, 3 Walker2d seeds:
Within this ablation, Ant shows no adaptive benefit. Walker2d has a higher adaptive mean with large variation: 2 zero-bonus seeds fall early; 2 adaptive seeds stay upright throughout. Fixed coefficients match mean gates (1.2/1.1), without a tuning sweep. Fixed has a higher Ant mean and similar Walker2d mean.
Q4. Multiple policies with candidates/critics fixed
This has been explained in detail with experimentations in W4.
We tried to answer all the feedbacks to the best of our ability. To adhere to the character count we kept the answers concise and happy to explain more if needed. We sincerely hope that you would reconsider your rating.