- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes GemNES, an online meta-black-box optimizer for many-task optimization built on separable NES. Existing NES's high-variance natural gradient estimates hinder the scaling, resulting in slow convergence. Proposed GemNES adopt the dual-level IGC scheme, which encode cross-task agreement through natural gradient updates on meta policy and task-distributions. Experiments cover many-task and brax-MuJoCo benchmarks demonstrate faster and sustained convergence compared with representative NES-based methods and classical RL methods.
Critical strengths
-
I like the idea using concurrent task trajectories as meta training signal and as variance-reduction evidence. I think this is reasonable design and meta-reward reuses evaluated batches, so it costs no extra function evaluations.
-
The paper gives through and candid evaluation results, including per-instance tables with confidence interval, runtime analysis, and multiple ablation and sweeps.
Critical weaknesses
-
The core claimed mechanism (dual-level IGC) contributes little. the gains come mainly from meta-learned control, specifically step-size adaptation. In Table 2, without meta-control setup is significantly worse on synthetic instances and MuJoCo benchmark. Removing cross-task consensus and affinity filter matters on only few instances and tasks, so the evidence thus points to online meta-learned learning rate control helps SNES rather than 'consensus accelerate evolution'. The authors should either reframe the contribution or show settings where consensus yields a large significant benefit over w/o-transfer setup.
-
The comparison with RL baselines in Table 1 is not matched in budgt. The paper states that proposed GemNES attains significantly higher reward than classical RL baselines, PPO and SAC, on 6 and 8 of 10 tasks. But each ES method and RL methods different environment per steps, and GemNES is therefore compared against methods given up to 5-25 times fewer environment interactions. Could you verify the RL setups and either compare at matched environment steps or remove the claims of superiority over classical RL?
Questions for authors
-
How does GemNES compare to (a) a frozen, randomly initialized meta-policy, (b) random control perturbations, and (c) SNES with a tuned learning rate or standard step-size adaptation? This is needed to attribute gains to meta-learning.
-
Can you confirm the PPO/SAC configurations and provide a comparison at matched environment steps?
6 comments
Weakness 1 & Question 1 (2/2). Local adds gains beyond learned controls
Thank you for asking where Local consensus provides a substantial benefit. w/o-transfer retains base-control learning and switches off Local transfer.
In the original 10-run policy-search experiment, GemNES significantly improves mean reward over w/o-transfer by 11.4% on HalfCheetah and 30.3% on Ant.
| Variant | Tasks | + / - / = vs GemNES |
|---|---|---|
| w/o-transfer | 10 | 0 / 3 / 7 |
Full taskwise reward means (SDs) and significance results appear in Appendix F.4, Table A-XIV. Counts + / - / = indicate that the comparator is significantly better / worse / not significantly different from GemNES (two-sided Wilcoxon rank-sum, alpha = 0.05). Removing affinity filtering also significantly lowers Ant reward from 2658.3 (691.0) to 2250.2 (604.0).
Eq. (12) makes the link between Global and Local explicit. Global's learned step sizes enter the base and transfer directions, and combines its learned rate with Local's filtered, suppressed direction. The following diagnostics test the gains of these realized updates.
To check whether Local helps simply by making updates larger, we added an equal-length comparison. We compared the learned-rate update with zero transfer and with an isolated update rescaled to the full step's Euclidean length in coordinates. Gradient estimates and learned base controls are identical in these one-step comparisons. In Corollary 2, Eq. (15), measures the alignment gained by changing direction. Uniform positive rescaling gives .
We used CEC19-MaTSO1 (high global similarity, Rosenbrock tasks) and WCCI22-MaTSO10 (low global similarity, mixed-function tasks). With 5 trajectories, 50 tasks, and 20/50/80% checkpoints, we obtain 750 states per instance. To test whether the gains last, we also compared learned with zero transfer over 40 further all-task updates from 20/50% checkpoints. Both branches follow the same Global learning rule, with controls adapting to their own states. One-step and fresh endpoint evaluations use 3 matched antithetic batches of 50 samples.
| Comparison | Instance | Relative objective gain (%), mean (SD) | P / Z / N |
|---|---|---|---|
| Learned vs zero, one step | CEC19-MaTSO1 | 0.0061 (0.0036) | 700 / 50 / 0 |
| Learned vs zero, one step | WCCI22-MaTSO10 | 0.0505 (0.0944) | 413 / 335 / 2 |
| Full vs equal-length isolated | CEC19-MaTSO1 | 0.0059 (0.0035) | 700 / 50 / 0 |
| Full vs equal-length isolated | WCCI22-MaTSO10 | 0.0452 (0.0753) | 403 / 344 / 3 |
| Learned vs zero, 40 updates | CEC19-MaTSO1 | 5.6869 (6.5431) | 467 / 33 / 0 |
| Learned vs zero, 40 updates | WCCI22-MaTSO10 | 8.2545 (7.8960) | 378 / 114 / 8 |
Positive gain means a lower objective value than the reference. Entries are mean (SD) across 5 trajectory medians. P/Z/N count intervals above zero, containing zero, or below zero among 750 one-step or 500 endpoint states. These unadjusted pointwise 95% intervals use 1000 paired bootstrap resamples.
To test affinity separately, we compared filtered and unfiltered updates using the same samples and controls. Filtering yields positive 95% intervals for the change in measured , the alignment term in Theorem 1, Eq. (13), in 87.9% (14.2%) and 56.7% (27.0%) of states. Percentages are mean (SD) across trajectories (Appendix G.1).
After measuring objective gains in , we tested the theoretical gain condition using frozen-rank utility at 3 preselected tasks and the 20/50% checkpoints of the same 5 trajectories. We fix Global's learned step sizes and the realized Local direction, retaining each reference batch's ranks and sample-dependent shaping along the path. Corollary 3 tests the utility gain at learned relative to zero transfer. A positive slope bound certifies gain throughout by Eq. (20). This holds at 13/27 and 20/21 active states, with 95% simultaneous coverage over the tested active subset. The other 12 states have . Appendix G.2, Table A-XXIV reports counts by checkpoint.
The equal-length tests support Local's directional contribution, and the 40-update tests show short-term gains as Global continues to adapt. Eq. (20) connects this evidence to theory by certifying positive surrogate gains from the combination of learned Global controls and Local directions at the passing states.
Weakness 2 & Question 2. ES/RL interaction budgets and configurations
Weakness 2
The comparison with RL baselines in Table 1 is not matched in budgt. The paper states that proposed GemNES attains significantly higher reward than classical RL baselines, PPO and SAC, on 6 and 8 of 10 tasks. But each ES method and RL methods different environment per steps, and GemNES is therefore compared against methods given up to 5-25 times fewer environment interactions. Could you verify the RL setups and either compare at matched environment steps or remove the claims of superiority over classical RL?
Question 2
Can you confirm the PPO/SAC configurations and provide a comparison at matched environment steps?
Thank you for raising this concern. ES and PPO/SAC use data and computation differently, so fairness depends on which cost is matched. GemNES aggregates returns from parallel policy rollouts, whereas PPO/SAC reuse transitions for policy and value updates. Equal environment steps compare interaction efficiency. Wall-clock time also includes the different update costs.
Prior ES–RL work distinguishes sample efficiency from wall-clock performance (Salimans et al., 2017). LES (Lange et al., 2023) included PPO reference scores from Lu et al. (2022), obtained with default Brax settings, alongside ES learning curves.
GemNES uses a nominal budget of 50,000 policy evaluations per task, each averaging 5 rollouts of up to 1000 steps. Incumbent reevaluations count toward the same budget. GemNES checks the budget after each synchronized all-task generation and stops when it is reached. Early termination shortens the episodes.
Following your suggestion, we removed the general superiority claim over PPO/SAC and their win/loss summaries. Table 1 now compares ES methods. Table A-XII retains PPO/SAC taskwise means and standard deviations as reference results.
The mean end-to-end wall-clock times per run, averaged over 10 runs, are 8.04 h for GemNES, approximately 8.9 h for PPO, and 11.1 h for SAC (Figure A-1(b)). These training times are of a similar order, making terminal rewards useful references under the reported setup.
Both RL implementations use Brax 0.13.0, with the following settings.
| Setting | PPO | SAC |
|---|---|---|
| Configured training steps | 50,000,000 | 10,000,000 |
| Final logged training steps, every run | 56,197,120 | 10,005,760 |
| Runs per task | 10 | 10 |
| Policy hidden layers / activation | 4 × 32 / tanh | 4 × 32 / tanh |
| Learning rate / discount | 3e−4 / 0.99 | 3e−4 / 0.99 |
| Training environments / batch size | 2,048 / 1,024 | 128 / 256 |
| Maximum episode horizon | 1,000 | 1,000 |
| Recorded evaluation points / evaluation environments | 50 / 5 | 50 / 5 |
| Observation normalization / reward scaling | Enabled / 0.1 | Enabled / 0.1 |
| Evaluation policy setting in runner | Deterministic | Stochastic |
Section 4.1, Appendix F.3, and Table A-V state these configurations and the revised scope. The revised comparison provides performance references at these reported runtimes and method-specific budgets.
We sincerely thank you for the suggested experiments. We have updated and uploaded the manuscript, with all changes marked in blue.
Weakness 1 & Question 1 (1/2). Global IGC and attribution controls
Weakness 1
Question 1
Meta-learning provides the architecture, while Global IGC specifies how tasks jointly train the controller. In Section 3.1, each candidate meta-policy receives a score pooling normalized progress across tasks (Eq. (4)). This is the global consensus signal, favoring control rules that improve pooled task progress. The score's rank utility drives a natural-gradient update of the shared meta-policy distribution (Eq. (5)), following its Fisher geometry. The policy then maps each task's state to its own step sizes, shaping exponent, and ρ.
Step-size adaptation is therefore one effect of Global IGC. Local IGC forms consensus at the task-distribution level by aggregating affinity-selected peer natural gradients (Eqs. (6)–(7)). Together, they coordinate controls and directions using the same search trajectories.
1. Finer-grained ablation
The original w/o-meta disables learned base controls and Local transfer together, losing significantly on 14 synthetic instances and 6 MuJoCo tasks. We added rho-only to retain adaptive Local transfer with fixed base controls. The original w/o-transfer retains base-control learning and removes Local.
Both rho-only and w/o-meta use GemNES's base learning rates with mean/scale multipliers and shaping exponent fixed to (1,1,1). Rho-only retains affinity and suppression and learns ρ through Global progress. w/o-meta sets ρ=0.
We used 16 synthetic instances with 30 runs each. Each has 50 tasks of dimension 50, using 50 samples per update and 50,000 evaluations per task. In both tables, + / - / = counts comparators significantly better / worse / not significantly different from GemNES (two-sided Wilcoxon rank-sum, alpha = 0.05). Average Friedman ranks are within each 4-method comparison. Lower is better.
Full results are in Appendix F.4, Table A-XV. GemNES ranks first. Comparisons with rho-only and w/o-transfer assess base-control learning and adaptive Local transfer, respectively. With base controls fixed, rho-only improves mean UOV over w/o-meta on 12/16 instances and ranks better (2.56 vs 3.56), supporting adaptive Local transfer with fixed step sizes and shaping.
In Table 2, freezing step-size, shaping, and transfer-rate adaptation causes 8, 2, and 3 significant losses, respectively. w/o-ada-transfer fixes ρ=0.25, with 0/3/13 (+ / - / =). These results support all three controls.
2. Frozen, random, and tuned controls
The three requested controls use the same 16 instances and 30 evaluation runs each.
Full results are in Appendix F.5, Table A-XVI.
(a) FrozenRandom. Freezing the initial meta-policy weights preserves architecture, state feedback, and Local IGC. The results support online adaptation.
(b) RandomControl. We sample reward-independent outputs from a fixed bank of 256 joint control vectors per task, calibrated to the initial policy distribution. Local IGC is retained throughout. The results support learning from progress feedback.
(c) SNES-Tuned. We tested 9 mean/scale multiplier pairs in 0.5,1,22, using 3 development runs on WCCI22-MaTSO4 and 50,000 evaluations per task. Selection minimized log terminal error averaged across tasks and runs.
Appendix C.2 defines the variants, and Appendix F.5 details tuning. Frozen and random controls support online meta-policy adaptation within Global IGC. SNES-Tuned supports the combined framework against tuned fixed rates.