acceptodds
Under review as a conference paper at ICLR 2027

MIMUS: Post-Hoc Refinement of Frozen Rewards via Residual Evolution

Abstract

Reinforcement learning systems often rely on rewards that are costly to redesign and may not fully capture task objectives. Inspired by inheritance and variation in natural evolution, we introduce MIMUS, a post-hoc reward refinement framework that freezes an existing base reward and evolves a compact, low-rank additive neural residual. The residual is initialized to output zero, ensuring that evolution starts exactly from the base reward. Before evolution, a Signal Gate checks whether candidate reward comparisons reveal a reliable optimization signal amid policy-training noise. The residual is subsequently evolved using Covariance Matrix Adaptation Evolution Strategy (CMA-ES), guided by teaching fitness computed from the native returns or task-aligned metrics of policies trained under candidate rewards. This approach enables reward optimization without residual supervision or differentiation through policy training. Beyond the evolutionary search, long-horizon audits guide reward selection for final policy training. We evaluate MIMUS on four tasks in Gymnax and MJX with handcrafted, LLM-generated, and learned neural base rewards. Across these tasks and reward sources, MIMUS improves downstream policy performance while leaving the base rewards unchanged, demonstrating that residual evolution offers a practical complement to reward design.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Reward Optimization, Residual Reward Learning, Evolution Strategies, Black-Box Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1p1UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MIMUS for post-hoc reward refinement without modifying an existing reward function. The method freezes an arbitrary-form base reward RbaseR_{\mathrm{base}} and adds a learnable low-rank neural residual on top of it; the residual output head is zero-initialized, so the initial composite reward is functionally identical to the base reward. At the outer level, CMA-ES evolves the residual parameters; at the inner level, PPO is run for each candidate reward, with the performance of the post-training policy on task-level metrics used as the teaching fitness. The authors also introduce Signal Gate to check whether reward perturbations are distinguishable from PPO randomness, and use a shared anchor, multiple training seeds, and a long-horizon S→HS \rightarrow H audit to select the final reward. The experiments cover Reacher-misc and DeepSea-bsuite in Gymnax, and PointMass and FingerTurnEasy in MuJoCo Playground/MJX. The base rewards include native handcrafted rewards, rewards learned by Reward Networks (RN), and program rewards generated by Eureka. The main results report final population mean fitness values of 0.257, 0.923, 0.319, and 0.499 on the handcrafted rewards, respectively. Most results change only slightly in the long-horizon audit, although the best candidate on FingerTurnEasy decreases from 0.543 to 0.526. RN+MIMUS outperforms the RN reference on all four tasks, while the gains from Eureka+MIMUS are mainly visible for the best candidate and are small in several cases (e.g., 0.3437→0.3450 on PointMass). Overall, the paper presents a clear framework and a fairly detailed appendix, but the current evidence remains insufficient to support strong claims about search reliability and cross-task practicality.

Critical strengths
  1. The problem definition and motivation are clear. Freezing the base reward preserves its original formulation, while zero-output initialization exactly recovers it at the initial center. The shared policy anchor and the Rbase+αΔRR_{\mathrm{base}} + \alpha\Delta R interface provide a controlled starting point for refinement, although they do not guarantee preservation of policy performance after evolution.

  2. In the native handcrafted-reward experiments, the evaluation protocol accounts for the high variance of nested RL. Signal Gate, a shared seed bank across candidates, center-reward audits (with separate center-training seeds on three tasks in Table 6), and the long-horizon HH audit provide checks from short-term selection signals to performance at the declared longer training horizons. The appendix also reports a failed Signal Gate on WalkerWalk-MJX and returns the frozen base reward when the signal is insufficient, which is more honest than reporting successful cases only.

  3. The experimental and implementation details are relatively complete. The paper reports the observation/action/transition features, reward formulas, PPO configurations, candidate counts, training seeds, search dimensions, and residual scales for all four tasks, and provides ablations on residual/scratch, CMA-ES/OpenAI-ES, zero/random heads, and low-rank/dense parameterizations. The tables in Appendices B, C, D, and E allow readers to audit the computational budget and the meaning of the different estimands with reasonable clarity.

Critical weaknesses
  1. MIMUS has several distinctive features: it requires no residual labels, freezes an arbitrary-form base reward, and supports representation-agnostic refinement. However, its additional algorithmic novelty beyond this problem setting and the reliability protocol is less clear. As Section 2 notes, Reward Networks (Ferreira et al., 2022) also evaluate evolved neural rewards through the policies they teach, whereas the residual-reward methods discussed there use different supervision. A sharper comparison of this distinction and the added value of the refinement protocol would strengthen the novelty argument.

  2. Appendix D, Table 11 shows that the smallest passing scale for Reacher-misc is σ0=0.1\sigma_{0} = 0.1, whereas the main search reported in Appendix C.2/Table 5 and Figure 7 starts from σ0=0.02\sigma_{0} = 0.02; the authors also explicitly acknowledge that at least one criterion fails at 0.02. In addition, signal diagnostics/preflight are explicitly disabled for RN refinement on PointMass. Consequently, the reliability checks provided by Signal Gate were not applied consistently across the reported experiments, and the main Reacher result uses a configuration that does not pass the gate.

  3. The experiments cover four simulated tasks. The main residuals use width 16 and rank 4, with additional width-64 low-rank/dense comparisons. External fitness is task-specific: continuous-control tasks use designed success/quality metrics, while DeepSea uses success rate with a progress tie-break during selection. The paper itself acknowledges horizon reversal on FingerTurnEasy and that WalkerWalk cannot pass the gate. Improvements over RN/Eureka are reported mainly as the final population mean/best candidate rather than a uniform independent final test, and the wrapper/identity of some base artifacts still contain “open” fields in the appendix. The additive interface supports different base-reward representations, but evidence for reliable practical gains beyond the tested tasks, bases, and budgets remains limited.

Questions for authors
  1. Why does the main Reacher search use σ0=0.02\sigma_{0} = 0.02 despite failing Signal Gate, and would using the passing scale of 0.1 change the conclusions?

  2. How do residual rank, width, α\alpha, and input features affect correction capacity, and how do you detect residuals that undermine the base reward's intended incentives?

  3. Is comparison reliability rechecked as CMA-ES moves away from its initial center, and what triggers additional evaluation or termination when candidate–seed interactions dominate?

  4. What quantitative rule defines unsupported improvement or degradation, and how does it apply to FingerTurnEasy's observed decline from SS to HH?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 1p1U

Response to Reviewer 1p1U

Thank you for your careful, encouraging review and for recognizing our clear problem definition, variance-aware protocol and honest WalkerWalk report. All revisions are highlighted in blue in the updated PDF.


Weakness 1:

Novelty beyond the problem setting and protocol is unclear relative to Reward Networks and residual-reward methods.

Response:

We agree that MIMUS shares the outer loop of Reward Networks (RN; Ferreira et al., 2022) and the additive form of residual-reward methods (Cao et al., 2025; Hatgis-Kessell et al., 2026). What is new is their combination: post-hoc refinement of any existing reward without labels (New content highlighted; see Related Work and Appendix "Detailed Comparison with Prior Work", Table 2):

Table R1RNResidual rewardsMIMUS
Basenative rewardfixed proxyany frozen reward (program, LLM, learned)
Start of searchrandom network—exactly the base (zero head)
Search spacefull networkfull networklow-rank gene (280–2,297 dims)

1. It refines the outputs of other reward-design methods. On fresh seeds, MIMUS improves a frozen RN artifact (Reacher +0.027 to +0.049 at H, 4 outer runs) and a frozen Eureka program (+0.042 to +0.063) (Appendix "Fresh-Seed Reward Audits", Table 17).

2. Starting exactly at the base matters. A random head drops the initial Reacher fitness from 0.148 to 0.006, and removing the base collapses the search (final mean 0.008 vs 0.199) (Ablation Studies).


Weakness 2 & Question 1:

Why was Reacher run at σ₀ = 0.02, which fails the gate, and would 0.1 change the conclusions?

Response:

This inconsistency was real: σ₀ = 0.02 came from an early native-return sweep and was not re-calibrated for the task fitness. All Reacher results except the width-64 pair are re-run at σ₀ = 0.1 (Revised; see Table 1, Figures 2–4, and Appendix "Fixed-Population Signal Diagnostics"):

Table R2 (Reacher)Signal CheckFinal meanFresh-seed Δ at H
σ₀ = 0.02 (submitted)weak0.257+0.051
σ₀ = 0.1 (revised)strong0.199+0.028 to +0.051 (3 outer)

Conclusions do not change; the weak-signal σ₀ = 0.02 even searches better. PointMass is re-checked with its search fitness, and PointMass-RN has its own check (Appendix "Confirmation and Interpretation"). We therefore revised the procedure: the Signal Check now guides σ₀, K and N as an auxiliary diagnostic, not a pass/fail gate, and the fresh-seed test decides whether a reward is kept (Revised; see "Signal Check Before Evolution" and Algorithm 1).


Weakness 3 & Question 4:

No uniform independent test for RN/Eureka; what rule defines degradation, e.g., for FingerTurnEasy?

Response:

1. Uniform test. Every task–base pair is retrained against its base on unused seeds at S and H (New content highlighted; see Table 1 and Appendix "Fresh-Seed Reward Audits", Tables 16–17).

2. Rule. If the paired 95% CI of Δ = R* − base at H lies above 0, the gain is supported; below 0, it is a degradation; otherwise it is unsupported and the base is kept (New content highlighted; see "Long-Horizon Reward Audit and Selection").

3. FingerTurnEasy. Δ is +0.099 [+0.091, +0.106] at S and +0.000 [−0.010, +0.011] at H, so the H-stage gain is unsupported: R* does not degrade (0.562 → 0.572), but the base catches up (0.463 → 0.571).

Artifact identities are complete (New content highlighted; see Appendix "RN Configuration", Table 10); claims are limited to the four tasks.


Question 2:

How do rank, width, α and features affect capacity, and how are harmful residuals detected?

Response:

Rank-4 beats same-width dense residuals on at least one statistic on every task with 4.9–12.3× fewer parameters, while widening from 16 to 64 helps on two tasks and hurts on one (Appendix "Component and Representation Ablations", Tables 20–21). We did not vary α or the input features separately (Limitations). Harmful residuals are visible because the evaluator excludes the residual and every component is reported: e.g., PointMass success rises 0.80 → 0.98 while native return drops 0.5% (New content highlighted; see Appendix "Fresh-Seed Reward Audits", Table 18).


Question 3:

Is reliability rechecked as CMA-ES moves away from the initial center?

Response:

Not in the submission; we now recompute it at every generation from archived scores (New content highlighted; see Appendix "Reliability Along the Search", Table 14). The candidate/interaction ratio stays above 1 in all 350 native and RN Reacher generations, but falls below 1 in 89/200 Eureka generations and in all σ₀ = 0.02 generations. Such searches are flagged for more seeds (Algorithm 1).


We sincerely thank Reviewer 1p1U for the constructive review and would gladly discuss any remaining questions.

Reviewer_GJHUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MIMUS, a reward-refinement approach that freezes an existing base reward and adds a bounded neural residual with low-rank hidden layers and zero-output initialization. CMA-ES optimizes this residual based on the task-level performance of policies trained under each candidate reward. To reduce comparison noise and validate outcomes, the method uses shared policy anchors and training seeds, a preliminary Signal Gate to assess whether candidate differences are reproducible, and longer training runs to audit selected rewards. The authors evaluate the approach across four simulated tasks using native, learned (Reward Networks), and LLM-generated (Eureka) base rewards. While the residual formulation is sensible, the reported experiments show significant discrepancies from the stated validation protocol and do not yet establish reliable improvement, long-horizon persistence, or consistent refinement across reward sources.

Critical strengths

Practical residual interface and attention to experimental difficulties: Adding a separate residual with zero-output initialization accommodates both reward programs and neural rewards without modifying their underlying implementation, while guaranteeing exact recovery of the base reward at the initial center. Furthermore, the framework directly targets genuine experimental challenges in reward evolution—such as noisy inner-loop training, short-horizon selection bias, perturbation scale, and learner state restoration—and establishes a valuable conceptual distinction between selection, validation, and independent testing (Appendix A.4).

Transparent reporting of qualifications and negative results: The appendices openly document confounded ablations, protocol exceptions, mixed low-rank results, and an unsuccessful WalkerWalk experiment. This transparency is crucial for understanding the behavior and current limitations of the method, even though several of these qualifications are not adequately reflected in the main text's conclusions.

Critical weaknesses

Insufficient independent validation and incomplete long-horizon controls: The empirical evaluation relies on a single outer CMA-ES search seed per task (with figure bands reflecting variation across per-seed population means rather than outer-loop reproducibility) and fails to evaluate fixed selected rewards on untouched policy-training seeds. In FingerTurnEasy, training seeds 101–104 are reused for both candidate search and center evaluation. For RN and Eureka refinement, claims rely on population averages and optimistic maximums rather than independent testing of a frozen selected reward. Furthermore, the long-horizon audits (Table 1) omit the matched base reward at the long horizon, making it impossible to verify whether the base policy catches up or overtakes the refined reward—a critical gap given the FingerTurnEasy horizon reversal acknowledged in Appendix F.

Protocol discrepancies in the Signal Gate and missing equal-compute baselines: The reported experiments frequently deviate from the advertised Signal Gate protocol: Reacher-misc uses an initial scale (0.02) that fails the gate rather than the passing scale (0.1); PointMass evaluates native return under an earlier protocol during gating while optimizing a time/effort fitness in the main search; and PointMass RN refinement disables signal diagnostics and preflight entirely. Because the gate is not consistently applied to the actual search configurations, the experiments cannot establish that it drives reliable refinement. Additionally, given the substantial compute cost (e.g., 12,800 candidate-seed trainings and ~1.05 trillion transitions for one Reacher search) and prior work on Reward Networks, the paper lacks necessary equal-compute search baselines (such as spending equivalent added budget on baseline RN/Eureka optimization, random search in the residual space, or parameter-matched dense/linear residuals) and clean ablations (the frozen-prior ablation confounds base removal with output-head initialization changes).

Questions for authors

Independent Testing and Outer Search Repetitions: What are the final, untouched-test results for a single frozen retained reward in each task and reward-source combination, and how many independent outer evolutionary searches were performed (as referenced in Appendix E.2 but missing from the text)?

Long-Horizon Base Controls and Reversals: How does the refined reward compare against the matched base reward trained to the exact same long horizon on each task (across external fitness, native return, and component metrics)? What precisely is the FingerTurnEasy horizon reversal mentioned in Appendix F, and how does it affect the claim in Section 5.2 that teaching quality survives convergence?

Signal Gate Consistency: Which reported runs strictly followed the complete Signal Gate procedure without exceptions, and can you provide validation diagnostics for the actual search configurations used in the main Reacher and PointMass experiments?

Provenance, Contradictions, and Compute Accounting: Can you provide the exact frozen RN artifact identities and reward compositions omitted in Appendix C.5, clarify the contradictions between the prose and tables (specifically Appendix E.5 vs. Table 16 regarding KL/clip fractions, and Appendix E.6 vs. Table 15 regarding separable vs. full covariance), and report end-to-end wall-clock times and GPU-hours?

2 comments
AuthorsOpenReview ↗

Response to Reviewer GJHU

Response to Reviewer GJHU

Thank you for your rigorous review and for valuing our transparent reporting of negative results, which we kept throughout. All revisions are highlighted in blue in the updated PDF.


Weakness 1 & Questions 1, 2:

Single outer search per task, no untouched-seed test of a frozen reward (FingerTurnEasy reuses seeds 101–104), and no matched base at the long horizon.

Response:

We fully agree, and we corrected the FingerTurnEasy description.

1. Untouched tests. Every task–base pair now freezes R* (final CMA-ES center) and retrains it and its matched base from the same anchor on unused seeds (FingerTurnEasy: 5001–5020) (New content highlighted; see Table 1 and Appendix "Fresh-Seed Reward Audits", Tables 16–17):

Table R1: Δ at HNativeRNEureka
Reacher-misc+0.028 to +0.051+0.027 to +0.049+0.042 to +0.063
PointMass+0.069+0.048, +0.068−0.003, +0.005
FingerTurnEasy+0.000−0.024−0.005
DeepSea (success)0 → 10.5 → 1.01.0 → 1.0

2. Outer searches. Reacher has 3 / 4 / 4 searches for the native / RN / Eureka bases, all positive at H; PointMass has 2 per learned base. FingerTurnEasy and DeepSea repetitions finish tomorrow and will follow here (New content highlighted; see Appendix "Independent Outer Searches", Table 24).

3. Matched base at H. Table 1 lists base and R* at S and H, with native return and components in Table 18: e.g., PointMass success rises 0.80 → 0.98 while native return drops 0.5%.

4. FingerTurnEasy reversal. On 20 fresh seeds, R* leads at S (+0.099 [+0.091, +0.106]) and still improves from S to H (0.562 → 0.572), but the base catches up (+0.000 [−0.010, +0.011]). Refinement speeds up learning without changing the converged quality; we removed the claim that teaching quality survives convergence on every task (Revised; see "Refinement of Handcrafted Rewards").


Weakness 2 & Question 3:

Signal Gate deviations (Reacher σ₀ = 0.02, PointMass native-return gating, PointMass-RN disabled); which runs followed it?

Response:

Thank you for this precise audit. In the submission no run followed the procedure without exception. We re-ran Reacher at the check-selected σ₀ = 0.1, recomputed the PointMass check with its search fitness plus an independent confirmation, and added a PointMass-RN check. Each search is now listed with the check on its exact configuration (New content highlighted; see Appendix "Signal Check Coverage", Table 27), and the diagnostics are recomputed at every generation of every archived search (Appendix "Reliability Along the Search", Table 14). The check is now an auxiliary diagnostic; the fresh-seed audit decides whether a reward is kept (Revised; see "Signal Check Before Evolution").


Weakness 2 (baselines and ablations):

Missing equal-compute baselines and a clean base-removal ablation.

Response:

1. Random and null search. With the same K, N, budget, seeds and evaluator, the final population mean on PointMass is 0.319 (MIMUS) vs 0.212–0.218 (three random searches) and 0.223 (null), and on FingerTurnEasy 0.499 vs 0.369 (New content highlighted; see Appendix "Equal-Compute Random and Null Search", Table 25). Reacher and DeepSea random searches follow tomorrow.

2. Clean base removal. The random-head (base kept) and scratch (base removed) arms share a random output head of scale 0.5, so they differ only in the base: removing it drops AUC from 0.161 to 0.003 on Reacher and from 0.661 to 0.423 on DeepSea (New content highlighted; see Appendix "Component and Representation Ablations").

3. Not run. We did not extend RN/Eureka construction with the added budget (8–40× the Eureka search's policy trainings) or test parameter-matched residuals; both are stated as limitations (Limitations).


Question 4:

Artifact identities and compositions, the KL/clip and covariance contradictions, and compute.

Response:

Both contradictions were prose errors, now corrected.

  • Artifacts: SHA-256 identities of all eight frozen bases and the RN composition native + γΦ(s′) − Φ(s) (New content highlighted; see Appendix "RN Configuration", Table 10).
  • KL/clip: the table (Table 16 in the submission) now uses 20 fresh seeds: lower KL and clip fraction at S, comparable at H (Revised; see Appendix "Inner-Loop Training Diagnostics", Table 22).
  • Covariance: the table (Table 15 in the submission) was correct; only the DeepSea width-64 pair is separable (Revised; see Appendix "Component and Representation Ablations" and "Outer-Loop Optimizer Diagnostics").
  • Compute: one search takes 0.3 h (DeepSea) to 39.7 h (FingerTurnEasy) on 1–2 GPUs (New content highlighted; see Appendix "Compute and Wall-Clock Time", Table 26).

We sincerely thank Reviewer GJHU for the rigorous review and would gladly discuss any remaining questions.

AuthorsOpenReview ↗

Follow-up: remaining outer searches and random-search baselines

Follow-up: remaining outer searches and random-search baselines

As promised, the remaining runs have finished, and the updated PDF includes them (New content highlighted; see Appendix "Independent Outer Searches", Table 24, and Appendix "Equal-Compute Random and Null Search", Table 25).


1. Independent outer searches (Weakness 1 & Questions 1, 2)

Each repetition changes only the outer CMA-ES seed, and its final center is audited against the matched base on unused seeds:

Table R3: Δ at H, per outer searchNativeRNEureka
Reacher-misc+0.051, +0.049, +0.028+0.047, +0.032, +0.049, +0.027+0.052, +0.049, +0.063, +0.042
DeepSea (success)0 → 1 (all 3)0.5 → 1.0 (all 3)1.0 → 1.0 (all 3)
PointMass+0.069+0.048, +0.068−0.003, +0.005
FingerTurnEasy+0.000−0.024, −0.024−0.005, −0.056

Of the ten task–base settings with repeated searches, nine reproduce the first search: all 20 Reacher and DeepSea searches keep a positive or saturated endpoint, PointMass repeats its RN gain and Eureka parity, and FingerTurnEasy-RN repeats its gain at S (+0.084, +0.089) and its small decrease at H.

The exception is FingerTurnEasy-Eureka. The second search gives no gain at S (+0.001) and a degradation at H (−0.056 [−0.074, −0.040], 4/4 seeds negative). Our decision rule rejects this reward and keeps the Eureka base. The per-generation check had already flagged this setting as weak (Table 14), and the base is the strongest FingerTurnEasy reward, leaving little headroom. We report this case explicitly (Revised; see Appendix "Fresh-Seed Reward Audits", Table 17, and Limitations).

PointMass and FingerTurnEasy with the native base keep one search each (28–40 wall-clock hours per search), audited on 12 and 20 fresh seeds.


2. Equal-compute random search (Weakness 2)

Table R4: final population meanMIMUSRandom searchNull
Reacher-misc (base 0.180)0.162–0.202, AUC 0.171–0.196 (3 outer)0.165, AUC 0.163—
DeepSea (success)0.923–0.977 (3 outer)0.146—
PointMass0.3190.212–0.218 (3 seeds)0.223
FingerTurnEasy0.4990.369—

Null search is run on PointMass only: all candidates share the base reward and a fixed seed bank, so it has no spread by construction and reduces to one base training; random search carries the comparison on all four tasks. Random search samples around the base in every generation, so it measures what selection alone buys. Its population does not move (Reacher 0.163 → 0.165 over 50 generations, while every MIMUS search rises), and on DeepSea, PointMass and FingerTurnEasy it ends far below MIMUS. Its single best candidate stays well below the MIMUS best on PointMass and FingerTurnEasy, but is comparable on Reacher (best-ever 0.276 vs 0.278–0.296) and DeepSea (success 1 in both), where improving directions near the frozen base are easy to find. On fresh seeds, the Reacher random-search best also improves the base at H (+0.040 [+0.007, +0.070]), within the MIMUS range, so the base-anchored low-rank residual space itself holds usable directions. Its search-stage score is optimistic, though (0.276 → 0.206 on fresh seeds at S; MIMUS best 0.274 → 0.216), whereas the retained MIMUS center does not shrink (0.203 → 0.220). This is why MIMUS keeps the center and decides by the fresh-seed audit.


We hope these results complete the evidence you requested, and we would be glad to discuss any remaining concerns.

Reviewer_At2BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies post-hoc refinement of an existing reward function. MIMUS keeps a base reward frozen and adds a small neural residual, whose output head starts at zero so that the initial reward equals the base. CMA-ES searches the residual parameters, and each candidate is scored by training PPO policies from a shared starting point and measuring a task-specific "teaching fitness". Before the search, a Signal Gate checks on a fixed set of perturbations whether candidate rankings are reproducible across training seeds. After the search a long-horizon audit retrains the selected reward for longer. The experiments use four small tasks from Gymnax and MuJoCo Playground and three kinds of base reward: the native reward, a Reward Network (Ferreira et al., 2022) and a Eureka program generated with DeepSeek-V4.1-Flash. With the native reward as base, the final population mean ends above the base on all four tasks, for example 0.180 to 0.257 on Reacher and 0.000 to 0.923 on DeepSea. Gains are also reported when RN and Eureka rewards serve as the base, although for Eureka they are small.

Critical strengths

1. The problem setting is practical.

Many RL systems already have a reward that works reasonably well. Refining it through an additive residual, without touching the original program or network, gives one interface for handcrafted, learned and LLM-generated rewards.

2. The Signal Gate addresses a real weakness of black-box reward search.

Comparing candidate effects with candidate-by-seed interactions before spending the search budget is a sensible check. The WalkerWalk example in Appendix F shows the gate stopping a search that would otherwise have chased seed noise.

3. The setting is detailed and informative while the experimental results are good.

The appendix lists seed banks, budgets, the Eureka prompts and the cases where runs deviated from the protocol, which is what made most of the concerns below checkable.

Critical weaknesses

1. My main concern is that the gains are measured inside the search, and the retained reward is never tested against the base on fresh seeds.

The headline numbers are the final population mean and best candidate (lines 342 and 456–457). All stages reuse the same seed banks (Table 6), even the FingerTurnEasy audit that line 345 calls independent. The resulting optimism shows on Reacher, where the best candidate is close to 0.30 at the end of the search (Figure 2a) but scores 0.234 when retrained (Table 1). The fresh-seed test of Algorithm 1 (line 780) and the confidence intervals promised in Appendix E.2 (line 1486) do not appear, and Table 1 omits the base reward at H. The one comparison at H favors the base, with a native return of 425.0 against 410.7 for the center reward on FingerTurnEasy (Table 16), which conflicts with the claim that "the gains largely persist" (line 479).

2. The comparisons are not matched in compute, and the base references move with the protocol.

One MIMUS run on Reacher trains 12,800 PPO policies (about 10^12 environment transitions), and Eureka+MIMUS uses 8 to 40 times more policy trainings than the Eureka search that produced its base (Table 9), but the paper reports no total compute. A fair comparison would give Eureka, RN, PPO tuning or random search over the residual the same budget. The frozen Eureka reward for Reacher also scores 0.418 in its construction run (Appendix E.3) but 0.337 as the reference (line 464), a larger change than any gain reported for the Eureka bases.

3. The Signal Gate, a core part of the method, was not applied as described.

The Reacher search ran at σ0=0.02σ_0 = 0.02, where the gate fails all three criteria, instead of the gate-selected 0.1 (lines 1384–1388). The PointMass gate scored a different fitness from the one the search used (lines 1448–1450). And the PointMass RN run disabled the gate (line 1452). The gate also filtered the tasks, since Cartpole and WalkerWalk were tried and dropped (lines 1457 and 1681), so the claims should be limited to tasks where reward effects exceed PPO seed noise.

4. The paper presents the problem as new, but I look at this as a meta-learning of a reward signal, and that literature is barely discussed.

The inner loop trains a policy, the outer loop scores it with the task fitness, and CMA-ES updates the residual from this validation signal. This is the optimal reward problem of Singh et al. (2009, 2010), restricted to a base plus a residual. Apart from one sentence on meta-gradient rewards (lines 139–141), the paper omits work that meta-learns RL objectives with evolution strategies, such as Evolved Policy Gradients (Houthooft et al., 2018) and Discovered Policy Optimisation (Lu et al., 2022). Reward Networks (Ferreira et al., 2022), the closest prior work, already evolve neural rewards that augment the native reward, with the trained agent as fitness (line 143). The meta-learning view also explains point 1, because selection at an early endpoint is a truncated meta-objective, and such objectives are known to favor solutions that look good early (Wu et al., 2018).

5. The ablations give weak support to the choices that distinguish MIMUS from Reward Networks.

The random output head ends higher than the zero head on all four tasks (Table 14), the low-rank comparison uses width 64 instead of the width 16 of the main experiments, and OpenAI-ES beats CMA-ES on DeepSea. Each ablation is a single outer run, and many AUC differences are only 0.01 to 0.02.

6. The experiments are toy-scale compared with the robotics settings that motivate the paper.

All four tasks have two-dimensional actions, three have observations of 4 to 12 dimensions, and the fourth is a 16 × 16 grid (Table 3), while the one locomotion task that was tried, WalkerWalk, failed the gate. One quadruped, humanoid or dexterous-manipulation task, from MuJoCo Playground or the Eureka benchmark, would do more for the claim of "a practical complement to reward design" than further small tasks.

Questions for authors
  1. Could the authors report the retained reward R* against the base reward on fresh policy-training seeds and fresh evaluation seeds, with confidence intervals? Algorithm 1 (line 780) and Appendix E.2 (line 1486) refer to such results.

  2. What do the population-mean and best-candidate statistics look like for a null search, in which every candidate uses the base reward (zero residual) with the same K, N and seed banks? This would show how much of the best-candidate gain comes from selection alone.

  3. How were the S values in Table 1 obtained? For PointMass and FingerTurnEasy they equal the in-search peaks of Table 13 (0.344 and 0.543), while on Reacher the retrained best candidate is clearly below its in-search value.

  4. What is the total compute of each MIMUS run (policy trainings, environment steps, GPU-hours)? How do Eureka, RN, PPO tuning under the base reward and random search over the residual perform with the same number of policy trainings?

  5. Why was the Reacher search run at σ0 = 0.02 rather than the gate-selected 0.1, and what happens at 0.1? How were the gate thresholds chosen, and which tasks were attempted in total?

  6. What is the ratio of residual to base-reward magnitude for the final retained reward on each task?

  7. How does MIMUS compare with the additive variant of Reward Networks when both use the same optimizer and budget?

  8. Is it possible to have results on at least one higher-dimensional task, for example a quadruped, humanoid or dexterous-manipulation task from MuJoCo Playground or from the Eureka benchmark?

  9. How well do candidate rankings at S agree with rankings at H, for example over a small sample of candidates retrained to H? This would measure the short-horizon bias of selecting at S directly.

4 comments
AuthorsOpenReview ↗

Response to Reviewer At2B (1/2)

Response to Reviewer At2B (1/2)

Thank you for this exceptionally careful review. Your audit of the appendix found real inconsistencies, and fixing them made the paper substantially more rigorous. All revisions are highlighted in blue in the updated PDF.


Weakness 1 & Questions 1, 3:

The retained reward is never tested against the base on fresh seeds. ... Could the authors report R* against the base on fresh policy-training and evaluation seeds, with confidence intervals? How were the S values in Table 1 obtained?

Response:

1. Fresh-seed test (Q1). We fully agree. Every task–base pair now freezes R* (final CMA-ES center) and retrains it and its matched base from the same anchor on seeds unused by the search, with paired 95% bootstrap CIs; Table 1 now lists both arms at S and H. The FingerTurnEasy audit we called independent reused the search seeds 101–104; it now uses 5001–5020 (New content highlighted; see Table 1 and Appendix "Fresh-Seed Reward Audits", Tables 16–17):

Table R1: Δ at HNativeRNEureka
Reacher-misc+0.028 to +0.051 (3 outer)+0.027 to +0.049 (4)+0.042 to +0.063 (4)
PointMass+0.069 [+0.034, +0.107]+0.048, +0.068−0.003, +0.005
FingerTurnEasy+0.000 [−0.010, +0.011]−0.024, −0.024−0.005, −0.056
DeepSea (success)0 → 1 (3 outer)0.5 → 1.0 (3)1.0 → 1.0 (3)

Gains persist at H on Reacher with all three bases, on PointMass with the native and RN bases, and on DeepSea. FingerTurnEasy gains only at S, and one FingerTurnEasy-Eureka search degrades at H, so the decision rule keeps its base (New content highlighted; see Appendix "Independent Outer Searches", Table 24).

2. FingerTurnEasy at H. The conflict you found is real. The cited row of Table 16 was the first of four replicas, not their mean (427.1 vs 426.9), and we removed "the gains largely persist". On 20 fresh seeds, R* matches the base in fitness at H and has a higher native return (+13.2 [+4.1, +22.2]); its advantage is faster learning by S (+0.099 [+0.091, +0.106]).

3. S values in Table 1 (Q3). They mixed two protocols: the PointMass and FingerTurnEasy best-candidate cells were copied from the search, while Reacher used a retraining whose learning rate annealed to zero at update 200 (the search schedule is still at 60% there). The Reacher gap you noted is also selection optimism: at σ₀ = 0.1 the best candidate falls from 0.274 to 0.216 on fresh seeds at S, whereas the retained center does not shrink (0.203 → 0.220). Table 1 now reports only the fresh-seed audit (Revised; see Table 1).


Weakness 2 & Questions 2, 4:

The comparisons are not matched in compute, and the base references move with the protocol. ... null search, total compute, and equal-budget alternatives?

Response:

1. Null and random search (Q2). With the same K, N, budget, seeds and evaluator (New content highlighted; see Appendix "Equal-Compute Random and Null Search", Table 25):

Table R2: final population meanMIMUSRandomNull
Reacher (base 0.180)0.162–0.202 (3 outer)0.165—
DeepSea (success)0.923–0.977 (3 outer)0.146—
PointMass0.3190.212–0.218 (3 seeds)0.223
FingerTurnEasy0.4990.369—

Null search is run on PointMass only: all candidates share the base reward and a fixed seed bank, so it has no spread by construction and reduces to one base training; random search carries the comparison on all four tasks. The random-search population does not move (Reacher AUC 0.163 vs 0.171–0.196). Its best Reacher candidate still improves the base on fresh seeds (+0.040 [+0.007, +0.070] at H), so the base-anchored residual space itself contains usable directions, but its search score is just as optimistic (0.276 → 0.206).

2. Compute (Q4). One search takes 2,560–12,800 policy trainings and 0.3–39.7 wall-clock hours on 1–2 GPUs (New content highlighted; see Appendix "Compute and Wall-Clock Time", Table 26).

3. Budgets and references. Refinement uses 8–40× the policy trainings of the Eureka search that built its base, so we present RN/Eureka+MIMUS as a further refinement stage, not a cheaper alternative; we did not extend Eureka/RN construction or tune PPO at that budget (Limitations). The references do move: the frozen Reacher Eureka reward scored 0.418 as the construction-search maximum, 0.337 retrained on seeds 42–45, and 0.284 on fresh seeds at S. This is why every claim is now a paired comparison within one protocol.

AuthorsOpenReview ↗

Response to Reviewer At2B (2/2)

Response to Reviewer At2B (2/2)

Weakness 3 & Question 5:

The Signal Gate was not applied as described. ... Why σ₀ = 0.02 rather than 0.1, how were the thresholds chosen, and which tasks were attempted in total?

Response:

All three deviations were real and are corrected.

  • σ₀. 0.02 came from an early native-return sweep, never re-calibrated for the task fitness. All Reacher results except the width-64 pair are re-run at the check-selected 0.1 (Revised; see Table 1 and Appendix "Fixed-Population Signal Diagnostics"). The weak-signal 0.02 search ends higher (0.257 vs 0.199) with a similar fresh-seed gain (+0.051 vs +0.028 to +0.051), so the check now guides σ₀, K and N as an auxiliary diagnostic, not a gate (Revised; see "Signal Check Before Evolution").
  • PointMass. The check is recomputed with the search fitness, plus a confirmation; the PointMass-RN refinement, run with the check disabled, now has its own check (New content highlighted; see Appendix "Signal Check Coverage", Table 27).
  • Thresholds. ρ = 0.60 is a conventional strong correlation, 0.625 exceeds the 0.375 expected from random signs with N = 4, and a ratio of 1 means candidate effects match candidate–seed interactions; being heuristic, they are diagnostic only.
  • Tasks. With the full method, we ran three tasks beyond the four reported. WalkerWalk and Hopper (MJX) failed the Signal Check because PPO seed noise exceeded the reward effect; CartpoleBalanceSparse passed it, but its search population showed no consistent direction, so we stopped it. Candidate tasks are first admitted by a native-PPO audit (New content highlighted; see Limitations).

Weakness 4 & Question 9:

I look at this as a meta-learning of a reward signal ... optimal reward problem, EPG, DPO ... selection at an early endpoint is a truncated meta-objective (Wu et al., 2018). How well do candidate rankings at S agree with rankings at H?

Response:

1. Framing. We adopt it. MIMUS is an instance of the optimal reward problem [1, 2] that starts exactly at a frozen base, searches a low-rank gene by black-box ES rather than meta-gradients [3], and meta-learns a reward rather than a loss or update rule [4, 5] (New content highlighted; see Related Work and Appendix "Detailed Comparison with Prior Work").

2. Short-horizon bias (Q9). We retrained eight final-generation candidates to S and H (New content highlighted; see Appendix "S-to-H Rank Agreement", Table 28):

Table R3Spearman ρTop candidate at S
Reacher, σ₀ = 0.10.81top two stay top two
Reacher, σ₀ = 0.020.60kept
DeepSea0.66kept
PointMass-RN0.57sixth at H
FingerTurnEasy-RN0.40kept

S rankings are informative but imperfect, as [6] predicts, so the audit at H decides.


Weakness 5 & Question 7:

The ablations give weak support ... each is a single outer run. How does MIMUS compare with additive Reward Networks?

Response:

  • Zero vs random head. At σ₀ = 0.1 the zero head also ends higher on Reacher (0.199 vs 0.183); the random head starts from a corrupted reward (0.006 vs 0.148), and its retained center gains less at H on fresh seeds (+0.036 vs +0.051) (Appendix "Component and Representation Ablations").
  • Width. Dense is only 1.9× larger than rank 4 at width 16, so we compared at width 64 (4.9–12.3× smaller).
  • Single runs. We keep only effects beyond the outer-search spread: dropping the base collapses Reacher (0.008 vs 0.199); CMA-ES beats OpenAI-ES on three tasks, DeepSea (a 7–8× larger gene) being the exception.
  • Additive RN. Not run. Its random head and dense network are ablated with the same optimizer and budget; its state-potential input is not.

Weakness 6 & Question 8:

The experiments are toy-scale. ... at least one higher-dimensional task?

Response:

We agree but could not add one in time: a search costs 6,400–12,800 policy trainings, and WalkerWalk shows PPO seed noise can dominate reward effects in locomotion. We limit the claims to the four tasks and name a lower-variance learner (e.g., SAC) as the prerequisite (Limitations).


Question 6:

What is the ratio of residual to base-reward magnitude for the final retained reward?

Response:

State-dependent residual RMS over base RMS on fixed replay is 0.09 (Reacher), 0.39 (PointMass) and 0.51 (FingerTurnEasy). The tanh head bounds the residual by 1 per step, and its constant part does not change the optimal policy in these fixed-length episodes. The DeepSea residual is 1.7% of the goal reward (New content highlighted; see Appendix "Magnitude of Retained Residuals", Table 29).


Thank you again; we welcome any further questions.

[1, 2] Singh et al., CogSci 2009 / IEEE TAMD 2010. [3] Zheng et al., NeurIPS 2018. [4] Houthooft et al., NeurIPS 2018. [5] Lu et al., NeurIPS 2022. [6] Wu et al., ICLR 2018.

Thank you for the detailed rebuttal. Your reponse partially answer the questions, but they also narrow the result, so I will keep my score.

AuthorsOpenReview ↗

Response to Reviewer At2B's Follow-up

Response to Reviewer At2B's Follow-up

Thank you for reading our response and for your reply. We understand the concern that the revised claims are narrower. They are narrower because the test is the stricter one you asked for in Q1: the submitted numbers came from inside the search, whereas every number now comes from seeds never used by the search, paired against the matched base, with confidence intervals and repeated outer searches. The central claim, that a frozen reward can be refined post hoc without residual labels, survives this test for all three reward sources:

  • Reacher, all three bases. All 11 independent outer searches (3 native, 4 RN, 4 Eureka) improve their frozen base at H, by +0.027 to +0.063 (native base 0.180 → 0.231).
  • PointMass and DeepSea. The gain persists at H on PointMass with the native and RN bases (native +0.069 [+0.034, +0.107]; RN +0.048, +0.068), and on DeepSea refinement converts failing seeds to success in every outer search (native 0 → 1; RN 0.5 → 1.0).
  • Where the claim narrows. On FingerTurnEasy the refined reward learns faster (+0.099 [+0.091, +0.106] at S), but the base catches up at H, and the near-saturated Eureka bases on PointMass and FingerTurnEasy reach parity. In these cases the decision rule keeps the base; this includes the one search that degrades (−0.056).

We therefore see the revision as making the result clearer rather than weaker: MIMUS improves converged performance where the base leaves headroom, and the fresh-seed audit identifies and rejects the cases where it does not.

We also acknowledge that three requests were answered with analysis rather than new experiments: equal-budget Eureka/RN construction and PPO tuning (part of Q4), the additive Reward Network under the same optimizer and budget (Q7), and a higher-dimensional task (Q8). If other points remain insufficiently answered, we would be grateful if you could tell us which, and we will address them before the discussion closes.

Thank you again for your exceptionally careful examination of our paper. Your detailed audit of the appendix and the questions it raised have greatly helped us make the results of this work clearer and more rigorous.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.