- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MIMUS for post-hoc reward refinement without modifying an existing reward function. The method freezes an arbitrary-form base reward and adds a learnable low-rank neural residual on top of it; the residual output head is zero-initialized, so the initial composite reward is functionally identical to the base reward. At the outer level, CMA-ES evolves the residual parameters; at the inner level, PPO is run for each candidate reward, with the performance of the post-training policy on task-level metrics used as the teaching fitness. The authors also introduce Signal Gate to check whether reward perturbations are distinguishable from PPO randomness, and use a shared anchor, multiple training seeds, and a long-horizon audit to select the final reward. The experiments cover Reacher-misc and DeepSea-bsuite in Gymnax, and PointMass and FingerTurnEasy in MuJoCo Playground/MJX. The base rewards include native handcrafted rewards, rewards learned by Reward Networks (RN), and program rewards generated by Eureka. The main results report final population mean fitness values of 0.257, 0.923, 0.319, and 0.499 on the handcrafted rewards, respectively. Most results change only slightly in the long-horizon audit, although the best candidate on FingerTurnEasy decreases from 0.543 to 0.526. RN+MIMUS outperforms the RN reference on all four tasks, while the gains from Eureka+MIMUS are mainly visible for the best candidate and are small in several cases (e.g., 0.3437→0.3450 on PointMass). Overall, the paper presents a clear framework and a fairly detailed appendix, but the current evidence remains insufficient to support strong claims about search reliability and cross-task practicality.
Critical strengths
-
The problem definition and motivation are clear. Freezing the base reward preserves its original formulation, while zero-output initialization exactly recovers it at the initial center. The shared policy anchor and the interface provide a controlled starting point for refinement, although they do not guarantee preservation of policy performance after evolution.
-
In the native handcrafted-reward experiments, the evaluation protocol accounts for the high variance of nested RL. Signal Gate, a shared seed bank across candidates, center-reward audits (with separate center-training seeds on three tasks in Table 6), and the long-horizon audit provide checks from short-term selection signals to performance at the declared longer training horizons. The appendix also reports a failed Signal Gate on WalkerWalk-MJX and returns the frozen base reward when the signal is insufficient, which is more honest than reporting successful cases only.
-
The experimental and implementation details are relatively complete. The paper reports the observation/action/transition features, reward formulas, PPO configurations, candidate counts, training seeds, search dimensions, and residual scales for all four tasks, and provides ablations on residual/scratch, CMA-ES/OpenAI-ES, zero/random heads, and low-rank/dense parameterizations. The tables in Appendices B, C, D, and E allow readers to audit the computational budget and the meaning of the different estimands with reasonable clarity.
Critical weaknesses
-
MIMUS has several distinctive features: it requires no residual labels, freezes an arbitrary-form base reward, and supports representation-agnostic refinement. However, its additional algorithmic novelty beyond this problem setting and the reliability protocol is less clear. As Section 2 notes, Reward Networks (Ferreira et al., 2022) also evaluate evolved neural rewards through the policies they teach, whereas the residual-reward methods discussed there use different supervision. A sharper comparison of this distinction and the added value of the refinement protocol would strengthen the novelty argument.
-
Appendix D, Table 11 shows that the smallest passing scale for Reacher-misc is , whereas the main search reported in Appendix C.2/Table 5 and Figure 7 starts from ; the authors also explicitly acknowledge that at least one criterion fails at 0.02. In addition, signal diagnostics/preflight are explicitly disabled for RN refinement on PointMass. Consequently, the reliability checks provided by Signal Gate were not applied consistently across the reported experiments, and the main Reacher result uses a configuration that does not pass the gate.
-
The experiments cover four simulated tasks. The main residuals use width 16 and rank 4, with additional width-64 low-rank/dense comparisons. External fitness is task-specific: continuous-control tasks use designed success/quality metrics, while DeepSea uses success rate with a progress tie-break during selection. The paper itself acknowledges horizon reversal on FingerTurnEasy and that WalkerWalk cannot pass the gate. Improvements over RN/Eureka are reported mainly as the final population mean/best candidate rather than a uniform independent final test, and the wrapper/identity of some base artifacts still contain “open” fields in the appendix. The additive interface supports different base-reward representations, but evidence for reliable practical gains beyond the tested tasks, bases, and budgets remains limited.
Questions for authors
-
Why does the main Reacher search use despite failing Signal Gate, and would using the passing scale of 0.1 change the conclusions?
-
How do residual rank, width, , and input features affect correction capacity, and how do you detect residuals that undermine the base reward's intended incentives?
-
Is comparison reliability rechecked as CMA-ES moves away from its initial center, and what triggers additional evaluation or termination when candidate–seed interactions dominate?
-
What quantitative rule defines unsupported improvement or degradation, and how does it apply to FingerTurnEasy's observed decline from to ?
Response to Reviewer 1p1U
Response to Reviewer 1p1U
Thank you for your careful, encouraging review and for recognizing our clear problem definition, variance-aware protocol and honest WalkerWalk report. All revisions are highlighted in blue in the updated PDF.
Weakness 1:
Response:
We agree that MIMUS shares the outer loop of Reward Networks (RN; Ferreira et al., 2022) and the additive form of residual-reward methods (Cao et al., 2025; Hatgis-Kessell et al., 2026). What is new is their combination: post-hoc refinement of any existing reward without labels
(New content highlighted; see Related Work and Appendix "Detailed Comparison with Prior Work", Table 2):1. It refines the outputs of other reward-design methods. On fresh seeds, MIMUS improves a frozen RN artifact (Reacher +0.027 to +0.049 at H, 4 outer runs) and a frozen Eureka program (+0.042 to +0.063)
(Appendix "Fresh-Seed Reward Audits", Table 17).2. Starting exactly at the base matters. A random head drops the initial Reacher fitness from 0.148 to 0.006, and removing the base collapses the search (final mean 0.008 vs 0.199)
(Ablation Studies).Weakness 2 & Question 1:
Response:
This inconsistency was real: σ₀ = 0.02 came from an early native-return sweep and was not re-calibrated for the task fitness. All Reacher results except the width-64 pair are re-run at σ₀ = 0.1
(Revised; see Table 1, Figures 2–4, and Appendix "Fixed-Population Signal Diagnostics"):Conclusions do not change; the weak-signal σ₀ = 0.02 even searches better. PointMass is re-checked with its search fitness, and PointMass-RN has its own check
(Appendix "Confirmation and Interpretation"). We therefore revised the procedure: the Signal Check now guides σ₀, K and N as an auxiliary diagnostic, not a pass/fail gate, and the fresh-seed test decides whether a reward is kept(Revised; see "Signal Check Before Evolution" and Algorithm 1).Weakness 3 & Question 4:
Response:
1. Uniform test. Every task–base pair is retrained against its base on unused seeds at S and H
(New content highlighted; see Table 1 and Appendix "Fresh-Seed Reward Audits", Tables 16–17).2. Rule. If the paired 95% CI of Δ = R* − base at H lies above 0, the gain is supported; below 0, it is a degradation; otherwise it is unsupported and the base is kept
(New content highlighted; see "Long-Horizon Reward Audit and Selection").3. FingerTurnEasy. Δ is +0.099 [+0.091, +0.106] at S and +0.000 [−0.010, +0.011] at H, so the H-stage gain is unsupported: R* does not degrade (0.562 → 0.572), but the base catches up (0.463 → 0.571).
Artifact identities are complete
(New content highlighted; see Appendix "RN Configuration", Table 10); claims are limited to the four tasks.Question 2:
Response:
Rank-4 beats same-width dense residuals on at least one statistic on every task with 4.9–12.3× fewer parameters, while widening from 16 to 64 helps on two tasks and hurts on one
(Appendix "Component and Representation Ablations", Tables 20–21). We did not vary α or the input features separately(Limitations). Harmful residuals are visible because the evaluator excludes the residual and every component is reported: e.g., PointMass success rises 0.80 → 0.98 while native return drops 0.5%(New content highlighted; see Appendix "Fresh-Seed Reward Audits", Table 18).Question 3:
Response:
Not in the submission; we now recompute it at every generation from archived scores
(New content highlighted; see Appendix "Reliability Along the Search", Table 14). The candidate/interaction ratio stays above 1 in all 350 native and RN Reacher generations, but falls below 1 in 89/200 Eureka generations and in all σ₀ = 0.02 generations. Such searches are flagged for more seeds(Algorithm 1).We sincerely thank Reviewer 1p1U for the constructive review and would gladly discuss any remaining questions.