- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
HER is an LLM-driven program optimization framework that organizes candidates into hierarchical populations with promotion, demotion, and elimination. Experiments cover six mathematical and five systems benchmarks, with multiple baselines, ablations, and an approximate token-matched comparison. The inconsistencies in task validity and algorithm specification undermine confidence in the claims.
Critical strengths
- Concrete and well-motivated design. Returning stagnant candidates to exploratory layers provides a plausible mechanism for combining sustained refinement with renewed exploration.
- Useful ablations. Tables 4–6 examine hierarchy, rewriting, initialization, inertia, and reference selection. Transferring HER’s initialization to OpenEvolve also addresses an important alternative explanation.
Critical weaknesses
- The signal-processing examples appear inconsistent with the stated causal constraints. Appendix C.1 describes real-time, causal, low-latency filtering. However, Appendix P includes backward RTS smoothing and forward–backward filtering, with zero-phase processing enabled by default. If the evaluator supplies complete sequences and scores the resulting outputs directly, these examples do not establish performance under the stated online constraints. The evaluation protocol and its relationship to the reported results require clarification. This concern applies specifically to this task and its case studies.
- Reference-selection probabilities are specified inconsistently. Equations (9)–(10) define reference inclusion probabilities as and , whereas Appendix D states that implementation conditions and yield probabilities and . These define different algorithms. Under the latter interpretation, the additional factor of 0.5 in Equation (16) increases global-reference inclusion, contrary to its stated purpose of maintaining layer-level dominance. This ambiguity affects a central mechanism and reproducibility.
- The novelty claim and attribution to migration need sharper support. The introduction’s characterization of prior approaches as using uniform resource allocation is too broad: AdaEvolve already dynamically allocates resources across populations. HER’s more distinctive contribution is its ordered candidate hierarchy and bidirectional migration. Although the hierarchy ablations are informative, the description does not establish whether layer-specific operators, prompts, and sampling policies remain controlled when hierarchy is removed. Consequently, the independent contribution of migration remains unclear.
Questions for authors
- How does the signal-processing evaluator enforce causality, and which programs produced the reported scores? Would those programs pass a test that changes future inputs while holding the observed prefix fixed?
- Which reference-inclusion rule was used in the experiments? How should Equations (9)–(10) and the explanation of Equation (16) be corrected?
- With layer-specific operators and prompting retained, how much performance is lost when migration or stagnation-triggered demotion is disabled?
3 comments
Comment 2 (W2 / Q2): “Reference-selection probabilities are specified inconsistently. Equations (9)–(10) define reference inclusion probabilities as (ϕ) and (ψ), whereas Appendix D states that implementation conditions (r>ϕ) and (r>ψ) yield probabilities (1-ϕ) and (1-ψ). These define different algorithms. Under the latter interpretation, the additional factor of 0.5 in Equation (16) increases global-reference inclusion, contrary to its stated purpose of maintaining layer-level dominance. This ambiguity affects a central mechanism and reproducibility.”
Response: We thank the reviewers for raising this point. Equations (9)–(10) as written do not accurately describe the implementation, and we will correct them in the revision. We respond to each point below.
-
Which rule was used. All reported experiments used the rule described in Appendix D: a value is drawn, and a reference is included when exceeds the mapping value, giving effective inclusion probabilities (personal best), (layer best), and (global best). A larger fitness gap thus makes a reference less likely to be included. In the revision we will (i) correct Equations (9)–(10) to and , (ii) refer to these mappings as inclusion thresholds throughout the paper, and (iii) add a clarifying sentence in Section 3 so the main text matches Appendix D and the code that produced the reported results.
-
The factor 0.5 in Equation (16). Under the implemented rule, this factor halves the threshold for the global best and thereby increases its inclusion probability, exactly as the reviewer notes. For example, in L2 the layer best is included with probability 0.6–0.8 and the global best with 0.8–0.9; the two typically appear together in the prompt. Our statement that the factor maintains "layer-level guidance dominant" was written under the mistaken reading of and as probabilities and will be replaced with an accurate description of the implemented behaviour (L1 never sees the global best; L2 and L3 reference it frequently, with the layer best still included at a high rate). The ablations in Tables 4–6 used this same rule, so their conclusions are unaffected. We are additionally running a control with the 0.5 factor removed () and will report it in the revision.
Comment 3 (W3 / Q3): “The novelty claim and attribution to migration need sharper support. The introduction’s characterization of prior approaches as using uniform resource allocation is too broad: AdaEvolve already dynamically allocates resources across populations. HER’s more distinctive contribution is its ordered candidate hierarchy and bidirectional migration. Although the hierarchy ablations are informative, the description does not establish whether layer-specific operators, prompts, and sampling policies remain controlled when hierarchy is removed. Consequently, the independent contribution of migration remains unclear.”
Response: Thank you for raising this point. We agree that the characterization of prior approaches as using uniform resource allocation is inaccurate, since AdaEvolve already performs dynamic allocation across populations. We will revise the Introduction accordingly. To isolate the independent contribution of migration, we conducted two controlled ablations in which all layer-specific operators, prompts, sampling policies, and computational budgets were held fixed. One disables only stagnation-triggered demotion, while the other disables all cross-layer migration.
| Method | Circle P. (Sq.) | Prism | TXN |
|---|---|---|---|
| HER | 2.6354 0.0010 | 26.50 0.34 | 4256.7 349.4 |
| w/o stagnation demotion | 2.6337 0.0026 | 26.44 0.34 | 4162.9 112.5 |
| w/o migration | 2.6314 0.0005 | 26.37 0.10 | 4121.1 188.0 |
Disabling stagnation-triggered demotion reduces performance on all three tasks, by 0.0017 on Circle Packing, 0.06 on Prism, and 93.8 on TXN. Removing migration entirely causes larger drops of 0.0040, 0.13, and 135.6, respectively. These controlled results show that migration provides gains beyond the layer-specific search policies, while stagnation-triggered demotion makes a complementary contribution by returning stalled candidates to more exploratory layers. We will add these results and clarify in the revision that HER's distinctive contribution is the ordered candidate hierarchy with bidirectional migration, rather than dynamic resource allocation alone.
Reminder: Discussion Period Ending in a Few Days
Dear Reviewer TmZ8,
Just a brief reminder that the author–reviewer discussion period concludes in a few days.
Thank you for your insightful questions, which pushed us to verify several details of our setup. Our response keeps the signal-processing question to that task and its case study. Under the prefix-fixed causal check, HER scores 0.5691 ± 0.0253 (best 0.5968), and the best program is a forward-only filter. We would also like to clarify that the reported runs already follow Appendix D: a reference is included when a uniform draw exceeds the threshold, and Equations (9)–(10) in the revised manuscript use the probabilities 1−ϕ and 1−ψ. Under that same rule, the factor 0.5 in Equation (16) increases global-best inclusion, and Tables 4–6 were run with it, so their conclusions stand. As for the ablation, with layer-specific operators, prompts, and budgets held fixed, full HER scores 2.6354, 26.50, and 4256.7 on Circle Packing (Square), Prism, and TXN. Removing stagnation-triggered demotion lowers these to 2.6337, 26.44, and 4162.9, and removing migration lowers them further to 2.6314, 26.37, and 4121.1. Taken together, we believe these results support framing the contribution as an ordered hierarchy with bidirectional migration, and the revised manuscript now describes it in these terms. These updates are marked in blue.
We hope this addresses your concerns, but if any point remains open, we would be glad to discuss it further before the discussion ends.
Thank you again for the time and effort you put into the review.
Best regards,
Authors
Thank you for your thoughtful comments and for the time devoted to reviewing our manuscript. We address each point in detail below.
Comment 1 (W1 / Q1): “The signal-processing examples appear inconsistent with the stated causal constraints. Appendix C.1 describes real-time, causal, low-latency filtering. However, Appendix P includes backward RTS smoothing and forward–backward filtering, with zero-phase processing enabled by default. If the evaluator supplies complete sequences and scores the resulting outputs directly, these examples do not establish performance under the stated online constraints. The evaluation protocol and its relationship to the reported results require clarification. This concern applies specifically to this task and its case studies.”
Response: We are grateful for these precise questions. They led us to re-examine the signal-processing evaluator and uncovered a real inconsistency between the task description and the evaluation code that we had missed. We answer the questions in order, and the short version is: the evaluator does not enforce causality, the showcased programs would not pass the test the reviewer describes, and we have re-run the comparison with that test enforced.
How does the evaluator enforce causality? We have to admit that it does not. The evaluator hands the complete noisy sequence to the evolved program and scores the returned filtered sequence directly. There is no check on whether a given output sample depends on later input samples. The task, including both the evaluator and the description, was taken as is from the SkyDiscover suite [1]. Its description speaks of real-time, causal filtering, but its evaluator does not enforce this, and the causal / low-latency wording in Appendix C.1 was carried over from that description without our noticing the mismatch. All five methods in Table 2 were evaluated under this same full-sequence protocol, so the comparison there is fair within the protocol, but it is not evidence on a causal task. We should have stated this clearly and we will.
Which programs produced the reported scores? The Signal Processing entry in Table 2 is the mean over 3 seeds of the final best program of each HER run. The two programs in Appendix P are the iteration-42 diff and the iteration-156 full rewrite on the lineage of the final best program of one of these runs. They are intermediate ancestors rather than the scored program itself, but the final best programs of all three runs likewise rely on RTS smoothing or forward-backward filtering.
Would they pass a prefix-fixed test? Unfortunately they would not. We implemented exactly this test as a corrected evaluator. It requires that the output has one sample per complete window (input length minus window size plus one), and that running the program on input prefixes of 50%, 70% and 85% of the sequence reproduces the corresponding prefix of the full-sequence output within 1e-6. Any violation scores 0. The final best programs of all three HER runs behind Table 2 fail this test, two because the prefix output changes with future samples and one because of a wrong output length, and so do the Appendix P programs.
With this evaluator we re-ran all five methods with 3 seeds each. The model, iteration budget and all other settings are as in the paper, and the causal requirement is also stated in the task prompt for this experiment:
All best programs here pass the causal check, and HER's best program under this protocol is a purely forward adaptive Kalman / trend filter with no backward pass. Mean scores drop for every method relative to Table 2, as expected. HER keeps the highest mean and two of its three seeds (0.5635, 0.5968) exceed every baseline run (best 0.5520).
In the revision we will (1) rewrite Appendix C.1 to describe the task as actually evaluated and drop the causal / low-latency claims from the benchmark description, the case study and the original task prompt, (2) add the causal-enforced experiment and evaluator description as a new appendix next to Table 2, with a footnote on the Signal Processing row pointing to it, (3) mark the Appendix P programs as non-causal and add the best causal HER program as a case study, and (4) release the corrected evaluator with our code. We agree this concern is specific to this task and its case study, and we thank the reviewer for pinpointing it.
[1] Liu et al. "SkyDiscover: A Flexible, Adaptive Framework for AI-Driven Scientific and Algorithmic Discovery."