- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how multiple LLM-based discovery processes can share useful discoveries instead of evolving independently. PE maintains task-specific searches connected through a shared ledger, where executable interventions discovered on one task are replayed and evaluated on other tasks. The paper also introduces hierarchical screening for expensive discovery, and evaluates these ideas on recommendation, RL, LLM, and VLM tasks. Overall, the experiments suggest that cross-instance collaboration can improve discovery outcomes under the considered settings.
Critical strengths
The main strength is the idea of sharing executable discoveries across task-specific searches. This is more concrete than simply sharing textual descriptions, since discoveries can be directly replayed on other instances and the resulting evidence can affect subsequent search. The paper also provides a broad evaluation across recommendation, RL, LLM, and VLM discovery. The reported gains, particularly the difference in successful full-target candidates, provide encouraging evidence that the framework can be useful in practice.
Critical weaknesses
- The empirical comparison with closely related evolutionary multitask approaches is missing. A central idea of PE is to share discoveries across task-specific searches, while recent work such as EMO-STA also uses cross-task information sharing for LLM-guided evolutionary discovery. The current experiments mainly compare PE with standalone/direct evolution, which shows that collaboration can be useful, but does not clarify how much PE improves over existing cross-task evolutionary approaches.
- The contribution of the different components of PE is not sufficiently isolated. PE combines cross-instance replay, peer evaluation, and hierarchical screening/promotion. The overall results show that the combined framework works well, but it is difficult to tell which part is responsible for the improvement. In particular, the current experiments do not clearly separate the benefit of cross-instance collaboration itself from the additional feedback and selection introduced by the screening procedure.
Questions for authors
1.PE combines cross-instance replay with hierarchical screening and peer evaluation. How much does each component contribute to the final improvement? In particular, would cross-instance collaboration still provide a similar gain without the hierarchical screening procedure? 2.The Track A comparison uses the same number of source iterations, which seems reasonable for evaluating the search process. However, PE also spends computation on replaying candidates on peer instances. Could the authors clarify how large this additional cost is and how the results should be interpreted when considering source iterations versus total compute? 3.The differences in successful full-target candidates (13 vs. 0 and 57 vs. 12) are quite striking. Were additional independent runs or seeds tried for these experiments? If so, do they show a similar trend? 4.The benefit of replay may depend on how related the peer instances are. Does the paper provide any evidence on when replay is helpful or harmful as the similarity between instances changes? I would be interested to know whether the gain mainly comes from sharing between closely related tasks. 5.PE seems closely related to evolutionary multitask approaches such as EMO-STA, which also transfer information across task-specific searches. Given this similarity, I think a direct EMO-STA baseline under a comparable setting would be useful for assessing the advantage of PE over existing evolutionary multitask approaches.
7 comments
W2. Contributions of replay, shared evidence, and screening
Original comment.
The contribution of the different components of PE is not sufficiently isolated. PE combines cross-instance replay, peer evaluation, and hierarchical screening/promotion. The overall results show that the combined framework works ... In particular, the current experiments do not clearly separate the benefit of cross-instance collaboration itself from the additional feedback and selection introduced by the screening procedure.
Our response.
Thank you for asking us to separate these effects. We added controls for direct reuse, the information supplied to subsequent proposals, and screening decisions. We also compare shared context with a control that retains the subset-to-full screening pipeline.
A. Direct reuse and the local reporting metric. Both S and PE report the best candidate proposed by each local search in the main local comparison. Replayed candidates are excluded from that primary metric. Each PE search selects parents from its own code archive; foreign replayed programs are not directly inserted into that archive. For proposal generation, peer results provide shared evidence that can inform subsequent local edits. Replay also supplies evidence for screening and promotion.
The table summarizes the original paper's results at 100 source iterations, comparing mean best local PE gains with those obtained when locally proposed and peer-replayed candidates are combined. Every replayed candidate is retrained and evaluated on the receiving source. Gains are relative to each source's reference.
| Family | PE local mean | Including replay | Sources improved |
|---|---|---|---|
| Watch-time | 3.85% | 3.95% | 1/5 |
| Ranking | 8.97% | 11.29% | 3/5 |
These additional gains are reported separately from the main comparison of locally proposed designs. Appendix C gives the per-source results.
B. Shared information: experiment setting. We reuse the S and PE results from the main paper and add two PE variants with different shared inputs. All four configurations use five watch-time sources and 100 iterations per source. S uses independent source histories and evaluates the top two candidates per source at 27K. PE and both variants retain local evaluation feedback and the two-of-four peer gate before larger-scale evaluation.
| Configuration | Mean best local gain | Successful / evaluated at 27K | Best larger-scale gain |
|---|---|---|---|
| S | 2.84% | 4/10 | 1.77% |
| PE | 3.85% | 5/7 | 2.17% |
| PE – Same-format summaries without outcomes | 4.25% | 16/28 | 2.00% |
| PE – Code only | 3.59% | 2/2 | 1.35% |
Success denotes at least a 0.5% relative MAE reduction on KuaiRand-27K.
The summaries-without-outcomes variant keeps the summary template but removes source, peer-replay, and 27K scores and pass/fail fields from the shared context, retrieving recent records instead of ranking them by outcome. Local evaluation feedback and outcome-based publication and promotion remain active. Code-only sharing supplies code snippets from other source searches without mechanism summaries or evaluation outcomes. Appendix E.2 describes the exact retrieval and formatting rules.
The two summary-based configurations reach best larger-scale gains of 2.17% and 2.00%, compared with 1.35% for code-only sharing. PE has the highest best larger-scale gain, while summaries without outcomes have the highest local mean (4.25%). These results suggest that informative mechanism summaries and measured outcomes help identify stronger larger-scale designs.
C. Screening analysis across two reduced-cost LLM PE runs. We report full-target outcomes for promoted and rejected candidates to examine which stronger improvements the screening rules retain or miss. We use a 2% gain threshold for this analysis.
| Run / candidate group | Full-target evaluations | Gains ≥2% | Finding |
|---|---|---|---|
| Run 1: peer rejects | 44 | 29 | 29 successful variants concentrate in one related D5-B optimizer family |
| Run 1: intermediate rejects | 9 | 0 | None reaches the stronger-gain threshold |
| Run 2: all rejects | 120 | 0 | All 120 fall below the stronger-gain threshold |
Across two reduced-cost LLM PE runs, 144 of 173 evaluated rejected candidates (83.24%) fall below the 2% gain threshold, including all 120 in Run 2. The 29 exceptions are related variants of one D5-B Muon-update and parameter-routing direction in Run 1. Excluding D5-B, all 11 remaining peer rejects in Run 1 fall below 1%, and all eight intermediate rejects fall below 2%. We verified that, although the D5-B variants were screened out, PE still discovered the same promising change through other sources. This highlights PE's strength in discovering promising designs through exploration across multiple sources. The source-stratified rejected-candidate counts are reported in Appendix G.3.
D. Shared context with screening retained: reduced-cost LLM pretraining. We use the same approximately 185M-parameter Gemma 3-style model code on FineWeb-Edu, with a full target of approximately 1B training tokens and one H100 per evaluation. PE and PE without shared proposal context use matched total training-and-evaluation GPU compute and the same source subsets, peer replay, intermediate screening, and evaluation pipeline. The only intervention is to disable the shared evidence supplied to subsequent proposals. The metric is the mean accuracy across seven downstream tasks; gain is the relative improvement over the common reference of 35.18%. The table aggregates full-target evaluations across three independent discovery runs; the detailed training setup is given in Q3.
| Configuration | Full-target evaluations | Gains ≥1% | Gain ≥2% | Best full-target gain |
|---|---|---|---|---|
| PE | 79 | 14 | 4 | 2.98% |
| PE without shared proposal context | 2 | 2 | 1 | 2.80% |
Across the three discovery repetitions, PE yields 14 full-target records with gains of at least 1% with shared context, compared with two without it. Best gains are 2.98% and 2.80%, respectively.
At the ≥2% threshold, PE produces four full-target evaluations (1, 0, and 3 across Runs 1–3), compared with one for Direct.
These results support sharing within the subset-and-screening pipeline; Appendix G.4 gives the run-level outcomes.
In the revised paper, we organize the component analyses in Section 5, with shared-input controls in Appendix E.2 and screening/shared-context analyses in Appendices G.3–G.4.
Q1. Component contributions and collaboration without hierarchy
Original question.
PE combines cross-instance replay with hierarchical screening and peer evaluation. How much does each component contribute to the final improvement? In particular, would cross-instance collaboration still provide a similar gain without the hierarchical screening procedure?
Our response.
Yes: with promotion gates disabled, shared context yields a best larger-scale gain of 2.25%, versus 1.86% without shared context, under the same top-two-per-source evaluation rule (ten candidates per configuration).
Experiment setting. Both ungated configurations use the five watch-time sources and 100 iterations per source. Promotion gates are disabled in both; one configuration retains shared proposal context and the other removes it. After discovery, each evaluates its top two candidates per source, ranked by local performance, on KuaiRand-27K: ten candidates per configuration. Success requires at least a 0.5% relative MAE reduction.
Results. Mean local gain averages the best locally proposed gain on each source. The table reports the additional ungated comparison.
| Configuration | Mean local gain | Successful / evaluated at 27K | Best larger-scale gain |
|---|---|---|---|
| Ungated PE, shared context | 3.98% | 8/10 | 2.25% |
| Ungated PE, no shared context | 4.21% | 8/10 | 1.86% |
Both configurations yield 8/10 successful evaluations; the no-context configuration has the higher local mean (4.21% versus 3.98%). This supports the usefulness of collaboration for finding a stronger larger-scale design without hierarchical screening.
These additional ungated searches are separate from the original S-versus-gated-PE experiment. The original gated PE evaluates seven promoted candidates and reaches a best gain of 2.17%, while the ungated shared-context run evaluates ten locally selected candidates and reaches 2.25%. Best gain is the maximum over the evaluated candidates, so comparisons depend on both the size and composition of that pool. Differences in discovery trajectories and selection rules further prevent attributing the 0.08-percentage-point difference to gating alone. Under the same ungated evaluation quota, shared context improves best gain from 1.86% to 2.25%, directly supporting the benefit of collaboration.
Author Response
Thank you for recognizing the value of sharing executable discoveries and for the constructive suggestions. We added a direct EMO-STA comparison, component controls, computation accounting, independent discovery repetitions, and transfer analyses. For each point, we describe the purpose, setup, results, and their implications below.
W1. Comparison with existing evolutionary multitask approaches
Original comment.
The empirical comparison with closely related evolutionary multitask approaches is missing. A central idea of PE is to share discoveries across task-specific searches, while recent work such as EMO-STA also uses cross-task information sharing for LLM-guided evolutionary discovery. The current experiments mainly compare PE with standalone/direct evolution, which shows that collaboration can be useful, but does not clarify how much PE improves over existing cross-task evolutionary approaches.
Our response. Thank you for suggesting this baseline. We added EMO-STA and compared it with PE under the same OpenEvolve harness.
Experiment setting. Both methods use the same five watch-time sources, reference models, training evaluator, and GPT-5.5-high proposer. Both methods use a 100-iteration source-search budget per task: EMO-STA allocates 50 iterations to its shared phase and 50 to adaptation, while PE uses 100 source iterations. KuaiRand-27K is used for larger-scale evaluation. EMO-STA evaluates the top two adapted candidates per source; PE uses peer-based promotion.
Results. Local gain is the mean best relative MAE reduction across the five sources. Success in larger-scale evaluation requires at least a 0.5% relative MAE reduction.
The EMO-STA source and larger-scale results come from the same search. Appendix G.1 describes the implementation.
Per-source local gains (%). Both methods use the same five watch-time datasets; each entry reports the best locally proposed gain on that source.
PE achieves higher best local gains on all five source datasets, with a mean of 2.39% versus 1.72% for adapted EMO-STA. In larger-scale evaluation on KuaiRand-27K, PE obtains 11 successful evaluations out of 11, compared with 6 out of 10 for EMO-STA, and a higher best gain (1.75% versus 1.55%). These results directly support PE's usefulness relative to an existing cross-task evolutionary method in this watch-time setting.
In the revised paper, we add the direct EMO-STA comparison in Section 4.2 and Appendix G.1.