acceptodds
Under review as a conference paper at ICLR 2027

Discovered, Not Designed: Population Evolution for Collaborative and Compute-Intensive Model Discovery

Abstract

LLM-driven evolution enables iterative model development, but two practical goals remain underexplored: finding model designs that transfer across related tasks and sustaining improvement when training is expensive. We introduce Population Evolution (PE), a collaborative, hierarchical framework that connects ongoing local searches through shared experimental evidence. PE evaluates code changes across related training instances and shares the results to guide subsequent proposals and promotion to larger training scales. For expensive targets, PE searches small training subsets and screens candidates through peer and intermediate evaluations before full-target training. We introduce RMD-Bench to evaluate both settings across ranking, watch-time prediction, RL algorithm discovery, and LLM/VLM pretraining. Compared with standalone evolution at matched source iterations, PE raises mean best local gains from 7.01% to 8.97% in ranking and from 2.84% to 3.85% in watch-time, while improving the best larger-scale outcome in all three joint-discovery families. In watch-time discovery, PE improves best larger-scale gains with four of five harnesses and all four proposers. On new recommendation datasets under shared target-side calibration, every evaluated PE design improves over the reference in mean performance. Under matched total GPU compute, completed LLM discovery runs yield a best relative accuracy gain of 2.48% and 13 successful candidates for PE, versus 0.92% and none for direct evolution. VLM loss reduction reaches 8.78% versus 5.05% under matched total GPU compute.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Automated Model Discovery, Recursive Self-Improvement, Self-Evolving Systems, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_uz4VOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how multiple LLM-based discovery processes can share useful discoveries instead of evolving independently. PE maintains task-specific searches connected through a shared ledger, where executable interventions discovered on one task are replayed and evaluated on other tasks. The paper also introduces hierarchical screening for expensive discovery, and evaluates these ideas on recommendation, RL, LLM, and VLM tasks. Overall, the experiments suggest that cross-instance collaboration can improve discovery outcomes under the considered settings.

Critical strengths

The main strength is the idea of sharing executable discoveries across task-specific searches. This is more concrete than simply sharing textual descriptions, since discoveries can be directly replayed on other instances and the resulting evidence can affect subsequent search. The paper also provides a broad evaluation across recommendation, RL, LLM, and VLM discovery. The reported gains, particularly the difference in successful full-target candidates, provide encouraging evidence that the framework can be useful in practice.

Critical weaknesses
  1. The empirical comparison with closely related evolutionary multitask approaches is missing. A central idea of PE is to share discoveries across task-specific searches, while recent work such as EMO-STA also uses cross-task information sharing for LLM-guided evolutionary discovery. The current experiments mainly compare PE with standalone/direct evolution, which shows that collaboration can be useful, but does not clarify how much PE improves over existing cross-task evolutionary approaches.
  2. The contribution of the different components of PE is not sufficiently isolated. PE combines cross-instance replay, peer evaluation, and hierarchical screening/promotion. The overall results show that the combined framework works well, but it is difficult to tell which part is responsible for the improvement. In particular, the current experiments do not clearly separate the benefit of cross-instance collaboration itself from the additional feedback and selection introduced by the screening procedure.
Questions for authors

1.PE combines cross-instance replay with hierarchical screening and peer evaluation. How much does each component contribute to the final improvement? In particular, would cross-instance collaboration still provide a similar gain without the hierarchical screening procedure? 2.The Track A comparison uses the same number of source iterations, which seems reasonable for evaluating the search process. However, PE also spends computation on replaying candidates on peer instances. Could the authors clarify how large this additional cost is and how the results should be interpreted when considering source iterations versus total compute? 3.The differences in successful full-target candidates (13 vs. 0 and 57 vs. 12) are quite striking. Were additional independent runs or seeds tried for these experiments? If so, do they show a similar trend? 4.The benefit of replay may depend on how related the peer instances are. Does the paper provide any evidence on when replay is helpful or harmful as the similarity between instances changes? I would be interested to know whether the gain mainly comes from sharing between closely related tasks. 5.PE seems closely related to evolutionary multitask approaches such as EMO-STA, which also transfer information across task-specific searches. Given this similarity, I think a direct EMO-STA baseline under a comparable setting would be useful for assessing the advantage of PE over existing evolutionary multitask approaches.

7 comments
AuthorsOpenReview ↗

Author Response

Thank you for recognizing the value of sharing executable discoveries and for the constructive suggestions. We added a direct EMO-STA comparison, component controls, computation accounting, independent discovery repetitions, and transfer analyses. For each point, we describe the purpose, setup, results, and their implications below.

W1. Comparison with existing evolutionary multitask approaches

Original comment.

The empirical comparison with closely related evolutionary multitask approaches is missing. A central idea of PE is to share discoveries across task-specific searches, while recent work such as EMO-STA also uses cross-task information sharing for LLM-guided evolutionary discovery. The current experiments mainly compare PE with standalone/direct evolution, which shows that collaboration can be useful, but does not clarify how much PE improves over existing cross-task evolutionary approaches.

Our response. Thank you for suggesting this baseline. We added EMO-STA and compared it with PE under the same OpenEvolve harness.

Experiment setting. Both methods use the same five watch-time sources, reference models, training evaluator, and GPT-5.5-high proposer. Both methods use a 100-iteration source-search budget per task: EMO-STA allocates 50 iterations to its shared phase and 50 to adaptation, while PE uses 100 source iterations. KuaiRand-27K is used for larger-scale evaluation. EMO-STA evaluates the top two adapted candidates per source; PE uses peer-based promotion.

Results. Local gain is the mean best relative MAE reduction across the five sources. Success in larger-scale evaluation requires at least a 0.5% relative MAE reduction.

MethodHarnessMean best local gainSuccessful / evaluated at 27KBest larger-scale gain
EMO-STA, after adaptationOpenEvolve1.72%6/101.55%
PEOpenEvolve2.39%11/111.75%

The EMO-STA source and larger-scale results come from the same search. Appendix G.1 describes the implementation.

Per-source local gains (%). Both methods use the same five watch-time datasets; each entry reports the best locally proposed gain on that source.

DatasetEMO-STA, after adaptationPE, OpenEvolve
CIKM164.26%5.72%
KuaiRec0.24%0.56%
WeChat0.29%0.59%
KuaiRand-Pure0.27%0.59%
KuaiRand-1K3.57%4.47%
Mean1.72%2.39%

PE achieves higher best local gains on all five source datasets, with a mean of 2.39% versus 1.72% for adapted EMO-STA. In larger-scale evaluation on KuaiRand-27K, PE obtains 11 successful evaluations out of 11, compared with 6 out of 10 for EMO-STA, and a higher best gain (1.75% versus 1.55%). These results directly support PE's usefulness relative to an existing cross-task evolutionary method in this watch-time setting.

In the revised paper, we add the direct EMO-STA comparison in Section 4.2 and Appendix G.1.

W2. Contributions of replay, shared evidence, and screening

Original comment.

The contribution of the different components of PE is not sufficiently isolated. PE combines cross-instance replay, peer evaluation, and hierarchical screening/promotion. The overall results show that the combined framework works ... In particular, the current experiments do not clearly separate the benefit of cross-instance collaboration itself from the additional feedback and selection introduced by the screening procedure.

Our response.

Thank you for asking us to separate these effects. We added controls for direct reuse, the information supplied to subsequent proposals, and screening decisions. We also compare shared context with a control that retains the subset-to-full screening pipeline.

A. Direct reuse and the local reporting metric. Both S and PE report the best candidate proposed by each local search in the main local comparison. Replayed candidates are excluded from that primary metric. Each PE search selects parents from its own code archive; foreign replayed programs are not directly inserted into that archive. For proposal generation, peer results provide shared evidence that can inform subsequent local edits. Replay also supplies evidence for screening and promotion.

The table summarizes the original paper's results at 100 source iterations, comparing mean best local PE gains with those obtained when locally proposed and peer-replayed candidates are combined. Every replayed candidate is retrained and evaluated on the receiving source. Gains are relative to each source's reference.

FamilyPE local meanIncluding replaySources improved
Watch-time3.85%3.95%1/5
Ranking8.97%11.29%3/5

These additional gains are reported separately from the main comparison of locally proposed designs. Appendix C gives the per-source results.

B. Shared information: experiment setting. We reuse the S and PE results from the main paper and add two PE variants with different shared inputs. All four configurations use five watch-time sources and 100 iterations per source. S uses independent source histories and evaluates the top two candidates per source at 27K. PE and both variants retain local evaluation feedback and the two-of-four peer gate before larger-scale evaluation.

ConfigurationMean best local gainSuccessful / evaluated at 27KBest larger-scale gain
S2.84%4/101.77%
PE3.85%5/72.17%
PE – Same-format summaries without outcomes4.25%16/282.00%
PE – Code only3.59%2/21.35%

Success denotes at least a 0.5% relative MAE reduction on KuaiRand-27K.

The summaries-without-outcomes variant keeps the summary template but removes source, peer-replay, and 27K scores and pass/fail fields from the shared context, retrieving recent records instead of ranking them by outcome. Local evaluation feedback and outcome-based publication and promotion remain active. Code-only sharing supplies code snippets from other source searches without mechanism summaries or evaluation outcomes. Appendix E.2 describes the exact retrieval and formatting rules.

The two summary-based configurations reach best larger-scale gains of 2.17% and 2.00%, compared with 1.35% for code-only sharing. PE has the highest best larger-scale gain, while summaries without outcomes have the highest local mean (4.25%). These results suggest that informative mechanism summaries and measured outcomes help identify stronger larger-scale designs.

C. Screening analysis across two reduced-cost LLM PE runs. We report full-target outcomes for promoted and rejected candidates to examine which stronger improvements the screening rules retain or miss. We use a 2% gain threshold for this analysis.

Run / candidate groupFull-target evaluationsGains ≥2%Finding
Run 1: peer rejects442929 successful variants concentrate in one related D5-B optimizer family
Run 1: intermediate rejects90None reaches the stronger-gain threshold
Run 2: all rejects1200All 120 fall below the stronger-gain threshold

Across two reduced-cost LLM PE runs, 144 of 173 evaluated rejected candidates (83.24%) fall below the 2% gain threshold, including all 120 in Run 2. The 29 exceptions are related variants of one D5-B Muon-update and parameter-routing direction in Run 1. Excluding D5-B, all 11 remaining peer rejects in Run 1 fall below 1%, and all eight intermediate rejects fall below 2%. We verified that, although the D5-B variants were screened out, PE still discovered the same promising change through other sources. This highlights PE's strength in discovering promising designs through exploration across multiple sources. The source-stratified rejected-candidate counts are reported in Appendix G.3.

D. Shared context with screening retained: reduced-cost LLM pretraining. We use the same approximately 185M-parameter Gemma 3-style model code on FineWeb-Edu, with a full target of approximately 1B training tokens and one H100 per evaluation. PE and PE without shared proposal context use matched total training-and-evaluation GPU compute and the same source subsets, peer replay, intermediate screening, and evaluation pipeline. The only intervention is to disable the shared evidence supplied to subsequent proposals. The metric is the mean accuracy across seven downstream tasks; gain is the relative improvement over the common reference of 35.18%. The table aggregates full-target evaluations across three independent discovery runs; the detailed training setup is given in Q3.

ConfigurationFull-target evaluationsGains ≥1%Gain ≥2%Best full-target gain
PE791442.98%
PE without shared proposal context2212.80%

Across the three discovery repetitions, PE yields 14 full-target records with gains of at least 1% with shared context, compared with two without it. Best gains are 2.98% and 2.80%, respectively.

At the ≥2% threshold, PE produces four full-target evaluations (1, 0, and 3 across Runs 1–3), compared with one for Direct.

These results support sharing within the subset-and-screening pipeline; Appendix G.4 gives the run-level outcomes.

In the revised paper, we organize the component analyses in Section 5, with shared-input controls in Appendix E.2 and screening/shared-context analyses in Appendices G.3–G.4.

Q1. Component contributions and collaboration without hierarchy

Original question.

PE combines cross-instance replay with hierarchical screening and peer evaluation. How much does each component contribute to the final improvement? In particular, would cross-instance collaboration still provide a similar gain without the hierarchical screening procedure?

Our response.

Yes: with promotion gates disabled, shared context yields a best larger-scale gain of 2.25%, versus 1.86% without shared context, under the same top-two-per-source evaluation rule (ten candidates per configuration).

Experiment setting. Both ungated configurations use the five watch-time sources and 100 iterations per source. Promotion gates are disabled in both; one configuration retains shared proposal context and the other removes it. After discovery, each evaluates its top two candidates per source, ranked by local performance, on KuaiRand-27K: ten candidates per configuration. Success requires at least a 0.5% relative MAE reduction.

Results. Mean local gain averages the best locally proposed gain on each source. The table reports the additional ungated comparison.

ConfigurationMean local gainSuccessful / evaluated at 27KBest larger-scale gain
Ungated PE, shared context3.98%8/102.25%
Ungated PE, no shared context4.21%8/101.86%

Both configurations yield 8/10 successful evaluations; the no-context configuration has the higher local mean (4.21% versus 3.98%). This supports the usefulness of collaboration for finding a stronger larger-scale design without hierarchical screening.

These additional ungated searches are separate from the original S-versus-gated-PE experiment. The original gated PE evaluates seven promoted candidates and reaches a best gain of 2.17%, while the ungated shared-context run evaluates ten locally selected candidates and reaches 2.25%. Best gain is the maximum over the evaluated candidates, so comparisons depend on both the size and composition of that pool. Differences in discovery trajectories and selection rules further prevent attributing the 0.08-percentage-point difference to gating alone. Under the same ungated evaluation quota, shared context improves best gain from 1.86% to 2.25%, directly supporting the benefit of collaboration.

Reviewer_9ABtOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
3(+1)
Read review
Paper summary

This paper introduces Population Evolution (PE), a framework that lets LLM-driven searches on related tasks share code changes and evaluation results to guide further improvements. For expensive training tasks, PE searches on small data subsets first and screens promising candidates before training them at larger scales. The authors also introduce RMD-Bench, covering recommendation, RL algorithm discovery, and LLM/VLM pretraining. Experiments show higher average local recommendation gains than independent searches at matched search iterations, and better LLM/VLM performance with more successful candidates than direct full-scale search under matched GPU budgets.

Critical strengths
  1. The paper addresses sharing useful discoveries across related tasks and making search affordable when evaluating each candidate requires expensive training. Their new RMD-Bench provides a useful setting for studying these problems across recommendation, RL algorithm discovery, and LLM/VLM pretraining.

  2. Experimental scale is satisfactory, and in the reported LLM/VLM experiments, PE finds better-performing designs than direct full-scale search under matched total GPU budgets. The larger-scale watch-time improvements hold across several search harnesses and proposal models, and the transfer experiments show that some discovered designs remain useful across datasets and training scales.

Critical weaknesses
  1. In the LLM/VLM experiments (Track B), the evaluation does not isolate the benefit of collaboration from the benefit of cheaper subset search and hierarchical screening. Track A includes standalone searches without shared context, but Track B compares PE only with direct full-target search. I think a more informative baseline would use the same subsets/peer-intermediate screening rules/total GPU budget while disabling shared proposal context. This would help establish how much collaboration contributes to the reported gains.

  2. Appendix B.5 reports only one complete discovery run per method–configuration. I think repeated evaluations of selected candidates do not establish how consistently the discovery procedure finds good designs. I understand that full runs are expensive, but reporting the preliminary independent runs or repeating comparisons in a less expensive setting would strengthen the evidence. Generalization results are also mixed. As acknowledged in Appendix F.1, the recommendation transfer experiments use the same test split to select hyperparameters and training duration and to report final performance. Although all candidates share the selected configuration, there is no independent test after calibration. In the held-out RL evaluation, the best PE candidate also falls slightly below the unmodified reference mean, so the gains observed during discovery do not clearly carry over to new tasks.

  3. The presentation is unnecessarily dense and hard to follow, especially for first-time readers. The abstract packs in numerous results using terms such as “best local gains” and “shared target-side calibration” without making their meaning clear. The introduction describes "intervention", “joint discovery” and “decomposed discovery” without a concrete example showing how a proposed code change is shared and used by another search. Figure 1 further mixes the overall workflow with detailed subset labels such as D2-A, D5-C, and D10+C, making it difficult to follow without first understanding the experimental setup. I think the abstract, introduction, and Figure 1 could be more digestible.

Questions for authors
  1. As a suggestion, I would be interested in an evaluation on MLS-Bench’s adaptive LLM-pretraining setting. This seems relevant to Track B, although the collaborating instances and screening stages would need to be specified. It could help establish whether PE’s gains extend beyond the discovery protocols introduced in this paper.

  2. Do candidates that perform well in peer and intermediate evaluations also tend to do well at full scale? Have you tried evaluating a representative sample of rejected candidates at full scale to see whether screening misses promising designs?

9 comments

Thank you for the careful review. We have added repeated discovery runs, shared-context controls, calibration and transfer checks, an external pretraining task, and full-scale analyses of rejected candidates. We address each concern below.

W1. Isolating collaboration from subset search and screening

Original comment.

In the LLM/VLM experiments (Track B), the evaluation does not isolate the benefit of collaboration from the benefit of cheaper subset search and hierarchical screening. Track A includes standalone searches without shared context, but Track B compares PE only with direct full-target search. I think a more informative baseline would use the same subsets/peer-intermediate screening rules/total GPU budget while disabling shared proposal context. This would help establish how much collaboration contributes to the reported gains.

Our response. Across three independent discovery runs, with the subset-search and screening pipeline held fixed, PE produces 14 full-target evaluations with gains of at least 1%, versus 2 without shared context, and a best gain of 2.98% versus 2.80%. The control retains subset search, peer replay, and intermediate screening, removing only the shared evidence supplied to later proposals.

Experiment setting. We compare PE, PE without shared proposal context, and Direct at matched total training-and-evaluation GPU compute:

  • PE: subset search and screening, with shared proposal context.
  • PE without shared proposal context: the same complete discovery and evaluation pipeline and a matched total GPU budget as PE, with only shared proposal context disabled. Each search proposes from its own history; peer replay still supplies the scores needed for screening.
  • Direct: full-target search without the subset-to-full pipeline.

Given the cost of the original campaigns, we use a reduced-cost LLM task. The full target is reduced from 100.06B to about 1B training tokens: 2,774 rather than 34,700 steps, global batch 352 rather than 2,816 sequences, sequence length 1,024, and one H100 per evaluation instead of eight H200s. We retain the same approximately 185M-parameter Gemma 3-style model code, editable interface, corpus, and seven-task downstream-accuracy evaluator. Gains are relative improvements in mean downstream accuracy against the common discovery reference 35.18%. The table aggregates full-target evaluations across three independent discovery runs.

Results.

MethodFull-target evaluationsGain ≥1%Gain ≥1.5%Gain ≥2%Best gain
PE7914642.98%
PE without shared proposal context22212.80%
Direct24016412.37%

At the ≥2% threshold, PE produces four full-target evaluations (1, 0, and 3 across Runs 1–3), compared with one for Direct.

These results support the usefulness of shared context within this pipeline. In a separate comparison with Direct, PE produces more outcomes meeting the 1.5% gain threshold and achieves the highest observed gain.

In the revised paper, we summarize the shared-context control in Section 4.4 and present its setting and results in Appendix G.4.

W2a. Independent repetitions of the discovery procedure

Original comment.

Appendix B.5 reports only one complete discovery run per method–configuration. I think repeated evaluations of selected candidates do not establish how consistently the discovery procedure finds good designs. I understand that full runs are expensive, but reporting the preliminary independent runs or repeating comparisons in a less expensive setting would strengthen the evidence.

Our response. Thank you for emphasizing the repeatability of discovery. We report an additional independent PE search on watch-time prediction and independent searches in a reduced-cost LLM setting, following your suggested lower-cost alternative.

Experiment setting. The additional PE search starts afresh on the same five watch-time sources and uses 100 iterations per source. KuaiRand-27K provides the larger-scale evaluation; success means at least a 0.5% relative MAE reduction. The table compares PE from the main paper with PE repeat under this protocol.

Results: PE discovery and independent repeat.

MethodMean best local gainSuccessful / evaluated at 27KBest larger-scale gain
PE3.85%5/72.17%
PE repeat3.29%34/342.22%

PE repeat finds a design with a 2.22% gain in larger-scale evaluation, close to the original 2.17%, and produces 34 successful evaluations. This additional search demonstrates that PE can again discover a useful larger-scale model.

Results: reduced-cost LLM discovery. Each of the three independent PE discovery runs from W1 finds at least one design exceeding the 1% full-target gain threshold. Together they produce 14 evaluations with gains of at least 1%; the best gain is 2.98%. These are independent discovery runs that generate new designs. Four evaluations reach the ≥2% threshold (1, 0, and 3 across Runs 1–3). Appendix G.4 reports their protocols and results.

In the revised paper, we present the watch-time PE repeat and reduced-cost LLM discovery results in Appendices G.1 and G.4.

W2b. Validation-only recalibration and fresh-sample evaluation

Original comment.

Generalization results are also mixed. As acknowledged in Appendix F.1, the recommendation transfer experiments use the same test split to select hyperparameters and training duration and to report final performance. Although all candidates share the selected configuration, there is no independent test after calibration.

Our response. We address the calibration concern by repeating the recommendation-transfer evaluations with settings selected on validation data. We also evaluate frozen designs on a new KuaiRand-27K sample to check whether their improvements persist on different evaluation data.

Fresh-sample evaluation. We test whether the gains of fixed discovered programs persist on a different sample of KuaiRand-27K interactions. The original watch-time benchmark uses a 10% sample of the raw logs. We construct the fresh sample using only records from the remaining 90%, excluding every record in the original sampled corpus. We apply the same preprocessing and training/evaluation pipeline to the new sample and measure relative MAE reductions against its reference.

Candidate selection. For each method, we select the top three distinct designs by original KuaiRand-27K gain from its already evaluated pool in the original AdaEvolve/GPT-5.5 watch-time run and freeze them before fresh-data evaluation. PE's pool consists of seven gate-promoted candidates; three behaviorally identical programs, each with an original 1.07% gain, count as one design. S's pool consists of the original ten candidates selected by taking the top two by local performance from each source.

Results.

Frozen poolCandidatesBest gain on the original sampleBest new-sample gain
PE3+2.17%+2.85%
S3+1.77%+2.61%

Findings. All three frozen designs from each method exceed the 0.5% success threshold on the fresh sample. PE achieves a best fresh-sample gain of 2.85% and a mean of 2.53%, compared with S's 2.61% best and 2.12% mean. PE's selected designs generalize to the new sample, yielding higher best and mean gains than S in this frozen pool. Appendix G.5 gives the selected designs and their individual old- and new-sample gains.

Validation-only recalibration. We test whether the discovered designs retain their gains when reference hyperparameters and training duration are selected using validation data rather than test data. The designs remain fixed. On VK-LSVD, we train the reference under six predefined configurations for ten epochs and select the configuration and epoch with the lowest MAE on the released validation split. This selects the third configuration after one epoch. On KuaiVideo, we reserve the final 20% of each user's training requests for validation and choose among six predefined configurations. We then freeze one configuration and training duration per dataset, retrain the reference and every candidate from scratch under that same setting, and evaluate on the existing test split.

Target and methodDesigns / evaluationsMean gain
VK-LSVD, PE3 distinct designs1.66% MAE reduction
VK-LSVD, S5 distinct designs1.25% MAE reduction
KuaiVideo, PE33 evaluations; 32 improved+0.08 pp AUC
KuaiVideo, S45 evaluations; 30 improved+0.03 pp AUC

On VK-LSVD, mean relative MAE reduction is 1.66% for PE and 1.25% for S, averaging equally over the three and five distinct designs, respectively. These validation-only results retain positive transfer gains on both datasets. The separate fresh-sample experiment evaluates fixed candidates on new KuaiRand-27K data.

In the revised paper, we summarize these evaluation checks in Section 4.2 and give their protocols and results in Appendix G.5, distinguishing validation-only recalibration from evaluation on a fresh target sample.

Reviewer_ZsK6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Population Evolution (PE), a collaborative strategy for automated deep learning (AutoDL). By coordinating multiple otherwise independent search processes, PE achieves improvements in overall search performance.

Critical strengths

The paper is clearly written and well motivated, making it easy for readers to understand.

I find the methodological direction and overall narrative appealing. The authors incorporate the idea of population evolution into the search process by expanding collaboration among individual searches and demonstrate its effectiveness. Although this general idea is not new, I appreciate this research direction.

The experiments cover a diverse range of tasks and evaluation perspectives, helping readers understand the method’s effectiveness.

Critical weaknesses
  1. Presentation of Table 1. The selected dimensions emphasize features of the proposed benchmark and closely align with its design, which is insufficient to establish the overall strengths and weaknesses of the different benchmarks. The table only indicates whether existing benchmarks explicitly provide the evaluation support emphasized by the authors. Whether these capabilities are more valuable than those prioritized by other benchmarks remains an open question. From an informational standpoint, most entries for the compared benchmarks are “—”, which makes the table’s added value seem limited.

  2. Results in Table 2. Overall, PE improves over S. However, a closer examination reveals cases where the improvement is marginal or performance even declines, such as WeChat and KuaiRand-Pure in Table 2B. Could the authors explain why collaboration yields limited benefits or negative effects in these cases?

  3. Generalizability across Dataset Selections. How sensitive are the conclusions to the particular set of source datasets selected? Would PE consistently outperform S if a different set of related datasets were used?

  4. Scaling the Number of Local Searches. How do performance and computational costs change as the number of local searches increases? In particular, what are the additional token usage, GPU compute, and wall-clock time associated with adding one local search to PE?

  5. Figure Presentation. As a minor suggestion, incorporating more visual illustrations and reducing the amount of text could make the figures more engaging and easier to follow.

Questions for authors

All questions are detailed in the weaknesses above.

4 comments

Thank you for your careful review and for recognizing the motivation and breadth of the evaluation. Your comments prompted us to clarify the benchmark's purpose, examine where collaboration helps, and measure its additional cost. We address each point below.

Throughout this response, watch-time gain is the relative MAE reduction against the fixed reference. A successful KuaiRand-27K evaluation improves MAE by at least 0.5%.

W1. Presentation of the benchmark comparison

Reviewer comment.

Presentation of Table 1. The selected dimensions emphasize features of the proposed benchmark and closely align with its design, which is insufficient to establish the overall strengths and weaknesses of the different benchmarks. The table only indicates whether existing benchmarks explicitly provide the evaluation support emphasized by the authors. Whether these capabilities are more valuable than those prioritized by other benchmarks remains an open question. From an informational standpoint, most entries for the compared benchmarks are “—”, which makes the table’s added value seem limited.

Response.

Thank you; we agree that the original checklist did not provide a balanced comparison of benchmarks with different goals. We have removed this table from both the main paper and appendix. Section 6 now explains the complementary evaluation purposes of the related benchmarks without implying an overall ranking.

The replacement Table 1 serves a different, concrete purpose: it maps each task family to its discovery sources, larger-scale/full target, and transfer targets. This also helps readers follow the two tracks. We have clarified that RMD-Bench is designed to study sharing across related searches and subset-to-full discovery; these capabilities complement the objectives of existing benchmarks.

In the revised paper, we replace the benchmark checklist with the instance map in Table 1 and discuss the complementary purposes of existing benchmarks in Section 6.

W2. Limited or negative benefits on WeChat and KuaiRand-Pure

Reviewer comment.

Results in Table 2. Overall, PE improves over S. However, a closer examination reveals cases where the improvement is marginal or performance even declines, such as WeChat and KuaiRand-Pure in Table 2B. Could the authors explain why collaboration yields limited benefits or negative effects in these cases?

Response.

Thank you for highlighting these cases. WeChat is essentially tied (0.80% for S versus 0.81% for PE), while KuaiRand-Pure remains below S even after replay (PE: 0.16% → 0.65%; S: 0.75%). Sharing therefore does not guarantee improvement on every source. Table 2 reports the best designs proposed locally by each source; the comparison below also includes candidates replayed from other sources in the same original run.

DatasetS: best locally proposed gainPE: best locally proposed gainPE: best gain including replayed candidates
WeChat0.80%0.81%0.81%
KuaiRand-Pure0.75%0.16%0.65%

KuaiRand-Pure. Replay supplies a useful design that the local PE search did not find: a duration-conditioned Laplace-mixture head with a near-median readout, discovered on WeChat and retrained on Pure.

Different search trajectories are one possible explanation for the local gap, but this comparison does not isolate its cause. We therefore distinguish the best locally proposed design from the best design available after replay.

In the revised paper, we clarify the distinction between locally proposed and replayed designs in Section 5. The same WeChat-discovered head achieves a 2.17% gain on KuaiRand-27K, compared with the original S best of 1.77%, showing that a source can contribute a transferable design even when its own local result is essentially tied.

W3. Generalizability across source-dataset selections

Reviewer comment.

Generalizability across Dataset Selections. How sensitive are the conclusions to the particular set of source datasets selected? Would PE consistently outperform S if a different set of related datasets were used?

Response.

Thank you. In the added source-composition comparisons, PE has a higher best larger-scale gain than S in two of the three four-source settings. These experiments test whether useful discovery depends on the original five-source set.

Experiment setting. We remove CIKM16, KuaiRand-Pure, or WeChat in turn and rerun PE for 100 iterations per remaining source. Each four-source configuration retains the five-source setting's requirement of at least two peer improvements of 0.5% or more, now among three peers. We also evaluate a three-source population containing CIKM16, KuaiRec, and WeChat, requiring this improvement on at least one of two peers. All configurations use KuaiRand-27K for larger-scale evaluation and the same 0.5% success criterion. These configurations vary both source composition and population size.

Results.

PE's best larger-scale gain remains positive for every tested source set, including after removing CIKM16, which contributes much of the original local improvement. Useful discovery therefore persists across the tested source compositions.

Removed sourcePE best gainS best gain
CIKM161.74%1.77%
KuaiRand-Pure2.14%1.59%
WeChat1.80%1.77%
Three sources1.89%0.497%

W4. Scaling the number of local searches

Reviewer comment.

Scaling the Number of Local Searches. How do performance and computational costs change as the number of local searches increases? In particular, what are the additional token usage, GPU compute, and wall-clock time associated with adding one local search to PE?

Response.

Thank you. We examine how adding local searches changes discovery outcomes, token usage, and GPU work. In accordance with our resource-reporting policy, we provide approximate GPU-hour estimates.

Experiment setting. We compare three-, four-, and five-source watch-time configurations, with 100 search iterations per source and KuaiRand-27K as the larger-scale target. The three-source configuration uses CIKM16, KuaiRec, and WeChat; the four-source configurations omit one source from the original five-source set. A peer improvement counts toward promotion only if its gain is at least 0.5%. Promotion requires one of two peer improvements with three sources, two of three with four sources, and two of four with five sources. These experiments characterize practical population changes, which also change source composition.

Best larger-scale gains.

Population / configurationBest larger-scale gain
Three sources1.89%
Four sources, remove CIKM161.74%
Four sources, remove KuaiRand-Pure2.14%
Four sources, remove WeChat1.80%
Five sources, original run2.17%
Five sources, repeat2.22%

The five-source entries are the original PE result and an independent repeat. Appendices G.1–G.2 report the best larger-scale gains for these configurations.

Best larger-scale gains are positive for all tested configurations: 1.89% with three sources, 1.74–2.14% with four sources, and 2.17% and 2.22% in the two five-source runs. These results show that PE finds useful larger-scale designs across the tested populations, with outcomes varying by source composition.

Approximate token use.

PopulationApprox. input tokens (M)Approx. visible output tokens (M)
Three sources32–3418–19
Four sources4724
Five sources6030

Token estimates include proposer and summarizer calls. Across these configurations, a rough budgeting increment per added source is approximately 13–15M input tokens and 6M visible output tokens.

Additional GPU work: Adding a source incurs its own 100-iteration search, which is on the order of ten GPU-hours in this watch-time setting, plus peer evaluations in both directions: its discoveries are tested on existing peers, and existing discoveries are tested on the new source. Total peer-evaluation work is approximately 3–15, 15–40, and 30–60 GPU-hours for the three-, four-, and five-source configurations, respectively.

Wall-clock time: We do not have a controlled comparison of the marginal elapsed time from adding one local search: concurrency and scheduling vary across configurations, making comparable measurements difficult to collect. In this deployment, PE's wall-clock hours and total GPU-hours are of the same broad order.

In the revised paper, we report these larger-scale gains in Appendices G.1–G.2 and approximate resource estimates in Appendix G.2.

W5. Figure presentation

Reviewer comment.

Figure Presentation. As a minor suggestion, incorporating more visual illustrations and reducing the amount of text could make the figures more engaging and easier to follow.

Response.

Thank you for this suggestion. We simplify Figure 1 to show the shared discovery loop and two tracks without experimental subset labels, and add Figure 2 to illustrate the subset hierarchy separately. The existing five-family comparison is moved into the main text as Figure 3. We retain the receiving-instance transfer profiles in Figure 4 and clarify their caption and discuss negative transfer in the accompanying text. Figure 5 is redrawn as a timeline and training window to explain the checkpoint-averaging example. We retain the original finding-led organization of the result paragraphs and remove repeated numerical details.

In the revised paper, these targeted figure, caption, and text changes clarify the workflow, experimental setup, and main findings.

We sincerely appreciate the time and effort you put into reviewing our paper. Your feedback helped us improve the manuscript. Thank you!

Reviewer_beFMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The author introduced Population Evolution (PE), a framework that improved model evolution across instances and training scales by enabling shared experimental evidence to be passed across instances (tasks). They demonstrate its effectiveness in two settings: collaborative discovery across instances and discovery transfer from small subsets to full data. They also introduced an RMD-Bench to evaluate the framework in two settings.

Critical strengths
  1. The paper discovered that sharing experimental feedback between local searches (from my interpretation, a local search is like an auto-research repo) can guide subsequent searches and improve the discovery results. It also shows that the feedback or discovered model designs can be transferred from a small subset to full dataset.
  2. They designed a benchmark covering five task families across two settings and conducted extensive experiments to validate the idea.
Critical weaknesses
  1. Presentation: The experimental sections, including the benchmark section, are somewhat difficult to follow.
    1. Many datasets and subset identifiers are introduced across different tracks. Repeated references to these names require frequent cross-checking.
    2. Track A and Track B are presented separately in the methodology, but their connection to the results in Sections 4 and 5 could be clearer. Sections 4.2 and 4.3 seem to correspond to Tracks A and B, respectively; explicitly including the track names in their headings would improve readability. Section 5 could also be stated more clearly.
    3. A long space is spent reporting changes in statistics. It would be better to put more emphasis on a clearer track specification, the overall trends across tasks, and the underlying message.
    4. A bit too frequent references to the appendices.
  2. Motivation for benchmark dataset selection: The reason for selecting these five task families and assigning different families to the two tracks could be better justified. Evaluating both tracks within at least one common task family would also help assess whether the findings hold across both settings. For instance, introducing a subset-to-full dataset search pipeline within a Track A task could provide a useful comparison.
  3. Limited ablations: Lines 157-158 suggest using a summary of mechanisms and outcomes. Does PE’s improvement come from sharing mechanisms, sharing measured outcomes, or their combination? Further analysis in Section 5 would also help explain the improvement. For instance, if the improvements come mainly from architecture changes, optimizer modifications, or other training interventions.
Questions for authors

Main questions refer to Critical Weaknesses. Two minor questions:

  1. If the findings also hold for other task families, specifically Track B when using different harness and proposal models (extension to 4.4)?
  2. How sensitive is PE to the wording of the shared-context prompt, the summarized level of mechanism, and the amount of shared evidence?
11 comments
AuthorsOpenReview ↗

Author Response

Thank you for your careful review and constructive suggestions. We have reorganized the presentation and added shared-information controls, component reversions, and configuration checks. We address each point below.

W1 — Overall presentation

Original comment.

Presentation: The experimental sections, including the benchmark section, are somewhat difficult to follow.

Our response.

We agree. The revision addresses each of your four presentation suggestions below.

W1a — Dataset and subset identifiers

Original comment.

Many datasets and subset identifiers are introduced across different tracks. Repeated references to these names require frequent cross-checking.

Our response.

Thank you for identifying this source of difficulty. We now collect the instance names and their roles in a single main-text table, presented below, so readers do not need to reconstruct the setup across sections.

Table R1. Task families and instance roles (revised manuscript Table 1).

TrackFamilyDiscovery sourcesLarger scale / full targetTransfer beyond discovery
ARankingKuaiLive, KuaiRand-1K, KuaiRand-Pure, KuaiRec, KuaiSARKuaiRand-27KKuaiVideo (CTR)
AWatch-timeCIKM16, KuaiRec, WeChat, KuaiRand-Pure, KuaiRand-1KKuaiRand-27KVK-LSVD
ARL algorithmSame task, five seeds; 1.5B policy3B policy; LCB-423LCB-632
BLLM pretrainingD2-A/B (2%); D5-A/B/C (5%)Complete training corpus (100%)—
BVLM pretrainingD2-A/B (2%); D5-A/B/C (5%)Complete training corpus (100%)—

Track A uses five sources and one larger-scale instance per family. Each Track B family uses five sources, two intermediate views, and one full target. Transfer targets are additional to these 34 core instances. LCB denotes LiveCodeBench.

Table R1 describes the original benchmark. The additional VK-LSVD subset-to-full experiment is reported in W2b below.

For Track B, D5-A/B/C are disjoint 5% subsets; D2-A and D2-B are nested inside D5-A and D5-B, respectively. The intermediate views are D10 clean (approximately 10%, disjoint from the source subsets) and D10+C (D5-C plus a fresh 5%). Revised Figure 2 illustrates this hierarchy separately from the overview. Figure 1 now explains only the workflow, and the introductory example uses descriptive subset sizes rather than identifiers.

W1b — Track names and experimental organization

Original comment.

Track A and Track B are presented separately in the methodology, but their connection to the results in Sections 4 and 5 could be clearer. Sections 4.2 and 4.3 seem to correspond to Tracks A and B, respectively; explicitly including the track names in their headings would improve readability. Section 5 could also be stated more clearly.

Our response.

Following your suggestion, we now name the tracks in the headings and place the two Track A subsections together before Track B. The revised structure is:

  • §4.2 Track A: Local Improvement and Design Transfer
  • §4.3 Track A: Search Harnesses and Proposers
  • §4.4 Track B: Full-Target Quality and Discovery Yield
  • §5 Analysis of Transfer and Discovered Designs, organized around direct reuse and transfer across instances (§5.1), shared evidence and later refinements (§5.2), and component reversions and shared-context controls (§5.3).

Section 5 now opens with the following roadmap:

We first measure whether designs found by one search improve another instance. We then trace an example of shared evidence and later refinements. Finally, we test which code changes and shared-context inputs matter in selected experiments.

Section 4 organizes the overall results by track. Section 5 organizes the analysis by question and identifies the task setting within each experiment or example.

W1c — Overall findings and numerical detail

Original comment.

A long space is spent reporting changes in statistics. It would be better to put more emphasis on a clearer track specification, the overall trends across tasks, and the underlying message.

Our response.

We agree that the main text should make the overall finding easier to identify. We retain the original conclusion-led organization, remove repeated numerical details, and bring the five-family comparison into the main text as Figure 3. Its results are presented below in table form.

Table R2. Best final-stage gains in the original completed discovery runs (revised manuscript Figure 3, presented here as a table).

TrackFamilyEvaluation stageS / Direct best gainPE best gain
ARankingLarger-scale evaluation4.70%6.19%
AWatch-timeLarger-scale evaluation1.77%2.17%
ARL algorithmLarger-scale evaluation1.95%6.64%
BLLM pretrainingFull-target evaluation0.92%2.48%
BVLM pretrainingFull-target evaluation5.05%8.78%

Track A compares PE with standalone search (S) at matched source iterations. Track B compares PE with Direct under matched recorded training–evaluation GPU-hours. Each row reports the completed run's best final-stage gain using that family's metric and fixed reference.

The original completed runs show higher best final-stage gains across all five families. The revised text separates this overall finding from local improvement and transfer to new instances, with detailed measurements in the table and figure.

W1d — Appendix references

Original comment.

A bit too frequent references to the appendices.

Our response.

We have consolidated repeated appendix pointers and placed the information needed to follow the experiments directly in the main text: the instance map in Table 1 (shown as Table R1 above), the subset hierarchy in Figure 2, the comparison protocols in §4.1, and the five-family overview in Figure 3 (Table R2 above). The appendices retain extended measurements and implementation details. This reduces the need to leave the main text while following the experimental setup and principal findings.

W2a — Task-family selection and track allocation

Original comment.

Motivation for benchmark dataset selection: The reason for selecting these five task families and assigning different families to the two tracks could be better justified.

Our response.

Thank you for this suggestion. Our selection follows two considerations: whether a task supports the discovery setting being studied, and whether improving its models or training procedures is practically meaningful. We therefore focus on well-studied workloads with executable design spaces and clear evaluation protocols.

Track A: related instances with affordable repeated evaluation. We require compatible code interfaces across instances, meaningful variation across datasets or independent runs, and sufficiently affordable evaluations for searches to exchange measured results. Ranking and watch-time prediction provide related recommendation datasets with reusable model components. RL algorithm discovery complements these settings with independent seeded searches over a shared policy-training interface.

Track B: expensive full-target evaluation with affordable smaller-scale experiments. We require smaller training settings that preserve the objective and editable code interface, together with full-target evaluation to assess whether discoveries remain useful at scale. LLM and VLM pretraining naturally fit these requirements because evaluating each design through full training is expensive.

These five families combine compatibility with the evaluation protocols and practical relevance: improvements can benefit recommendation models, policy training, and language or vision-language pretraining. The same selection principles can also guide extensions to other task families with suitable interfaces and evaluation settings.

Track assignment depends on the workload and evaluation protocol. A family can support both tracks. For example, ranking or watch-time tasks can use Track A when repeated evaluations on related instances are affordable, and a Track B subset-to-full pipeline when training on a larger dataset makes direct search expensive. Following your suggestion, we added precisely such an extension on VK-LSVD: it evaluates watch-time prediction under Track B, complementing the original Track A experiments within the same family. We describe its setup and results in W2b below.

W2b — Evaluating both tracks within one family

Original comment.

Evaluating both tracks within at least one common task family would also help assess whether the findings hold across both settings. For instance, introducing a subset-to-full dataset search pipeline within a Track A task could provide a useful comparison.

Our response.

Thank you for this suggestion. We added subset-to-full discovery on VK-LSVD, extending the watch-time prediction family, already evaluated under Track A, to Track B.

Experiment setting. To evaluate both tracks within the same task family, we add a subset-to-full watch-time discovery experiment on VK-LSVD. PE searches on smaller data subsets and evaluates promising designs on the full VK-LSVD target, while Direct searches directly on the full target. Both methods use a matched total compute budget. Direct performs five full-target searches with 24 iterations each, totaling 120 iterations.

Hierarchy and promotion. We use the same subset proportions and intermediate-view construction as in LLM/VLM pretraining: three disjoint 5% source subsets, with two 2% source subsets nested within two of them. The intermediate views are a disjoint 10% subset and the third 5% source subset combined with a fresh 5%. All improvement gates use the watch-time threshold of 0.5% relative MAE reduction: source publication, improvement on at least two of four peers, and improvement on each intermediate view.

Reference and evaluation. We follow the same Track B reference-calibration, split-construction, and evaluation protocol as in LLM/VLM pretraining, using MAE for watch-time prediction. Reference configurations are established before discovery and held fixed across methods. The full VK-LSVD reference MAE is 15.25 seconds. Gains are relative MAE reductions against the corresponding fixed reference.

Table R2b. Subset-to-full watch-time discovery on VK-LSVD.

MethodFull-target evaluation attemptsGain ≥1%Gain ≥1.5%Gain ≥2%Best gain
PE35322701.91%
Direct12030601.77%

Gain is the relative MAE reduction against the reference; threshold columns count full-target evaluation records.

Findings. Under the matched total compute budget, PE achieves a higher best full-target gain (1.91% versus 1.77%) and more evaluations reaching gains of at least 1% (32 versus 30) and 1.5% (27 versus 6). PE obtains these results with 35 full-target evaluation attempts, compared with 120 for Direct. This experiment supports subset-to-full discovery for finding stronger watch-time designs within the same total compute budget.

Together with the original Track A watch-time experiments, this provides the suggested common-family evaluation and extends subset-to-full discovery beyond LLM/VLM pretraining. It supports the selection principle explained above: track allocation reflects the workload and evaluation protocol, rather than an intrinsic restriction on the task family.

In the revised paper, we briefly summarize the VK-LSVD experiment and its main finding in Section 4.4 and present the results table in Appendix G.7.

Comments on the paper

6 comments

We thank all reviewers for their time and constructive feedback. We have updated the paper, with revisions highlighted in red, to improve presentation and clarify the benchmark protocols. We have also added ablations of discovered model components and PE's sharing and screening mechanisms, along with comparisons against EMO-STA and longer standalone search. We further report independent discovery repetitions, analyses of idea transferability, and extensions to MLS-Bench and to Track B within the watch-time prediction family. Together, these additions strengthen the evidence for PE's ability to discover better designs through cross-source evolution.

We welcome any further feedback during the remaining discussion period and would be happy to clarify any outstanding questions. We hope these revisions address your concerns and would be grateful if you would reconsider your assessment in light of the additional evidence and clarifications.

We sincerely appreciate the thoughtful comments and constructive feedback from you. We hope our responses have adequately addressed your questions and concerns. As the rebuttal period deadline approaches, we look forward to your feedback on our responses and would be happy to clarify any additional questions you may have.

Thank you for your thoughtful comments and constructive feedback. Since our initial response, we have further revised the abstract, the introduction, and our description of the experimental settings to improve readability. As the discussion period draws to a close, we would welcome any further feedback and would be happy to clarify any remaining points.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.