acceptodds
Under review as a conference paper at ICLR 2027

ROAR: Unifying Runs across Heterogeneous AI-Driven Research Systems

Abstract

Each run of an AI-driven research system (ADRS) is an expensive search over a vast solution space, and dependable evaluation requires many runs, making run data both costly to produce and valuable to retain for large-scale analysis. Yet this data remains fragmented: teams operate in isolation, ADRS frameworks emit results in different formats, and no shared infrastructure exists to aggregate or compare runs across problems and systems. We present ROAR, a solution for systematically unifying and analyzing heterogeneous ADRS outputs. ROAR addresses two challenges: reconciling heterogeneous ADRS outputs and enabling analytics across runs with different objectives and scoring functions. We achieve this through a relational schema and parsing layer that normalize heterogeneous ADRS outputs while preserving data lineage and temporal structure, and accommodating new systems without requiring schema modifications. From building a corpus of more than 900 runs from multiple ADRS, we show how pooled data can reveal properties of problem landscapes that are difficult to observe. Consistent with prior work, runs with identical configurations may converge to different scores. We find that many runs realize most gains early, and that the effectiveness of different strategies for incorporating prior solutions into the search process varies across problems. We further show that the pooled corpus is actionable and not merely analytical by using ROAR to configure ADRS runs. Together, these results illustrate how pooled ADRS data can expose problem-dependent structure in search behavior that is difficult to detect from any single system, team, or benchmark. Such cross-cutting insights are difficult to obtain while runs remain siloed; ROAR is the first infrastructure designed to unify them.

Primary area:
datasets and benchmarks
Keywords:
AI driven research systems

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qMkLOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ROAR, which is a shared infrastructure for collecting and analyzing runs from heterogeneous AI-driven research systems. ROAR maps different frameworks into a common relational representation of campaigns, candidate solutions, evaluations, and parent/context relationships, while retaining references to the original artifacts. The goal is to make expensive research runs reusable rather than leaving them isolated within individual projects.

In the paper, the authors use a pooled corpus to study variation in final outcomes, the likelihood of improvement after stagnation, and associations between candidate diversity, context composition, and search progress. They also fit a budget-allocation policy that decides when to stop a campaign and restart. The paper’s contribution is in providing an infrastructure and the contributed run corpus, as well as the exploratory analyses demonstrating possible uses of that corpus.

Critical strengths
  1. The motivation is timely in a sense that many current systems record candidates, evaluations, and search history in different ways, which makes prior runs difficult to reuse. ROAR’s distinction between mutation ancestry and examples supplied as context addresses this point by preserving information about how a candidate was produced.
Critical weaknesses
  1. Some conclusions go beyond what the results show. Runs ending with different scores do not necessarily mean they found different local optima, but some may simply have produced invalid solutions or used different settings. Likewise, finding that certain context examples are associated with improvement does not show that those examples caused it. The analyses are interesting, but I would be more comfortable treating them as patterns to investigate rather than guidance on how to design a search system.
  2. The benefit of the budget policy needs a stronger test. The policy is evaluated by replaying existing runs rather than launching new ones, and those existing runs also help choose the policy. This may make the improvement look larger than it would be in practice. It is also unclear whether the learned policy beats a simple approach of running several shorter searches, or whether matching iteration counts really matches the cost of running them.
Questions for authors
  • Does the learned policy do better than simply splitting the budget across several shorter runs?
  • How can the authors ensure that different final scores reflect different local optima, rather than some runs producing invalid solutions? - Have the authors try comparing the solutions themselves?
  • Have the authors tried the policy on new runs that were not used to develop it? (Does the main results generalize?)
  • Is the Palindrome Hamiltonian Path (290 runs) excluded from all analyses? (in Table 3)
  • What does "context" mean for Best-of-N candidates?
  • Does code diversity correlate with code length, and do the RQ3 results hold after controlling for it?
  • How was the correctness of agent-written parsers validated beyond Pydantic type checks? In other words, how can we ensure that it has mapped the fields correctly?
  • How are failed or invalid candidates scored? If you remove runs whose final score reflects evaluator failure, does the bimodality in RQ1 survive?
4 comments

We thank the reviewer for identifying several places where the claims and validation procedures require greater precision. We address each concern below.

Scope of the conclusions. Multimodal final scores show different run outcomes, but do not establish distinct local optima. Similarly, RQ3 identifies associations between context composition and subsequent improvement, not causal effects. We will correct the claims and recommendations accordingly.

Budget-policy evaluation. Appendix J reports a disjoint-pool sensitivity analysis in which the policy is fit on half of each configuration's campaigns and relaunches are drawn from the other half. The estimated gain is smaller but remains positive in 10 of 12 configurations. We are extending this evaluation with repeated disjoint splits. Under the same 50-iteration budget, we are also adding equal-split baselines, including 2×25 and 5×10, and pre-specified fixed-patience baselines that do not use configuration-specific fitting. These comparisons separate the benefit of multiple starts, conditional stopping, and configuration-specific policy fitting.

Invalid candidates and RQ1 sensitivity. Candidates that fail to compile, crash, time out, or return an infeasible output receive a score of zero and remain part of the search trajectory. However, we agree that an all-invalid campaign may represent search failure rather than a low-scoring optimization basin. We will therefore report the number of campaigns containing no valid candidate and repeat the RQ1 analysis conditional on each campaign producing at least one valid candidate. This will show whether the reported multimodality reflects variation among valid solutions, a mixture of successful and failed searches, or both. ROAR preserves contributed evaluator outputs, but it does not independently certify evaluator correctness; we will state this limitation separately.

Palindrome Hamiltonian Path. All 290 Palindrome Hamiltonian Path campaigns have a final score of zero because no candidate passed the evaluator. Individual failed candidates remain informative observations of search behavior, but this problem has no within-problem final-score variation for RQ1, no positive improvement events for RQ3, and no empirical headroom for RQ4 policy comparison. For RQ2, the empirical escape probability is identically zero. We will state this explicitly in the dataset description and relevant results sections.

Meaning of context for Best-of-N. In these runs, the Best-of-N mechanism selects the highest-scoring program found so far as the parent and holds it fixed for the next N=5 successfully evaluated candidates. Each prompt asks the model to improve that parent. It also includes four context programs sampled afresh each iteration from the ten highest-scoring programs, excluding the parent, plus a short summary of recent high-scoring attempts. After each block, the mechanism again selects the best program found so far and repeats. We will document this implementation and distinguish the mechanism's N from the campaign-allocation parameter n used in RQ4.

Code diversity and code length. We agree that code length may confound embedding-based diversity. We will report the correlation between diversity and token length and refit the RQ3 mixed-effects models with standardized token length as an additional fixed effect, retaining campaign random intercepts and iteration phase. We will report whether the direction and magnitude of the diversity associations remain stable.

Validation of agent-written parsers. The schema enforces structural and type consistency, while semantic correctness is assessed by manually comparing parsed samples with the corresponding raw logs before a parser is merged. We will describe this validation procedure, including its scope and limitations, more clearly in the paper. We are also investigating comparison with an existing deterministic parser as an additional check on field-mapping accuracy.

Reviewer_qMkLOpenReview ↗

Response to the Authors

Thank you for outlining the planned revisions! However, since the response does not include any preliminary results showing whether the results will generalize or actually support the expected outcomes, I am unable to revise my original assessment at this time. I believe the paper would benefit from these revisions once they are implemented.

Update: This response reports the previously promised analyses and respective paper revisions.

Budget-policy evaluation. The comparison of the fitted (kstop,n)(k_{\mathrm{stop}}, n) policy with equal splits of the budget and prespecified fixed-patience rules, all scored on campaigns that played no part in fitting, is reported in the common comment above together with the resulting restatement of RQ4 in the revised paper. We refer the reviewer to it.

Code diversity and code length. Code diversity falls with code length, so both RQ3 analyses in the revised paper (Section 4.4) now control for code token length. The numbers shift slightly but no conclusion changes. The early-diversity correlations are unchanged (Polynomino Packing ρ=0.16\rho = 0.16, p=0.004p = 0.004, Bounded 2D Knapsack ρ=0.04\rho = 0.04, not significant). In the context-composition models, every significant factor on Polynomino Packing and CUDA Attention keeps its sign and magnitude. The one casualty is on Bounded 2D Knapsack, where the better context ratio goes from a small negative effect (−0.09-0.09) to no significant effect (+0.02+0.02, p=0.72p = 0.72), and score diversity shrinks from −0.21-0.21 to −0.13-0.13 while keeping its sign. The problem-dependent direction of the effects, and the Knapsack and Polynomino Packing readings built on it, stand as submitted.

Validation of agent-written parsers. We validated the parsed records against the raw run directories with an independent reader, written from scratch and sharing no code with the parsers. It re-read the raw run directories of 911 campaigns on the three combinatorial problems and recovered the same 47,168 candidates with the same scores and iteration indices as ROAR, and the same best score and iteration count for every campaign. The remaining 17 campaigns of the corpus are not covered by this check. This check is reported in the revised paper in the appendix "Validation of Parsed Records".

Scope of the conclusions. The revised paper no longer treats multimodal final scores as evidence of distinct local optima. RQ1 now describes what is measured, the shape of the final-score distribution across identical configurations, and the inference to basins is stated once as a hypothesis with its limits spelled out in a new appendix paragraph and figure ("Basins versus modes"). The sentences that read a continuous spread as "multiple local optima" or a binary spread as a "binary basin structure" have been reworded in RQ1, RQ2 and RQ3 to speak of continuous and binary outcome distributions. All of these edits are highlighted in blue in the revised paper.

Reviewer_qMkLOpenReview ↗

Response to the Authors

Thank you for the additional experiments and analyses! They address several of my concerns:

  • The budget policy evaluation comparing equal split vs. specified fixed rules and the restated RQ4 clarified my confusion.
  • The robustness analyses for code diversity and code length and the validation of parsers helped address some of my concerns.
  • The reformulated RQ2 was also helpful and I believe this is a better reflective of the paper's research contribution.

Taking these updates, the revised paper, and also reading other reviews into account, I have increased my score from 1 to 2.

Reviewer_y6UtOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents ROAR (Repository of ADRS Runs), a solution for systematically unifying and analyzing outputs from heterogeneous AI-Driven Research Systems (ADRS). It aims to enable analytics across runs with different objectives and scoring functions. They describe their main contribution as the first general data aggregation infrastructure for ADRS: a unified relational schema with a parser layer, enabling the cross-problem and cross-system analysis. They also contribute a dataset with more than 900 ADRS runs spanning 7 problems, 14 generator models and 6 optimization mechanisms across multiple ADRS frameworks. They also do an exploratory data analysis to answer four research questions:

  1. Whether runs with identical configurations can end at different scores.
  2. Whether the rule-of-3 stopping heuristic fits ADRS runs.
  3. Whether diversity and the choice of past programs shown to the LLM predict results.
  4. Whether the pooled data can be used to configure new runs with a budget policy.
Critical strengths
  1. The paper proposes a schema and a parser to unify runs (outputs) on different ADRS, both on the same problem and across problems. This unification can help researchers avoid repeating expensive runs.
  2. The research questions raised and answered using the dataset created by the paper can give us insights on various ADRS and the kinds of problems being solved. The budget policy (RQ4) is an interesting direction: ADRS will eventually have to run under a budget, and thinking in terms of a budget policy is a good way to approach that.
  3. The dataset created by the paper would be a great resource for people to study and come up with newer insights.
Critical weaknesses
  1. For RQ4, the paper reports "the size of the gain is likely optimistic", since the budget policy is tested on the same runs it was learned from (line 1236). The RQ4 gains need to be re-done, on new ADRS runs or on runs held out from fitting, for us to confirm them.
  2. The paper states the dataset (lines 492–493) will be made available, but at the time of review the repository only has code, with no dataset files or license, so others cannot use or verify the dataset. This also matters for judging the contribution: it is unclear how many of the runs are new and how many were already analyzed in Ellis & Castro (2026). Building on earlier runs is reasonable, but the paper should state the overlap, since a large overlap would make the dataset a much smaller contribution.
  3. The majority of the runs (914 of 928) come from three problems, but the main analysis focuses on only two of them (Knapsack and Polyomino). Palindrome Ham. Path (290 runs) is not analyzed, which raises the question of what the results would be on the third problem.
  4. The promise of ROAR is that it unifies runs across problems, but it is not clear what value is added by pooling across problems. Cross-problem analysis is limited in answering the RQs. Only RQ2 aggregates three problems, and one of the problems has just 3 runs. The other analyses are reported per problem.
  5. Of the four research questions, RQ3 appears to be the main new analysis: RQ1 confirms prior work, and the RQ4 gains need to be re-evaluated. ( RQ1 repeats prior work (Ellis & Castro, 2026) on the same two problems, and while the authors explicitly call our attention to RQ1, they also state it confirms the prior work.)
Questions for authors
  1. How much of the corpus overlaps with the runs analyzed in Ellis and Castro (2026)?
  2. Have any new ADRS runs been executed with the fitted policy?
  3. Where is the dataset hosted, and under what license?
  4. Is there a reason why the Palindrome Ham. Path (290 runs) is not analyzed? Do the findings hold for it?
  5. Is there any advantage seen by pooling across problems? Does it reveal anything that per-problem analysis doesn't?
5 comments

We thank the reviewer for the careful assessment of the corpus, its relationship to prior work, and the scope of the analyses. We address each concern below.

Independent evaluation of RQ4. We agree that the main replay estimate is likely optimistic because trajectories used to fit the policy can also supply relaunches. Appendix J reports a disjoint-pool sensitivity analysis in which the policy is fit on half of each configuration's campaigns and relaunch trajectories are drawn from the other half. The estimated gain is smaller but remains positive in 10 of 12 configurations. We are extending this to repeated disjoint splits and will report uncertainty across splits. We are also adding equal-split baselines, including 2×25 and 5×10, and prespecified fixed-patience rules under the same 50-iteration budget. These comparisons separately test the benefit of replication, conditional stopping, and learning a configuration-specific policy. Prospective runs with the fitted policy are also in progress; we will report them separately if completed during discussion.

Dataset location, license, and overlap with Ellis and Castro (2026). The dataset was available through the anonymous Hugging Face link in the repository README, but this route was not sufficiently prominent, and the license was not stated. We have made the dataset link explicit in the repository and will state the applicable license in both the repository and paper. Of the submitted corpus's 928 campaigns, 914 were included in the corpus analyzed by Ellis and Castro (2026), while 14 were not previously analyzed there. The overlap is concentrated in three combinatorial problems (Polyomino Packing, Palindrome Hamiltonian Path, and Bounded 2D Knapsack), all run with SkyDiscover using the EvoX, AdaEvolve, and best-of-N search strategies. Since submission, we have added 144 further campaigns across OpenEvolve, GEPA, and ShinkaEvolve, none of which overlap with Ellis and Castro (2026). The unified schema, parser layer, and public release of the normalized and raw run corpus are new contributions of this work. We will add a provenance breakdown to Appendix B.

Palindrome Hamiltonian Path. All 290 Palindrome Hamiltonian Path campaigns have a final score of zero because no candidate passed the evaluator. Therefore, there is no final-score variation for RQ1, no observed improvement events for RQ3, and no empirical headroom for policy comparison in RQ4. For RQ2, the escape rate is identically zero. We will state this explicitly in the dataset description and the relevant results sections rather than leaving the exclusion unexplained.

Value across problems. We agree that the submitted paper primarily demonstrates a common representation and analysis pipeline across problems, not a policy that transfers across unrelated problems. Pooling currently provides two benefits: it makes the same ordinal and artifact-based analyses executable despite incompatible score scales, and it exposes problem-level heterogeneity that would be invisible from one benchmark alone. For example, the direction of the context-composition associations differs across Knapsack, Polyomino Packing, and CUDA Attention. The RQ4 policy itself is fit within configuration, not across problems. We will more clearly distinguish what the present results demonstrate versus the opportunities for future work on general cross-problem policy transfer.

Novelty relative to prior work. We agree that RQ1 is an independent replication and extension of the basin-structure result in Ellis and Castro (2026). The new contributions are the general cross-system data infrastructure and public corpus, the conditional-escape analysis in RQ2, the context-composition analysis in RQ3, and the budget-allocation formulation in RQ4. We will qualify the empirical RQ4 claim based on the fully disjoint evaluation rather than presenting all four research questions as equally established new findings.

Thank you for the detailed response and clarifications. The dataset is now accessible from the repository README, and I appreciate it being made explicit; a license is still needed. The promised RQ4 analyses would strengthen that part of the paper. The 144 new campaigns are not part of the submitted paper; if included, they should be folded into the paper and its analyses.

However, the main weaknesses remain. 914 of the 928 campaigns were already analyzed in Ellis & Castro (2026), which strengthens my concern about the dataset as a contribution: the schema and parser layer may be new, but the data largely is not, and RQ1 is largely a replication. Since all 290 Palindrome Ham. Path runs score zero, they carry little information for RQ1–RQ4, so the part of the corpus that contributes to the results is smaller than the headline number suggests. For these reasons, my score remains unchanged.

Update: This response reports the previously promised analyses and respective paper revisions.

Independent evaluation of RQ4. The repeated disjoint splits, the equal-split and fixed-patience baselines, and evaluation on independent runs are reported in the common comment above, together with the respective updates to RQ4 (§4.5) in the revised paper. We refer the reviewer to it.

Novelty of the dataset relative to prior work. We have made the distinction between ROAR’s contribution and its application to existing datasets, including their provenance, clearer and more prominent throughout the revised paper. The revised contribution claims focus on what ROAR adds: the first public, unified release of these runs and the schema and ingestion layer that preserve candidate histories, measurements, lineage, and artifacts for common analysis across ADRS. Revisions are highlighted in blue.

Dataset location and license. The dataset is released under the Creative Commons Attribution 4.0 International license (CC BY 4.0), and the license is stated in both the repository and the revised paper.

Reviewer_whDPOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

ROAR provides a unified infrastructure for collecting and analyzing runs from heterogeneous AI-driven research systems. Its schema and parsing layer preserve candidate lineage, contextual relationships, measurements, and raw artifacts. The authors assemble 928 runs containing 48,210 candidates and demonstrate analyses of search outcomes, stagnation, context composition, and budget allocation.

Critical strengths
  1. The unified schema preserves candidate lineage and context relationships across heterogeneous ADRS frameworks.
  2. The corpus of 928 runs provides a useful resource for repeated-run and cross-system analysis.
  3. The budget-allocation study offers promising evidence that pooled history can guide practical decisions.
Critical weaknesses
  1. The budget-allocation evaluation would be stronger with simple fixed-restart baselines and fully separate fitting and replay pools. These comparisons would clarify the additional benefit of learning policies from pooled history.

  2. Some analytical interpretations need qualification. In particular, observing escape rates below a probability upper bound does not itself establish miscalibration. The stagnation results would be more convincing when presented as empirical conditional escape probabilities.

Questions for authors

Could you compare the adaptive allocation policy against fixed restart strategies under the same total budget, to isolate the benefit of adaptation?

3 comments

We thank the reviewer for identifying two important limitations: the current RQ4 comparison does not separate the benefit of restarting from the additional benefit of learning a configuration-specific policy, and the rule-of-three upper bound should not be treated as a calibration target.

Simple restart baselines. We agree. The submitted result establishes that the combined ROAR policy outperforms one uninterrupted 50-iteration campaign, but it does not isolate which part of the gain comes from replication versus learned adaptation. Under the same 50-iteration budget, we are adding: (i) equal splits across shorter runs, including 2×25 and 5×10, and (ii) prespecified patience rules that restart after k∈{5,10,20}k\in{5,10,20} non-improving iterations. We will compare these directly with the learned, configuration-specific (kstop,n)(k_{\mathrm{stop}},n) policy. This distinguishes the benefit of multiple starts, conditional stopping, and adaptation from pooled history.

Separate fitting and evaluation pools. We also agree that reusing trajectories for fitting and replay can make the estimated gain optimistic. The submitted appendix includes a disjoint half-split sensitivity analysis: fitting on one half of each configuration's campaigns and drawing replay trajectories only from the other half reduces the estimated gain, but it remains positive in 10 of 12 configurations. We are extending this to repeated disjoint splits so that every reported evaluation is out of sample with respect to policy selection. Because some configurations contain only 8–33 campaigns, these estimates will have wider uncertainty, which we will report explicitly. We are also launching prospective runs with the fitted policy; if they do not finish during discussion, we will clearly distinguish completed out-of-sample replay results from pending prospective validation.

Stagnation interpretation. We agree that the current "miscalibration" language is incorrect. The rule of three gives an approximate 95% upper confidence bound under an i.i.d. Bernoulli model, not a predicted escape rate, so observing empirical rates below 63% does not demonstrate miscalibration. We will remove that claim and more precisely show RQ2 is estimating empirical conditional escape probabilities:

Pr⁡(improvement in the next ⌊k/3⌋ iterations∣k consecutive non-improving iterations).\Pr(\text{improvement in the next }\lfloor k/3\rfloor\text{ iterations}\mid k\text{ consecutive non-improving iterations}).

We retain the ⌊k/3⌋\lfloor k/3\rfloor horizon only for continuity with the original construction, not as a calibration target. We will revise the RQ2 question, Figure 2, the results discussion, and the conclusion accordingly, and report campaign-level uncertainty and the treatment of censored episodes.

Thank you to the authors for the rebuttal and the additional clarifications. While I appreciate the effort put into addressing my questions, the primary weaknesses raised in my review still persist. As a result, I am inclined to keep my current score.

Update: This response reports the previously promised analyses and respective paper revisions.

Simple restart baselines and separate fitting and evaluation pools. Both analyses are reported in the common comment above, which compares the fitted (kstop,n)(k_{\mathrm{stop}}, n) policy with equal splits and fixed patience rules under the same budget, with every number scored on campaigns that played no part in fitting. We refer the reviewer to it for the results and respective updates to RQ4 (§4.5).

Stagnation interpretation. As promised, the "miscalibration" language has been removed from the revised paper. The rule of three gives an approximate 95% upper confidence bound under an i.i.d. Bernoulli model, not a predicted escape rate, so empirical rates below 63% do not demonstrate miscalibration, and the paper no longer says they do. RQ2 is now stated as estimating empirical conditional escape probabilities.

Comments on the paper

2 comments

Several reviews raised complementary questions about the RQ4 evaluation; we report the completed analyses together here and address their implications in each reviewer thread.

Restart baselines with disjoint fitting and evaluation pools. Every RQ4 number below is scored on campaigns that played no part in fitting the policy. The single half-split of Appendix K is replaced by repeated random half-splits of each configuration's campaigns. The (kstop,n)(k_{\mathrm{stop}}, n) policy is fit on one half, and the evaluated campaign and every relaunch trajectory come from the other half. Alongside the fitted policy we score five prespecified rules that use no fitting: two equal splits of the budget (2×25 and 5×10) and three fixed patience rules that restart after 5, 10 or 20 non-improving iterations. All rules run under the same 50-iteration budget, on the same held-out campaigns, with the same relaunch draws, and against the same reference: the best score the same campaign reached within 50 uninterrupted iterations. A rule's score is the best score found across its relaunches within that 50-iteration budget. We evaluate the 12 RQ4 configurations in our corpus, comprising 152 campaigns with 8–33 campaigns per configuration, using 200 random splits. We then apply the same procedure to the harness-generalization corpus of Gupta et al., released August 14, 2026, comprising 2,107 runs across 27 configurations with 10–342 runs per configuration, using 40 random splits. For each configuration and rule, we average the gain over held-out evaluations and random splits. “Positive configurations” counts configurations with positive mean gain; “worst configuration” is the minimum configuration-level mean gain; and “mean gain” is the unweighted mean across configurations.

RuleOur corpus: positive (of 12)Our corpus: best (of 12)Our corpus: mean gain (points)External corpus: positive (of 27)External corpus: best (of 27)External corpus: mean gain (points)
2×25122+9.2243+2.8
5×10112+12.4192+1.4
Restart after 5105+12.22112+2.0
Restart after 10111+11.7276+3.9
Restart after 20121+9.5262+3.1
Fitted (kstop,n)(k_{\mathrm{stop}}, n)101+8.2212+2.8

Reallocating the budget across restarts improves the best score reached, on both corpora every fixed rule and fitted policy have a positive mean gain. The best rule differs between configurations. Restarting after 5 non-improving iterations is best most often, but it is negative in 2 configurations on our corpus and 6 on the external corpus, where restarting after 10 is negative in 1 and none. The fitted policy has a lower mean gain than fixed rules on our corpus and than restarting after 10 on the external corpus, and is the best choice in only three configurations overall.

Fixed rule is the practical recommendation. Which fixed rule is best depends on the configuration, and no rule is best in more than half of the configurations on either corpus, so whether a rule transfers can only be judged by comparing rules over many configurations, which the pooled corpus allows. RQ4 is restated accordingly in the revised paper, and language implying that a learned policy is required or superior has been removed.

The revised paper reports these results in RQ4 (§4.5) and removes language implying that a learned policy is necessary or uniformly superior. We welcome clarification if any aspect of the shared RQ4 concern or any other concern remains unresolved.

Several reviewers raised concerns about coverage beyond data collected by Ellis and Castro, we have extended the coverage of the analysis:

Coverage beyond Ellis and Castro (2026). We agree that RQ1 and RQ2 on the submitted corpus rest largely on runs that Ellis and Castro collected. The revised paper therefore adds the corpus of Gupta et al. (2026), with the authors' permission, 3,269 campaigns on problems that Ellis and Castro did not study, collected by an independent team with a different search mechanism and different models. As a result, RQ1 and RQ2 analyze two additional problems, circle packing and the Heilbronn triangle problem. RQ3 is omitted from this extension because the new campaigns' mechanism largely does not use context programs.

The results are in the revised paper. For RQ1, the run-to-run spread on circle packing and the Heilbronn triangle is as wide as on 2D Knapsack and Polynomino Packing. The same configuration ends anywhere between 0.5 and 1.0 on circle packing. However, Multimodality is rare. On circle packing and the Heilbronn triangle, 1 of 16 configurations is confirmed bimodal. On 2D Knapsack and Polynomino Packing, 12 of 24 are. One confirmed difference between the two sets of problems is the seed program. Every 2D Knapsack and Polynomino Packing campaign starts from a program that does not pass the evaluator, and 18% of them never fix it. Circle packing and Heilbronn triangle campaigns start from a working program. We report this as a possible reason for the gap, not a confirmed one. Nonetheless, we still observe large variation in score outcomes with identical configurations.

For RQ2, escape rates stay far below the 63% bound on all five problems with the new problems being slightly higher but never exceeding around 20%. On 2D Knapsack and Polynomino Packing, the escape rate does not change with stagnation length. On circle packing and the Heilbronn triangle, it rises. The paper traces this to the seed program and to the number of candidates per iteration. The mechanism used for circle packing and the Heilbronn triangle generates between 1 and 16 candidates per iteration, and the mechanisms used for 2D Knapsack and Polynomino Packing generate one per iteration. When we restrict to circle packing and Heilbronn triangle campaigns with one candidate per iteration, and to 2D Knapsack and Polynomino Packing campaigns that found a working solution, the escape rates are close. Therefore, the chance of escaping when stagnating is still rare.

Applying the unchanged RQ1, RQ2, and RQ4 analyses to the independently produced Gupta et al. corpus demonstrates ROAR’s central contribution: a common representation that supports the same analyses across heterogeneous teams, frameworks, formats, and configurations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.