- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces ROAR, which is a shared infrastructure for collecting and analyzing runs from heterogeneous AI-driven research systems. ROAR maps different frameworks into a common relational representation of campaigns, candidate solutions, evaluations, and parent/context relationships, while retaining references to the original artifacts. The goal is to make expensive research runs reusable rather than leaving them isolated within individual projects.
In the paper, the authors use a pooled corpus to study variation in final outcomes, the likelihood of improvement after stagnation, and associations between candidate diversity, context composition, and search progress. They also fit a budget-allocation policy that decides when to stop a campaign and restart. The paper’s contribution is in providing an infrastructure and the contributed run corpus, as well as the exploratory analyses demonstrating possible uses of that corpus.
Critical strengths
- The motivation is timely in a sense that many current systems record candidates, evaluations, and search history in different ways, which makes prior runs difficult to reuse. ROAR’s distinction between mutation ancestry and examples supplied as context addresses this point by preserving information about how a candidate was produced.
Critical weaknesses
- Some conclusions go beyond what the results show. Runs ending with different scores do not necessarily mean they found different local optima, but some may simply have produced invalid solutions or used different settings. Likewise, finding that certain context examples are associated with improvement does not show that those examples caused it. The analyses are interesting, but I would be more comfortable treating them as patterns to investigate rather than guidance on how to design a search system.
- The benefit of the budget policy needs a stronger test. The policy is evaluated by replaying existing runs rather than launching new ones, and those existing runs also help choose the policy. This may make the improvement look larger than it would be in practice. It is also unclear whether the learned policy beats a simple approach of running several shorter searches, or whether matching iteration counts really matches the cost of running them.
Questions for authors
- Does the learned policy do better than simply splitting the budget across several shorter runs?
- How can the authors ensure that different final scores reflect different local optima, rather than some runs producing invalid solutions? - Have the authors try comparing the solutions themselves?
- Have the authors tried the policy on new runs that were not used to develop it? (Does the main results generalize?)
- Is the Palindrome Hamiltonian Path (290 runs) excluded from all analyses? (in Table 3)
- What does "context" mean for Best-of-N candidates?
- Does code diversity correlate with code length, and do the RQ3 results hold after controlling for it?
- How was the correctness of agent-written parsers validated beyond Pydantic type checks? In other words, how can we ensure that it has mapped the fields correctly?
- How are failed or invalid candidates scored? If you remove runs whose final score reflects evaluator failure, does the bimodality in RQ1 survive?
4 comments
Response to the Authors
Thank you for outlining the planned revisions! However, since the response does not include any preliminary results showing whether the results will generalize or actually support the expected outcomes, I am unable to revise my original assessment at this time. I believe the paper would benefit from these revisions once they are implemented.
Update: This response reports the previously promised analyses and respective paper revisions.
Budget-policy evaluation. The comparison of the fitted policy with equal splits of the budget and prespecified fixed-patience rules, all scored on campaigns that played no part in fitting, is reported in the common comment above together with the resulting restatement of RQ4 in the revised paper. We refer the reviewer to it.
Code diversity and code length. Code diversity falls with code length, so both RQ3 analyses in the revised paper (Section 4.4) now control for code token length. The numbers shift slightly but no conclusion changes. The early-diversity correlations are unchanged (Polynomino Packing , , Bounded 2D Knapsack , not significant). In the context-composition models, every significant factor on Polynomino Packing and CUDA Attention keeps its sign and magnitude. The one casualty is on Bounded 2D Knapsack, where the better context ratio goes from a small negative effect () to no significant effect (, ), and score diversity shrinks from to while keeping its sign. The problem-dependent direction of the effects, and the Knapsack and Polynomino Packing readings built on it, stand as submitted.
Validation of agent-written parsers. We validated the parsed records against the raw run directories with an independent reader, written from scratch and sharing no code with the parsers. It re-read the raw run directories of 911 campaigns on the three combinatorial problems and recovered the same 47,168 candidates with the same scores and iteration indices as ROAR, and the same best score and iteration count for every campaign. The remaining 17 campaigns of the corpus are not covered by this check. This check is reported in the revised paper in the appendix "Validation of Parsed Records".
Scope of the conclusions. The revised paper no longer treats multimodal final scores as evidence of distinct local optima. RQ1 now describes what is measured, the shape of the final-score distribution across identical configurations, and the inference to basins is stated once as a hypothesis with its limits spelled out in a new appendix paragraph and figure ("Basins versus modes"). The sentences that read a continuous spread as "multiple local optima" or a binary spread as a "binary basin structure" have been reworded in RQ1, RQ2 and RQ3 to speak of continuous and binary outcome distributions. All of these edits are highlighted in blue in the revised paper.
Response to the Authors
Thank you for the additional experiments and analyses! They address several of my concerns:
- The budget policy evaluation comparing equal split vs. specified fixed rules and the restated RQ4 clarified my confusion.
- The robustness analyses for code diversity and code length and the validation of parsers helped address some of my concerns.
- The reformulated RQ2 was also helpful and I believe this is a better reflective of the paper's research contribution.
Taking these updates, the revised paper, and also reading other reviews into account, I have increased my score from 1 to 2.
We thank the reviewer for identifying several places where the claims and validation procedures require greater precision. We address each concern below.
Scope of the conclusions. Multimodal final scores show different run outcomes, but do not establish distinct local optima. Similarly, RQ3 identifies associations between context composition and subsequent improvement, not causal effects. We will correct the claims and recommendations accordingly.
Budget-policy evaluation. Appendix J reports a disjoint-pool sensitivity analysis in which the policy is fit on half of each configuration's campaigns and relaunches are drawn from the other half. The estimated gain is smaller but remains positive in 10 of 12 configurations. We are extending this evaluation with repeated disjoint splits. Under the same 50-iteration budget, we are also adding equal-split baselines, including 2×25 and 5×10, and pre-specified fixed-patience baselines that do not use configuration-specific fitting. These comparisons separate the benefit of multiple starts, conditional stopping, and configuration-specific policy fitting.
Invalid candidates and RQ1 sensitivity. Candidates that fail to compile, crash, time out, or return an infeasible output receive a score of zero and remain part of the search trajectory. However, we agree that an all-invalid campaign may represent search failure rather than a low-scoring optimization basin. We will therefore report the number of campaigns containing no valid candidate and repeat the RQ1 analysis conditional on each campaign producing at least one valid candidate. This will show whether the reported multimodality reflects variation among valid solutions, a mixture of successful and failed searches, or both. ROAR preserves contributed evaluator outputs, but it does not independently certify evaluator correctness; we will state this limitation separately.
Palindrome Hamiltonian Path. All 290 Palindrome Hamiltonian Path campaigns have a final score of zero because no candidate passed the evaluator. Individual failed candidates remain informative observations of search behavior, but this problem has no within-problem final-score variation for RQ1, no positive improvement events for RQ3, and no empirical headroom for RQ4 policy comparison. For RQ2, the empirical escape probability is identically zero. We will state this explicitly in the dataset description and relevant results sections.
Meaning of context for Best-of-N. In these runs, the Best-of-N mechanism selects the highest-scoring program found so far as the parent and holds it fixed for the next N=5 successfully evaluated candidates. Each prompt asks the model to improve that parent. It also includes four context programs sampled afresh each iteration from the ten highest-scoring programs, excluding the parent, plus a short summary of recent high-scoring attempts. After each block, the mechanism again selects the best program found so far and repeats. We will document this implementation and distinguish the mechanism's N from the campaign-allocation parameter n used in RQ4.
Code diversity and code length. We agree that code length may confound embedding-based diversity. We will report the correlation between diversity and token length and refit the RQ3 mixed-effects models with standardized token length as an additional fixed effect, retaining campaign random intercepts and iteration phase. We will report whether the direction and magnitude of the diversity associations remain stable.
Validation of agent-written parsers. The schema enforces structural and type consistency, while semantic correctness is assessed by manually comparing parsed samples with the corresponding raw logs before a parser is merged. We will describe this validation procedure, including its scope and limitations, more clearly in the paper. We are also investigating comparison with an existing deterministic parser as an additional check on field-mapping accuracy.