acceptodds
Under review as a conference paper at ICLR 2027

SysLoopBench: Evaluating Long-Horizon System Evolution by Autonomous Agents

Abstract

Autonomous agents hold promise for accelerating the development of scientific and engineering systems. However, existing benchmarks provide limited insight into whether agents can independently recreate the capabilities accumulated in expert-developed systems and advance beyond them. To address this gap, we introduce Sysloopbench, a benchmark that challenges agents to recreate and surpass expert-developed capabilities through long-horizon evolution of basic starting systems, retaining the original objectives and interfaces while independently developing their own algorithms and designs. Sysloopbench comprises 30 carefully designed tasks across mathematics, physics, biology, chemistry, engineering, and computer science. We construct each task from an expert-developed real-world system, extracting target capabilities, preparing a basic starting system and development environments, and establishing executable references for evaluation. Agents iteratively test and revise their systems using execution feedback, and we evaluate the resulting systems against expert-developed references on held-out workloads. Our evaluation reveals both the potential of long-horizon evolution to advance systems beyond expert references and the challenges of turning iterative improvements into lasting, generalizable gains across performance and efficiency.

Primary area:
datasets and benchmarks
Keywords:
Agent Evaluation, Benchmark, Evolution, Harness

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5zX6OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper introduces SysLoopBench, a benchmark of 30 scientific and engineering software development tasks across six domains. It evaluates whether agents can iteratively improve systems toward or beyond expert-developed references. Experiments with six models examine final performance, efficiency, progress, regressions, and recovery.

Critical strengths
  1. The paper studies a central question: whether agents can keep improving scientific and engineering systems while retaining earlier gains. The benchmark provides a useful addition to existing evaluations of this process.
  2. The benchmark constructions and experiments involve substantial work, and the analysis is thorough. The findings about progress, regressions, recovery are informative. The paper is also clearly written.
Critical weaknesses
  1. Long development runs may vary considerably, but each model–task pair is run only once. It is therefore unclear how stable the scores and trajectory findings are. Given the cost, could the authors repeat a set of tasks with the best- and worst-performing models, or just the best-performing model?
  2. All results use OpenCode, so it is unclear whether the model differences and development trajectories depend on this choice. Since the harness can affect model performance, could the authors test another harness with the best- and worst-performing models, or just the worst-performing model?
Questions for authors

Please address the questions raised in the weaknesses above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 5zX6

Thanks for the constructive feedback. We provide a point-to-point response as follows.

W1 & Q1. Stability of final scores and development trajectories across independent runs

Thank you for asking about the stability of the scores and trajectory findings across independent runs.

We conducted repeatability experiments and confirmed that our trajectory findings hold across independent runs, with the benchmark scores of the reruns shown below falling within a modest range for each task.

Specifically, we report three independent reruns of GPT-5.6 Sol (the best-performing model) for each of three tasks from different domains. All other settings were the same as in the main evaluation. The results are shown below.

TaskPaper held-outRerun held-outHeld-out mean ± SDDev. R1Dev. R5Dev. R10
Wind turbine control96.7292.72 / 98.40 / 92.3595.05 ± 2.9843.53 ± 6.7250.25 ± 4.1051.81 ± 4.38
Order-to-Cash reconciliation60.6856.75 / 59.42 / 59.2459.02 ± 1.640.00 ± 0.0057.11 ± 2.0759.97 ± 0.98
Peptide-spectrum matching71.0173.22 / 79.22 / 78.4175.46 ± 3.9939.27 ± 27.0370.08 ± 7.8474.73 ± 4.34

(1) In terms of final scores, the reported reruns attain comparable final performance despite differences in their development paths, showing that the corresponding paper results can be reproduced through independent development. (2) Regarding the trajectory findings, the repeats show comparable overall development trends despite differences at individual rounds, consistent with the qualitative trajectory findings reported in the paper.

Our benchmark design supports repeatability through fixed data, standardized evaluation, and traceable development records. We keep task data, starting implementations, and budgets fixed across runs to make outcomes comparable, and validate measurement procedures to limit scoring noise. We freeze each submitted system and retain its feedback, allowing each point in the development trajectory to be traced to a specific implementation and evaluation record.


W2 & Q1. Dependence of model differences and development trajectories on the harness

Thank you for suggesting an evaluation with an alternative harness.

We conducted cross-harness experiments using Pi and found that the model ordering and broad development trends remain consistent, although absolute scores and individual trajectories vary with the harness.

Specifically, we compared GPT-5.6 Sol (the best-performing model) and Qwen3.8 Max (the lowest-scoring model in the first round) on two tasks from different domains using OpenCode and Pi. All other task and evaluation settings were kept consistent with the main evaluation. The results are shown below.

TaskModelOpenCode held-outPi held-outOpenCode dev. R1 → R5 → R10Pi dev. R1 → R5 → R10
Wind Turbine ControlGPT-5.6 Sol96.7289.34 (↓)49.36 → 56.17 → 51.9248.21 → 47.60 → 48.89
Order-to-Cash ReconciliationGPT-5.6 Sol60.6865.04 (↑)0.00 → 54.81 → 60.750.00 → 64.99 → 64.89
Wind Turbine ControlQwen3.8 Max53.2467.90 (↑)38.68 → 60.18 → 60.8246.33 → 46.88 → 54.86
Order-to-Cash ReconciliationQwen3.8 Max73.8065.87 (↓)0.00 → 73.57 → 74.710.00 → 64.57 → 65.24

(1) In terms of model differences, the model comparisons remain consistent across harnesses. Both harnesses preserve the taskwise ordering, and GPT retains the higher mean held-out score across the two tasks. Harness choice can raise or lower absolute scores depending on the model-task combination, without changing the direction of these comparisons.

(2) Regarding the trajectory findings, the broad development trends and differences in model improvement persist across harnesses. Harness choice changes the timing and magnitude of progress. The same model can improve mainly in earlier rounds under one harness and later rounds under another, producing different development paths.

Therefore, we use a common harness in the main evaluation to compare all models under the same development conditions. Our benchmark keeps the task definition, initial system, evaluator, and budget fixed when changing the harness, supporting comparisons under a consistent task protocol.


Thank you again for your constructive feedback, which has helped us strengthen the work. We have incorporated all the clarifications and additional results discussed above into the revised manuscript.

Thank you for the additional experiments. They address my concerns. I have also reviewed the comments from the other reviewers and your responses, and I see that you have added a substantial amount of experimental work. I think you have done a very good job. Given that negative scores still exist for the paper, and to fully express my recognition of your work, I have decided to raise my score from 3 to 4.

AuthorsOpenReview ↗

Thank You for Your Support

Thank you very much for your strong support and encouraging feedback! We are glad that our responses have addressed your concerns. We sincerely appreciate your time and effort in reviewing our manuscript.

Reviewer_KWrPOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

SYSLOOPBENCH evaluates whether autonomous agents can improve basic scientific and engineering systems over many development rounds. It includes 30 tasks across six domains. Agents can change algorithms and system structure, while the task goals and interfaces stay fixed. The benchmark records ten development rounds and evaluates the final system on held-out workloads. Scores measure both solution quality and efficiency against expert-developed references. Results show clear gains during development, but also regressions and gaps on held-out data. Some systems beat the reference on selected tasks or metrics.

Critical strengths
  1. The benchmark studies a useful problem. Improving a working system over time requires linked design choices. The fixed interface and open implementation space give agents room to make these choices.

  2. Many tasks have careful validity checks. Examples include exact lattice checks, numerical residuals, real compressed bitstreams, and fresh-process decoding. These checks help ensure that a high score comes from a usable system.

  3. Several data splits test meaningful changes. Examples include molecular-scaffold splits for ADMET, separate receptors for docking, unseen network topologies for grid restoration, and held-out query structures for SQL. These are stronger than only changing random seeds.

  4. The analysis goes beyond final scores. It separates new gains from recovery of earlier losses and studies quality–efficiency trade-offs. The wind turbine case also reports worse loads and latency alongside better control accuracy.

Critical weaknesses
  1. Data construction needs more detail. Several tasks lack exact data versions, case IDs, generation settings, or split rules, making reproducibility hard to assess.

  2. Data overlap is not fully checked or reported. The paper should clarify speaker overlap in LibriMix, protein similarity in CLEAN-derived splits, and shared molecular inputs across chemistry tasks.

  3. Some test sets are small or use limited distribution shifts. For example, coupled-cluster tasks have eight cases per split, while road matching uses new journeys within shared cities and road networks.

Questions for authors
  1. Can you release a data manifest for all tasks, including source versions, case counts, case IDs, generation settings, and exact split rules?

  2. What overlap checks are used for each task? Please clarify speaker and recording overlap in LibriMix, protein similarity in CLEAN-derived splits, and source overlap in image compression.

  3. Do Hartree–Fock, CCSD, and triples correction share molecules, geometries, or basis sets? Are there other shared inputs across tasks?

  4. For road matching, can you add a held-out-city test to measure transfer beyond shared road networks?

6 comments
AuthorsOpenReview ↗

Responses to Reviewer KWrP (1/3)

Thank you for the constructive review. We address each point below.

W1 & Q1. A reproducible data manifest for all 30 tasks

Thank you for asking for more detailed data construction records.

We have added a complete data manifest for all 30 tasks to the revised manuscript, specifying source versions, case counts and IDs, generation settings, and exact split rules. For ease of review, we also include three tasks below.

TaskData sourcesNumber of casesCase constructionSplit construction
RNA Secondary StructureCircular RNA sequences using ViennaRNA 2.7.216 development and 16 held-out sequencesFour sequence families, each covering lengths of 64, 96, 144, and 192 nucleotidesDevelopment and held-out sequences are generated independently, while retaining the same family and length coverage
Learned Image CompressionCLIC 2020 photographs with generated graphics24 training, 24 development, and 24 held-out imagesSix content families: professional photographs, mobile photographs, documents, line drawings, textures, and mixed contentSource photographs are assigned to disjoint splits; generated graphics use separate seeds for each split
SQL Cardinality PlanningA frozen JOB/IMDB snapshot with TPC-H and TPC-DS databases using DuckDB 1.5.521 development and 40 held-out queriesQueries specify tables, filters, and join relationships across the three databasesHeld-out queries include eight JOB template groups excluded from development, new TPC-H query structures, and the entire TPC-DS database

We built reproducibility into each task from the outset by fixing its source data, construction procedures, and split assignments. For existing datasets, we fix the source versions, selected cases, and preprocessing steps. For generated workloads, we fix the generator versions, parameters, and random seeds. We freeze the resulting workloads before agent evaluation and preserve their case-level split assignments, enabling reconstruction of the same inputs used in our experiments.


W2.1 & Q2. Overlap between splits within each task

Thank you for asking about the overlap checks used for the data splits.

To clarify, we checked for data overlap during dataset construction and kept the relevant training, development, and held-out splits disjoint at the sample or source level required by each task. Our checks used sample identities or content hashes, source provenance, and domain-specific criteria, tracing derived examples back to their original sources. Below, we use the tasks you highlighted to illustrate these checks and their results.

Task 1: Blind audio separation (LibriMix). Held-out speakers and recordings are disjoint from both training and development. Training and development both draw from LibriSpeech dev-clean: all 35 development speakers and 43 recording chapters also appear in training, although the splits share NO constituent utterances. We traced each mixture back to its speaker, chapter and utterance IDs, and compared the underlying noise files for noisy mixtures. No pair of splits shares mixture IDs or WHAM noise files.

Task 2: Enzyme function prediction (CLEAN). The candidate’s data splits contain NO shared complete protein sequences, and high-identity matches to the training data are rare. We compared every development and held-out sequence with the full fitting and calibration pool using MMseqs2. Their median identities to the highest-scoring local matches were 28.40% and 32.54%, respectively. Requiring at least 80% alignment coverage of both sequences, only 1.10% of development sequences and 1.37% of held-out sequences had a match at ≥50% identity. Matches at ≥70% identity were confined to 11-22 aa held-out peptides, with none among query sequences of at least 50 aa.

Task 3: Learned image compression. The training, development and held-out splits use the same CLIC 2020 source collection, but each uses a distinct set of source photographs. We traced all output images to their source records, including photographs embedded in mixed graphics. Each pair of splits had zero shared source-photo hashes, output PNG hashes or generation seeds. These checks establish separation by source file.

AuthorsOpenReview ↗

Responses to Reviewer KWrP (2/3)

W2.2 & Q3. Inputs shared across tasks

Thank you for asking about shared inputs across tasks.

We reuse some physical inputs across the three quantum-chemistry tasks, each with a distinct computational objective and an independent evaluation. The remaining tasks do not share task inputs.

CCSD and perturbative triples correction share frozen molecular-orbital integrals, while Hartree-Fock has partial overlap with them in molecules, geometries, and basis sets. The CCSD and triples integrals are generated from identical molecules, geometries, and basis sets. Hartree-Fock shares some molecular species and the cc-pVDZ basis with these tasks, including the same equilibrium-water geometry; the other overlapping species use different geometries.

Hartree-Fock, CCSD, and perturbative triples correction pose fundamentally different numerical problems and require distinct solver designs. Hartree-Fock tests the agent’s ability to design iterative optimization and convergence strategies for finding a stable, self-consistent electronic state. CCSD tests its ability to solve nonlinear amplitude equations accurately and efficiently. Perturbative triples correction tests its ability to implement and optimize high-order tensor contractions for an additional energy correction.

We keep task development and evaluation independent, preventing solutions or feedback from carrying over between tasks that share physical inputs. We run each model-task pair in its own agent episode, with a separate workspace, conversation, submissions, and development feedback. We assess the final frozen implementation using that task’s verifier, reference, and held-out workload.


W3.1. Coverage of the coupled-cluster evaluation cases

Thank you for highlighting the importance of coverage in the coupled-cluster test sets.

The eight-case CCSD and perturbative triples test sets support meaningful solver comparisons by covering complementary molecular systems, geometries, and basis sets. Specifically, the held-out set combines LiH, BH, F₂ and O₂ with stretched N₂, H₂ and HF, plus an H₄ system. The development set also includes atomic Be and Ne and polyatomic H₂O, NH₃ and CH₄; together, the development and held-out workloads span two basis sets. These variations test nonlinear-solver convergence and energy accuracy in CCSD, and accurate, efficient tensor contractions across different molecular configurations and tensor sizes in triples correction.

Calibration during task construction demonstrates that the CCSD and perturbative triples workloads distinguish simplified approximations from the corresponding native expert implementations. The MP2/zero-singles control for CCSD and the zero-triples control for triples correction scored substantially below their respective native PySCF references. We assess numerical fidelity through correlation-energy accuracy and amplitude-equation residuals for CCSD, and energy-correction accuracy for triples. We also compare per-case solve times against native PySCF, evaluating whether implementations deliver accurate results efficiently across the molecular problems covered.

AuthorsOpenReview ↗

Responses to Reviewer KWrP (3/3)

W3.2 & Q4. Road matching on an unseen city

Thank you for suggesting a held-out-city evaluation.

We conducted an additional held-out-city evaluation and found that our conclusions about beyond-reference engineering gains and quality-efficiency trade-offs also hold under a cross-city distribution shift.

Specifically, we directly evaluated the frozen final implementations from all six models on Monaco, whose road network was absent from development. We used the same route-generation procedure and six GPS observation conditions as in Utrecht, generating 60 distinct journeys with 360 trace variants and 27,052 GPS observations. All other evaluation protocol settings remained unchanged and the results are shown below.

Frozen matcherHof/Utrecht score ↑Utrecht quality ↑Monaco quality ↑Monaco latency (ms/GPS point) ↓Monaco score ↑
GPT-5.6 Sol113.120.96060.85158.82496.45
Claude Opus 4.8112.190.92890.78752.42998.52
GLM-5.386.400.84750.90574.480107.76
Kimi K3103.290.79280.84901.032108.08
DeepSeek V4 Pro113.540.93450.92154.253109.26
Qwen3.8 Max112.980.92930.73532.45792.40
Native HMM100.000.94700.928210.464100.00

The Monaco evaluation shows that our findings on beyond-reference engineering gains and quality-efficiency trade-offs extend to an unseen road network, while improvements generalize unevenly across models. DeepSeek and Kimi retain their advantage over the native reference without further tuning. The coexistence of speed advantages and matching-quality gaps reinforces the need to assess efficiency jointly with solution quality. Changes in which implementations remain above the reference further demonstrate that engineering gains transfer differently across models.


Thank you again for your detailed review and helpful suggestions, which have improved the clarity and completeness of the manuscript. All the explanations, analyses, and additional results discussed above are included in the revised manuscript.

Reviewer_tcT8OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

SysLoopBench evaluates autonomous agents developing scientific and engineering systems through successive revisions in a persistent workspace. It contains 30 tasks across six domains, each grounded in an expert-developed reference system. Agents receive a basic starting implementation, a fixed objective and interface, development tools, and execution feedback. Final systems are evaluated on held-out workloads.

The evaluation covers six models, with one ten-round episode per model–task pair and up to 45 minutes of agent execution per round. The scoring framework combines functional validity, reference-relative performance, and efficiency, assigning the expert reference a score of 100. The paper analyzes final capability, development trajectories, recovery from regressions, and performance–efficiency trade-offs.

The results show substantial improvements during development, but lower final held-out scores and persistent gaps to the references overall. Selected systems improve particular engineering outcomes, including low-bit CPU computation and wind turbine control.

I find the benchmark direction worthwhile and the task collection potentially useful. However, the current evaluation provides stronger descriptive evidence about these particular runs than evidence isolating the benefits of long-horizon development or establishing reliable model comparisons.

Critical strengths

Strength 1: The benchmark evaluates useful executable systems rather than treating functional validity as the endpoint.

The tasks require agents to reconcile application quality, computational cost, and operating constraints. The distinction between producing a valid system and matching a strong reference is useful: most final submissions are valid, yet many remain substantially below reference performance.

The task descriptions also include meaningful domain-specific checks. Examples include exact lattice preservation, Lean-certified proofs, numerical reference checks, fresh-process image decoding, and closed-loop control evaluation. These choices make the benchmark more informative than a collection of loosely specified optimization objectives.

Separating development feedback from final held-out assessment is another important design choice. The difference between development and final scores exposes limitations that would be missed by evaluating only on the workloads used during revision.

Strength 2: The trajectory and performance–efficiency analyses reveal useful distinctions beyond a single final score.

The paper records immutable submitted systems and examines regressions, recovery, plateaus, and retained gains. This distinguishes discovering an improvement from preserving it through later revisions. The task-level results and complete development scores make the reported patterns easier to inspect.

The separate performance and efficiency axes are also valuable. In particular, the paper shows that efficiency coverage changes when minimum performance requirements are imposed, and that aggregate trade-offs can conceal individual systems dominated on both axes.

The engineering examples are presented with relevant qualifications. The low-bit kernel improves latency while using more memory. The turbine controller improves speed tracking and energy capture but increases some loads, platform motion, and latency, yielding an overall score below 100. Reporting these trade-offs is more informative than presenting isolated gains alone.

Critical weaknesses

Weakness 1: The experiments do not isolate the contribution of persistent long-horizon development from additional computation and search.

The central motivation concerns sustained, interdependent system development. However, comparing early and late rounds of the same run also compares different amounts of computation, feedback, and search. The paper does not include an equal-budget restart baseline, independent shorter attempts with development-based selection, or a control that removes conversation history while preserving the current code.

Consequently, the results show that continued execution often improves development scores, but do not establish how much of that benefit specifically depends on accumulating system-level progress in a persistent development process. The trajectory statistics are informative descriptions, but are not a substitute for such a comparison.

Only the final system is evaluated on held-out workloads. This establishes final generalization, but leaves unclear whether gains observed between rounds 1, 5, and 10 also represent improvements on unseen workloads. Evaluating frozen intermediate systems after development concludes would address this without exposing held-out feedback to the agent.

Weakness 2: One episode per model–task pair is insufficient to establish the reliability of several model comparisons and trajectory findings.

The evaluation contains 180 episodes, but only one realization of each model–task pair. This provides broad coverage while leaving within-task variability unmeasured. For example, the final mean scores of GPT-5.6 Sol and Kimi K3 differ by only 0.33 points. The paper appropriately describes them as closely matched, but repeated runs are still needed to determine how stable their domain profiles, breakthrough rates, and gain-retention differences are.

Some trajectory analyses use small score thresholds to identify plateaus, recovery, and renewed progress. Task validation checks measurement consistency, but the paper does not quantify evaluation noise relative to these thresholds. Re-evaluating unchanged snapshots would help distinguish genuine revisions from measurement variation.

The shared wall-clock allowance is a reasonable practical budget, but actual model tokens, development computation, and API costs are not reported. Default reasoning and sampling settings can also differ across models. These choices do not invalidate the evaluation, but the conclusions should be framed as comparisons of the specified model–harness configurations under this execution policy.

Weakness 3: Reference calibration, scoring choices, and evaluation-access boundaries need more concrete specification.

Assigning every reference a score of 100 creates a convenient reporting convention, but does not establish comparable difficulty or reference strength across tasks. The starting systems also differ substantially in what they already implement. Initial-system scores and more explicit justification of the chosen reference operating points would help distinguish agent capability from task construction effects.

The scoring framework uses task-specific metric transforms, numerical floors, weights, and normalization parameters. Appendix C explains the procedure, but the actual task-by-task values are not enumerated in the manuscript. Since the headline results average these normalized scores, readers need the concrete configurations and sensitivity analyses to assess how strongly conclusions depend on these choices.

The access policy also needs clarification. The protocol states that held-out workloads are unavailable to agents and that submitted systems run separately. However, it does not fully specify what agents can inspect or execute concerning reference implementations, evaluator code, workload generators, installed packages, and external resources. This matters particularly for the claim that agents independently recreate expert-developed capabilities. I am not asserting leakage or misuse; the issue is that the documented access boundary is not precise enough to assess these possibilities.

The task cards provide substantial conceptual coverage, but reproducing this benchmark requires executable packages, exact prompts, reference versions, scoring configurations, and environment specifications. Those artifacts are central to evaluating the benchmark contribution.

Questions for authors

Clarification requests (answers could change my assessment).

  1. Under the same total development budget, how does persistent ten-round development compare with independent shorter attempts selected using development scores? What changes when conversation history is reset while the current code is retained?

  2. Can the frozen round-1 and round-5 systems be evaluated on held-out workloads after development concludes? Do the development improvements and apparent narrowing of model gaps persist under this comparison?

  3. How variable are the results across repeated episodes? Could you repeat a representative subset of tasks and re-evaluate unchanged snapshots to quantify both agent variability and measurement noise?

  4. Please provide the task-specific scoring parameters, initial-system scores, and reference versions and configurations. How stable are the aggregate conclusions under alternative reasonable metric and axis weights?

  5. What exactly can agents access during development? Please specify the rules for reference code and binaries, evaluator code, workload generators, installed dependencies, and network access, together with the mechanisms enforcing held-out isolation.

Suggestions for revision.

  1. Separate descriptive trajectory findings from claims about the distinctive value of persistent development. The current evidence supports the former more directly.

  2. Release versioned executable task packages, prompts, scoring configurations, environment definitions, submitted systems, and interaction records. Clearly identify which artifacts are available for review.

  3. Report actual development resource use, including elapsed time, model tokens, evaluation calls, and computational resources. Tool-call counts alone do not characterize the cost of an episode.

  4. Include uncertainty estimates for aggregate and domain-level results. Distinguish uncertainty across tasks from variability across repeated runs of the same task.

  5. Keep the beyond-reference claims tied to the measured objectives. The turbine example demonstrates improvements on selected control metrics rather than overall superiority under the benchmark's own score. The main text acknowledges this; the abstract and introductory framing should maintain the same qualification.

10 comments
AuthorsOpenReview ↗

Responses to Reviewer tcT8 (1/7)

We appreciate your detailed review and constructive suggestions. We address your comments and questions point by point below.

W1.1 & Q1 & Q6. Continuous development under the same budget

Thank you for suggesting controls to assess the contribution of persistent development.

Our equal-budget control experiments support the value of sustained revision and retained conversation history beyond making additional independent attempts.

Specifically, we use GPT-5.6 Sol with OpenCode on three tasks from different domains. We compare continuous ten-round development with the five retained independent two-round attempts per task, each starting from the original implementation. We select the short-run system by its final development score within this group. Each round follows the same development allowance and evaluation protocol as the main evaluation. The results are shown below.

TaskSettingDevHeld-out
SQL Cardinality Planning5 × 2 rounds40.74, 41.05, 40.34, 47.82, 50.2318.49
SQL Cardinality Planning1 × 10 rounds43.0619.19
Wind Turbine Control5 × 2 rounds37.18, 43.76, 37.06, 40.16, 32.8172.72
Wind Turbine Control1 × 10 rounds49.7192.72
Peptide-Spectrum Matching5 × 2 rounds40.86, 24.09, 22.60, 39.48, 21.3042.93
Peptide-Spectrum Matching1 × 10 rounds78.6981.21

The restart comparison supports the value of refining an evolving implementation rather than allocating the same allowance entirely to independent search. Continuous development achieves stronger development and held-out performance on Wind Turbine Control and Peptide-Spectrum Matching, while SQL Cardinality Planning reaches similar held-out performance. On the first two tasks, selecting among the short attempts does not recover the gains obtained when later revisions can build on earlier implementation work and feedback.

We further compare ten-round runs with conversation history retained or reset between rounds. In the reset setting, the current codebase, installed dependencies, and immediately preceding development feedback remain available. All other settings stay the same.

TaskHistory retained: DevHistory retained: Held-outHistory reset: DevHistory reset: Held-out
SQL Cardinality Planning43.0619.1948.6016.32
Wind Turbine Control49.7192.7238.9286.65
Peptide-Spectrum Matching78.6981.2125.8226.70

The history-reset comparison supports the value of earlier conversations beyond the preserved code and latest development feedback. All three tasks finish with lower held-out scores after history reset. For SQL Cardinality Planning, the higher development score does not translate into stronger held-out performance. The code preserves the current implementation, while conversation history also carries earlier experiments, decisions, and feedback that can guide subsequent revisions.

Our trajectory findings describe development dynamics, while the restart and history-reset controls provide evidence about the value of sustained revision and retained conversation history. In the revised manuscript, we make this distinction explicit and ground claims about the contribution of continuous development in these controlled comparisons.

AuthorsOpenReview ↗

Responses to Reviewer tcT8 (2/7)

W1.2 & Q2. Generalization of development gains

Thank you for suggesting held-out evaluations of frozen intermediate systems.

Our evaluations of frozen intermediate systems show that development gains extend to unseen workloads and that the models become more closely matched on average as development proceeds.

Specifically, we evaluate the frozen R1 and R5 systems of GPT-5.6 Sol and Qwen3.8 Max on three tasks from three different domains, alongside the reported R10 results. We perform these held-out evaluations after development has concluded, without returning held-out feedback to the agents. We use the original task protocols, workloads, references, and scoring configurations. The results are shown below.

TaskModelDev R1Dev R5Dev R10Held-out R1Held-out R5Held-out R10 (paper)
Orbit DeterminationGPT-5.6 Sol52.46101.10104.8769.58103.77107.04
Orbit DeterminationQwen3.8 Max5.54113.22115.743.33118.09120.71
Peptide-Spectrum MatchingGPT-5.6 Sol0.0060.6470.220.0061.6471.01
Peptide-Spectrum MatchingQwen3.8 Max0.040.0058.880.040.0060.35
Wind Turbine ControlGPT-5.6 Sol49.3656.1751.9285.2098.0896.72
Wind Turbine ControlQwen3.8 Max38.6860.1860.8244.6150.9753.24

The intermediate held-out evaluations show that development gains transfer to unseen workloads at different stages of development. Both models’ substantial early improvements on Orbit Determination are reflected in held-out performance, as is Qwen’s later improvement on Peptide-Spectrum Matching. This correspondence shows that the observed development trajectories capture improvements that extend beyond the workloads providing feedback.

Across the tasks, GPT and Qwen become more closely matched in mean held-out performance while retaining different task strengths. Qwen overtakes GPT on Orbit Determination and substantially reduces its R5 deficit on Peptide-Spectrum Matching, while GPT retains its advantage on Wind Turbine Control. The narrowing overall gap therefore reflects different development gains across tasks, rather than convergence to the same capability profile.


W2.1 & Q3.1. Variability across independent runs

Thank you for asking about the reliability of results across independent runs.

Our repeatability experiments show consistent final performance and broad development trends across the reported reruns.

Specifically, we report three retained independent reruns of GPT-5.6 Sol on each of three tasks from different domains. All other settings remain consistent with the main evaluation. We summarize final held-out scores and development scores at rounds 1, 5, and 10 below.

TaskPaper held-outRerun held-outHeld-out mean ± SDDev. R1Dev. R5Dev. R10
Wind turbine control96.7292.72 / 98.40 / 92.3595.05 ± 2.9843.53 ± 6.7250.25 ± 4.1051.81 ± 4.38
Order-to-Cash reconciliation60.6856.75 / 59.42 / 59.2459.02 ± 1.640.00 ± 0.0057.11 ± 2.0759.97 ± 0.98
Peptide-spectrum matching71.0173.22 / 79.22 / 78.4175.46 ± 3.9939.27 ± 27.0370.08 ± 7.8474.73 ± 4.34

(1) In terms of final scores, the reported reruns attain comparable final performance despite differences in their development paths, consistent with the corresponding paper results. (2) Regarding the trajectory findings, the repeats show comparable overall development trends despite differences at individual rounds, consistent with the qualitative trajectory findings reported in the paper.

Our benchmark design supports repeatability through fixed data, standardized evaluation, and traceable development records. We keep task data, starting implementations, and budgets fixed across runs to make outcomes comparable, and validate measurement procedures to limit scoring noise. We freeze each submitted system and retain its feedback, allowing each point in the development trajectory to be traced to a specific implementation and evaluation record.

AuthorsOpenReview ↗

Responses to Reviewer tcT8 (3/7)

W2.2 & Q3.2. Measurement noise in repeated scoring

Thank you for suggesting repeated scoring of unchanged systems to quantify measurement noise.

Repeated scoring yields consistent results, with fluctuations below the threshold used in our trajectory analyses.

Specifically, we repeat development scoring of frozen R10 submissions. We keep the implementation, workloads, reference data, and scoring configuration fixed. This isolates evaluation noise from changes introduced during development. The results are shown below.

TaskPaper R10 devDev scoresMean ± sample SD
SQL Cardinality Planning64.177564.2080 / 64.6726 / 64.353164.4112 ± 0.2377
Peptide-Spectrum Matching70.220470.7206 / 70.7717 / 70.812270.7682 ± 0.0459
Wind Turbine Control51.918751.5702 / 52.1121 / 52.074051.9187 ± 0.3025

The repeated-scoring results support the reliability of our performance measurements and trajectory analysis. The frozen systems receive scores close to their reported R10 values. This shows that the reported performance levels are reproducible across evaluations. The observed fluctuations also remain below the one-point threshold. Small numerical movements therefore do not, by themselves, qualify as renewed progress. A larger improvement is required before the analysis assigns that label. This supports interpreting the identified gains as development progress rather than routine measurement variation.


W2.3 & Q8. Evaluation settings and resource use

Thank you for asking for more detailed evaluation settings and resource statistics.

We report explicit model settings and actual resource use for all six models. We use a shared harness and execution policy across the evaluation.

Model and harness settings

ModelOpenCode versionThinking / reasoningTemperatureTop-pMax output tokens
GPT-5.6 Sol1.18.4On (high)1.00.95128k
Kimi K31.18.4On (high)1.00.95128k
Claude Opus 4.81.18.4On (high)1.00.95128k
DeepSeek V4 Pro1.18.4On (high)1.00.95128k
GLM-5.31.18.4On (high)1.00.95128k
Qwen3.8 Max1.18.4On (high)1.00.95128k

Recorded resource use

ModelGateway tokensGateway API requestsGateway actual charges (US$)Average episode span (min)
GPT-5.6 Sol519,302,9797,464471.43136.09
Kimi K31,287,245,95815,5181,236.41413.99
Claude Opus 4.8722,723,29212,8344,017.19440.14
DeepSeek V4 Pro2,224,606,25011,942195.42209.48
GLM-5.31,754,856,90816,129704.49350.95
Qwen3.8 Max1,150,594,7239,889600.43391.95
Total7,659,330,11073,7767,225.37323.77

The six models differ in actual resource use under the same evaluation settings. GPT and Kimi achieve similar overall capability. GPT uses fewer tokens and API requests. Its billed cost and average episode span are also lower. The ranking by billed cost differs from the ranking by token use. These differences show why capability should be interpreted alongside actual resource consumption. Our comparisons concern the specified models under the shared harness and execution policy.

Comments on the paper

1 comment
AuthorsOpenReview ↗

A General Response by the Authors

Dear Area Chairs and Reviewers,

Thank you very much for the time and effort you have devoted to our submission.

We are pleased that all three reviews recognized strengths of our work, with scores of 4, 3, and 2. We provide a summary below to assist with your evaluation.

(1) Overview. We propose SysLoopBench to evaluate whether autonomous agents can develop scientific and engineering systems that match or surpass expert references.

  • A benchmark grounded in expert-developed systems. We derive 30 tasks across six domains from expert-developed systems. Agents independently develop the target capabilities from a basic implementation. The corresponding expert system serves as an executable reference.
  • Long-horizon evaluation. We evaluate six agents on all 30 tasks in 180 ten-round episodes. The records show what agents achieve initially and what they gain through long-horizon development. Held-out tests compare final quality and efficiency with expert performance.
  • Findings on system development. Our evaluation reveals both the potential for agents to advance beyond expert-developed capabilities and the difficulty of turning sustained development into lasting, generalizable gains.

(2) Strengths. All three reviewers highlight the value of our work:

  • Meaningful task design. SysLoopBench studies whether agents can keep improving a working system while retaining earlier gains. Its task cards provide useful and detailed information. (5zX6, KWrP, tcT8)
  • Practical and rigorous evaluation. SysLoopBench uses domain-specific validity checks, a common time budget, and executable expert references. Held-out workloads reveal limitations that development scores can miss. (KWrP, tcT8)
  • Thorough trajectory analysis. Our round-by-round evaluation record distinguishes discovering an improvement from retaining it and reveals when apparent progress recovers an earlier loss. (5zX6, KWrP, tcT8)
  • Informative engineering trade-offs. Our work shows how efficiency advantages change as quality requirements tighten and when one system outperforms another in both quality and efficiency. (KWrP, tcT8)
  • Clear presentation. Our work pairs the benchmark-wide findings with concrete engineering cases, making it easy to follow how individual system changes affect the reported outcomes. (5zX6, KWrP)

(3) Responses. We incorporated our point-by-point clarifications and additional experimental results into the revised manuscript.

  • Testing sustained development and generalization. We compared continuous development with equal-budget restarts and tested history resets while retaining code. Frozen intermediate systems showed that gains transferred to held-out workloads. (tcT8 W1.1 & Q1 & Q6, W1.2 & Q2)
  • Checking repeatability and harness dependence. We reran three tasks independently. Rescoring fixed systems gave standard deviations below one point. OpenCode and Pi preserved the model ordering on both tasks tested despite changes in scores and trajectories. (5zX6 W1 & W2 & Q1, tcT8 W2.1 & Q3.1, W2.2 & Q3.2)
  • Documenting data construction and validating workload coverage. We added a 30-task manifest, checked audio, protein, and image splits for overlap, clarified shared inputs and molecular case coverage, and tested six frozen road matchers on Monaco’s unseen network. (KWrP W1 & Q1, W2.1 & Q2, W2.2 & Q3, W3.1, W3.2 & Q4)
  • Clarifying reference calibration and scoring. We documented starting systems, initial scores, expert configurations, and task-specific scoring. We tested alternative weights, estimated domain-level uncertainty, and distinguished overall from metric-specific gains over experts. (tcT8 W3.1 & Q4.2, W3.2 & Q4.1, Q9, Q10)
  • Specifying resource use and evaluation access. We reported model settings, tokens, API requests, charges, and episode durations. We also specified which resources agents can access during development and how separate agent and verifier containers enforce held-out isolation. (tcT8 W2.3 & Q8, W3.3 & Q5)
  • Supporting reproducibility. We identified the materials available for review and described the planned versioned release of executable task packages, prompts, environments, scoring configurations, frozen submissions, and interaction and evaluation records. (tcT8 W3.4 & Q7)

(4) Reviewer feedback.

  • Reviewer 5zX6 (3 → 4) confirmed that our experiments addressed the initial concerns. After reading the other reviews and our responses, 5zX6 praised the substantial additional work and raised the score to express strong support.
  • Reviewer KWrP (initial score 3) recognized our contribution and gave 3 for Presentation, Informativeness, and Soundness.
  • Reviewer tcT8 (initial score 2) stated that our answers could change the assessment. We replied point by point and have yet to receive a response.

Thank you again for your time and consideration.

Sincerely,
The Authors of #40805

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.