- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper introduces SysLoopBench, a benchmark of 30 scientific and engineering software development tasks across six domains. It evaluates whether agents can iteratively improve systems toward or beyond expert-developed references. Experiments with six models examine final performance, efficiency, progress, regressions, and recovery.
Critical strengths
- The paper studies a central question: whether agents can keep improving scientific and engineering systems while retaining earlier gains. The benchmark provides a useful addition to existing evaluations of this process.
- The benchmark constructions and experiments involve substantial work, and the analysis is thorough. The findings about progress, regressions, recovery are informative. The paper is also clearly written.
Critical weaknesses
- Long development runs may vary considerably, but each model–task pair is run only once. It is therefore unclear how stable the scores and trajectory findings are. Given the cost, could the authors repeat a set of tasks with the best- and worst-performing models, or just the best-performing model?
- All results use OpenCode, so it is unclear whether the model differences and development trajectories depend on this choice. Since the harness can affect model performance, could the authors test another harness with the best- and worst-performing models, or just the worst-performing model?
Questions for authors
Please address the questions raised in the weaknesses above.
3 comments
Thank you for the additional experiments. They address my concerns. I have also reviewed the comments from the other reviewers and your responses, and I see that you have added a substantial amount of experimental work. I think you have done a very good job. Given that negative scores still exist for the paper, and to fully express my recognition of your work, I have decided to raise my score from 3 to 4.
Thank You for Your Support
Thank you very much for your strong support and encouraging feedback! We are glad that our responses have addressed your concerns. We sincerely appreciate your time and effort in reviewing our manuscript.
Response to Reviewer 5zX6
Thanks for the constructive feedback. We provide a point-to-point response as follows.
Thank you for asking about the stability of the scores and trajectory findings across independent runs.
We conducted repeatability experiments and confirmed that our trajectory findings hold across independent runs, with the benchmark scores of the reruns shown below falling within a modest range for each task.
Specifically, we report three independent reruns of GPT-5.6 Sol (the best-performing model) for each of three tasks from different domains. All other settings were the same as in the main evaluation. The results are shown below.
(1) In terms of final scores, the reported reruns attain comparable final performance despite differences in their development paths, showing that the corresponding paper results can be reproduced through independent development. (2) Regarding the trajectory findings, the repeats show comparable overall development trends despite differences at individual rounds, consistent with the qualitative trajectory findings reported in the paper.
Our benchmark design supports repeatability through fixed data, standardized evaluation, and traceable development records. We keep task data, starting implementations, and budgets fixed across runs to make outcomes comparable, and validate measurement procedures to limit scoring noise. We freeze each submitted system and retain its feedback, allowing each point in the development trajectory to be traced to a specific implementation and evaluation record.
Thank you for suggesting an evaluation with an alternative harness.
We conducted cross-harness experiments using Pi and found that the model ordering and broad development trends remain consistent, although absolute scores and individual trajectories vary with the harness.
Specifically, we compared GPT-5.6 Sol (the best-performing model) and Qwen3.8 Max (the lowest-scoring model in the first round) on two tasks from different domains using OpenCode and Pi. All other task and evaluation settings were kept consistent with the main evaluation. The results are shown below.
(1) In terms of model differences, the model comparisons remain consistent across harnesses. Both harnesses preserve the taskwise ordering, and GPT retains the higher mean held-out score across the two tasks. Harness choice can raise or lower absolute scores depending on the model-task combination, without changing the direction of these comparisons.
(2) Regarding the trajectory findings, the broad development trends and differences in model improvement persist across harnesses. Harness choice changes the timing and magnitude of progress. The same model can improve mainly in earlier rounds under one harness and later rounds under another, producing different development paths.
Therefore, we use a common harness in the main evaluation to compare all models under the same development conditions. Our benchmark keeps the task definition, initial system, evaluator, and budget fixed when changing the harness, supporting comparisons under a consistent task protocol.
Thank you again for your constructive feedback, which has helped us strengthen the work. We have incorporated all the clarifications and additional results discussed above into the revised manuscript.