acceptodds
Under review as a conference paper at ICLR 2027

CPSE-Bench: Benchmarking Coding Agents on Cyber-Physical Systems Engineering

Abstract

Coding agents perform strongly on generic coding benchmarks, but it remains unknown whether they can reliably engineer cyber-physical systems (CPS), such as driving simulators, robotics platforms, and autonomy stacks whose correctness requires closed-loop validation rather than unit testing alone. We introduce CPSE-Bench, a repository-level, execution-grounded benchmark of real cyber-physical engineering tasks across thirteen production repositories spanning driving, robotics, and aerial autonomy. Each task pairs a historical issue with a containerized behavioral reproducer verified against the buggy version and maintainer repair. We organize the released tasks into eight literature-grounded cyber-physical capability suites and evaluate nine models with mini-SWE-agent and find that even the strongest resolves only 73.5% of tasks. Our failure investigation reveals errors in enforcing physical constraints and propagating state across components, motivating models and scaffolds that reason about and validate the behavioral effects of code changes. We will publicly release the benchmark's public split and toolbox, comprising the collection pipeline, evaluation suite, and diagnostic toolbox, to facilitate further research in this field.

Primary area:
datasets and benchmarks
Keywords:
Coding Agents Benchmarks

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GDvQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CPSE-Bench adapts SWE-bench-style repair evaluation for cyber-physical systems (CPS) by collecting 400 issue-fix pairs from 13 open-source repositories and using containerized behavioral reproducer for evaluation through closed-loop checks. Tasks are categorized into eight capability suites such as physical dynamics and control. Evaluating nine models, the best one resolves only 73.5% of tasks, with determinism (57.1%) and physical dynamics (63.1%) being the weakest areas. The authors identify three failure modes through patch-level analysis and compare against the OpenHands scaffold, revealing limited model adaptability, particularly with GPT-5.6 Sol, which significantly decreased functionality. A public release will include a 40-task split and respective tools, while 360 tasks remain held back.

Critical strengths
  • The manuscript is well-motivated.
  • The failure analysis is the most valuable part of the manuscript
Critical weaknesses
  • The statistical analysis is thin.
  • "Closed-loop validation" is oversold.
Questions for authors
  1. The three failure modes come from 8 hand-picked cases (2 per leading model). Was the taxonomy validated by quantitatively classifying all failed patches, or could it be an artifact of case-selection bias?

  2. A few related work should be discussed, e.g.,

  • Ali, Shaukat, Paolo Arcaini, and Aitor Arrieta. "Foundation models for the digital twins creation of cyber-physical systems." In International Symposium on Leveraging Applications of Formal Methods, pp. 9-26. Cham: Springer Nature Switzerland, 2024.
  • X. Xie and L. Ma, "Marvel: Effective AI-Enabled Cyber-Physical System Falsification Guided by Novelty Detection," in IEEE Transactions on Reliability, vol. 75, pp. 2247-2259, 2026, doi: 10.1109/TR.2026.3705058.
  1. HWE-Bench (native simulation flows for hardware) and SWE-bench Science (scientific repositories) already do domain-specialized, execution-native repository-level repair. Beyond the domain and the behavioral-check emphasis, what in the construction or evaluation methodology is essentially new and not portable from those works?
Reviewer_T7rJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CPSE-Bench, a benchmark designed to evaluate coding agents on real-world cyber-physical systems (CPS) engineering tasks. The benchmark includes tasks from 13 production repositories spanning autonomous driving, robotics, and aerial autonomy. It introduces a closed-loop evaluation framework better suited to CPS, assessing the effectiveness of code repairs through execution-grounded behavioral validation rather than relying solely on unit tests. The authors evaluate nine models and find that even the strongest model achieves a task resolution rate of 73.5%. Further failure analysis reveals that existing coding agents still struggle with enforcing physical constraints and propagating state across components.

Critical strengths
  1. The domain of the benchmark is novel. This is the first benchmark that evaluates agents' ability on closed-loop CPS tasks.

  2. The motivation of the closed-loop setup is reasonable for CPS tasks.

  3. Most of the evaluation results are comprehensive. The evaluation contains most of the powerful models.

  4. The paper is easy to follow.

Critical weaknesses
  1. The added value of closed-loop evaluation is not sufficiently quantified.

    Appendix indicates that some checks are essentially component-level tests (open-loop). The paper does not report the proportion of tasks that involve genuine closed-loop evaluation, nor does it quantify how many behavioral defects the additional checks can detect beyond the original repository tests. How do the authors ensure that all tests are conducted in a closed-loop setting?

  2. The failure analysis does not sufficiently support the attribution of failures to CPS-specific capabilities.

    Several failures involve undefined interfaces, compilation errors, or incomplete patches, which may reflect general software engineering or execution issues rather than CPS-specific challenges. Although the appendix acknowledges the limitations of the case studies, the paper lacks a quantitative breakdown distinguishing general engineering failures from those related to physical constraints or temporal behaviors.

  3. The robustness of some evaluation results and the reliability of the corresponding conclusions remain unclear.

    While the main leaderboard evaluates all 400 tasks, the scaffold comparison and public set use only 40 tasks, with five tasks per category and a single attempt per task. Given that some results differ by only one or two solved tasks, the lack of repeated experiments and uncertainty estimates makes it difficult to assess whether these differences are meaningful.

Questions for authors
  1. Beyond the maintainer-provided reference patches, have the authors tested alternative correct implementations to assess the risk of false rejections? The appendix acknowledges that validation of alternative correct fixes has not been quantified. Could the authors provide additional evidence?

  2. How many tasks are evaluated entirely through closed-loop execution, rather than component-level tests?

  3. Could the authors clarify the criteria used to assign capability labels and report the proportions of failures attributable to general software engineering issues versus genuine violations of physical or temporal constraints?

  4. Could the authors repeat the 40-task scaffold experiments and report uncertainty estimates (like avg@k, pass@k), or expand the task set, to verify whether the observed differences are robust?

  5. Could the authors report the average input/output token usage, and API cost for each model?

  6. If incomplete patches and interface omissions decrease significantly under an agent framework with a more complete harness, these failures become harder to attribute to the model’s lack of CPS capabilities.

Reviewer_w5PdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CPSE-Bench introduces 400 repair and feature-implementation tasks from 13 cyber-physical software repositories spanning driving, robotics, and aerial autonomy. Tasks cover eight capability groups and evaluate agent patches through containerized behavioral checks and required repository tests. The paper provides a benchmark construction toolbox and evaluates nine models with mini-SWE-agent, with the strongest resolving 73.5% of tasks. It also analyzes failed repairs and compares four models under OpenHands on a 40-task subset to examine how agent workflows affect performance.

Critical strengths
  1. The paper addresses an important problem in CPS software repair through a benchmark spanning autonomous driving, robotics, and aerial autonomy, including both simulators and autonomy software stacks. Repairing these systems requires code changes that preserve the intended physical and temporal behavior.

  2. The benchmark is carefully designed around real-world maintenance issues. Each issue is validated against both the buggy version and the maintainer's fix, and agent-generated patches are evaluated through execution rather than code inspection alone. The prepared environments, hidden tests, and restricted network access also help ensure a fair and controlled evaluation.

  3. The experimental results provide useful insights into where current agents struggle. Even when agents identify the correct code changes, they often fail to fully implement or validate the repair at the system level. The comparison across agent scaffolds also suggests that simply providing more tools is not enough. These findings highlight the need for better agent workflows that can reliably complete repairs and verify their effects on system behavior.

Critical weaknesses
  1. The reliability of the agent-generated evaluation tests is not sufficiently established. The construction agent receives both the issue and the maintainer patch, and the tests are validated by checking that they distinguish the buggy version from the reference fix (Appendix A.2–A.3). This does not establish that they cover the full requirement or reject other incorrect repairs. Independent review of test coverage, together with tests of deliberately incorrect patches and alternative correct repairs, would strengthen confidence that benchmark scores reflect genuine task completion rather than loopholes in the generated tests.

  2. The closed-loop claim is broader than the evaluation described. Figure 2 emphasizes feedback between software actions and evolving system states, but some tasks use component-level checks that do not require such a loop (Appendix A.1). The paper should report how many tasks actually involve closed-loop execution. Showing how often patches pass existing repository tests but fail the added behavioral checks would also clarify the benchmark’s advantage over conventional testing.

  3. The failure analysis does not clearly distinguish CPS-specific reasoning errors from general implementation failures. Several examples involve undefined methods or incomplete implementations, and four of the eight attempts timed out. Separating these failures from executable patches that violate CPS behavior would clarify the source of difficulty. An analysis of which domain knowledge or core capabilities, such as coordinate transformations, physical constraints, or temporal dependencies, most often contribute to failures would make the findings more useful.

  4. The model comparison would be more informative with task-level results. Table 1 reports percentages from a single attempt per task, so it is difficult to tell whether models struggle with the same problems or have different strengths. Report solved-task counts and compare successes and failures across models. Repeated runs would also help determine whether small score differences are consistent.

  5. The evaluation relies mainly on mini-SWE-agent, with OpenHands tested only on the 40-task proxy set. Including additional frameworks, such as full SWE-agent or Aider, and evaluating them on a broader task set would help determine whether the observed failures persist across different editing and testing workflows. Using the same models and comparable budgets would better distinguish model limitations from scaffold effects. The analysis should also separate failures to submit a patch from submitted repairs that fail the CPS requirements, since these point to different problems.

  6. The paper does not sufficiently discuss how tasks were selected and filtered, their distribution across repositories and dates, and how potential training exposure to historical fixes is assessed. Since only 40 of 400 tasks are planned for public release, it should also explain how researchers can reproduce full-benchmark scores and audit disputed test outcomes.

Questions for authors
  1. Do the generated tests reject incorrect repairs and accept alternative correct ones? Could the authors report an independent audit, including whether the PX4 geofence test rejects a patch that always uses current position and breaks default prediction.

  2. How many tasks actually require closed-loop execution?

  3. Which failures are due to missing CPS knowledge, rather than unfinished code? Separate build failures and incomplete repairs from patches that run but produce incorrect behavior. It would be useful to identify which domain knowledge or capabilities agents struggle with most and how often these lead to failure.

  4. How can independent researchers reproduce the full 400-task results and challenge incorrect verdicts when only 40 tasks are public? Specify the evaluation process and release task-selection statistics, repository/date distributions, and an assessment of potential training exposure.

  5. As far as I could tell, everything was done in simulation. As a benchmark idea, that is okay, but are any of these also tested in hardware? The physical side of things is very important in actual CPS, and things that work in simulation, often do not work in reality, and illustrating sim-to-real transfer for some would be a strong demonstration (and could be done eg with HITL instead of only some of the SITL). Without this, one could view this in essence as some nice but fancy game playing RL, in essence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.