- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CPSE-Bench adapts SWE-bench-style repair evaluation for cyber-physical systems (CPS) by collecting 400 issue-fix pairs from 13 open-source repositories and using containerized behavioral reproducer for evaluation through closed-loop checks. Tasks are categorized into eight capability suites such as physical dynamics and control. Evaluating nine models, the best one resolves only 73.5% of tasks, with determinism (57.1%) and physical dynamics (63.1%) being the weakest areas. The authors identify three failure modes through patch-level analysis and compare against the OpenHands scaffold, revealing limited model adaptability, particularly with GPT-5.6 Sol, which significantly decreased functionality. A public release will include a 40-task split and respective tools, while 360 tasks remain held back.
Critical strengths
- The manuscript is well-motivated.
- The failure analysis is the most valuable part of the manuscript
Critical weaknesses
- The statistical analysis is thin.
- "Closed-loop validation" is oversold.
Questions for authors
-
The three failure modes come from 8 hand-picked cases (2 per leading model). Was the taxonomy validated by quantitatively classifying all failed patches, or could it be an artifact of case-selection bias?
-
A few related work should be discussed, e.g.,
- Ali, Shaukat, Paolo Arcaini, and Aitor Arrieta. "Foundation models for the digital twins creation of cyber-physical systems." In International Symposium on Leveraging Applications of Formal Methods, pp. 9-26. Cham: Springer Nature Switzerland, 2024.
- X. Xie and L. Ma, "Marvel: Effective AI-Enabled Cyber-Physical System Falsification Guided by Novelty Detection," in IEEE Transactions on Reliability, vol. 75, pp. 2247-2259, 2026, doi: 10.1109/TR.2026.3705058.
- HWE-Bench (native simulation flows for hardware) and SWE-bench Science (scientific repositories) already do domain-specialized, execution-native repository-level repair. Beyond the domain and the behavioral-check emphasis, what in the construction or evaluation methodology is essentially new and not portable from those works?