- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces a benchmark called Mainframe-Gym that contains problems for maintaining a telecommunications billing system and aims to evaluate coding agents’ ability to solve tasks involving COBOL estates. A COBOL estate is an industrial software system built around COBOL programs, including source codes, job configs, data definitions, and databases. Unlike an isolated programming problem, a task here requires multilevel interaction with the system.
The benchmark is built on a retired production system, but the authors construct the maintenance tasks and their tests because historical tickets and test suites are unavailable. They introduce HARWELL to run the system outside a mainframe. They also develop TASKFORGE, a workflow in which engineers use AI agents to help construct tasks, with a domain expert consulted on business rules.
In total, the benchmark contains 27 tasks with 308 test cases with each task receiving human review. They also evaluated 5 model-harness pairs each on 8 of the tasks, with the best pair solving 5 tasks at least once. A separate pilot examines failures across the broader task set and finds that agents often locate relevant code but leave the overall repair incomplete.
Critical strengths
The paper introduces a setting for evaluating coding agents on legacy software maintenance. In particular, the authors contribute HARWELL, which enables executable evaluation of maintenance tasks within a COBOL estate. It supports COBOL programs along with JCL jobs and database operations, which allows agents to test their changes and the benchmark to evaluate the resulting business behavior.
Compared with previous mainframe benchmarks, this enables a broader maintenance setting. MainframeBench [1] evaluates knowledge and COBOL summarization. COBOLEval [2] executes COBOL solutions, but problems there are isolated programs adapted from HumanEval. In the estate studied here, an agent must determine which programs actually execute before making a repair. Job configurations can select archived versions, while different billing plans can use separate program copies. The mobile-billing example illustrates this challenge: its reference fix changes eight files, including an archived version. DevOps-Gym [3] also evaluates agents on larger software workflows, so system-level evaluation is not itself new. HARWELL’s contribution is providing the execution support needed to evaluate these legacy mainframe workflows. This makes it possible to test maintenance abilities that isolated COBOL problems do not exercise.
Ref.: [1] Dau et al. XMainframe: A Large Language Model for Mainframe Modernization, 2024. [2] Bloop. COBOLEval, 2024. [3] Tang et al. DevOps-Gym: Benchmarking AI Agents in Software DevOps Cycle, 2026.
The authors also contribute TASKFORGE, which is the workflow they use to construct the tasks for this benchmark. The engineers use AI agents to help construct benchmark tasks and consult a domain expert on business rules. Its validation process checks that the reference fix passes and that the unchanged system fails. It also tests whether the grader rejects deliberately incorrect repairs that still compile and run. The graders reject all 208 faults tested.
This is a different starting condition from approaches that reuse existing tests. SWE-smith [4] introduces faults into repositories and retains candidates that break previously passing tests. DevOps-Gym’s issue-resolving pipeline [3] draws on pull requests with test-validated fixes. Mainframe-Gym’s estate came without those records or a test suite, so the authors had to construct the evaluation criteria themselves.
The value of this contribution is the attempt to make task construction verifiable despite the missing development history. The fault checks support the graders’ ability to detect the tested mistakes.
Ref.: [4] Yang et al. SWE-smith: Scaling Data for Software Engineering Agents, 2025. [3] Tang et al. DevOps-Gym: Benchmarking AI Agents in Software DevOps Cycle, 2026.
Critical weaknesses
- Validation I have some concerns about HARWELL. The validation does not fully establish whether the benchmark measures correct maintenance of the original system. The expected outputs are recorded by running the reference fixes through HARWELL, but HARWELL itself has not been compared against execution on a mainframe.
For the benchmark grader, there is uncertainty about whether a solution could be rejected because it differs from the reference fix despite satisfying the task. Table 7 in the appendix reports that some cases labeled pass-to-pass already fail on the unchanged system. I would also like to understand how these FTP/PTP labels were actually assigned and checked.
- Strength of the empirical evidence In the evaluation section, I would like to see more comparisons on the tasks. The main comparison evaluates 5 model harness pairs on only 8 shared tasks from one estate, and combined with wide and overlapping confidence intervals, it's hard to discern what the implication behind a score is. It is also unclear how representative these tasks are of maintenance problems in other COBOL estates.
For TASKFORGE, the paper describes how the workflow was used but does not measure whether it reduces construction effort compared with other possible processes. For example, I would like to see how much time the engineer and the expert spent on crafting a task. The results show that the workflow can produce tasks, but provide limited evidence for the claimed benefit of scaling task construction while conserving expert time.
Questions for authors
How were these FTP/PTP labels assigned and checked, and why were cases that fail on the unchanged system retained as PTP? If PTP labeled cases failed under the unchanged system already, how would it affect the scores for the agents? 2. How much time did the engineers and domain expert spend constructing and reviewing the tasks? 3. What validation, beyond agreement with the reference fixes in HARWELL, supports that the graded outputs match the intended mainframe behavior?