acceptodds
Under review as a conference paper at ICLR 2027

MAINFRAME-GYM: Benchmarking Agents on the Maintenance of Production Legacy Estates

Abstract

Coding agents are being deployed on COBOL estates just as the engineers who maintain them retire, yet no benchmark measures whether an agent can maintain a production mainframe system. Existing mainframe benchmarks test knowledge or single functions, and software benchmarks are mined from public repositories whose history and tests a production estate lacks. We introduce Mainframe-Gym, a benchmark of maintenance tickets on the 432k-line decommissioned billing estate of a large telecommunications company, and release the two pieces of infrastructure it required. Harwell runs the estate's COBOL and JCL batch chains, with DB2 emulated, off the mainframe. TaskForge is a human-AI collaboration harness that scales task construction: engineers direct the work, agents draft and verify the tasks, and the domain expert, who does not work in English, is consulted in their own language and reviews tasks in full, so scarce expert time goes to judgment rather than writing. Every grader is tested before any agent runs. We test five frontier models, each in its own or a common harness. The best pair solves five of eight tickets at least once, and two tickets defeat every pair. In a closer pilot study, agents usually find the right programs but make incomplete changes, missing the clone programs and archived versions that carry the same logic, and their own tests do not catch it.

Primary area:
datasets and benchmarks
Keywords:
benchmarks and datasets, coding agents, LLM agents, mainframe, COBOL, legacy software, benchmark construction, human-in-the-loop

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_hxFvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a benchmark called Mainframe-Gym that contains problems for maintaining a telecommunications billing system and aims to evaluate coding agents’ ability to solve tasks involving COBOL estates. A COBOL estate is an industrial software system built around COBOL programs, including source codes, job configs, data definitions, and databases. Unlike an isolated programming problem, a task here requires multilevel interaction with the system.

The benchmark is built on a retired production system, but the authors construct the maintenance tasks and their tests because historical tickets and test suites are unavailable. They introduce HARWELL to run the system outside a mainframe. They also develop TASKFORGE, a workflow in which engineers use AI agents to help construct tasks, with a domain expert consulted on business rules.

In total, the benchmark contains 27 tasks with 308 test cases with each task receiving human review. They also evaluated 5 model-harness pairs each on 8 of the tasks, with the best pair solving 5 tasks at least once. A separate pilot examines failures across the broader task set and finds that agents often locate relevant code but leave the overall repair incomplete.

Critical strengths

The paper introduces a setting for evaluating coding agents on legacy software maintenance. In particular, the authors contribute HARWELL, which enables executable evaluation of maintenance tasks within a COBOL estate. It supports COBOL programs along with JCL jobs and database operations, which allows agents to test their changes and the benchmark to evaluate the resulting business behavior.

Compared with previous mainframe benchmarks, this enables a broader maintenance setting. MainframeBench [1] evaluates knowledge and COBOL summarization. COBOLEval [2] executes COBOL solutions, but problems there are isolated programs adapted from HumanEval. In the estate studied here, an agent must determine which programs actually execute before making a repair. Job configurations can select archived versions, while different billing plans can use separate program copies. The mobile-billing example illustrates this challenge: its reference fix changes eight files, including an archived version. DevOps-Gym [3] also evaluates agents on larger software workflows, so system-level evaluation is not itself new. HARWELL’s contribution is providing the execution support needed to evaluate these legacy mainframe workflows. This makes it possible to test maintenance abilities that isolated COBOL problems do not exercise.

Ref.: [1] Dau et al. XMainframe: A Large Language Model for Mainframe Modernization, 2024. [2] Bloop. COBOLEval, 2024. [3] Tang et al. DevOps-Gym: Benchmarking AI Agents in Software DevOps Cycle, 2026.

The authors also contribute TASKFORGE, which is the workflow they use to construct the tasks for this benchmark. The engineers use AI agents to help construct benchmark tasks and consult a domain expert on business rules. Its validation process checks that the reference fix passes and that the unchanged system fails. It also tests whether the grader rejects deliberately incorrect repairs that still compile and run. The graders reject all 208 faults tested.

This is a different starting condition from approaches that reuse existing tests. SWE-smith [4] introduces faults into repositories and retains candidates that break previously passing tests. DevOps-Gym’s issue-resolving pipeline [3] draws on pull requests with test-validated fixes. Mainframe-Gym’s estate came without those records or a test suite, so the authors had to construct the evaluation criteria themselves.

The value of this contribution is the attempt to make task construction verifiable despite the missing development history. The fault checks support the graders’ ability to detect the tested mistakes.

Ref.: [4] Yang et al. SWE-smith: Scaling Data for Software Engineering Agents, 2025. [3] Tang et al. DevOps-Gym: Benchmarking AI Agents in Software DevOps Cycle, 2026.

Critical weaknesses
  1. Validation I have some concerns about HARWELL. The validation does not fully establish whether the benchmark measures correct maintenance of the original system. The expected outputs are recorded by running the reference fixes through HARWELL, but HARWELL itself has not been compared against execution on a mainframe.

For the benchmark grader, there is uncertainty about whether a solution could be rejected because it differs from the reference fix despite satisfying the task. Table 7 in the appendix reports that some cases labeled pass-to-pass already fail on the unchanged system. I would also like to understand how these FTP/PTP labels were actually assigned and checked.

  1. Strength of the empirical evidence In the evaluation section, I would like to see more comparisons on the tasks. The main comparison evaluates 5 model harness pairs on only 8 shared tasks from one estate, and combined with wide and overlapping confidence intervals, it's hard to discern what the implication behind a score is. It is also unclear how representative these tasks are of maintenance problems in other COBOL estates.

For TASKFORGE, the paper describes how the workflow was used but does not measure whether it reduces construction effort compared with other possible processes. For example, I would like to see how much time the engineer and the expert spent on crafting a task. The results show that the workflow can produce tasks, but provide limited evidence for the claimed benefit of scaling task construction while conserving expert time.

Questions for authors

How were these FTP/PTP labels assigned and checked, and why were cases that fail on the unchanged system retained as PTP? If PTP labeled cases failed under the unchanged system already, how would it affect the scores for the agents? 2. How much time did the engineers and domain expert spend constructing and reviewing the tasks? 3. What validation, beyond agreement with the reference fixes in HARWELL, supports that the graded outputs match the intended mainframe behavior?

Reviewer_c1MEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MAINFRAME-GYM, a benchmark for evaluating coding agents on maintenance tasks in a real, decommissioned COBOL billing system. It contributes Harwell, a Linux-based execution environment for mainframe applications, and Task Forge, a human-guided, AI-assisted pipeline for constructing and validating maintenance tasks. The benchmark contains 27 tasks with 308 test cases, while the primary evaluation compares five model–harness pairs on eight common tasks. The results suggest that agents frequently identify relevant programs but struggle to complete modifications across all necessary locations.

Critical strengths
  1. Realistic evaluation setting: The benchmark uses a production-derived legacy codebase with complex dependencies, duplicated programs, and archived versions, providing a useful alternative to existing repository-wide code-fixing benchmarks.

  2. Substantial engineering contribution: Harwell integrates COBOL execution, JCL interpretation, database emulation, and mainframe-specific utilities, enabling execution-based evaluation without requiring proprietary mainframe infrastructure.

  3. Systematic benchmark construction and validation: Task Forge combines automated verification with domain-expert consultation. The authors validate all 27 task packs against reference fixes, unchanged states, and seeded faults, reporting that all 308 test cases pass under the reference fixes and all 208 seeded faults are detected.

  4. Useful empirical observations: The failure analysis highlights agents' tendency to make incomplete modifications across related programs and incorrectly conclude that their changes are sufficient.

Critical weaknesses
  1. Limited methodological novelty: The task formulation and execution-based grading largely follow existing repository-wide code-fixing benchmarks, such as SWE-bench. Although Task Forge introduces a human-guided workflow for constructing tasks without repository histories or existing tests, its methodological advances over prior task-synthesis frameworks are not sufficiently established.

  2. Limited experimental scale: Although the benchmark contains 27 tasks, the primary comparison across five model–harness pairs includes only eight common tasks. The resulting confidence intervals are wide and overlapping, limiting conclusions about comparative agent performance.

  3. Limited generalizable insights: The failure analysis identifies incomplete multi-program modifications as a major challenge, but the paper provides limited controlled evidence isolating the effects of duplicated programs, archived versions, and execution dependencies. Consequently, it remains unclear which characteristics of the benchmark are primarily responsible for agent failures.

  4. Uncertain execution fidelity: Harwell has approximately 80 documented deviations from mainframe behavior and has not been directly validated against execution on z/OS. Although the authors document workarounds and many deviations concern components outside the evaluated batch workload, the extent to which the remaining differences affect grading reliability is not established.

  5. Task validation concerns: Expert review initially identified inaccuracies in eight of 18 sampled pre-change states, only one of which had been flagged by the automated world judge. Most were subsequently corrected, but this highlights limitations in the automated validation process. Additionally, five admitted tasks were later found to contain defects, raising further questions about benchmark reliability.

Questions for authors

See Weaknesses.

Reviewer_Zf46OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper presents a benchmark testing coding agents ability to solve tasks from a COBOL mainframe environment taken from a proprietary decomissioned production system.

The workflow for creating tasks is described in depth; production data was not available, so tasks had to be created from scratch using a domain expert who worked with the system for 20 years.

The complete benchmark and evaluation was not completed at the time of submission. 27 tasks were validated, and five agents were evalauted on the 8-27 of the completed tasks with 1-7 seeds per task. Opus 5.5 performs the best.

Critical strengths

The benchmark is a novel, valuable measure of coding agent ability that existing benchmarks do not capture. Constructing a valid benchmark in this way would require a lot of engineering effort and would be valuable to the community.

Critical weaknesses
  • The benchmark has not been completed, and results presented in the paper are preliminary: only 8 of the 27 tasks have evaluations from all five presented models, and many of the task-model evaluations only have a single seed run, while others have up to 7 seeds run. A full evaluation would be 27 tasks, 5 models, and 5 trials per task. Almost no analysis is conducted of the evaluation results.
  • The paper has extreme issues with writing clarity. Very little of the benchmark construction process is understandable. The task-creation "funnel" taxonomy is not clearly explained. I believe a lot of the issues in clarity stem from the heavy use of AI to write the paper. There is an abundance of unexplained jargon throughout the paper which makes it hard to understand.
Questions for authors
  • Please explain the task collection funnel in clearer terms.
  • Please run evaluation for 3-5 trials for each method.
  • Please fix the lack of clarity throughout the paper.

The level of incompleteness of the current paper makes it hard for me to imagine recommending acceptance. Even if revisions include a fully completed benchmark, evaluation, analysis, and clearer manuscript, this would be such a drastic change from the current manuscript that a resubmission seems more appropriate

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.