acceptodds
Under review as a conference paper at ICLR 2027

Occam's Progressive Harness for Agentic Migration of Legacy Scientific Program

Abstract

Migrating legacy Fortran solvers to modern stacks like JAX manually is expensive, since one solver can reach tens of thousands of lines. Coding agents with a regular harness can pursue a goal autonomously. Such a harness carries the agent's notes, or a history kept by a manager, from one session to the next, and ends the work by decision of the agent, a judge or the manager. A successful migration, however, should preserve the full scientific functionality of the legacy solver. A harness that ends the work without identifying the missing requirements and the redundant code can deliver a program of low quality. We propose Occam's Progressive Harness (OP-Harness), which takes a minimal sufficient program as its target, a program that meets every requirement and contains nothing else, and manages one verified measurement, the distance from the candidate program to that target. A builder plans and builds toward the target from the last measurement. A verifier measures the distance again as the gaps and the redundancies of the program. The two alternate on the same candidate program until both counts are zero. Across six whole legacy Fortran solvers migrated to JAX under the same model and task information, OP-Harness consistently delivers more complete programs, with stronger performance on held-out tests and post-migration probes. These results suggest that explicitly managing the remaining distance to the target can improve the reliability of autonomous whole-solver migration, bringing autonomous modernization of legacy scientific software closer to practical use.

Primary area:
foundation or frontier models, including LLMs
Keywords:
coding agents, agent loops, verification, long-horizon tasks, state transition, scientific code migration, legacy scientific software

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wsdHOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2(+1)
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper studies autonomous migration of legacy Fortran scientific solvers to Python and JAX within specified task contracts. OP-Harness alternates fresh-context builder and verifier sessions: the verifier reports missing requirements and redundant code, and the builder uses these reports to plan subsequent changes. Migration ends when the verifier reports neither category or a resource limit is reached. The evaluation covers six solver tasks and five baseline harnesses under a shared model and task information, with held-out tests and post-migration probes. OP-Harness achieves the strongest aggregate behavioral results at a higher inference cost.

Critical strengths
  • The proposed progressive harness seems intuitive and correct in tasks like large legacy repo migration.
  • The reference-based evaluation is useful. The task packages include held-out tests and additional probes, with numerical tolerances and physical checks against the original Fortran solvers.
Critical weaknesses
  • The methodological advance over judged and hierarchical harnesses remains unclear. Tracking missing requirements and unnecessary code is an intuitive way to organize development. However, Sections 3.1–3.2 do not clearly establish what this representation adds beyond an explicit requirement checklist and repeated inspection of the current program. Appendix C describes the hierarchical baseline's auditor as examining files and tests and rebuilding task constraints. The paper needs to explain the essential distinction and demonstrate when the alternative progress representations fail while OP-Harness succeeds. The complete-harness comparisons show different outcomes, but do not isolate this distinction through matched workflows or component ablations.

  • The reliability and scalability of identifying gaps and redundancies are not sufficiently evaluated. Section 3.2 acknowledges that the requirement set is not explicitly enumerated and that the true distance cannot be computed directly; the verifier instead estimates the two lists. Determining that a requirement is missing, or that a component can be removed without compromising the task's scope, is itself a substantial verification problem. It is unclear how consistently the verifier defines and counts these items, how often it misses a requirement or incorrectly labels code redundant, and how its cost and accuracy change as the program and requirement scope grow. Figure 4 reports post-hoc counts and Figure 5 reports structural grades, but these do not establish the reliability of the proposed progress signal or zero-distance stopping rule. This is important because that signal is the central state passed between iterations.

  • The evaluation is too narrow to support the broader long-running progress-management claim. The six solvers cover different scientific domains, but all experiments concern the same family of contract-scoped Fortran-to-JAX migrations. The independent numerical checks are useful evidence for these tasks. However, the evaluation does not characterize which long-running-task conditions make the proposed progress representation beneficial, or show that the same benefit carries beyond this migration setup. If the main contribution is a general way to manage progress, the paper needs evidence organized around that claim, including targeted failure cases and controlled comparisons, rather than relying on aggregate migration success alone.

Questions for authors
  • What is the essential difference between OP-Harness and a judged or hierarchical workflow that already inspects the current program and maintains a requirement checklist? Can you show representative cases in which those workflows lose track of progress and explain how OP-Harness avoids the same failure? Please isolate the relevant change while keeping the underlying agent, tools, verification effort, and budget matched.
  • How are the units of a gap and a redundancy determined? How do you assess missed gaps, incorrect redundancy judgments, and agreement across repeated verifier calls? Among runs terminated by PASS, how often do independent tests or probes still reveal missing functionality? What evidence is available about verifier cost and reliability as code size and task scope increase?
  • With the same builder/verifier implementation, what changes when redundancy tracking, progressive ordering, or the structured report is removed individually? What task characteristics predict the benefit, and what evidence supports applying the proposed progress-management principle beyond legacy-code migration?
7 comments
AuthorsOpenReview ↗

Response to Reviewer wsdH (Part 1 / 5)

We thank the reviewer for the time devoted to our submission. We appreciate your assessment that the progressive harness is intuitive and correct for legacy migration, and that the evaluation against the reference solvers is useful.

In response to your questions, we added controlled migrations in which each variant changes one part of OP-Harness. We also measured the verifier's reliability directly. A new table in Appendix C of the revised manuscript compares the states of LH-Harness and OP-Harness.

We would be happy to discuss further and to clarify any point that remains unclear.

W1 & Q1(1) What is the essential difference between OP-Harness and a judged or hierarchical workflow that already inspects the current program and maintains a requirement checklist?

Response: Thank you for this important question. Although both types of harnesses have an examiner outside the working agent, the inspections and checklists of the judge and auditor are not the "explicit requirement checklist and repeated inspection of the current program" that the reviewer has in mind. The original submission already describes how these two types of harnesses operate and notes that neither takes measures to remove redundant code (Section 2.2, Appendix C).

First, the Judged harness does not inspect the program. The judge in Hermes Agent reads only the one-line task goal and the first 4,000 characters of the agent's latest reply to decide whether the agent should continue. It does not read the program or run tests, and it keeps no requirement checklist.

Second, the checklist of the Hierarchical harness is a predefined template for general tasks (computer-use GUI agents or terminal coding). The LH-Harness auditor examines files and tests and rebuilds a list of task constraints in ten categories (see Table 9 in revised Appendix C for details). The manager also keeps a list of acceptance constraints. Both lists are drawn up from the text of the task, while the requirements of a legacy solver are embodied in the behavior of the reference program and not written out one by one in that text. The template does not ask for a comparison with the reference program, and no role is responsible for finding the code in the program that exceeds the requirements.

Third, the list that the auditor rebuilds does not reach the manager. LH-Harness, designed for general tasks, caps each report of the auditor at 2,500 characters taken from its beginning and end, and the manager's total history at 100,000 characters. Both values are hard-coded in the source code of LH-Harness 0.1.7. All baselines in the main experiment of the paper are used with their default settings, and we did not change these values. In our migrations, this list lies in the middle of the report, exactly the part that is cut off, so none of the reports that the manager reads contains it.

Compared with the general tasks that LH-Harness targets, migration has a distinct structure that a harness can exploit. Although a legacy program has no written specification, an agent can measure gaps and redundancies against the reference program. The list of gaps covers requirements that the text of the task does not state, and the list of redundancies covers the parts of the program beyond the requirements and must be empty before a migration ends.

Therefore, the essential difference lies in the state that OP-Harness manages across sessions, namely the lists of gaps and redundancies of the candidate program relative to the minimal sufficient program.

Q1(2) Can you show representative cases in which those workflows lose track of progress and explain how OP-Harness avoids the same failure?

Response: Judged harnesses lose track of progress by ending too early, as our response to Q1(3) and Table R2 below show.

For Hierarchical harnesses, the task state that the manager keeps can fall behind the program, as the Introduction of the original submission points out and the records of the main experiment show (see revised Appendix C). In one migration of disort, this situation lasted from round 3 to the last round. At the start of round 12, the workspace already contained 26 commits that changed files, including the entry file and the source code of the numerical computation. The plan of the manager for this round still read "the state carrier is still empty and eleven rounds of wall-clock were lost", and the task it assigned was still to create and commit the first runnable entry file. In this migration, sessions timed out repeatedly and the auditor never produced a report. Because the manager does not inspect the program, the task state could not be updated and stayed at the point when the workspace was empty.

In contrast, the state of OP-Harness does not reuse the record of the previous iteration. In every iteration, the verifier measures the gaps and redundancies again from the current program, so the state does not fall behind the program.

AuthorsOpenReview ↗

Response to Reviewer wsdH (Part 2 / 5)

Q1(3) & Q3(1) Please isolate the relevant change while keeping the underlying agent, tools, verification effort, and budget matched. With the same builder/verifier implementation, what changes when redundancy tracking, progressive ordering, or the structured report is removed individually?

Response: We thank the reviewer for proposing these controlled comparisons, and we answer the two requests together. We added five groups of migrations, each of which changes one part of OP-Harness. Each new group is compared with OP-Harness repeated on DeepSeek-V4.1-Flash-1017, the version that the provider updated after the submission, and uses the same coding agent, model, tools, and limits of 16 iterations and 24 hours. Each group performed three migrations on each of the six solvers.

Two groups replace a part of OP-Harness with its counterpart in Judged and Hierarchical harnesses. In the Judged workflow group, a judge decides when the migration ends, and the pass rates on all three sets of tests dropped. The LH-Harness state group changes only the content of the verifier report from gaps and redundancies to the task state and the checklist of constraints that LH-Harness records. With this replacement, migrations ended earlier and the delivered programs had more redundancies. The pass rates on the held-out tests and the probes also dropped. The other three groups each remove one of the three designs, and Table R2 lists the results of all six groups.

For verification effort, OP-Harness and the three groups that change the report of the verifier, namely LH-Harness state, Gaps only, and Unstructured report, each ran one verifier session per iteration. However, Table R2 shows that their migrations ran different numbers of iterations and therefore different numbers of verifier sessions. In the Judged workflow group, by contrast, the judge uses no tools and performs no verification. The comparison of this group with OP-Harness isolates the replacement of verification with a judgment.

Migrations under the Judged workflow mostly ended in the iteration in which all visible tests passed, so they lost track of progress and the gaps that these tests do not cover remained in the delivered programs. In contrast, migrations of OP-Harness ran more than twice as many iterations, because the verifier kept listing gaps or redundancies and PASS requires both lists to be empty.

In the LH-Harness state group, the migration ends only when the task state has no unfinished item and the checklist has no blocking constraint. The verifier still checks and runs the program in every iteration. The builder still reads the full report, unlike the truncation in the original LH-Harness. All other settings are unchanged.

The task state that LH-Harness records does not record redundancies, so the reports of this group, like those of the Gaps only group in which the verifier lists only gaps, almost never identified a redundancy. Yet the LH-Harness state group ended earlier than the Gaps only group. These results show that once the state is replaced by the task state designed for general tasks, the parts of the program beyond the requirements are no longer measured. The migration then ends early, and redundancies remain in the delivered program.

Table R2. OP-Harness and five variants, each of which changes one part of OP-Harness (mean ± std over three trials, each averaged over the six solvers). The best value of each column with an arrow is in bold, and the arrow marks whether higher or lower is better.

VariantHeld-out ↑Probe ↑Edge cases ↑IterationsGaps ↓Redundancies ↓
Judged workflow0.70±0.030.69±0.060.41±0.114.56±0.926.50±1.533.28±0.59
LH-Harness state0.83±0.030.86±0.030.74±0.085.72±0.632.22±0.543.56±0.51
Gaps only0.93±0.030.92±0.030.81±0.027.11±0.481.56±0.793.22±1.67
Unstructured report0.82±0.040.82±0.070.66±0.088.50±0.602.89±1.641.89±0.84
No progressive ordering0.89±0.090.90±0.060.81±0.1213.39±0.511.50±0.441.89±0.51
OP-Harness (ours)0.92±0.060.93±0.010.85±0.0811.72±1.351.39±0.101.61±0.84

Note. Gaps and Redundancies are estimated in each delivered program after the migration by an agent with the verifier prompt of OP-Harness and without the migration history.

  • The implementation details of each group are in Appendix G, with an overview in Table 23 and the prompts in Appendix G.1.
  • All metrics of each group are in Table 4 of the main text and Table 24 in Appendix G.2.
  • The edge cases are new held-out tests that we constructed during the discussion period and are listed in Appendix E.
  • The counting of Gaps, Redundancies, and the process metrics is described in Appendix G.2.
AuthorsOpenReview ↗

Response to Reviewer wsdH (Part 3 / 5)

Q3(1) With the same builder/verifier implementation, what changes when redundancy tracking, progressive ordering, or the structured report is removed individually?

Continued Response: In the Gaps only group, the verifier lists only gaps, and an empty list of gaps becomes the condition for ending the migration. All other settings are unchanged. Without the list of redundancies, the fraction of reports identifying redundancies fell from 83% to 3%, and the plan items of the builder that remove redundancies also fell to nearly zero. The delivered programs therefore kept more redundancies. As Table R2 above shows, the pass rate was slightly higher on the held-out tests, about the same on the probes, and slightly lower on the edge cases.

The Unstructured report group changes only how the verifier writes its report, from two separate lists to a narrative without sections, and no longer requires the items to be listed one by one with their evidence. With all other settings unchanged, the verifier still checks in both directions, gaps and redundancies, and the migration still ends only when the report finds neither. The fraction of reports identifying redundancies fell to 17%, and the plan items removing redundancies fell to 0.83 per migration. The pass rates on all three sets of tests also dropped, as Table R2 shows.

In the No progressive ordering group, the instructions that set the order of work are deleted from the prompt of the builder. The builder is no longer asked to order the remaining requirements by their importance to the correctness of the program or to finish one item before starting the next. The plan and the itemized commits are kept. Migrations took more iterations, and the number of migrations that ended with PASS within 16 iterations fell from 11 to 8 out of 18. The commit history of the builder explains the extra iterations. The builder returned more often to items that it had already committed, and the plan items committed again rose from 1.06 to 2.83 per migration. Table R2 shows slightly lower pass rates on all three sets of tests.

Although the pass rates of the Gaps only and No progressive ordering groups changed little, the effect of these two designs shows mainly in the process metrics of a migration, namely whether the verifier identifies redundancies, whether the builder removes them, and how many iterations the migration takes.

Q2(1) How are the units of a gap and a redundancy determined?

Response: Thank you for this precise question. Each requirement that the verifier judges to be missing enters the list of gaps in its report. A part of the program enters the list of redundancies if the verifier judges that the task remains complete without the part. Both definitions are written in the prompt of the verifier (Appendix D). Since the task description does not list its requirements one by one (Section 3.2 of the paper), the verifier decides the granularity of the items, and an item has no predefined unit. Nevertheless, we measured the size of an item. Taking redundancies as the example, in 8 of the 18 migrations of OP-Harness in the main experiment, the last report of the verifier identifies specific redundancies, 36 in total. Among these 36 redundancies, one redundancy names a median of 2 functions or other objects. In contrast, many gaps have no corresponding code yet, so we do not estimate the size of a gap. Since the unit is not predefined, the number of items is only an estimate of the number of gaps and redundancies. Our response to W2 & Q2(2), Q2(3), Q2(4) below reports whether this estimate is stable across repeated verifier sessions. Whether the verifier gives PASS depends only on whether both lists are empty, not on the number of items.

W2 Figure 4 reports post-hoc counts and Figure 5 reports structural grades, but these do not establish the reliability of the proposed progress signal or zero-distance stopping rule.

Response: We are grateful for this careful reading of Figures 4 and 5. The measurements in Table R3 below give the reliability of the estimate of gaps and redundancies. Figure 4 of the paper and the Gaps and Redundancies columns of Table R2 above use exactly this estimate. All of them come from agents equipped with the verifier prompt of OP-Harness. We use these estimates only to compare a harness or a variant with OP-Harness by their means over 18 migrations, and the conclusions of this paper do not rest on a difference of one or two gaps or redundancies.

Reviewer_9RfFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces Occam's Progressive Harness (OP-Harness) which is used to perform code migrations from legacy codebases (e.g., fortran) to modern systems (e.g., JAX). It does this by utilizing a builder which plans towards the target from the last measurement, and a verifier which measures the distance by accounting for both gaps and redundancies, these alternate until no gaps or redundancies remain.

Critical strengths
  • The performance of this method across 6 representative benchmark solvers (bellhop, cea, distort, mercury6, modflo88 and xfoil) beats other tested harnesses on challenging tasks, and meets them on others (where most hit 100%). Dampening this somewhat, It does require some significant increase in token use/cost.

  • Supplementary material provides an in depth analysis of experimental methods used as well as details about how the method operates, which involve rather extensive prompts.

Critical weaknesses
  • While this was mentioned in the limitations, it was the first thing that came to my mind while reading the introduction. The methodology defines a target as a minimal sufficient program (meeting every requirement of the task and containing nothing else), however for continuous simulations and other solvers which can involve complex dynamics it may not be possible to define a fixed set of tasks as there may be edge cases where the potentially very interesting/weird dynamics arise. If only a fixed grid of input/output targets are provided as tasks these important areas could be missed (which also are most likely the most challenging to migrate). Softening this however, even in these scenarios the methodology outperforms other harnesses so the work provides a positive step forward in this area.

  • Not critical, but the title should be Programs (plural) instead of Program for the last word.

Questions for authors

My major question regards to the major weakness finding. How can this methodology handle edge cases and other complex dynamics in solvers -- or is this left for future work? This seems critical for "true" ports of these legacy code bases, and I felt the work should have spent more time addressing this.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 9RfF (Part 1 / 2)

We sincerely thank the reviewer for the careful reading of our paper and its supplementary material, and for the encouraging assessment of our results. We are grateful for your note that our work "provides a positive step forward in this area."

Your question about edge cases and complex dynamics helped us strengthen the evidence on inputs that the visible tests do not cover.

During the rebuttal, we constructed 49 edge cases across the six solvers and compared the delivered XFOIL programs with wind tunnel measurements. We would be glad to discuss further and to clarify anything that remains unclear.


W1 ...for continuous simulations and other solvers which can involve complex dynamics it may not be possible to define a fixed set of tasks as there may be edge cases where the potentially very interesting/weird dynamics arise. If only a fixed grid of input/output targets are provided as tasks these important areas could be missed...

Q1 How can this methodology handle edge cases and other complex dynamics in solvers -- or is this left for future work? This seems critical for "true" ports of these legacy code bases, and I felt the work should have spent more time addressing this.

Response: Thank you for this insightful question. We agree that the original submission lacked a discussion of edge cases and complex dynamics. Accordingly, we have conducted new experiments and added Appendix E, Appendix F, and a dedicated paragraph in the Limitations section of the revised manuscript.

First, we would like to clarify the definition of the target in our methodology. We agree that if the target were defined merely as a fixed grid of input/output pairs, important areas with interesting or weird dynamics could indeed be missed. In our work, however, the target is defined over the input domain specified by the task contract, while the visible tests serve only as a subset of sample cases within that domain. As noted in Section 2.2 of the paper, requirements not covered by visible tests are easily overlooked in subsequent sessions under existing harnesses. Although we cannot provide theoretical guarantees, preventing such uncovered requirements from being missed is precisely what motivated the design of OP-Harness. Both the builder and the verifier work against the reference program, which already contains the branches corresponding to edge cases and complex dynamics. Ideally, therefore, these branches fall within the scope of requirements even when they are not covered by visible tests.

To empirically investigate how well the current methodology handles edge cases and complex dynamics, we constructed 49 new edge cases across the six solvers and evaluated all 108 programs delivered in the main experiment on them, without running new migrations. These edge cases were selected based on physical regimes, such as laminar separation bubbles at low Reynolds numbers and angles of attack near stall in XFOIL. All inputs lie within the task contract domain, execute code paths in the reference programs that were never triggered by the visible tests, and were completely withheld from all harnesses during migration. Across these 49 edge cases, OP-Harness achieves an average pass rate of 0.90 ± 0.02 across the six solvers, outperforming every baseline (which range from 0.21 to 0.66). The per-solver breakdown is shown in Table R1. The largest margin appears on cea, where OP-Harness achieves 0.73 ± 0.06 compared to 0.00 to 0.27 for the five baselines.

Table R1. Pass rates on 49 edge cases (per-solver counts in parentheses): mean ± std over three migrations.

Harnessbellhop (10)cea (10)disort (5)mercury6 (5)modflow88 (10)xfoil (9)Mean
Codex0.60±0.530.23±0.060.80±0.001.00±0.000.63±0.150.15±0.260.57±0.07
OpenClaw0.20±0.200.20±0.170.67±0.311.00±0.000.67±0.210.00±0.000.46±0.02
LH-Harness0.13±0.230.13±0.150.73±0.121.00±0.000.83±0.290.15±0.260.50±0.08
Terminus 20.10±0.170.00±0.000.00±0.001.00±0.000.17±0.060.00±0.000.21±0.03
Hermes Agent0.50±0.440.27±0.210.60±0.351.00±0.000.83±0.290.78±0.190.66±0.02
OP-Harness (ours)1.00±0.000.73±0.060.73±0.121.00±0.001.00±0.000.96±0.060.90±0.02

The physical regimes, the input features, and examples of the reference subprograms that each of the 49 edge cases executes, together with Table R1, are provided in Appendix E of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 9RfF (Part 2 / 2)

W1 ...for continuous simulations and other solvers which can involve complex dynamics it may not be possible to define a fixed set of tasks as there may be edge cases where the potentially very interesting/weird dynamics arise. If only a fixed grid of input/output targets are provided as tasks these important areas could be missed...

Q1 How can this methodology handle edge cases and other complex dynamics in solvers -- or is this left for future work? This seems critical for "true" ports of these legacy code bases, and I felt the work should have spent more time addressing this.

Continued Response: The 49 edge cases check the delivered programs against the reference programs. We further compared the delivered programs against experimental data, running them on conditions outside the evaluation cases for which wind tunnel measurements are available. We performed this evaluation on XFOIL, the largest of the six solvers (90,616 lines of reference code). We compiled 86 experimental data points across three airfoils from the existing literature, including angles of attack near stall and laminar separation bubbles. On the lift measurements for which a program returns a value, two of the three programs delivered by OP-Harness matched the lift coefficients of the original XFOIL within 3.5×10−133.5 \times 10^{-13}. The third program matched within 1×10−51 \times 10^{-5} across all points except for a single post stall condition. The mean absolute lift coefficient deviations from the wind tunnel measurements for the three delivered programs are 0.0898, 0.0918, and 0.0894, respectively, compared to 0.0898, 0.0898, and 0.0894 for the original XFOIL on the exact same points. Comparison plots across all airfoils are provided in Appendix F of the revised manuscript.

The 49 edge cases and the wind tunnel comparison show that the current method already handles most of the edge cases we tested. For regimes that no test covers, one possible approach is to validate candidate programs using physical invariants, such as conservation laws and analytical solutions in known asymptotic limits, which do not depend on test coverage of the input space. Finally, we have added a paragraph in the Limitations section of the revised manuscript explicitly stating that in regimes where correct variants of the reference program disagree on an output quantity (such as chaotic orbits), the evaluation compares only the quantities on which the variants agree, that coverage of edge cases lacks theoretical guarantees, and that the target remains bounded by the specified task contract.


W2 Not critical, but the title should be Programs (plural) instead of Program for the last word.

We thank the reviewer for catching this. We have updated the last word of the title to the plural form, Programs, in the revised manuscript.


We thank the reviewer again and welcome further questions during the discussion.

Reviewer_Y7VrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Modernizing legacy solvers (e.g., Fortran to JAX) is expensive. One can potentially speed this up by the use of AI (autonomous coding agents). The results have been mixed though, with conventional inference time harnesses delivering low-quality programs. One of the error events is not identifying missing requirements or redundant code. This paper works to address this lacuna. The authors propose Occam’s Progressive Harness (OP-Harness), which continuously measures the distance between a candidate program and a minimal, yet sufficient, target. By alternating a builder and a verifier to eliminate code gaps and redundancies, this method aims to delivers complete, high-performing programs across entire solver migrations.

Critical strengths
  • More complete programs: works to meet necessary scientific requirements without missing vital functionality.
  • Stronger performance: migrated code does better on held-out tests and post-migration validation modules.
  • Improved reliability: enhances the overall dependability of code migration processes.

I liked the idea of "minimal sufficient target" and managing the remaining distance via iterations between builder and verifier modules.

Critical weaknesses
  • Given how practical the problem is and how clean the method is, I wish the authors actually did "production grade" work and launch in public and get practical traction "in the wild". Far superior "peer review" than here at ICLR. I strongly recommend taking this approach out in the public and working to get "reviews in the wild" before bringing it to a conference such as ICLR. As is there are too many submissions and the methods are not "breakthrough" or anything remote, the only real test is scalability in real world environments.
Questions for authors

see "weakness" section above.

1 comment
AuthorsOpenReview ↗

Response to Reviewer Y7Vr

We thank the reviewer for the time and effort devoted to our submission. We appreciate that you found the problem practical and the method clean, and that you liked the idea of a minimal sufficient target. Your emphasis on practical use led us to test the delivered programs on conditions outside our benchmark.

During the rebuttal, we compared the delivered XFOIL programs with wind tunnel measurements, and our responses also clarify the contributions of the paper and the scale of its tasks. We would be happy to discuss further and to address any remaining concern.


W1 & Q1(1) I wish the authors actually did "production grade" work and launch in public and get practical traction "in the wild"... I strongly recommend taking this approach out in the public and working to get "reviews in the wild" before bringing it to a conference such as ICLR.

Response: Thank you for the important suggestion to productize this approach, release it publicly, and test it in real usage. However, we believe that establishing a benchmark and conducting controlled evaluations under peer review is just as essential as public deployment. Controlled evaluations and peer review serve to validate the soundness of the core methodology itself, a step that should naturally precede productization. Accordingly, our roadmap prioritizes rigorous peer review before moving toward full production. While a production release is currently in preparation as future work, our harness, task packages, and evaluation scripts are already publicly available and ready to use in our anonymous repository, enabling the reader to reproduce our results or apply them to their own solvers.

To further substantiate the reliability of the delivered programs in practical scenarios, we conducted an additional experiment evaluating them on operating conditions backed by wind tunnel experimental data outside our benchmark suite. We performed this evaluation on XFOIL, the largest of the six solvers (90,616 lines of reference code). We compiled 86 experimental data points across three airfoils from the existing literature. On the lift measurements for which a program returns a value, two of the three programs delivered by OP-Harness matched the lift coefficients of the original XFOIL within 3.5×10−133.5 \times 10^{-13}. The third program matched within 1×10−51 \times 10^{-5} across all points except for a single post stall condition. The mean absolute lift coefficient deviations from the wind tunnel measurements for the three delivered programs are 0.0898, 0.0918, and 0.0894, respectively, compared to 0.0898, 0.0898, and 0.0894 for the original XFOIL on the exact same points. These results show that even on conditions outside the evaluation cases, the delivered programs remain consistent with the original XFOIL and exhibit comparable agreement with physical measurements. Comparison plots across all airfoils are provided in Appendix F of the revised manuscript.


Q1(2) As is there are too many submissions and the methods are not "breakthrough" or anything remote, the only real test is scalability in real world environments.

Response: Regarding novelty and scalability, our work makes three distinct research contributions.

First, we formalize the migration objective as a minimal sufficient program. Second, we define the cross-session state managed by the harness as the remaining gaps and redundancies relative to this target, remeasured against the task in every round. Third, we introduce the first benchmark for agentic migration of complete solvers from Fortran to JAX.

In terms of scale, every task in our benchmark requires migrating a complete solver, with reference codebases ranging from 7,015 to 90,616 lines of code.


We thank the reviewer again and welcome further questions during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.