- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2(+1)
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
The paper studies autonomous migration of legacy Fortran scientific solvers to Python and JAX within specified task contracts. OP-Harness alternates fresh-context builder and verifier sessions: the verifier reports missing requirements and redundant code, and the builder uses these reports to plan subsequent changes. Migration ends when the verifier reports neither category or a resource limit is reached. The evaluation covers six solver tasks and five baseline harnesses under a shared model and task information, with held-out tests and post-migration probes. OP-Harness achieves the strongest aggregate behavioral results at a higher inference cost.
Critical strengths
- The proposed progressive harness seems intuitive and correct in tasks like large legacy repo migration.
- The reference-based evaluation is useful. The task packages include held-out tests and additional probes, with numerical tolerances and physical checks against the original Fortran solvers.
Critical weaknesses
-
The methodological advance over judged and hierarchical harnesses remains unclear. Tracking missing requirements and unnecessary code is an intuitive way to organize development. However, Sections 3.1–3.2 do not clearly establish what this representation adds beyond an explicit requirement checklist and repeated inspection of the current program. Appendix C describes the hierarchical baseline's auditor as examining files and tests and rebuilding task constraints. The paper needs to explain the essential distinction and demonstrate when the alternative progress representations fail while OP-Harness succeeds. The complete-harness comparisons show different outcomes, but do not isolate this distinction through matched workflows or component ablations.
-
The reliability and scalability of identifying gaps and redundancies are not sufficiently evaluated. Section 3.2 acknowledges that the requirement set is not explicitly enumerated and that the true distance cannot be computed directly; the verifier instead estimates the two lists. Determining that a requirement is missing, or that a component can be removed without compromising the task's scope, is itself a substantial verification problem. It is unclear how consistently the verifier defines and counts these items, how often it misses a requirement or incorrectly labels code redundant, and how its cost and accuracy change as the program and requirement scope grow. Figure 4 reports post-hoc counts and Figure 5 reports structural grades, but these do not establish the reliability of the proposed progress signal or zero-distance stopping rule. This is important because that signal is the central state passed between iterations.
-
The evaluation is too narrow to support the broader long-running progress-management claim. The six solvers cover different scientific domains, but all experiments concern the same family of contract-scoped Fortran-to-JAX migrations. The independent numerical checks are useful evidence for these tasks. However, the evaluation does not characterize which long-running-task conditions make the proposed progress representation beneficial, or show that the same benefit carries beyond this migration setup. If the main contribution is a general way to manage progress, the paper needs evidence organized around that claim, including targeted failure cases and controlled comparisons, rather than relying on aggregate migration success alone.
Questions for authors
- What is the essential difference between OP-Harness and a judged or hierarchical workflow that already inspects the current program and maintains a requirement checklist? Can you show representative cases in which those workflows lose track of progress and explain how OP-Harness avoids the same failure? Please isolate the relevant change while keeping the underlying agent, tools, verification effort, and budget matched.
- How are the units of a gap and a redundancy determined? How do you assess missed gaps, incorrect redundancy judgments, and agreement across repeated verifier calls? Among runs terminated by PASS, how often do independent tests or probes still reveal missing functionality? What evidence is available about verifier cost and reliability as code size and task scope increase?
- With the same builder/verifier implementation, what changes when redundancy tracking, progressive ordering, or the structured report is removed individually? What task characteristics predict the benefit, and what evidence supports applying the proposed progress-management principle beyond legacy-code migration?
7 comments
Response to Reviewer wsdH (Part 2 / 5)
Q1(3) & Q3(1) Please isolate the relevant change while keeping the underlying agent, tools, verification effort, and budget matched. With the same builder/verifier implementation, what changes when redundancy tracking, progressive ordering, or the structured report is removed individually?
Response: We thank the reviewer for proposing these controlled comparisons, and we answer the two requests together. We added five groups of migrations, each of which changes one part of OP-Harness. Each new group is compared with OP-Harness repeated on DeepSeek-V4.1-Flash-1017, the version that the provider updated after the submission, and uses the same coding agent, model, tools, and limits of 16 iterations and 24 hours. Each group performed three migrations on each of the six solvers.
Two groups replace a part of OP-Harness with its counterpart in Judged and Hierarchical harnesses. In the Judged workflow group, a judge decides when the migration ends, and the pass rates on all three sets of tests dropped. The LH-Harness state group changes only the content of the verifier report from gaps and redundancies to the task state and the checklist of constraints that LH-Harness records. With this replacement, migrations ended earlier and the delivered programs had more redundancies. The pass rates on the held-out tests and the probes also dropped. The other three groups each remove one of the three designs, and Table R2 lists the results of all six groups.
For verification effort, OP-Harness and the three groups that change the report of the verifier, namely LH-Harness state, Gaps only, and Unstructured report, each ran one verifier session per iteration. However, Table R2 shows that their migrations ran different numbers of iterations and therefore different numbers of verifier sessions. In the Judged workflow group, by contrast, the judge uses no tools and performs no verification. The comparison of this group with OP-Harness isolates the replacement of verification with a judgment.
Migrations under the Judged workflow mostly ended in the iteration in which all visible tests passed, so they lost track of progress and the gaps that these tests do not cover remained in the delivered programs. In contrast, migrations of OP-Harness ran more than twice as many iterations, because the verifier kept listing gaps or redundancies and PASS requires both lists to be empty.
In the LH-Harness state group, the migration ends only when the task state has no unfinished item and the checklist has no blocking constraint. The verifier still checks and runs the program in every iteration. The builder still reads the full report, unlike the truncation in the original LH-Harness. All other settings are unchanged.
The task state that LH-Harness records does not record redundancies, so the reports of this group, like those of the Gaps only group in which the verifier lists only gaps, almost never identified a redundancy. Yet the LH-Harness state group ended earlier than the Gaps only group. These results show that once the state is replaced by the task state designed for general tasks, the parts of the program beyond the requirements are no longer measured. The migration then ends early, and redundancies remain in the delivered program.
Table R2. OP-Harness and five variants, each of which changes one part of OP-Harness (mean ± std over three trials, each averaged over the six solvers). The best value of each column with an arrow is in bold, and the arrow marks whether higher or lower is better.
| Variant | Held-out ↑ | Probe ↑ | Edge cases ↑ | Iterations | Gaps ↓ | Redundancies ↓ |
|---|---|---|---|---|---|---|
| Judged workflow | 0.70±0.03 | 0.69±0.06 | 0.41±0.11 | 4.56±0.92 | 6.50±1.53 | 3.28±0.59 |
| LH-Harness state | 0.83±0.03 | 0.86±0.03 | 0.74±0.08 | 5.72±0.63 | 2.22±0.54 | 3.56±0.51 |
| Gaps only | 0.93±0.03 | 0.92±0.03 | 0.81±0.02 | 7.11±0.48 | 1.56±0.79 | 3.22±1.67 |
| Unstructured report | 0.82±0.04 | 0.82±0.07 | 0.66±0.08 | 8.50±0.60 | 2.89±1.64 | 1.89±0.84 |
| No progressive ordering | 0.89±0.09 | 0.90±0.06 | 0.81±0.12 | 13.39±0.51 | 1.50±0.44 | 1.89±0.51 |
| OP-Harness (ours) | 0.92±0.06 | 0.93±0.01 | 0.85±0.08 | 11.72±1.35 | 1.39±0.10 | 1.61±0.84 |
Note. Gaps and Redundancies are estimated in each delivered program after the migration by an agent with the verifier prompt of OP-Harness and without the migration history.
- The implementation details of each group are in Appendix G, with an overview in Table 23 and the prompts in Appendix G.1.
- All metrics of each group are in Table 4 of the main text and Table 24 in Appendix G.2.
- The edge cases are new held-out tests that we constructed during the discussion period and are listed in Appendix E.
- The counting of Gaps, Redundancies, and the process metrics is described in Appendix G.2.
Response to Reviewer wsdH (Part 3 / 5)
Q3(1) With the same builder/verifier implementation, what changes when redundancy tracking, progressive ordering, or the structured report is removed individually?
Continued Response: In the Gaps only group, the verifier lists only gaps, and an empty list of gaps becomes the condition for ending the migration. All other settings are unchanged. Without the list of redundancies, the fraction of reports identifying redundancies fell from 83% to 3%, and the plan items of the builder that remove redundancies also fell to nearly zero. The delivered programs therefore kept more redundancies. As Table R2 above shows, the pass rate was slightly higher on the held-out tests, about the same on the probes, and slightly lower on the edge cases.
The Unstructured report group changes only how the verifier writes its report, from two separate lists to a narrative without sections, and no longer requires the items to be listed one by one with their evidence. With all other settings unchanged, the verifier still checks in both directions, gaps and redundancies, and the migration still ends only when the report finds neither. The fraction of reports identifying redundancies fell to 17%, and the plan items removing redundancies fell to 0.83 per migration. The pass rates on all three sets of tests also dropped, as Table R2 shows.
In the No progressive ordering group, the instructions that set the order of work are deleted from the prompt of the builder. The builder is no longer asked to order the remaining requirements by their importance to the correctness of the program or to finish one item before starting the next. The plan and the itemized commits are kept. Migrations took more iterations, and the number of migrations that ended with PASS within 16 iterations fell from 11 to 8 out of 18. The commit history of the builder explains the extra iterations. The builder returned more often to items that it had already committed, and the plan items committed again rose from 1.06 to 2.83 per migration. Table R2 shows slightly lower pass rates on all three sets of tests.
Although the pass rates of the Gaps only and No progressive ordering groups changed little, the effect of these two designs shows mainly in the process metrics of a migration, namely whether the verifier identifies redundancies, whether the builder removes them, and how many iterations the migration takes.
Q2(1) How are the units of a gap and a redundancy determined?
Response: Thank you for this precise question. Each requirement that the verifier judges to be missing enters the list of gaps in its report. A part of the program enters the list of redundancies if the verifier judges that the task remains complete without the part. Both definitions are written in the prompt of the verifier (Appendix D). Since the task description does not list its requirements one by one (Section 3.2 of the paper), the verifier decides the granularity of the items, and an item has no predefined unit. Nevertheless, we measured the size of an item. Taking redundancies as the example, in 8 of the 18 migrations of OP-Harness in the main experiment, the last report of the verifier identifies specific redundancies, 36 in total. Among these 36 redundancies, one redundancy names a median of 2 functions or other objects. In contrast, many gaps have no corresponding code yet, so we do not estimate the size of a gap. Since the unit is not predefined, the number of items is only an estimate of the number of gaps and redundancies. Our response to W2 & Q2(2), Q2(3), Q2(4) below reports whether this estimate is stable across repeated verifier sessions. Whether the verifier gives PASS depends only on whether both lists are empty, not on the number of items.
W2 Figure 4 reports post-hoc counts and Figure 5 reports structural grades, but these do not establish the reliability of the proposed progress signal or zero-distance stopping rule.
Response: We are grateful for this careful reading of Figures 4 and 5. The measurements in Table R3 below give the reliability of the estimate of gaps and redundancies. Figure 4 of the paper and the Gaps and Redundancies columns of Table R2 above use exactly this estimate. All of them come from agents equipped with the verifier prompt of OP-Harness. We use these estimates only to compare a harness or a variant with OP-Harness by their means over 18 migrations, and the conclusions of this paper do not rest on a difference of one or two gaps or redundancies.
Response to Reviewer wsdH (Part 1 / 5)
We thank the reviewer for the time devoted to our submission. We appreciate your assessment that the progressive harness is intuitive and correct for legacy migration, and that the evaluation against the reference solvers is useful.
In response to your questions, we added controlled migrations in which each variant changes one part of OP-Harness. We also measured the verifier's reliability directly. A new table in Appendix C of the revised manuscript compares the states of LH-Harness and OP-Harness.
We would be happy to discuss further and to clarify any point that remains unclear.
Response: Thank you for this important question. Although both types of harnesses have an examiner outside the working agent, the inspections and checklists of the judge and auditor are not the "explicit requirement checklist and repeated inspection of the current program" that the reviewer has in mind. The original submission already describes how these two types of harnesses operate and notes that neither takes measures to remove redundant code (Section 2.2, Appendix C).
First, the Judged harness does not inspect the program. The judge in Hermes Agent reads only the one-line task goal and the first 4,000 characters of the agent's latest reply to decide whether the agent should continue. It does not read the program or run tests, and it keeps no requirement checklist.
Second, the checklist of the Hierarchical harness is a predefined template for general tasks (computer-use GUI agents or terminal coding). The LH-Harness auditor examines files and tests and rebuilds a list of task constraints in ten categories (see Table 9 in revised Appendix C for details). The manager also keeps a list of acceptance constraints. Both lists are drawn up from the text of the task, while the requirements of a legacy solver are embodied in the behavior of the reference program and not written out one by one in that text. The template does not ask for a comparison with the reference program, and no role is responsible for finding the code in the program that exceeds the requirements.
Third, the list that the auditor rebuilds does not reach the manager. LH-Harness, designed for general tasks, caps each report of the auditor at 2,500 characters taken from its beginning and end, and the manager's total history at 100,000 characters. Both values are hard-coded in the source code of LH-Harness 0.1.7. All baselines in the main experiment of the paper are used with their default settings, and we did not change these values. In our migrations, this list lies in the middle of the report, exactly the part that is cut off, so none of the reports that the manager reads contains it.
Compared with the general tasks that LH-Harness targets, migration has a distinct structure that a harness can exploit. Although a legacy program has no written specification, an agent can measure gaps and redundancies against the reference program. The list of gaps covers requirements that the text of the task does not state, and the list of redundancies covers the parts of the program beyond the requirements and must be empty before a migration ends.
Therefore, the essential difference lies in the state that OP-Harness manages across sessions, namely the lists of gaps and redundancies of the candidate program relative to the minimal sufficient program.
Response: Judged harnesses lose track of progress by ending too early, as our response to Q1(3) and Table R2 below show.
For Hierarchical harnesses, the task state that the manager keeps can fall behind the program, as the Introduction of the original submission points out and the records of the main experiment show (see revised Appendix C). In one migration of disort, this situation lasted from round 3 to the last round. At the start of round 12, the workspace already contained 26 commits that changed files, including the entry file and the source code of the numerical computation. The plan of the manager for this round still read "the state carrier is still empty and eleven rounds of wall-clock were lost", and the task it assigned was still to create and commit the first runnable entry file. In this migration, sessions timed out repeatedly and the auditor never produced a report. Because the manager does not inspect the program, the task state could not be updated and stayed at the point when the workspace was empty.
In contrast, the state of OP-Harness does not reuse the record of the previous iteration. In every iteration, the verifier measures the gaps and redundancies again from the current program, so the state does not fall behind the program.