- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors’ goal is to support migrating software projects from legacy, hard-to-maintain languages such as COBOL to a language with a larger developer base, such as Rust. Therefore, they intend to extract information from the code using large language models. Because large language models are unreliable, they propose a process that extracts information from the code step by step, matches it against the specifications, and updates the specification until the match rate plateaus. Remaining errors are attributed to missing context, unclear scope, and language model errors.
The process first creates questions that are LLM-generated given the code, together with the code’s answers. These questions serve later as queries against the specifications. After querying the specifications using the questions, an LLM matches the results with the answers from the code. Here, the authors evaluate if and how the respective question is answered by the specification compared with the code.
In order to support the migration, the authors’ goal is an optimized list of specifications, so they add or correct specifications when the LLM finds a question in the code that cannot be answered, or is answered differently, by the specifications. The answers’ match rate between the code and the specifications is compared with the respective rate on holdout questions and answers generated by an LLM from the code before the process and never used for updating. They observe the rate plateau at about a 90% match rate. This means that about 10% of the holdout questions extracted from the code are not answered in agreement with the code by the specifications, even if the specifications are updated iteratively, which the authors attribute to missing context, unclear scope, or language model errors.
Critical strengths
The paper's proposed process aims to reduce errors when migrating software projects. The theory seems correct and states its assumptions, and the 37-page manuscript already addresses many minor critiques.
Critical weaknesses
Forward references and a barely readable or understandable manuscript with an extreme amount of non-linearity, broken storytelling, without a clear contribution beyond existing question-answering evaluation and iterative refinement, and a focus on answering minor and detailed critiques, leaving the most severe problems open.
The lack of a reproduction package paired with a missing evaluation of the process itself for the held-out questions of the main experiment: that the questions are meaningful, that the questions are meaningfully answered by the requirements, and an analysis of the remaining errors and ways to mitigate them.
Questions for authors
Are the held-out questions of the main experiment meaningful? Are the remaining problems fixable, and how? How well can language models improve specifications when measured independently of the fidelity metric? What is the ground truth for the specifications?
3 comments
We thank the authors for the detailed comments. When the reviewer downloaded the manuscript again, no changes were visible. The reviewer’s rating, therefore, cannot be changed in any way. What makes the debated manuscript convoluted are - and the reviewer is not giving a complete list here - forward references, such as the “observability rule” used in the abstract but defined in Section 3, the revision loop shown only in Appendix E (Figure 7), “the template comparison” cited in Section 3 before its introduction in Section 5.2, and “the dashboard” of Section 5.1 that exists only as Figure 9 in the appendix, and inconsistent terms, such as “curriculum”/”held-out” in the text versus “train”/”test”, “frozen test”, “test bank”, “probe bank” and “measurement bank” elsewhere, “judge” meaning sometimes answerer plus comparator and sometimes only the answerer, F† called “balance level”, “fixed-point estimate” and “stationary-state balance”, and results switching between 15, 13, 12 and 11 programs and between “pooled” and “program-mean” fidelity. Beyond the presentation, some of the reviewers' former inquiries remain open: the relevance of the questions was not assessed for the main experiment, only for 24 questions of a later run with a different model, of which eight were classified as technical noise; the reviewer did not find an analysis of the remaining failures of the main experiment, only a single example from a different run, and the proposed mitigation is untested. Furthermore, the reviewer found no assessment of the revised documents independent of the fidelity score, although five of 30 checked edits introduced incorrect claims. Finally, the pipeline is still unavailable.
Thank you for the specific examples. Regarding the following one:
When the reviewer downloaded the manuscript again, no changes were visible. The reviewer’s rating, therefore, cannot be changed in any way.
We have uploaded a revised manuscript with the changes marked in purple.
You mentioned:
... inconsistent terms, such as “curriculum”/”held-out” in the text versus “train”/”test”, “frozen test”, “test bank”, “probe bank” and “measurement bank” elsewhere, “judge” meaning sometimes answerer plus comparator and sometimes only the answerer, F† called “balance level”, “fixed-point estimate” and “stationary-state balance”
We have uploaded a revised manuscript with the changes marked in purple. Section 3 now connects the "train" and "test" labels in the figures to the "curriculum" and "held-out questions" and distinguishes the answerer from the comparator.
We agree that consistent wording makes the paper easier to follow. We see these as minor terminology changes. They do not change how the method works, how we evaluated it, or the value of the contribution: a failed probe identifies a specific discrepancy between a requirement and the code that a reviewer can inspect and the reviser can address.
... some of the reviewers' former inquiries remain open: the relevance of the questions was not assessed for the main experiment, ...
We addressed each of your evaluation questions in items 1~4 of our rebuttal and described the available materials in the reproduction-package paragraph. If a specific point needs further clarification, we would appreciate knowing which one.
We appreciate your questions:
1. Are the held-out questions meaningful? An expert checked 120 held-out questions from the main CardDemo experiment against the COBOL source and confirmed 119 reference answers. The expert also read the requirements and agreed with 93 of the judge's 120 verdicts on the initial documents and 109 of 120 on the final documents. The sample was stratified by program and final verdict. The expert could inspect and override AI-drafted assessments (Section 5.6, Table 7 and Appendix L, Check 1).
We assessed relevance to migration in a later CardDemo run. Three mainframe specialists reviewed 24 questions and classified eight as technical noise by majority vote (Appendix L, Check 4). We did not assess relevance across all 911 main-run questions.
2. Can the remaining errors be fixed? Some errors come from the LLM judge. In Appendix F, the judge accepts a revision that allows a return to the originating screen, even though the caller sets the destination to sign-on. One option is to have several models judge the same answer, or to sample several judgments from one model at a nonzero temperature. If the judgments disagree, a reviewer can check the source before accepting the edit. Section 5.4 compares individual judges. We have not tested whether combining or resampling their judgments reduces errors. But this is a well-studied self-consistency topic, which is orthogonal to this study. Questions about which screen behaviors to preserve still require a decision from the migration team.
3. How much did the specifications improve independently of the fidelity score? An expert checked 30 applied edits against the source. Of those, 25 were faithful and five introduced an incorrect claim (Table 7). We have no independent before-and-after assessment of the complete documents.
4. What is the ground truth? The source code is the reference for what the existing system does. The migration team decides which behaviors the replacement should preserve. We evaluated individual requirements against the code and had no gold-standard document for the revised requirements.
Contribution and presentation. Could you point to a specific passage in the main text where the presentation made the argument difficult to follow and tell us what was unclear?
Reproduction package. The submission includes the prompts, graph extraction and sampling procedure. The reproducibility statement lists the pipeline, configurations, probes, document versions, and results planned for release for CardDemo, Redis, and PetClinic. Industrial code and requirements are confidential.