acceptodds
Under review as a conference paper at ICLR 2027

Fidelity Probes for Specification-Code Alignment

Abstract

Software modernization often relies on natural-language specifications recovered from an existing implementation. Incorrect or missing requirements can lead to defects in the modernized system. We introduce *fidelity probes* to check a specification against the code and guide its revision. Each probe pairs a question about program behaviour with a reference answer derived from the code. A language model answers the question using only the specification, and the fraction of agreeing answers defines *fidelity*. Disagreements flag conflicting claims or missing information and guide proposed corrections and additions to the requirements. An LLM generates probes directly from code or phrases facts selected through control-flow, data-flow, and call-graph analysis. We use an observability rule to focus probe questions on behaviour that the modernized system is intended to preserve, including user-visible outputs, changes to stored business data, and interactions with other systems. We revise the specification using fresh probes and evaluate each version on a fixed held-out set. A repair–regression model describes how new failures can offset the gains from repairs. On CardDemo, held-out fidelity improves from 0.59 to 0.89, exceeding free-form revision from source code. We also evaluate the method on industrial systems and public documentation, and assess probe quality and reported defects through human reviews and comparison with an independent requirements audit.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
fidelity probes, specification-code alignment, neurosymbolic probing, program analysis, requirements engineering, legacy code modernization, COBOL, iterative refinement, specification repair

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Fj7eOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors’ goal is to support migrating software projects from legacy, hard-to-maintain languages such as COBOL to a language with a larger developer base, such as Rust. Therefore, they intend to extract information from the code using large language models. Because large language models are unreliable, they propose a process that extracts information from the code step by step, matches it against the specifications, and updates the specification until the match rate plateaus. Remaining errors are attributed to missing context, unclear scope, and language model errors.

The process first creates questions that are LLM-generated given the code, together with the code’s answers. These questions serve later as queries against the specifications. After querying the specifications using the questions, an LLM matches the results with the answers from the code. Here, the authors evaluate if and how the respective question is answered by the specification compared with the code.

In order to support the migration, the authors’ goal is an optimized list of specifications, so they add or correct specifications when the LLM finds a question in the code that cannot be answered, or is answered differently, by the specifications. The answers’ match rate between the code and the specifications is compared with the respective rate on holdout questions and answers generated by an LLM from the code before the process and never used for updating. They observe the rate plateau at about a 90% match rate. This means that about 10% of the holdout questions extracted from the code are not answered in agreement with the code by the specifications, even if the specifications are updated iteratively, which the authors attribute to missing context, unclear scope, or language model errors.

Critical strengths

The paper's proposed process aims to reduce errors when migrating software projects. The theory seems correct and states its assumptions, and the 37-page manuscript already addresses many minor critiques.

Critical weaknesses

Forward references and a barely readable or understandable manuscript with an extreme amount of non-linearity, broken storytelling, without a clear contribution beyond existing question-answering evaluation and iterative refinement, and a focus on answering minor and detailed critiques, leaving the most severe problems open.

The lack of a reproduction package paired with a missing evaluation of the process itself for the held-out questions of the main experiment: that the questions are meaningful, that the questions are meaningfully answered by the requirements, and an analysis of the remaining errors and ways to mitigate them.

Questions for authors

Are the held-out questions of the main experiment meaningful? Are the remaining problems fixable, and how? How well can language models improve specifications when measured independently of the fidelity metric? What is the ground truth for the specifications?

3 comments

We appreciate your questions:

1. Are the held-out questions meaningful? An expert checked 120 held-out questions from the main CardDemo experiment against the COBOL source and confirmed 119 reference answers. The expert also read the requirements and agreed with 93 of the judge's 120 verdicts on the initial documents and 109 of 120 on the final documents. The sample was stratified by program and final verdict. The expert could inspect and override AI-drafted assessments (Section 5.6, Table 7 and Appendix L, Check 1).

We assessed relevance to migration in a later CardDemo run. Three mainframe specialists reviewed 24 questions and classified eight as technical noise by majority vote (Appendix L, Check 4). We did not assess relevance across all 911 main-run questions.

2. Can the remaining errors be fixed? Some errors come from the LLM judge. In Appendix F, the judge accepts a revision that allows a return to the originating screen, even though the caller sets the destination to sign-on. One option is to have several models judge the same answer, or to sample several judgments from one model at a nonzero temperature. If the judgments disagree, a reviewer can check the source before accepting the edit. Section 5.4 compares individual judges. We have not tested whether combining or resampling their judgments reduces errors. But this is a well-studied self-consistency topic, which is orthogonal to this study. Questions about which screen behaviors to preserve still require a decision from the migration team.

3. How much did the specifications improve independently of the fidelity score? An expert checked 30 applied edits against the source. Of those, 25 were faithful and five introduced an incorrect claim (Table 7). We have no independent before-and-after assessment of the complete documents.

4. What is the ground truth? The source code is the reference for what the existing system does. The migration team decides which behaviors the replacement should preserve. We evaluated individual requirements against the code and had no gold-standard document for the revised requirements.

Contribution and presentation. Could you point to a specific passage in the main text where the presentation made the argument difficult to follow and tell us what was unclear?

Reproduction package. The submission includes the prompts, graph extraction and sampling procedure. The reproducibility statement lists the pipeline, configurations, probes, document versions, and results planned for release for CardDemo, Redis, and PetClinic. Industrial code and requirements are confidential.

The lack of a reproduction package paired with a missing evaluation of the process itself for the held-out questions of the main experiment: that the questions are meaningful, that the questions are meaningfully answered by the requirements, and an analysis of the remaining errors and ways to mitigate them.

  • Held-out questions and answers. An expert confirmed 119 of 120 reference answers against the code. After reading the requirements, the expert agreed with 93 of 120 initial verdicts and 109 of 120 final verdicts (Table 7).
  • Relevance to migration. Three specialists assessed 24 questions from a later run and classified eight as technical noise by majority vote. We did not rate all 911 main-run questions for relevance (Appendix L, Check 4).
  • Remaining errors. In Appendix F, the caller sets the destination to sign-on. Checking that assignment exposes the error in the revised requirement. Disagreement between judges could flag such edits for review. We have not tested that mitigation.
  • Reproduction package. The submission includes the prompts, graph procedure, anonymous panel ratings, and panel analysis script. The full pipeline is part of the planned public release.

We thank the authors for the detailed comments. When the reviewer downloaded the manuscript again, no changes were visible. The reviewer’s rating, therefore, cannot be changed in any way. What makes the debated manuscript convoluted are - and the reviewer is not giving a complete list here - forward references, such as the “observability rule” used in the abstract but defined in Section 3, the revision loop shown only in Appendix E (Figure 7), “the template comparison” cited in Section 3 before its introduction in Section 5.2, and “the dashboard” of Section 5.1 that exists only as Figure 9 in the appendix, and inconsistent terms, such as “curriculum”/”held-out” in the text versus “train”/”test”, “frozen test”, “test bank”, “probe bank” and “measurement bank” elsewhere, “judge” meaning sometimes answerer plus comparator and sometimes only the answerer, F† called “balance level”, “fixed-point estimate” and “stationary-state balance”, and results switching between 15, 13, 12 and 11 programs and between “pooled” and “program-mean” fidelity. Beyond the presentation, some of the reviewers' former inquiries remain open: the relevance of the questions was not assessed for the main experiment, only for 24 questions of a later run with a different model, of which eight were classified as technical noise; the reviewer did not find an analysis of the remaining failures of the main experiment, only a single example from a different run, and the proposed mitigation is untested. Furthermore, the reviewer found no assessment of the revised documents independent of the fidelity score, although five of 30 checked edits introduced incorrect claims. Finally, the pipeline is still unavailable.

Thank you for the specific examples. Regarding the following one:

When the reviewer downloaded the manuscript again, no changes were visible. The reviewer’s rating, therefore, cannot be changed in any way.

We have uploaded a revised manuscript with the changes marked in purple.

You mentioned:

... inconsistent terms, such as “curriculum”/”held-out” in the text versus “train”/”test”, “frozen test”, “test bank”, “probe bank” and “measurement bank” elsewhere, “judge” meaning sometimes answerer plus comparator and sometimes only the answerer, F† called “balance level”, “fixed-point estimate” and “stationary-state balance”

We have uploaded a revised manuscript with the changes marked in purple. Section 3 now connects the "train" and "test" labels in the figures to the "curriculum" and "held-out questions" and distinguishes the answerer from the comparator.

We agree that consistent wording makes the paper easier to follow. We see these as minor terminology changes. They do not change how the method works, how we evaluated it, or the value of the contribution: a failed probe identifies a specific discrepancy between a requirement and the code that a reviewer can inspect and the reviser can address.

... some of the reviewers' former inquiries remain open: the relevance of the questions was not assessed for the main experiment, ...

We addressed each of your evaluation questions in items 1~4 of our rebuttal and described the available materials in the reproduction-package paragraph. If a specific point needs further clarification, we would appreciate knowing which one.

Reviewer_YXBBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents an unnamed system that uses fidelity probes, questions and answers generated from the source code, to iteratively refine natural language program specifications. These probes can be generated by an LLM, a control flow graph, a data flow graph, or a call graph. Specifications were revised using fresh probes and evaluated with a fixed held-out set. On CardDemo, held-out fidelity improves from 0.59 to 0.89.

Critical strengths
  1. The paper addresses an important problem of generating program specifications for modernizing legacy software.
Critical weaknesses
  1. Evaluation is flawed. First, there are no baselines from the related work section, making the system's efficacy compared to SOTA systems unknown. Second, there is no evidence that the fidelity probes are complete. If the fidelity probes miss a significant portion of program behavior, the system may be learning and emphasizing a part of the code when the LLM self-refinement is covering the spec more thoroughly. Third, even with the fidelity score as an evaluation metric, comparing results after 7 iterations is not fair. The comparison does not include computational budget, and the paper does not show whether the results were saturated (i.e., the rank may change after 100 iterations). Lastly, the main experiment only runs the system once. There is no evidence that the results will hold after repeated experiments.

  2. Real-world effectiveness is not proven. The paper does not establish which aspects of the specification are most important when modernizing legacy software. It may be coverage, accuracy, or both. The human audits found errors in the specifications, which may be more damaging than the lack of coverage. Also, the size and variety of programs used for evaluation are limited.

Questions for authors
  1. Why were methods mentioned in the related work section not used as baselines? Is the system better than SOTA systems?
  2. Can you prove that the fidelity probes cover all important program behaviors?
  3. Why were the iterations limited to seven? Is there strong proof that the performance was saturated? Do the results hold after a lot more iterations?
  4. Is the system more effective when including computational budget?
  5. Can you repeat the experiment multiple times and report statistics?
  6. What is the conclusion of Section 5.3? Which method is more effective? Why?
  7. What aspects of the specification are most important when using it to modernize legacy software?
  8. Human audits found errors in the specifications. Is there evidence that these are less damaging than missing specifications?
  9. Do the revised specifications lead to better software modernization?
  10. How well does your system generalize to larger and more diverse real-world programs?
3 comments

We appreciate your questions about the evaluation.

1. Baselines. We adapted Self-Refine from Section 2 to test whether probe feedback helps a model revise requirements. In free-form revision, the model reads the source and the requirements and finds discrepancies itself. In probe-guided revision, the same model also receives feedback on specific omissions and contradictions. Both methods produce revised requirements. The QA metrics in Section 2 score text, and the formal methods check programs or contracts. None of these methods directly quantifies how faithfully a natural-language requirements document describes the behavior of the source code.

We evaluated the revised documents on the same 911 held-out questions across 15 programs. After seven revisions, agreement reached 81.7% with free-form revision, 88.5% with resampled probes, and 90.8% with fixed probes (Section 5.2, Table 3). Resampled probe feedback improved agreement by 6.8 percentage points over our Self-Refine baseline.

2. Coverage. When we generate a question from a graph, we mark the source lines that support its answer. In production, we use this coverage signal to decide whether we have generated enough questions. If part of the code has too few questions, we generate more for that part. Section 3 describes how we select branches, data transformations, and calls. The paper omits the details of how we track the lines. A marked branch can still have outcomes that no question asks about, so line coverage alone cannot establish that we cover every important behavior.

3. Seven revisions. In production, we typically use two or three revisions. We used seven in the study to compare methods over the same number of rounds and track repairs and regressions in the analysis (Sections 4 and 5.1). On the 785 probes in Table 2, held-out agreement rose from 59.0% to 89.4% after two revisions, 90.6% after three, and 93.2% after seven. We did not establish saturation or test the ranking at 100 revisions.

4. Computational budget. Tables 13 and 14 report evaluation tokens, calls, and runtime for the direct run and three graph mixtures. They exclude revision calls, so an equal-cost ranking remains open. We compared document quality using the same held-out questions for Self-Refine and our method. We could not identify an independent metric for overall fidelity. We would welcome a metric you recommend for both methods that does not rely on an LLM giving the document a score.

5. Repeated runs. We ran each main condition once, so we cannot report variation across independent runs. Figure 5 answers a separate question about the judge. On the same 785 questions, the gain after three revisions was 31.6 percentage points under Opus, 25.5 under Qwen, and 27.3 under GPT-OSS.

6. Which method works best in Section 5.3? The balanced graph mixture had the largest gain on its own held-out questions, from 0.54 to 0.84. Control-flow probes alone went from 0.64 to a best score of 0.83, and the half-direct, half-control-flow mixture went from 0.64 to 0.80 (Table 5). Each mixture asks different questions, so these gains do not tell us which method produced the best document overall. When we evaluated graph-guided revisions on direct LLM questions across the 11 shared programs, the gains were smaller (Appendix J, Figure 14).

7. What matters in modernization? We focus on behavior the replacement must preserve, such as the decision a user receives, the data an update saves, and the information passed to another system (Section 3). Priorities depend on the migration. In Appendix L, reviewers unanimously confirmed six of seven findings about data and business rules. They disagreed more often about navigation and screen behavior.

8. Incorrect requirements versus omissions. We have no evidence that an incorrect requirement is less damaging than an omission. Five of 30 audited edits introduced an error (Table 7). In Appendix F, an edit changed the correct sign-on destination to allow a return to the originating screen. Our score counts each question equally and does not measure the consequences of that error.

9. Modernization outcomes. We measured whether the revised requirements better describe the existing code. We did not build and test a replacement from those requirements, so the effect on the resulting software remains open.

10. Scale and variety. We revised requirements for 32 industrial PL/1 programs, where agreement rose from 960/1,668 to 1,427/1,668 after one revision. Across four Java functional areas covering 25 PetClinic files, it rose from 151/203 to 201/203 after three revisions. We also evaluated specifications for 352 production COBOL programs on 23526 probes and human-written Redis documentation against six C modules (Table 6, Appendix N). The 352-program study evaluated requirements without revising them. We have also used the analysis on larger industrial codebases (tens of million lines) whose sizes are confidential.

Thank you for your response. I find only half of the questions addressed. My biggest concern is question 5. Without statistical evidence, I find it hard to trust the result. I will maintain my score.

Thank you for clarifying your main concern. We ran each CardDemo configuration once. An early edit changes the document that later rounds evaluate and repair. To measure run-to-run variance, we would need to repeat each configuration from the initial specification through its final revision.

Beyond the main CardDemo run, we conducted eight additional revision runs relevant to this question. Three varied the mix of graph-derived and direct LLM questions. Documents revised with graph-derived questions also improved on separately generated direct LLM questions (Figure 14). Three one-step runs varied the reviser. Opus, Qwen, and GPT-OSS each improved held-out agreement. In a separate check, all three models measured gains when judging the same CardDemo documents (Figure 5). The other two runs used different codebases and their own evaluation questions. Held-out agreement rose from 58% to 86% on industrial PL/1 programs and from 74% to 99% on Java PetClinic (Table 6).

We also tested how well the evaluation scales on System A, a production COBOL system with 352 programs and about 480,000 lines of code including copybooks. We evaluated its requirements with 23,526 probes in a single pass (Table 6, Appendix N).

We conducted five human evaluations (Section 5.6, Table 7 and Appendix L). On CardDemo, an expert confirmed 119 of 120 reference answers against the code, checked the judge's verdicts on the initial and final specifications, and reviewed 30 edits. Of those edits, 25 were faithful and five introduced an error. A second reviewer assessed whether 53 pairs of probes had the same meaning without seeing the comparator's decisions. On the industrial PL/1 system, failing probes were more likely to identify requirements that professional analysts had independently flagged as defective. Two panels examined later CardDemo runs. Three mainframe specialists assessed 24 questions for relevance and classified eight as technical noise. Five specialists reviewed 24 reported findings, confirming 15, rejecting seven, and splitting evenly on two.

Taken together, these results give us confidence that revision can improve agreement with the code across different question sets, models, and codebases. The human evaluations also show why individual edits need review: a judge can accept a change that makes a previously correct requirement wrong.

Reviewer_6bJCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a question-answering framework that is designed to iteratively revise natural-language software specifications against legacy source code. Each probe pairs a behavioral question with a reference answer extracted from the code via either direct LLM prompting or static program analysis graphs translated into business terminology. A separate LLM answers the question using only the candidate specification, and a comparator model flags contradictions or coverage gaps to guide targeted document edits. The authors shows the effectiveness of this revision loop on 15 COBOL programs from the public AWS CardDemo benchmark.

Critical strengths
  1. Aligning natural-language requirements with legacy code is a major bottleneck in software development where formal verification cannot be directly applied. The paper's strength lies in its unusually candid and thorough evaluation across public and industrial mainframes, including five distinct human validation checks that openly document when and how the automated pipeline fails.

  2. Separating specification failures into contradictions and coverage gaps linked to stable requirement identifiers provides concrete evidence for localized edits. On the CardDemo benchmark, this structured feedback allows the resampled revision loop to reach 88.5% held-out fidelity.

Critical weaknesses
  1. The comparator prompt allows vague specification text (such as "validates the limit") to agree with exact numeric thresholds in the code, which undercuts the goal of catching boundary bugs.

  2. Optimizing for probe agreement may introduce errors into the specification. Appendix F shows the pipeline overwriting a correct requirement with a flawed code answer while recording a higher fidelity score.

  3. Despite the complexity of extracting control-flow, data-flow, and call graphs, revisions driven by symbolic probes generalize poorly when tested on direct LLM probes across the shared 11-program cohort?

  4. The repair-regression convergence model in Section 4 relies on constant transition rates that may not hold empirically, as repair and regression rates vary a lot across iterations. Therefore, the formal analysis serves as a post-hoc accounting check rather than a predictive model of revision progress.

Questions for authors

Why does the comparator prompt in Appendix allow general statements in the specification ("validates the limit") to agree with specific numeric thresholds from the code? How can the system reliably catch boundary errors, like the motivating discount tier example in Section 1, if omits or generalizations of exact values are permitted to pass as agreements?

I'd also love to see author's response to weaknesses above.

1 comment

Thank you for pointing out the problem with the numeric-threshold example in our comparator prompt.

1. Numeric thresholds.

In mainframe modernization, the replacement can use a different storage mechanism and still preserve the same business rule. A requirement can therefore say that the system saves a record without naming the mainframe file. We allow that difference in wording. A discount threshold, however, determines what the customer receives.

At exactly 1,000, the code and REQ-DISC-002 in Figure 1 both give a 15% base discount. If a specification instead gives 20%, as Section 1 considers, the comparator is instructed to reject the conflicting values. For a boundary question, the comparator should count an answer as agreement only when the specification determines the outcome at the stated input. We use different prompt versions in production and continue to revise them. Appendix O.5 gives the version we used in these experiments. It can give credit for "validates the limit" even when the specification omits the threshold.

The probes complement formal verification and testing by checking the natural-language requirements before the replacement is built (Sections 1 and 2). An incorrect threshold in those requirements can otherwise carry into both the implementation and its acceptance tests. As Section 6 states, "High fidelity does not certify behavioural equivalence."

2. Incorrect edits.

Of the 30 edits checked against the source, 25 were faithful and five introduced an incorrect claim (Section 5.6, Table 7). The exit path in Appendix F is an example of the problem you describe. The code sets the destination to sign-on, and the original requirement says so. The revision allows a return to the originating screen. The comparator accepted that revision even though the original requirement was correct for this path.

Across the 911 held-out CardDemo probes, agreements rose from 535 to 806, contradictions fell from 120 to 46, and gaps fell from 256 to 59. For the exit requirement, checking the destination assignment against the revised text reveals the error.

3. Transfer from graph probes to direct LLM probes.

The gains on direct questions were smaller than the gains on each mixture's own held-out questions. Section 5.3 points to Figure 14 in Appendix J (p. 24) for the trajectories. For the table below, we pooled the saved results over 746 direct LLM probes across the 11 shared programs.

Revision probesInitialFinalGain
CFG0.6260.6936.7 points
Half direct / half CFG0.6330.6895.6 points
Balanced CFG / DFG / CallGraph0.6460.7207.4 points

We also tested the reverse direction. From the initial document to the sixth revision, documents revised using direct probes gained 6.0 points on CFG questions and 5.3 points on the balanced mixture. Their score on the half-direct/half-CFG questions fell by 1.4 points.

We use graphs to choose what to ask about before the LLM writes the question. For example, we can select a guard that controls whether a business record is updated. The guard and its location in the code provide evidence for the reference answer. Someone checking a proposed edit can follow the question back to that guard. We use the same approach to select data transformations and calls between programs (Section 3, Table 1 and Appendix K).

The wording can still change the question. For the same 90 CFG facts, the judge accepted 20 answers to template questions and 40 to LLM questions, with the documents held fixed (Section 5.2, Table 4 and Appendix M). In some cases, the template asked which condition triggers an action and the LLM asked what happens under that condition.

4. Transition analysis.

We agree that we have not shown that the model predicts the next revision's score. It does explain why the score stayed flat in the final revision. In the final step across all 15 programs, 21 previously failing probes passed and 21 previously passing probes failed. The overall score stayed the same (Section 5.1, lines 272-273). Theorem 4.1 gives convergence to repair_rate / (repair_rate + regression_rate) if the rates are constant and the recurrence is a contraction (Appendix A.2). The observed rates vary. To predict the next score, we would need to show that rates estimated from earlier revisions remain stable.

Comments on the paper

3 comments

Thank you for your detailed review.

1. Should we prefer resampling? This run gives us no reason to prefer resampling for held-out fidelity. Fixed probes scored 90.8% and resampled probes scored 88.5% (Table 3). We agree that "overfits" was too strong. The fixed run scored higher on its revision questions than on held-out questions, but that gap alone does not show that reusing questions hurt performance.

2. Did we test both final documents on new questions from another model? No. The closest test of shared-model bias changed the judge while keeping the documents, questions, and reference answers fixed. On 785 probes, the gain through revision three was 31.6 percentage points with Opus, 25.5 with Qwen, and 27.3 with GPT-OSS (Figure 5). Figure 14 in Appendix J also tests transfer between graph and direct LLM probes. Whether resampling beats fixed probes on fresh questions from another model remains open.

3. What happens when the LLM checks the code and requirements directly? Our free-form baseline does this at every revision. The model reads both documents, corrects contradictions, adds missing behavior, and returns the revised requirements in one call. It reached 81.7% after seven revisions on the same 911 questions, compared with 88.5% with resampled probe feedback (Section 5.2, Table 3). We did not test a separate first call that lists discrepancies before revision or compare with a practical requirements-recovery tool.

4. Are the 911 held-out probes independent? They come from the same source programs as the revision questions. We generated them separately and used their verdicts only for evaluation. In the saved resampled run, 258 of the 911 held-out questions also appeared verbatim in a revision round for the same program. On the other 653 questions, agreement rose from 377/653 (57.7%) to 578/653 (88.5%). This check counts exact wording only. We have not quantified overlap for the fixed curriculum. The free-form baseline uses no revision probes.

5. Did we repeat the main variants? Each ran once. We have no across-run means, variances, or confidence intervals.

6. Did experts evaluate the complete revised CardDemo documents? No. An expert checked 120 held-out probes and 30 edits against the code, with AI-drafted assessments available for review. Of the 30 edits, 25 were faithful and five introduced an incorrect claim (Section 5.6, Table 7). This gives us evidence about individual edits. Improvement in the complete documents remains unmeasured by experts.

7. Does the System B odds ratio hold on complete documents? In the excerpts we tested, failing probes were more likely to point to requirements the analysts had flagged. If an omitted paragraph answered a probe's question, the probe could report a gap that the full document would resolve. We did not rerun the comparison on the full documents, so we do not know whether the association would be as strong (Appendix L, Check 3).

8. Did the analysts review the revised System B document for new errors? No. Their audit covered the earlier requirements. After revision, agreement rose from 960 to 1427 on the same 1668 probes (Appendix N). We have no post-revision analyst assessment of newly introduced errors.

I thank the authors for providing the detailed clarification, which helped answer majority of my earlier concerns. Since my original score is already leaning on the positive side, I will retain my original score.

Thank you for your review and for considering our response. We appreciate your support.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.