acceptodds
Under review as a conference paper at ICLR 2027

ORSMT: Unleashing the Potential of Satisfiability Modulo Theories for Semantic Verification and Repair in LLM-Based Optimization

Abstract

Large language models (LLMs) are increasingly used to automate optimization modeling from natural-language problem descriptions. However, existing verification methods often focus on the final reference solution, potentially overlooking inconsistencies when the mathematical formulation and solver code represent different optimization problems yet yield the same objective value. We propose ORSMT, an SMT-based framework for verifying and repairing optimization models generated by LLMs. For verification, ORSMT encodes the mathematical formulation and solver model in SMT and checks their feasible regions and objective preferences, producing either certificates of consistency or counterexamples. For repair, ORSMT reconstructs an inconsistent model from its usable counterpart and leverages the original problem specification and counterexamples to correct errors shared by both models. Experiments on 3,034 problems from 12 benchmarks and 6 model sizes show that ORSMT raises model consistency from 43.2% to 86.4% on average, significantly improving agreement with the reference objective value. As a plug-and-play module, ORSMT further improves the performance of two existing auto-formulation systems.

Primary area:
optimization
Keywords:
LLM-based optimization, Auto formulation, Optimization verification

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_kAHpOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper is part of the recent line of research that aims to use LLMs for automated modeling of mathematical optimization problems---specifically linear programs and mixed integer linear programs. The benchmarks for these problems, such as NLP4LP, OptiBench, etc., typically consist of a natural language description of the problem, a mathematical formulation or “written mathematics” (the paper never spells out what exactly this means, but I presume it refers to a LaTeX formulation, for example?), and solver code. It’s important that these three elements all represent the same underlying optimization problem. This paper aims to use an SMT-based verification to diagnose mismatches between these three types of representations and then repair these representations using feedback from the diagnoses. The experiments apparently uncover many such mismatches (e.g., a match rate of 43% across 3034 problems), and the repair step repairs them (increasing the match rate to 88%).

Critical strengths

I have not seen SMT solvers used in the context of automated MILP formulation, so the paper introduces a novel method for an important problem. The problem itself is important because using LLMs for automated math optimization modeling is becoming an increasingly important approach in industry, and many benchmarks have been developed, as illustrated by Table 1. It’s important that these benchmarks are accurate (i.e., that the natural language descriptions and solver code represent the same problem).

Critical weaknesses
  • I have been working on ML for integer programming for almost ten years, so I would expect a paper like this to be extremely easy for me to read and understand. On the contrary, I found much of it to be inscrutable. For example:
    • Undefined mathematical notation or terms:
      • Line 113: What is H\mathcal{H}?
      • Line 240: What is kk? What is an applicability condition and a logical obligation? Examples would help immensely.
      • Line 261: What do you mean by reported coordinates?
      • Line 279: What are DAD_A and KAK_A? You say they’re “defined above,” but I don’t know what you’re referring to.
      • Line 365: The table reports “Redundant constraint on this side only” and “Constraint absent from the other side.” What do “this side” and “the other side” refer to?
    • Descriptions of methods that are far too high-level:
      • Line 149: “The translator L\mathcal{L} lowers admitted mathematical formulation into the model representation AA defined in Section 3.” What does it mean to “lower” a mathematical formulation into a model representation? How does the translator work---i.e., what is the underlying methodology? How do we know that the mathematical representation is faithful to the original formulation?
      • Line 188: “The normalization may make implicit domains explicit and derive redundant constraints and auxiliary definitions needed for compilation, while preserving the semantic model objects used for verification.” How does the normalization operator work---i.e., what is the underlying methodology? What does it mean to derive redundant constraints and auxiliary definitions? How does it preserve the semantic model objects?
      • Line 249: “Row entailment applies the same violation test to individual mathematical and executable restrictions. A written row violated by an implemented decision is missing from the implementation.” I have no idea what these two sentences mean.
    • Sloppy writing:
      • Line 206: Why aren’t you using the LaTeX proposition environment?
  • Although the use of SMT solvers seems novel in this context, I walk away with almost no understanding of why SMT solvers are the right tool for this problem and what key challenges the authors faced and overcame for their application in this context.
Questions for authors

Unfortunately, this paper would need a complete rewrite before I'd change my assessment.

2 comments

Thank you for the detailed comments and for recognizing the value of applying SMT to optimization modeling. We have rewritten the problem setting and method sections and added a short SMT background. Below we follow the organization of your comments. Original line numbers identify your comments, and the corresponding revised line numbers are given in parentheses.

Undefined mathematical notation or terms

  • Line 113: Runtime extraction (revised lines 140–145 and 234–238). The extraction operator HH executes the solver program P^\hat{P} and captures the variables, constraints, and objective constructed at the solver call. Its output is the instantiated solver model B=H(P^)B=H(\hat{P}). Written mathematics means the mathematical formulation M, expressed as equations or structured declarations, which is translated into A=L(M)A=L(M). Section 4 and Figure 2 make this distinction explicit.

  • Line 240: Applicability conditions and logical obligations (revised lines 154–156 and 276–281). Here k indexes a formal requirement. In (gk,ψk,Sk)(g_k,\psi_k,S_k), gkg_k specifies when the requirement applies, ψk\psi_k states what must hold, and SkS_k identifies the supporting source text. For example, if an active facility must have positive capacity, the applicability condition is that the facility is active and the obligation is that its capacity is positive. A violation satisfies gkg_k but not ψk\psi_k. Definition 1 and Eq. (8) now state these roles explicitly.

  • Line 261: Reported coordinates (revised lines 282–289). We mean the decision variables whose values appear in the solver report. The set O indexes these variables. Other variables may be unreported, so Eq. (9) checks whether the reported values can be extended to a feasible assignment of the instantiated solver model. Section 5.2 now uses the term reported decision variables.

  • Line 279: Variable domains and constraint sets (revised lines 295–303). DAD_A collects variable domains and bounds, while KAK_A contains the constraints of the mathematical model A. In Eq. (11), we retain DAD_A and all constraints except the one being tested, then ask whether that constraint can be violated. This establishes whether it is redundant. Section 5.2 defines both sets at their use.

  • Line 365: The two model sides (revised lines 390–395, Table 2). The two sides are the mathematical model A and the instantiated solver model B. We now name them explicitly in the table. The redundancy row counts constraints redundant only within the named model. The non-implication row counts constraints of that model that are not implied by the other model.

Descriptions of methods

  • Line 149: Translation methodology and fidelity (revised lines 140–149, 215–216 and 232–233). Thank you for asking how translation is performed and checked. We have replaced the term lowering with an explicit description of translation in Section 5.1. The translator reads the formulation M into the mathematical model A, recording decision and auxiliary variables, domains, constraints, the objective, and its optimization direction. Structured declarations are parsed directly. For mathematical text, an LLM supplies declarations that undergo the same parsing checks. The parser checks unique variable names and resolved references and stores finite decimal coefficients as exact rationals. Appendix A.2 specifies the supported formats, and Figure 2 shows the translation on an example. The original formulation is retained for inspecting the transcription, while requirements extracted from the problem statement are checked separately against A.

  • Line 188: Normalization and semantic preservation (revised lines 240–247 and 298–303). Thank you for highlighting the need to explain semantic preservation. The revised Section 5.1 specifies normalization as removal of constraints proved redundant. For each constraint, ORSMT retains the variable domains and other constraints and asks whether the selected constraint can be violated. UNSAT establishes that the remaining model already implies it, permitting removal. Each subsequent removal is tested against the constraints still retained. Before construction, ORSMT checks that normalization preserves both the feasible decisions and the pairs of decisions and objective values, after projecting out auxiliary variables. These are the two preservation conditions stated in Section 5.1 and justified in Appendix B.3.

  • Line 249: Row entailment (revised lines 295–298). Thank you for pointing out that this description was unclear. We now explain a row as an individual constraint and state both checking directions in Section 5.2. For a constraint c of the mathematical model A over corresponding decision variables, ORSMT checks whether the instantiated solver model B admits a decision violating c. SAT supplies such a decision and identifies the constraint not enforced by B. UNSAT establishes that B implies c, even if it implements the restriction through different constraints. Conversely, testing a constraint of B against A identifies an additional restriction imposed by the implementation. These checks locate the specific constraint responsible for a disagreement and supply the counterexample used for revision.

Writing and proposition formatting

  • Line 206: Proposition environment (revised lines 346–349). Thank you for pointing this out. Certified adoption is now presented using a numbered proposition environment. Proposition 1 in Section 5.4 states that adoption requires all policy checks to pass and that the original artifacts are retained when no candidate is adopted.

Why SMT is suitable and which challenges are addressed

Thank you for raising this question. LLM checking can miss a violated constraint, mistake an equivalent formulation for an error, or generate an explanation unsupported by the models. Repeated LLM review also adds generation cost without establishing whether the required relation holds. SMT addresses these problems by checking the relation itself. For optimization models, domains and constraints define feasible decisions, and objectives define their preferences. A query asks whether these decisions or preferences violate the relation being checked. SAT returns a concrete counterexample, which ORSMT traces to the implicated constraints or domains and their recorded source links. UNSAT certifies the checked relation. For these violation queries, the diagnosis is supported by a checkable assignment and acceptance by a formal check. The checks require no LLM generation and can be repeated efficiently after revision.

The difficulty is to compare the mathematical model with the model actually constructed by code, despite differences in syntax, redundant constraints, and auxiliary variables. ORSMT captures the instantiated solver model, aligns decision variables, and projects out auxiliary variables. Reconstruction preserves feasible decisions and objective preferences, and query reduction preserves the checked relations. These steps allow the counterexample to identify a specific problem for revision and the same verifier to check whether it has been resolved. This is the advantage of using SMT for both diagnosis and verification before adoption, as described in Sections 5.1–5.4.

Reviewer_g21kOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ORSMT checks semantic relations among written optimization models, captured solver models, source requirements, and reported solutions. It uses exact SMT queries within an admitted affine rational representation, then reconstructs or revises artifacts and rechecks required relations before adoption. Across 18,204 generated outputs, protected selection increases model-code agreement from 43.2% to 88.2% and correct-and-matched outputs from 25.2% to 40.2%. Additional experiments examine repair baselines, solver agreement, scaling, and integration with existing modeling systems.

Critical strengths
  • Exact relation queries distinguish artifact agreement from answer correctness, addressing a concrete limitation of output-based evaluation.
  • Protected selection retains all originally correct outputs in Table 6, supporting the practical value of rechecking.
  • Broad artifact evaluations, host integrations, and independent solver comparisons support the framework's practical verification value.
Critical weaknesses
  • Exact guarantees depend on transcription and source admission whose fidelity is not quantitatively established end-to-end.
  • Reconstruction and candidate selection complicate attributing repair gains specifically to the proposed SMT diagnostic feedback.
  • Table 6's Opt-Verifier counts are inconsistent: newly correct plus retained outputs total 6,559, not 7,070.
Questions for authors
  1. Section 4.2 certifies formal requirements supplied by the source adapter. How often has this representation been independently checked for transcription accuracy and coverage of every original requirement?

  2. Section 5 obtains 7,857 additional matches by reconstructing one artifact from its counterpart. How much correctness improvement remains attributable to SMT feedback when reconstruction and candidate budgets are matched across methods?

  3. Figure 1's verdict labels do not explain what is certified or witnessed, making the workflow difficult to follow. Could the caption define UNSAT, SAT, and UNKNOWN and connect each outcome to the relevant checks and subsequent actions?

  4. Could you illustrate semantic agreement in Eq. (2) with a simple example showing equal feasible regions and identical preference orderings? Please clarify why this permits different objective values, which Eq. (6) checks separately.

  5. In Eq. (8), please parenthesize the indexed conjunction to distinguish it from the adjacent binary conjunctions.

  6. In Section 4.1, replace "contraints" and "contraint rows" with "constraints" and "constraint rows." In Figure 1's caption, replace "relaton" with "relation."

3 comments

Thank you for the careful review. Below we address the source evaluation, the controls for attributing gains, and the corrections in the revised manuscript.

Weaknesses

W1 / Q1. Source and transcription fidelity.

Thank you for raising this concern. We now describe the source review procedure and report coverage together with the downstream checking results. Section 5.1 specifies model translation, and Appendix A.2 identifies the origin and supporting text of the formal requirements. The full source evaluation covers all 3,034 problems: 2,207 (72.7%) have formalization proposals, and source review marks 1,701 (56.1%) as covering all requirements. Partial formalization supplies requirements that can be checked individually. Source review compares variable domains, units, coefficients, constraints, and objectives with the original text and records omissions and disputed interpretations. These review judgments are reported separately from SMT verdicts and reference-answer correctness.

We added an independent transcription audit on 100 randomly sampled problems, using only the original text and withholding candidate models and reference answers. Nine inputs have missing or conflicting information. On the same remaining 91 inputs, independent model review finds complete and correct formulations for 28 with Qwen2.5-32B, 41 with Qwen2.5-72B, and 90 with gpt-6.1-sol at high reasoning: 30.8%, 45.1%, and 98.9%, respectively, under the recorded interpretations. The Qwen comparison uses the same adapter protocol, while GPT directly formulates the original text.

Passing source checks is associated with higher answer correctness. Among the 9,525 repair candidates, the 680 for which both artifacts pass source checks contain 638 correct answers, or 93.8%, compared with 45.7% across the full candidate pool. These source checks are auxiliary to the main repair procedure. Our core contribution is to check semantic model agreement, diagnose discrepancies, and verify repairs before adoption. Reconstruction operates on the mathematical and instantiated solver models. Source formalization supplies additional requirements for checking these models. Improving transcription itself is not our core contribution.

W2 / Q2. Contributions of reconstruction and verification in selection.

Thank you for requesting these controls. The controls evaluate reconstruction and verification in selection separately.

The reconstruction gain persists under stricter preservation checks across all 72 combinations of generators and benchmarks. Requiring both preservation checks for every new match retains 15,468 matches among 18,204 outputs (84.97%).

Verification-based selection improves answer correctness without additional generation. With the same 9,525 candidates and their order fixed, selecting by model–code and source checks gives 936 correct answers, compared with 877 for the first candidate and 932 for model–code checks alone.

With reconstruction, selection rules, and five candidates per input fixed, combined model–code and source feedback raises correctness from 63.84% to 74.53% on the 589 inputs passing source review before generation, a gain of 10.70 percentage points over no feedback. Across all 1,905 inputs, adding source feedback raises correctness from 35.38% to 42.20%, a gain of 6.82 percentage points over model–code feedback alone. In the full population, combined feedback gives 42.20% correctness compared with 44.41% without feedback.

W3. Opt-Verifier counts in the original Table 6.

Thank you for identifying this inconsistency. The column was mislabeled: 2,139 is the net gain. The paired records contain 4,420 retained, 2,650 newly correct, and 511 lost outputs. Hence final correct = 4,420 + 2,650 = 7,070, while net gain = 2,650 − 511 = 2,139. The revised Table 8 labels this column “Net correct gain”, consistent with these paired counts. Questions

Q3. Meaning of SAT, UNSAT, and UNKNOWN in Figure 1.

Thank you for pointing this out. We have added a separate SMT Background section before Problem Setting. Section 3 introduces SMT and explains SAT, UNSAT, and UNKNOWN. The revised method then explains each verdict in terms of the query being checked. For a query asking whether a semantic violation exists, SAT supplies a concrete counterexample, UNSAT certifies that no such violation exists in the encoded relation, and UNKNOWN leaves the check undecided. Counterexamples guide revision. For a report extension query, SAT supplies values that complete the reported decisions consistently with the solver model. Sections 5.2 and 5.4 specify these meanings and actions, respectively. UNKNOWN or a time limit leaves a required check undecided, so the candidate is not adopted and the original artifacts are retained if no candidate passes.

Q4. Semantic agreement and different objective values.

Thank you for suggesting an example. Semantic agreement preserves feasible decisions and their preference ordering, so absolute objective values may differ. Consider two minimization models with the same feasible set x∈{0,1,2}x \in \{0,1,2\} and objectives f(x)=xf(x)=x and g(x)=2x+3g(x)=2x+3. Both rank every pair of feasible decisions identically and attain their optimum at x=0x=0, but their optimal values are 0 and 3. Definition 2 and Eq. (7) distinguish these two relations.

Q5. Parentheses around the indexed conjunction.

Thank you for catching this ambiguity. The revised Eq. (8) writes the violation query for each requirement separately. Eq. (3) states that satisfaction must hold for every requirement, removing the ambiguous combination of conjunctions.

Q6. Typographical corrections in the model representation section and Figure 1.

Thank you for the careful proofreading. We have corrected “contraints” to “constraints,” “contraint rows” to “constraint rows,” and “relaton” to “relation.”

Thank you for the extensive revision, which rewrites more than half of the main text, including Sections 3–5. The clearer presentation and corrected Opt-Verifier counts address several of my concerns, and I have adjusted my component scores accordingly.

I maintain my overall score because, with reconstruction, selection, and candidate budgets fixed, combined feedback achieves 42.20% accuracy on the 1,905 originally refuted outputs, compared with 44.41% without feedback. Accuracy rises from 63.84% to 74.53% on the 589 inputs that passed source review, but only 56.1% of the 3,034 problems were judged to have all requirements formalized. Please include these controls in the manuscript and specify the adapter model so readers can assess where diagnostic feedback helps.

Reviewer_MuspOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a semantic verification framework to check and repair the disagreement among natural language problem requirements, math model, solver code, and delivered reports. They evaluate their framework ORSMT on 12 benchmarks, showing that their method can largely improve the math model - solver code match rate.

Initial Summary before LLM polishing: The authors propose a semantic verification framework, called ORSMT, to check and repair the disagreement between natural language problem description, mathematical formulations, solver codes, and the final report. This framework was evaluated using 12 benchmarks, and the results show that ORSMT can largely improve the match rate between the math model and solver code.

Initial Critical Strengths before LLM polishing (For the authors, reviewers, chairs, and the AI-tools the chairs used to detect AI generated content):

Initial Critical Strengths before LLM polishing: First, The studied problem is important in optimization modeling, namely a right answer in matching objective value does not guarantee the generated math model can match the natural language problem requirement. Second, this framework can provide counterexamples which show why it is wrong and evidence why it is correct, compared to run only solvers. These tests may be valuable in practice.

Critical strengths
  1. The paper addresses an important problem: a correct numerical answer does not guarantee that the generated model and code represent the same optimization problem.

  2. SMT checks provide concrete counterexamples and stronger evidence of agreement than execution tests alone. The reconstruction results and integrations with existing modeling systems show practical value.

Initial Critical Strengths before LLM polishing (For the authors, reviewers, chairs, and the AI-tools the chairs used to detect AI generated content):

First, The studied problem is important in optimization modeling, namely a right answer in matching objective value does not guarantee the generated math model can match the natural language problem requirement. Second, Their framework can provide counterexamples which show why it is wrong and evidence why it is correct, compared to run only solvers. These tests may be valuable in practice.

Critical weaknesses
  1. The Source Adapter’s accuracy remains unclear. Appendix D.7 does not independently measure omission or mistranslation rates, and Appendix A.2 allows source contracts to come from reference formulations. This does not establish reliable translation of the original natural-language requirements.

  2. The source checks can miss unjustified constraints shared by both artifacts. For example, for “minimize x subject to x ≥ 0,” both artifacts could add x ≥ 1. They would agree and satisfy the source constraint under Eq. (7), yet return the wrong optimum. How reliably source review catches such errors remains untested.

  3. The repair gain over Opt-Verifier is inconclusive. On 1,905 refuted outputs, certified voting achieves 50.9% versus 49.4% correct-and-matched outputs. The paired gain is 1.42 percentage points, with a 95% confidence interval of [-0.26, 3.05] (Appendix Table 27), which does not establish superiority.

  4. The experiments do not clearly isolate SMT’s contribution to repair accuracy. Certified voting uses five candidates, and source review adds model calls. A comparison with the same generation, review, and selection budgets is needed to determine whether the gains come from SMT or the additional computation.

Initial Critical Weaknesses before LLM polishing (For the authors, reviewers, chairs, and the AI-tools the chairs used to detect AI generated content): First, we cannot know whether the source adapter is really needed as I found that the authors did not measure omission/mistranslation rates in the Appendix D.7. Moreover, it seems that source contracts maybe cannot come from the truth math formulations (in Appendix A.2). As a result, the translation from NL problem description may not be reliable. Second, if some unexiistng constraints are contained in both the math formluations and solver code, say minimize x subject to x >= 0 but the math formluation and solver code write that minimize x subject to x > 1, this cannot be detected because the original constraint is ok and the optimial value is wrong. This is not tested. Next, some gains are very small, which is 50.9% vs 49.4%, only a small gap of 1.42. Besides, this is with the CI, (see Table 27 in the appendix, between -0.26 and 3.05). Therefore, the superiority over Opt-Verifier may not be conclusive. Finally, some comparisions may not be fair, for example, the certified voting has 5 candidates and source review has more calls of models. They should control budget on generation, review and selection. In other words, are the gains of repair accuracy of SMT from more computations?

Questions for authors

Please check the weak points.

1 comment

Thank you for highlighting the importance of checking semantic agreement beyond numerical answers and the practical value of reconstruction. We have clarified the problem setting and method sections to make the representations, verification relations, and adoption procedure explicit. Below we connect these revisions to your concerns and the corresponding experimental evidence.

W1. Source formalization: reliability and role in verification.

Thank you for raising this concern. We evaluate source coverage and transcription accuracy separately. Across 3,034 problems, 2,207 (72.7%) have formalization proposals, with 1,701 (56.1%) marked complete by source review. These source checks refute both artifacts in 1,266 of the 7,867 outputs where the models agree (16.1%).

Source formalization can be improved through stronger LLMs. An independent model audit covers 100 randomly sampled problems. The formalization receives only the original problem text, without reference formulations or reference answers. On the 91 inputs with sufficient and consistent information, complete and correct formulations under the recorded interpretations reach 30.8% with Qwen2.5-32B and 45.1% with Qwen2.5-72B under the same adapter protocol, and 98.9% with GPT directly formalizing the original text. Source formalization is auxiliary to our core contribution: checking semantic model agreement, diagnosing discrepancies, and verifying repairs before adoption. Reconstruction operates on the mathematical and instantiated solver models, while source checks additionally detect shared violations of the formal requirements.

W2. Can both artifacts share an unjustified constraint?

Thank you for raising this concern. We would like to clarify that requirement satisfaction and feasible-region comparison answer different questions. Eq. (8) checks whether each model satisfies a formal requirement. Applying the two directions of Eq. (5) against the source model also identifies decisions allowed by that model but excluded by both artifacts, which detects the shared restrictions you describe. Our evaluation identifies this situation in 999 of the 7,867 outputs where the mathematical model and solver model agree (12.70%): both exclude decisions allowed by the source contract.

The results also show how these findings lead to corrections. In the MAMO Easy test instance 434, both artifacts impose exactly 500 trucks where the source specifies at most 500. Restoring the upper bound exposes the extra restriction and recovers the correct selected answer, changing the optimum from 7,000 to 5,000. Across all 18,204 outputs, 236 additional outputs have both artifacts satisfy the admitted formal requirements after reconstruction.

Detecting these shared restrictions requires the source formulation to include the relevant feasible decisions. The requirement adapter supplies this information, and SMT checks the resulting relations against both artifacts.

W3. Statistical evidence in the Opt-Verifier comparison.

We would like to clarify the purpose of the comparisons. Table 2 diagnoses artifact availability and constraint discrepancies. Appendix Table 27 tests relative repair performance on the refuted subset. Table 8 evaluates the central claim over all outputs: obtaining correct answers together with semantic model agreement and retaining originally correct answers.

For this purpose, Table 8 shows 3,501 more correct-and-matched outputs than Opt-Verifier, together with 511 more retained correct answers. These differences explain the practical advantage of checking agreement and verifying adoption, beyond comparing numerical answers alone. The paired repair estimate in Appendix Table 27 is a separate comparison, not evidence of a statistically established advantage on that subset.

W4. Could additional computation explain the gains?

We would like to clarify that SMT contributes both a diagnosis for revision and verification before adoption. Counterexamples identify the violated relation and implicated constraints, with available links to their source. This evidence guides a correction that the same checks can then verify.

With all 9,525 candidates and their order fixed, selecting the first candidate gives 877 correct answers, selection by model–code agreement gives 932, and adding source checks gives 936. Thus checking adds 59 correct selections without additional generation. Deterministic reconstruction separately adds 7,857 model–code matches without LLM repair generation.

The repair comparison also tests whether the benefit requires more LLM computation. On the same 1,905 refuted outputs with the same 14B reviser, Table 6 reports 1,607 restored model–code matches for ORSMT and 68 for Opt-Verifier. ORSMT uses 3,596 tokens per output versus 7,949 for Opt-Verifier. Thus the higher repair rate accompanies lower generation cost in this comparison.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.