acceptodds
Under review as a conference paper at ICLR 2027

VERITASMATH: Scoped Evidence and Bounded Authority for LLM Mathematical Reasoning

Abstract

Generated verifiers for large language model (LLM) mathematical reasoning can execute successfully while checking the wrong claim, omitting constraints, or covering only a finite subset of a universal statement. Reliability therefore depends not only on whether verification occurs, but also on what each piece of evidence is authorized to conclude. We introduce , an evidence-constrained reasoning architecture that operationalizes bounded epistemic authority through claim-level evidence contracts and an authority-constrained state machine. Each verification record captures the checked claim, polarity, domain, method, coverage, and provenance; executable conflict re-computation, selection-only arbitration, repair re-entry, and answer-locked rendering constrain how evidence can affect candidate selection and output, while an obligation graph tracks whether all requested deliverables are supported. On a pre-specified, latency-capped six-benchmark subset spanning final-answer and proof tasks, achieves a problem-weighted mixed aggregate, compared with for code-based self-verification, a -point improvement. In fixed-trajectory replays, the same contracts are associated with lower unsupported promotion and payload mutation, and with higher deliverable completeness.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Evidence-Gated Orchestration, LLM Mathematical Reasoning, Executable Re-computation / Verification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_c47fOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VERITASMATH, a framework that tries to make LLM math reasoning more trustworthy. The basic idea is that each verification result should only count for what it actually checked, so the framework records what was checked, how far it reached, and where it came from. It also keeps generating, verifying, selecting, repairing, and writing the final answer as separate steps, and it tracks whether every part of a problem has been covered.

The authors test it on math competition problems with the same language model and the same time limit. On a set of 99 problems from six benchmarks, it scores 96.01%, compared with 92.42% for code-based self-verification. Their ablations and replays also point to better evidence reliability, candidate selection, and answer completeness. The catch is that the main comparisons don’t use matched compute, so it’s hard to know how much of the gain really comes from the framework itself.

Critical strengths

1. Separating verification evidence from decision authority. I like how the paper handles verification scope. It distinguishes a successful code run (E1), a finite test tied to a specific claim (E2), and a valid counterexample (E3). This targets a real failure mode: passing a computational check doesn’t mean a mathematical claim is true. Not letting finite tests certify universal claims also makes sense.

2. The fixed-trajectory analysis says more than the headline accuracy. Table 5 is more useful than the overall accuracy gain. With the same candidates and checker outputs, the full evidence-authority contract cuts the false-support rate from 9.6% to 2.3% compared with flat evidence labels. The obligation audit also raises deliverable completeness from 86.4% to 94.2% compared with a flat audit. This shows which parts of the framework actually help reliability, instead of leaning only on end-to-end accuracy.

Critical weaknesses

1. The gains aren’t cleanly separated from extra compute. Both methods share the same 1,200-second limit, but VERITASMATH uses about 58,000 tokens per run versus 40,000 for code-based self-verification. With an improvement of only 3.59 points, it’s hard to tell how much comes from the framework and how much from simply spending more inference compute. The fixed-trajectory experiments help, but they don’t settle this for end-to-end performance.

2. The evaluation is small and near saturation. The controlled comparison has only 99 problems, and several benchmark scores are already at or close to 100%. The authors also admit that the model’s training cutoff is unknown and that there’s no contamination analysis. So it’s hard to say whether the gains would hold up on truly unseen, harder problems.

3. The novelty isn’t very convincing. Evidence tracking, constrained arbitration, repair re-entry, and answer locking all look a lot like ideas already used in verification pipelines and multi-agent systems. Combining them is reasonable, but the paper doesn’t make clear which part is a real step beyond existing work. The ablations show the components help within this system, but that doesn’t prove the framework adds a fundamentally new capability.

Questions for authors

see Weaknesses

3 comments
AuthorsOpenReview ↗

Response to W1-W3: Matched Compute, Held-out Evaluation, and the Action Contract Contribution

Thank you for recognizing the scope-carrying evidence idea and the diagnostic value of controlled comparisons. We address compute, evaluation scope and novelty below.

Full denotes VERITASMATH, Code-SV code-based self-verification, and FA final-answer accuracy.

W1: Matched compute. The corrected Controlled99 comparison uses the May 2026 Intern-S2-Preview-397B snapshot with a common 58,000-observable-token ceiling and 1,200-second ceiling for Full and Code-SV. Actual mean use is 56,780 vs. 55,940 tokens. The original 96.01%/92.42% mixed headline used a different best-effort proof interface without answer locking; it is not evidence for the corrected strict policy. Native proof credit, coverage and CCR favor Code-SV. We will replace the mixed-score interpretation with endpoint-specific results.

Private160 also matches token/time ceilings, tools and external-resource permissions within each backbone. All candidate generation, checking, judging, repair, retries and rechecks count toward the total; generation and repair stop at exhaustion. No unbudgeted extra checking improves native outputs. Intern-S2 and Qwen3 need not have identical absolute budgets, and we do not compare their absolute compute efficiency. Hidden internal reasoning is unobservable.

W2: Held-out evaluation and response-count effects. Controlled99 contains 99 problems with four nested attempts each, not 396 independent problems. Private160 adds 100 independently authored FA and 60 proof problems, frozen before model runs. On Intern-S2, Full vs. Code-SV has FA 370/400 (92.50%) vs. 350/400 (87.50%) and correct-and-complete rate (CCR) 397/640 (62.03%) vs. 369/640 (57.66%). Native coverage is 70.94% vs. 84.38%, proof credit 25.00% vs. 55.50%, and emitted-valid proofs 54/240 vs. 84/240.

The completed matched analysis uses only originally emitted candidates. Both methods select 360 FA responses (90% coverage) and 60 proof responses (25%). FA risk is 6/360 (1.67%) vs. 22/360 (6.11%); proof risk is 7/60 (11.67%) vs. 16/60 (26.67%). FA CCR is 347/400 vs. 328/400; proof CCR is 48/240 vs. 39/240. These point estimates compare equal response counts within each task, addressing the explanation that lower risk comes only from answering fewer questions or fewer proof tasks.

The shared ranking formula uses evidence support, obligation coverage and provenance, not correctness labels or method identity. Scores and hash tie-breaks were frozen before label disclosure. Coverage targets were selected after observing native coverage, so these are descriptive post-hoc analyses, not preregistered tests or significance claims. Both methods are scored blind on randomized, de-identified final outputs using mathematical equivalence for FA, complete validity for binary proofs, and a common partial-credit rubric. Non-emission scores zero under planned denominators.

On Qwen3, the auxiliary analysis selects from stored candidates including withheld responses. At 90% FA/50% proof coverage, Full has FA errors 6/360 vs. 48/360, but proof errors 60/120 vs. 45/120. Pooled risk is 66/480 vs. 93/480. This supports the FA direction on a second backbone under a different output policy; proof risk worsens. We make no uniform cross-model proof claim. Private authorship reduces exposure risk, but unknown training cutoffs prevent a zero-contamination claim.

W3: Contribution beyond familiar components. The contribution is the action contract binding each evidence record to the actions it can authorize. Decomposition, checking and repair are familiar; the contract constrains how their evidence changes decisions. On the same 1,200 events, erroneous authorization is 11/480 (2.29%) for Full vs. 47/480 (9.79%) for flat evidence, with TP 710 vs. 706. With the same structured records and initial candidate/evidence state online, enforcement reduces erroneous-authorization attempts from 28/396 to 12/396; later generation and repair may diverge. Full has lower coverage and CCR than the unenforced judge, so this isolates an authorization gain with a service cost.

At native operating points, AMTFV's official implementation and recommended configuration give FA/CCR/risk/false-support rates of 93.00%/72.47%/6.68%/6.79%, versus Full's 95.33%/76.77%/3.66%/2.59%, under matched model/tasks/budget ceilings. AMTFV retains higher proof scores and coverage. Together, the online intervention and held-out selection results support a measurable evidence-to-action contribution at specified operating points, not overall proof-solving dominance.

Thanks for the detailed response. W1 is largely addressed by the matched-compute results. The additional private evaluation also helps address W2, although the evaluation is still relatively small and the proof results are not fully consistent across models. The response to W3 also makes the contribution of the action-contract mechanism clearer. Overall, I think W1 is largely resolved, while W2 and W3 are substantially, though not fully, addressed. I will maintain my score.

Thank you for your thoughtful follow-up and for recognizing that our responses substantially address W1–W3. We appreciate your careful assessment and agree that the evaluation remains limited in size and that proof results are not fully consistent across models. We will keep these limitations clear in how we present our claims. Thank you again for your time and consideration.

Reviewer_zKMROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the reliability of verification in LLM mathematical reasoning, focusing on cases where generated verifiers execute successfully but fail to substantiate the intended claims. It proposes VERITASMATH, an evidence-constrained reasoning architecture that uses claim-level evidence contracts to record verification claims, scope, coverage, and provenance. An authority-constrained state machine governs conflict resolution, candidate selection, and repair, while a graph tracks evidence coverage for required deliverables and answer-locked rendering preserves the selected response. Experiments cover eight competition mathematics benchmarks with both final-answer and proof-based tasks.

Critical strengths

S1: The authors address an important and interesting problem: generated verifiers may execute successfully while checking the wrong claim, omitting constraints, or providing insufficient coverage.

S2: Using an obligation graph to track whether all requested deliverables are adequately supported could be a useful design choice.

S3: The paper provides broad experimental coverage across multiple competition mathematics benchmarks, spanning both final-answer and proof-based tasks.

Critical weaknesses

W1. The presentation requires improvement in clarity and organization.

W2. The methodological novelty requires clearer justification. Decomposing a problem and solution into components and checking them individually resembles a conventional divide-and-conquer strategy. More discussion of related work and direct experimental comparisons with existing approaches are needed to identify the contributions.

W3. Definitions are insufficiently clear. Illustrative examples can be helpful for better readability.

• W3.1. Several central concepts are insufficiently specified. For example, the meanings of “immutable payload” and “provenance” in candidate a_i, and the description of (y) as an “optional protected payload,” remain unclear. Clear definitions and examples are needed.

• W3.2. Although the evidence record < c,Q,D,M,K,P,R,S > appears comprehensive, its construction for realistic mathematical claims is insufficiently illustrated. It remains unclear how each field is identified, whether all fields can be reliably populated, and how missing information or shared evidence across overlapping claims is handled. A mathematical example would clarify its practical applicability.

• W3.3. The relationships between SUPPORTED, PARTIALLY SUPPORTED, UNRESOLVED, ABSTAINED, and E0-4 are unclear.

• W3.4. There are repeated notations: S denotes "evidence class" in Section 3.2 and denotes "select a final candidate" in Table 1.

• W3.5 Algorithm 1 lacks sufficient explanation in the main text. Following common academic practice, its operations and control flow should be explicitly described.

W4: The construction and use of the obligation graph could be clarified further. The graph links each required deliverable to supporting claims and evidence, but its construction, dependency validation, and use within the workflow remain unclear. In addition, if the graph is generated by an LLM, how are its accuracy and completeness validated? Further details and evaluation are needed.

W5: Human annotation in the mechanism study is helpful. However, the quality of informal-proof verification remains a major concern. Without the expert grading of the proofs (Tables 2-3), it is difficult to assess whether the method reliably evaluates proof correctness and whether the reported proof-credit gains on benchmarks such as Putnam 2025 and IMO 2025 reflect improved mathematical validity.

Questions for authors

N/A

2 comments
AuthorsOpenReview ↗

Response to W1-W3: Evidence-to-Action Contracts, Payload Provenance, and Record Classifications

Thank you for highlighting the importance of scope tracking and deliverable coverage. We address W1-W3 here and Algorithm 1, graph validation, and proof performance in Part 2.

Full denotes VERITASMATH, Code-SV code-based self-verification, and FA final-answer accuracy.

W1-W2: Presentation and contribution. Decomposition identifies the obligations to solve. Our evidence-to-action contract specifies which actions a check can authorize. A finite positive test cannot authorize an infinite-domain universal claim; a counterexample applies only to its linked claim and domain; arbitration returns an existing candidate; repair creates a new candidate whose evidence must be rebuilt. This action constraint is the contribution beyond decomposition and checking.

We will sharpen the related-work distinction: LLM-Modulo provides critic-based control, and Claim-Level Reliability Assessment targets decision-critical claims and falsification. Our evaluated addition carries admissible evidence scope into rejection, selection, repair and rendering.

Two completed comparisons test it directly. On the same 1,200 annotated claim-evidence-action events, Full has TP/FN/FP/TN = 710/10/11/469 versus 706/14/47/433 for flat evidence. Erroneous authorization is 11/480 = 2.29% versus 47/480 = 9.79%, without an increase in false-rejection counts (10/720 versus 14/720). In an online comparison with the same structured records, initial candidate/evidence state, tools, action space and budget, enforcement reduces attempts with erroneous authorization from 28/396 to 12/396. Later generation and repair can diverge, so this tests the running controller rather than fixed replay alone. Full has lower coverage and CCR than this judge control; the gain is authorization reliability.

We also compared with AMTFV using its official implementation and recommended configuration, matching model, tasks and budget ceilings without removing its verification or repair features. At native operating points, Full/AMTFV have FA 95.33%/93.00%, correct-and-complete rate (CCR) 76.77%/72.47%, risk 3.66%/6.68%, and false-support rate 2.59%/6.79%. AMTFV has higher coverage and proof scores. These results support a reliability/delivery contribution, not overall proof-solving dominance.

W3.1: Payload and provenance. A payload contains the candidate's answer, derivation and qualifications. "Immutable" means arbitration and rendering cannot edit it in place. Selecting candidate 7 copies its stored payload; a repaired derivation receives a new ID and fresh checks. Provenance identifies the source candidate, checker, executable artifact, input and seed when applicable. "Optional protected payload" means fail-closed output can return an evidence/status summary without a mathematical answer when support is insufficient.

W3.2: Constructing and sharing records. The record is e=<c,Q,D,M,K,P,R,S>: claim, logical form, domain, method, coverage, provenance, result and class. For "n^2+n+41 is prime for every nonnegative integer n," a trial-division check at n=0,...,39 records that exact claim, universal quantifier, nonnegative-integer domain, trial division, coverage {0,...,39}, checker/input provenance, and an all-prime result. Its class is E2. Testing n=40 yields 41^2, an admissible E3 counterexample. This is an illustration, not a benchmark success.

Missing or malformed fields produce E0/inconclusive evidence; an execution-only PASS is E1. A shared program supports several claims only through separately validated links. The audit finds valid claim/provenance fields in 1,819/1,847 records, logical form in 1,826/1,847, domain in 1,804/1,847, method/coverage in 1,791/1,847, and result/class in 1,831/1,847. Of 386 shared links, 362 are correct; 22 of 24 incorrect links were downgraded and two promoted. Nine erroneous promotions occur across field/link categories.

W3.3-W3.4: Classes, states and notation. E0-E4 classify individual checks: E2 is a claim-linked finite/heuristic test, E3 an admissible counterexample, and E4 complete finite-domain checking in this implementation. There is no separate E4-D class. SUPPORTED means all necessary obligations satisfy the configured policy with no blocking counterexample; PARTIALLY_SUPPORTED exposes named gaps; UNRESOLVED records a conflict; ABSTAINED emits no payload. These are operational states, not guarantees of semantic proof soundness. We will use Class for evidence class and Sel for selection permission to eliminate the notation overload.

AuthorsOpenReview ↗

Response to W3.5, W4, and W5: Algorithmic Workflow, Graph Validation, and Proof Guarantees

W3.5: Algorithm 1 and organization. We will explain five stages: compile deliverables and dependencies; generate candidates; check claims and record evidence; resolve conflicts and audit/repair under action-authority constraints; emit the selected payload.

Algorithm 1 initializes the obligation graph, candidate history and evidence ledger. Each bounded round generates a local pool, checks its claims, resolves conflicts under the authority policy, and audits the selected candidate's obligations. An unresolved claim or uncovered deliverable becomes a repair hint. Repair creates a new candidate and restarts checking without inherited support. The final policy selects a state, and the locked renderer copies the chosen payload only when emission is allowed. Budget exhaustion stops generation/repair but cannot upgrade unsupported evidence.

W4: Graph construction and validation. The graph represents requested deliverables, supporting claims/evidence, and dependencies. An extremal task includes a global bound and an admissible construction attaining it. "Determine all" creates validity and exhaustiveness obligations. Alternative valid proof structures are disjunctive paths, so one complete path should suffice. Rule-based and model-derived missing-obligation sets are combined to direct repair.

Against human-annotated obligations and dependencies, node precision is 612/651 = 94.01% and recall 612/671 = 91.21%; edge precision is 472/498 = 94.78% and recall 472/531 = 88.89%. The audit finds 9/146 excluded valid alternative paths, 6/412 erroneous SUPPORTED responses caused by omitted obligations, and 11/118 unnecessary repair rounds. A separate coverage-label audit over 1,006 eligible nodes gives TP/FP/FN/TN = 587/17/42/360, precision 97.19%, and recall 93.32%. These are different audit objects and denominators.

Graph extraction is fallible. The online flat-obligation variant makes 68 repairs versus Full's 47; its CCR difference is +1.52 points with a 95% interval [-1.8, +4.9]. Fewer repairs alone do not establish better end-to-end completeness.

W5: Guarantees and proof scores. The controller restricts actions conditional on extracted records and preserves the selected payload. It does not prove an arbitrary checker or graph mathematically sound. No formal prover is connected. E4 requires complete enumeration of the declared finite domain; finite sampling remains E2. None of 61 necessary infinite-domain claims in 58 proof attempts was marked SUPPORTED and emitted. We have no successful emitted infinite-domain certification example from these runs.

The corrected strict run emits 30 fully valid and eight partial proofs out of 96 planned proof attempts, giving 35.42% normalized proof credit and 30/96 binary validity. At the stored stage, 48/96 proofs are valid; 18 were withheld. The earlier 91.89% best-effort proof-credit figure is not strict delivered proof performance. We will separate the stages and replace the broad proof-gain interpretation.

Private160 supports a narrower positive conclusion. It contains 100 FA and 60 proof problems, each attempted four times. On Intern-S2, Full/Code-SV have FA 370/400 vs. 350/400 and CCR 397/640 vs. 369/640. Native coverage is 454/640 vs. 540/640, proof credit 25.00% vs. 55.50%, and emitted-valid proofs 54/240 vs. 84/240.

Selecting only from original emitted candidates, both methods retain 360 FA and 60 proof outputs. FA errors are 6/360 vs. 22/360; proof errors are 7/60 vs. 16/60. Matched-coverage risk point differences are -4.44 and -15.00 points. FA CCR is 347/400 vs. 328/400; proof CCR is 48/240 vs. 39/240. We report these as descriptive post-hoc estimates, without significance claims. Coverage targets were chosen after observing native coverage. Both methods use the same label-independent ranking formula and hash tie-break, frozen before correctness labels were revealed; no preregistered-confirmation claim is made.

Within each backbone, the two methods have equal token/time ceilings and tool/resource permissions; all generation, checking, judging, repair, retries and rechecks count toward the total. Absolute budgets need not match across backbones. Scoring is method-blind on randomized, de-identified final outputs. FA uses mathematical equivalence, only complete valid proofs are binary-valid, and partial credit follows a common rubric. Non-emission scores zero under planned denominators; risk uses selected/emitted outputs. The supported conclusion is improved evidence authorization and reliable selection at specified coverage, with explicit proof/coverage costs. We will clarify these definitions and incorporate the completed evidence.

Reviewer_E6CpOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces an inference-time framework for mathematical reasoning that records the scope of verification evidence and limits the authority of downstream components. It combines an obligation graph, evidence classification, candidate selection, rechecked repairs, and answer-preserving rendering. Experiments on competition mathematics and controlled policy replays show improved scores and fewer unsupported selections.

Critical strengths
  1. The paper addresses practical failure modes where a checker can execute successfully without checking the intended claim, and later editing can invalidate previously checked reasoning.
  2. The proposed framework explicitly tracks evidence scope and distinguishes finite positive tests from counterexamples.
  3. The fixed-artifact policy comparisons and human annotations provide more diagnostic information than aggregate accuracy alone.
Critical weaknesses
  1. I found the methodological novelty of this paper quite limited. Generator-verifier workflows, claim-level checking, and verification-guided correction are established directions. The paper itself discusses close work such as LLM-Modulo, Claim-Level Reliability Assessment, and AMTFV, but does not convincingly isolate an advance beyond combining evidence labels, obligation tracking, restricted selection, and output preservation. An engineering contribution can be valuable, but here the need for the full framework and its advantage over simpler combinations are not sufficiently demonstrated.
  2. While the paper proposes explicit evidence records and routing rules, the problem of determining whether a checker actually verifies the intended mathematical claim remains unresolved. It is unclear how general proof obligations over infinite domains receive sufficient support, or how this restriction relates to the reported proof-task performance. The paper needs a clearer separation between preserving a verification decision and establishing that the decision is mathematically justified.
  3. While the experiments show improvements over several baselines, they do not clearly isolate the benefit of the proposed design. The full system performs more reasoning and verification, so it is unclear how much of the gain comes from the proposed evidence and authority rules rather than additional computation. There is also no direct comparison with the closest claim-level verification methods, or a simpler baseline combining rechecking after repair, an obligation checklist, and immutable output.
  4. The controlled studies explain individual decisions, but do not establish their effect on the full workflow. Replaying fixed candidates and checking results misses how different decisions would change later reasoning and repair. Preventing answer edits also shows preservation, not correctness.
  5. The evaluation provides limited evidence of general reliability. The main comparison covers only 99 problems, including 24 proof problems, and combines answer accuracy with partial proof credit. The mechanism study overrepresents cases requiring intervention, so its error rates may not reflect ordinary use. Possible training exposure to the benchmarks also remains unchecked.
Questions for authors
  1. Can the authors give a proof example involving an infinite domain and explain what evidence allows the system to mark its universal claims as supported?
  2. Among the observed failures, which required the proposed scope-aware routing to resolve, rather than simply rerunning verification after repair?
2 comments
AuthorsOpenReview ↗

Response to W1/W3/W4 and Part 2: Value of Action Contracts and Empirical Results

Thank you for identifying the central question: whether the authority rules add value beyond combining familiar verification components. Part 1 addresses W1/W3/W4; Part 2 covers validity, held-out evaluation, and both questions.

Full denotes VERITASMATH, Code-SV code-based self-verification, and FA final-answer accuracy.

W1/W3: Contribution and simpler controls. Our contribution is the action contract between evidence and the controller. It records the checked claim, domain, coverage, method and provenance, and limits what action that evidence may authorize. Finite positive tests cannot authorize an infinite-domain universal claim; arbitration selects an existing candidate; repair creates a new candidate whose evidence must be rebuilt. The obligation graph separately tracks required deliverables and dependencies.

LLM-Modulo provides a critic-based control paradigm, while Claim-Level Reliability Assessment targets decision-critical claims and falsification. Our evaluated addition is an explicit scope-to-action contract across rejection, selection, repair and rendering. We will sharpen this distinction in the related work.

On 1,200 annotated claim-evidence-action events, Full has TP/FN/FP/TN = 710/10/11/469, versus 706/14/47/433 for flat evidence. Erroneous authorization is 11/480 (2.29%) versus 47/480 (9.79%), a paired difference of -7.50 percentage points (95% CI [-10.2, -4.9]). False rejection is 1.39% versus 1.94%; its difference interval includes zero. Lower erroneous authorization accompanies slightly more true positives, rather than blanket rejection. Domain-checking and risk-aware controls have 26 and 33 false authorizations on the same events. These are self-implemented controls.

We also tested enforcement online with the same structured record format, initial candidate/evidence state, tools, action space, and budget. The judge could select only an existing candidate or abstain; later generation and repair could diverge. Attempts with erroneous authorization fall from 28/396 (7.07%) with an unenforced judge to 12/396 (3.03%) with enforcement (-4.04 points; 95% CI [-6.6, -1.6]). Final-output risk is 12/328 (3.66%) vs. 27/372 (7.26%). Full has lower coverage (82.83% vs. 93.94%) and correct-and-complete rate (CCR; 76.77% vs. 82.32%). This identifies an authorization-reliability gain with a service-level cost.

The corrected Controlled99 comparison uses the same Intern-S2 snapshot, 58,000-observable-token ceiling, and 1,200-second ceiling for Full and Code-SV. Mean use is 56,780 vs. 55,940 tokens. Native risk/coverage/CCR are 3.66%/82.83%/76.77% for Full, 10.91%/97.22%/79.29% for Code-SV, 7.67%/95.45%/83.08% for domain checking, and 4.52%/89.39%/84.85% for Risk-aware. We claim risk control at specified operating points, not uniform score gains.

The reviewer's "AMTYP" refers to AMTFV in our references. We used its official implementation and recommended configuration, preserving its verification, repair and output mechanisms while matching the model, tasks and budget ceilings. At native operating points:

EndpointFullAMTFV
FA95.33%93.00%
CCR76.77%72.47%
Selective risk3.66%6.68%
Link-level false-support rate2.59%6.79%
Coverage82.83%94.44%
Proof credit35.42%79.50%
Emitted-valid proofs / planned proof attempts31.25%72.92%

Full improves FA and complete delivery and has fewer insufficient promotions, while AMTFV has higher coverage and proof scores. This comparison does not match coverage.

W4: Online effects and preservation versus correctness. The online flat-evidence ablation allows later reasoning, repair and stopping to change. False support is 13/502 (2.59%) for Full vs. 45/506 (8.89%) for flat evidence, but flat evidence has higher coverage (95.96%) and CCR (82.83%). The flat-obligation variant triggers 68 repairs vs. 47, yet its CCR difference is uncertain (+1.52 points; 95% CI [-1.8, +4.9]); repair counts alone do not establish an online completeness gain.

With both variants emitting 328 responses, answer locking gives 0/328 final-payload mutations versus 45/328 unlocked. CCR is 76.77% vs. 68.94% (+7.83 points; 95% CI [+4.6, +11.2]); risk is 3.66% vs. 7.93% (-4.27; [-6.9, -1.8]). Correctness is graded independently, so zero mutation alone is not treated as mathematical correctness. These online results complement the conditional fixed-artifact audit.

AuthorsOpenReview ↗

Response to W2/Q1, W5, and Q2: Infinite-Domain Soundness, Evaluation Details, and Scope-Aware Resolution

W2/Q1: Correctness validation and infinite domains. Routing asks whether evidence is admissible for a stated claim and may authorize an action; it does not make an arbitrary generated checker or obligation graph sound. The strict trace audit has 61 necessary infinite-domain claims across 58 proof attempts; none was marked SUPPORTED and emitted. No formal prover is connected. We cannot supply a successful benchmark example of an emitted general infinite-domain proof.

For illustration, testing that n^2+n+41 is prime for n=0,...,39 supplies E2, not universal support. At n=40 the value is 1681=41^2, an admissible E3 counterexample. E4 requires complete checking of the declared finite domain. Support for a finite subclaim cannot silently transfer to an infinite-domain claim. This illustrates routing, not a benchmark success. Audits found nine erroneous field/link promotions and six erroneous SUPPORTED states caused by omitted obligations among 412 responses. We will make this semantic-soundness boundary explicit.

W5: Evaluation size, scoring, and exposure. Controlled99 has four attempts per problem, not 396 independent problems. The original 96.01%/92.42% mixed headline used a different best-effort proof interface without answer locking. It is not evidence for the corrected strict policy; we will replace that interpretation. Native proof credit, coverage and CCR favor Code-SV and Risk-aware.

Private160 adds 160 independently authored problems (100 FA, 60 proof; four attempts each), frozen before model runs. Within each backbone, Full and Code-SV have equal observable-token/time ceilings and tool/resource permissions. All generation, checking, judging, repair, retries and rechecks consume the shared budget; generation/repair stop at exhaustion. Budgets need not be equal across Intern-S2 and Qwen3, so we make no cross-model efficiency comparison.

Scoring is method-blind on randomized, de-identified final outputs. FA uses mathematical equivalence; only complete valid proofs are binary-valid; partial credit follows a shared rubric. CCR requires correctness and all necessary deliverables. Non-emission, timeout or exhaustion without a response scores zero under planned denominators; risk uses selected/emitted outputs. Headline scores never substitute intermediate checks or best stored candidates for final delivery.

On Intern-S2, Full vs. Code-SV has FA 370/400 (92.50%) vs. 350/400 (87.50%) and CCR 397/640 (62.03%) vs. 369/640 (57.66%). Native coverage is lower (70.94% vs. 84.38%), as are proof credit (25.00% vs. 55.50%) and emitted-valid proofs (54/240 vs. 84/240).

The matched analysis uses only original emitted candidates. Both methods use the same label-independent selection-score formula, direction and artifact-hash tie-break, frozen before correctness labels were revealed. Coverage targets were chosen after observing native coverage: FA 90%, proof 25%. At these post-hoc targets:

TaskFull errors / selectedCode-SV errors / selected
FA6/360 = 1.67%22/360 = 6.11%
Proof7/60 = 11.67%16/60 = 26.67%

Risk point differences are -4.44 and -15.00 points. FA CCR is 347/400 vs. 328/400; proof CCR is 48/240 vs. 39/240. These descriptive output-selection results address response-count and task-mix explanations at the chosen coverage levels. They are not preregistered confirmation, online interventions, or significance claims.

The auxiliary Qwen3 analysis includes withheld stored candidates and therefore uses a different output policy. At FA 90%/proof 50% coverage, FA risk is 6/360 vs. 48/360, but proof risk is worse for Full: 60/120 vs. 45/120. Pooled risk is 66/480 vs. 93/480, driven by FA's 3:1 output weight. We claim no uniform cross-model proof gain. Training cutoffs are unknown; private authorship reduces exposure risk without establishing zero contamination. The enriched 412-response cohort does not estimate ordinary-use prevalence.

Q2: Scope-aware resolution versus rerunning. Rerunning a checker for n<=30 still yields only E2 for a universal claim (NAT-COV-022); scope-aware repair leaves it unresolved and emits no proof. In NAT-DOM-014, routing rejects a witness outside the nonzero-integer domain. Across 72 trigger-selected events from 58 problems, scope-aware continuation vs. rerun-only resolves 55/72 vs. 26/72, emits correct outputs in 52/72 vs. 37/72, and has CCR 50/72 vs. 26/72; coverage is 61/72 vs. 68/72. Both branches retain repair, obligations, answer locking and equal budgets. "Resolved" requires defect removal and a mathematically correct stored candidate, not safe abstention alone. These are conditional event results.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response to the Area Chair — Part 1/3: Main findings and fair evaluation

We thank the Area Chair and reviewers for identifying the practical value of scoped verification and for asking whether its benefits survive fairer controls. Across the responses to E6Cp, zKMR, and c47f, we clarified the contribution and added evidence at three levels: shared-event authorization, online controller interventions, and independently authored problems. The resulting claim is concrete: VERITASMATH improves evidence authorization and answer reliability at specified operating points, with measurable coverage and proof-quality tradeoffs.

Matched budgets address the compute concern. Full denotes VERITASMATH; Code-SV denotes code-based self-verification. On corrected Controlled99, both use the May 2026 Intern-S2-Preview-397B snapshot, a 58,000-observable-token ceiling, and a 1,200-second ceiling. Mean token use is 56,780 versus 55,940. Within each backbone, both methods have equal tools and resource permissions; generation, checking, judging, repair, retries, and rechecks all count toward the total. Exhaustion stops generation and repair, with no unbudgeted offline improvement. Private160 matches token/time ceilings between methods within each backbone; its caps are not assumed equal to Controlled99's or equal across backbones. Hidden internal reasoning is unobservable.

Controlled99 contains 99 problems with four attempts each. Its native final-output endpoints are:

PolicySelective riskCoverageCCR
Full3.66%82.83%76.77%
Code-SV10.91%97.22%79.29%
Domain checking7.67%95.45%83.08%
Risk-aware4.52%89.39%84.85%

CCR is correct-and-complete rate: the final response must be mathematically correct and contain every required deliverable. These results establish lower native risk for Full alongside lower coverage and CCR than the simpler controls. They do not establish uniform gains across endpoints.

Private160 adds evidence beyond saturated public benchmarks. This set contains 100 independently authored final-answer (FA) and 60 proof problems, frozen before model runs, with four attempts per problem. On Intern-S2:

Native endpointFullCode-SV
FA correct / planned370/400 (92.50%)350/400 (87.50%)
CCR / planned397/640 (62.03%)369/640 (57.66%)
Emitted / planned454/640 (70.94%)540/640 (84.38%)
Normalized proof credit25.00%55.50%
Emitted-valid proofs / planned54/24084/240

Thus, Full improves FA by 5.00 percentage points and CCR by 4.38 points under planned denominators, while coverage and proof scores favor Code-SV. Outputs are scored method-blind after randomization and de-identification. Non-emission scores zero under planned denominators; risk is conditional on emitted or selected responses.

Equal response counts test the abstention explanation. Selecting only from the original emitted pools, both methods retain 360 FA outputs and 60 proof outputs. Full/Code-SV risks are 6/360 (1.67%) versus 22/360 (6.11%) for FA and 7/60 (11.67%) versus 16/60 (26.67%) for proofs. Risk point differences are -4.44 and -15.00 points. FA CCR is 347/400 versus 328/400; proof CCR is 48/240 versus 39/240. The comparison holds selected counts equal within each task, so the observed risk gaps cannot be attributed simply to unequal response counts or task proportions at these service levels.

Both methods use the same ranking formula and artifact-hash tie-break, based on evidence support, obligation coverage, and provenance. Scores were frozen before correctness labels or matched-selection results were revealed. Coverage targets were chosen after observing native coverage. These are descriptive post-hoc selection results, without confidence intervals, significance claims, or preregistered-confirmation claims; they do not replace native online evaluation.

AuthorsOpenReview ↗

General Response to the Area Chair — Part 2/3: Contribution and mechanism evidence

This part summarizes the evidence addressing E6Cp's and c47f's questions about attribution and novelty, and zKMR's question about the contribution beyond decomposition. Full denotes VERITASMATH; Code-SV denotes code-based self-verification.

The contribution is an enforced evidence-to-action contract. A record identifies the checked claim, logical form, domain, checking method, coverage, provenance, result, and evidence class. Its scope constrains rejection, selection, repair, and rendering: finite positive tests cannot authorize infinite-domain universal claims; counterexamples apply to their linked claim and domain; arbitration selects an existing candidate; repairs create new candidates requiring fresh checks; and rendering preserves the selected payload. The obligation graph tracks required deliverables and dependencies. LLM-Modulo supplies a critic-based control paradigm, while Claim-Level Reliability Assessment targets decision-critical claims and falsification. Our evaluated addition makes admissible evidence scope govern downstream actions across the workflow.

Shared inputs isolate authorization quality. On 1,200 annotated claim-evidence-action events, Full has TP/FN/FP/TN = 710/10/11/469, versus 706/14/47/433 for flat evidence. Among the 480 events annotated as inadmissible, erroneous authorization is 11/480 (2.29%) versus 47/480 (9.79%): a paired difference of -7.50 points, 95% CI [-10.2, -4.9]. False rejection is 10/720 versus 14/720; its difference interval includes zero. Domain-checking and risk-aware controls yield 26 and 33 false authorizations on the same events. These controls are self-implemented. The reduction accompanies slightly more true positives, rather than blanket rejection.

Online interventions address the fixed-replay limitation. With the same structured records, initial candidate/evidence state, tools, action space, and budget, enforcing authority reduces attempts with erroneous authorization from 28/396 (7.07%) to 12/396 (3.03%), difference -4.04 points, 95% CI [-6.6, -1.6]. The control judge can select an existing candidate or abstain; later generation and repair may diverge. Final risk is 3.66% versus 7.26%, coverage 82.83% versus 93.94%, and CCR 76.77% versus 82.32%. This supports an authorization gain with a service cost in a running controller.

The online flat-evidence variant has false support 45/506 (8.89%) versus Full's 13/502 (2.59%), but higher coverage and CCR. The flat-obligation variant triggers 68 repairs versus 47, while its CCR contrast is +1.52 points with a 95% CI [-1.8, +4.9]. We do not infer a completeness gain from repair counts alone. Fixed-artifact replays remain complementary estimates of conditional direct effects, rather than substitutes for these online tests.

Answer locking is evaluated by independently graded correctness. Both variants emit 328 responses. Locked versus unlocked rendering yields 0 versus 45 final-payload mutations, CCR 76.77% versus 68.94% (+7.83 points; 95% CI [+4.6, +11.2]), and risk 3.66% versus 7.93% (-4.27 points; 95% CI [-6.9, -1.8]). Byte preservation alone is not counted as mathematical correctness.

Scope-aware continuation outperforms checker reruns on observed defects. Across 72 trigger-selected events from 58 problems, it resolves 55/72 versus 26/72 defects, emits correct outputs in 52/72 versus 37/72, and reaches CCR 50/72 versus 26/72; coverage is 61/72 versus 68/72. Both branches retain repair, obligations, answer locking, and equal budgets. Resolution requires defect removal and a mathematically correct stored candidate, not abstention alone. These conditional events do not estimate ordinary-use prevalence.

A direct comparison uses official AMTFV. The review's “AMTYP” refers to AMTFV. We use its official implementation and recommended configuration, retaining verification, repair, and output mechanisms while matching model, tasks, and budget ceilings. At native operating points:

EndpointFullAMTFV
FA95.33%93.00%
CCR76.77%72.47%
Selective risk3.66%6.68%
Link-level false-support rate2.59%6.79%
Coverage82.83%94.44%
Proof credit35.42%79.50%

This supports answer reliability and complete delivery relative to the official baseline at native operating points; AMTFV retains higher coverage and proof performance.

AuthorsOpenReview ↗

General Response to the Area Chair — Part 3/3: Soundness boundaries, clarity, and discussion status

This part covers E6Cp's and zKMR's soundness and clarity concerns, c47f's cross-model concern, and the revised claim.

Routing guarantees and mathematical soundness are separated. The controller restricts actions conditional on extracted records; it does not make an arbitrary generated checker or obligation graph sound. No formal prover is connected. E4 means complete checking of the declared finite domain, while finite sampling remains E2. None of 61 necessary infinite-domain claims across 58 audited proof attempts was marked SUPPORTED and emitted. We have no successful emitted infinite-domain certification example from these runs.

For illustration, testing primality of n^2+n+41 for n=0,...,39 gives E2, not universal support; n=40 gives the admissible E3 counterexample 1681=41^2. This illustrates routing, not a benchmark success. Rerunning bounded tests cannot establish a universal claim; an out-of-domain witness cannot refute it.

Graph and record construction are assessed against human annotations. Obligation-node precision/recall are 612/651 (94.01%) and 612/671 (91.21%); dependency-edge precision/recall are 472/498 (94.78%) and 472/531 (88.89%). A separate coverage-label audit has precision 97.19% and recall 93.32%. Valid field rates across 1,847 records range from 97.0% to 99.1%; 362/386 shared claim-evidence links are correct. Of 24 incorrect shared links, 22 were downgraded and two promoted. The audit also identifies 9/146 excluded valid alternative paths, 6/412 erroneous SUPPORTED responses caused by omitted obligations, and 11/118 unnecessary repairs. These audits quantify the remaining semantic representation errors.

The revised Method explains record fields, provenance, payloads, evidence classes versus output states, and Algorithm 1's bounded control flow. A repaired payload receives a new candidate identity and fresh checks. Missing or malformed record fields give inconclusive evidence; shared programs require separately validated claim links. Alternative valid proofs are disjunctive paths, so one complete valid path suffices. SUPPORTED is an operational policy state, not a formal soundness certificate.

Proof credit and final proof validity are reported separately. Corrected strict Controlled99 emits 30 fully valid and eight partial proofs among 96 planned attempts, yielding 35.42% normalized credit and 30/96 binary validity. At the stored stage, 48/96 proofs are valid and 18 are withheld. The earlier 91.89% proof-credit figure used best-effort output without answer locking. The original latency-capped scaffold and component-ablation tables are retained with their experimental conditions stated; the old mixed 96.01%/92.42% headline is not evidence for the corrected strict policy. Partial credit is separate from binary validity. FA uses mathematical equivalence; CCR requires correctness and all deliverables in the final response.

Cross-model evidence supports a bounded conclusion. On Qwen3, the auxiliary stored-candidate analysis includes originally withheld responses. At 90% FA and 50% proof coverage, Full/Code-SV errors are 6/360 versus 48/360 for FA, but 60/120 versus 45/120 for proof. Pooled risk is 66/480 versus 93/480, with a 3:1 FA:proof output weight. This reproduces lower FA risk on a second backbone, while proof risk reverses. Its policy differs from Private160's emitted-only analysis. Both use post-hoc targets and point estimates without significance claims; neither establishes a general cross-model proof gain.

Controlled99 and Private160 contain 99 and 160 independent problem clusters; four attempts are repeated measurements within each problem. Independent authorship and freezing before runs reduce exposure concerns, but unknown training-data cutoffs prevent a zero-contamination claim. The enriched 412-response cohort and trigger-selected failure events likewise do not estimate population failure rates.

The revised Method, Experiments, Discussion and Limitations, and appendices incorporate these distinctions.

In subsequent discussion, c4Tf stated that W1 was “largely resolved,” W2 and W3 were “substantially, though not fully, addressed,” and maintained their score. We appreciate this assessment and retain the noted limits on evaluation size and cross-model proof consistency. We ask the AC to assess the clarified contribution through the matched-budget results, shared-event and online authorization evidence, and held-out final-output results, with the coverage, proof-quality, and semantic-soundness boundaries stated explicitly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.