acceptodds
Under review as a conference paper at ICLR 2027

VASIC: Validation-Guided Adaptation for Symbolic Integration with LLMs and CAS

Abstract

AI systems are increasingly used in research and education, generating complex proofs, derivations, and results faster than experts can reliably validate. Interactive theorem provers such as Lean provide strong guarantees for formalized proofs, but fully formalizing the symbolic derivations and analytical calculations central to physics and engineering is often impractical. As a result, a growing volume of LLM-generated mathematics in these domains remains unverified. To bridge this gap, we propose a complementary, scalable method based on Computer Algebra Systems (CAS) to improve the reliability of LLM-generated mathematical outputs. As a proof of concept, we present VASIC, a validation-guided LLM-CAS protocol for symbolic integration. VASIC tests each candidate LLM solution using symbolic and numerical methods. Failed attempts are routed through feedback-guided adaptation and retried. We benchmark VASIC against standalone CAS and LLM baselines from a broad range of model families and versions. Compared with direct use of the same model, VASIC increases validated success rates by 3.5–10 percentage points for frontier models and by up to 36.2 percentage points for smaller and less expensive models. For example, DeepSeek Reasoner V3 achieves 84.0% validated success with full VASIC, close to DeepSeek V4's 85.2%, at roughly half the estimated model-inference cost, counting the additional LLM requests made by VASIC. Together, these results show that LLM-CAS hybrids can increase AI success rate and practical reliability in problems of general importance for science and engineering.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Large Language Models, Symbolic Integration, Computer Algebra Systems, Symbolic Verification, Mathematical Reasoning, Tool-Augmented Language Models, Neuro-Symbolic AI, AI for math

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LwYXOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents VASIC, a symbolic integration framework that combines large language models with computer algebra systems (CAS). The framework uses CAS to check candidate answers and proposed integral transformations, provides validation feedback to guide retries, and attempts transformations such as integration by parts and substitution when direct solving fails. The authors evaluate 12 LLMs on a benchmark of 1,445 integration problems and report improvements in validated success rates over direct solving. A matched-retry experiment supports the effectiveness of validation feedback. The paper also evaluates validator reliability through constructed test cases and cross-CAS rechecks. Validation combines symbolic and numerical checks, with numerical agreement alone sufficient to accept some definite-integral results; it therefore does not constitute formal proof.

Critical strengths
  1. The authors evaluate 12 LLMs on 1,445 integration problems, demonstrating the framework’s benefits across different models. Under the same retry budgets and time limits, retries with validation feedback outperform retries without feedback, supporting the usefulness of the feedback.

  2. Beyond evaluating problem-solving performance, the authors test the validator using constructed cases and cross-check results with another CAS, reporting both errors and inconclusive outcomes.

Critical weaknesses
  1. The methodological novelty appears limited. Prior work such as CRITIC, CSV, and AlphaIntegrator has explored tool-based feedback, verification and correction, or neural–symbolic integration. Although VASIC differs in its implementation, its contribution appears primarily to be the integration and adaptation of existing ideas. The paper does not yet clearly establish a distinctive methodological contribution.

  2. The matched-retry experiment supports the usefulness of feedback, but it does not isolate the contribution of integral transformations. Full VASIC allows up to 33 model calls per problem. Comparisons with continued feedback-guided retries under matched compute budgets are needed to distinguish the benefits of transformations from those of additional attempts.

  3. The reported success rate combines symbolic validation with numerical-only acceptance for some definite integrals. Although the authors acknowledge this distinction and audit the validator, the overall risk of accepting incorrect results remains unclear. Reporting success rates and gains separately by validation type would help clarify the reliability of the results.

  4. The cost analysis does not capture the complete end-to-end overhead. The paper reports model inference costs and some timing measurements, but a fuller assessment should include CAS validation, transformation attempts, and retries.

Questions for authors
  1. What is VASIC’s main methodological contribution beyond existing tool-based verification and feedback-driven refinement approaches?

  2. Under matched compute budgets, how does full VASIC compare with continued feedback-guided retries on the original integral? How much of the improvement comes specifically from integral transformations?

  3. Could the authors report success rates separately for symbolic validation and numerical-only acceptance, and further assess the risk of accepting incorrect results?

3 comments
AuthorsOpenReview ↗

Response to Reviewer LwYX

Thank you for the positive assessment and the precise questions.

Q1: Methodological contribution. VASIC differs from CRITIC, CSV and AlphaIntegrator in three ways (Sections 1–3):

  1. The validator gates transformations, not only answers. Each proposed rewrite of the integral must pass a CAS equivalence check before the model may solve it, and every final answer is checked against the original integral, so a flawed rewrite cannot by itself produce an accepted answer.
  2. The validator is independent of the generating model. It does not run model-written code, and the model cannot accept its own answer. CRITIC and CSV keep the model in the verification loop.
  3. No training. Unlike AlphaIntegrator, VASIC needs no integration-specific training and applies unchanged to any LLM; we evaluate 12.

Evaluation follows the same principle: every answer is checked against the original integral, not against a stored reference. This matters in practice: 284 of the 1,445 stored references receive no CAS pass, and both backends reject 23 of them (Tables 8–9).

Q2: Matched compute; share from transformations. The share is in Table 2. Transformations add most for weaker direct solvers, for example 30.0 of DeepSeek Reasoner V3's 36.2 points, while feedback dominates for the three strongest models. The feedback comparison uses the same retry cap and time limits in both branches (41.6% vs. 21.9% on 512 non-passes from three models; Section 4.4). Full VASIC uses 2.4–4.1 calls per problem for the strongest models and 13–18 for the others, against the cap of 33 (Table 13). We are running the matched-compute comparison against continued feedback retries that you suggest, and will post the results in this thread before the discussion period ends (see the general response).

Q3: Success by validation type; risk of incorrect acceptance. Table 17 splits each model's passes into symbolic and numerical-only, for definite and indefinite integrals. Numerical-only acceptance accounts for 32–75% of accepted definite answers, about three-quarters for the three strongest models, and for at most 1.2% of accepted indefinite answers. It arises mainly for definite integrals without a closed-form CAS value, under the tolerance and sampling criteria in Appendix B.6. On risk, both audits from the general response apply: among 400 labelled cases, none of the 100 incorrect answers and none of the 100 non-equivalent transformations passed (Table 15). The cross-CAS recheck of 1,037 accepted records confirmed 794, was inconclusive on 240 and contradicted 3, of which one was an incorrect, numerically accepted answer (Section 4.5, Table 16).

End-to-end overhead. Model costs cover every request, including transformation proposals, retries and failed attempts (Table 13); CAS execution runs locally and is not priced. Table 14 gives median solve times per solved problem with CAS validation included: a direct pass takes under 2.5 minutes for every model, and a successful transformation route 2.3–18.1 minutes.

We hope this addresses your concerns and are happy to clarify any point further.

Matched-compute results. Under the budget of one VASIC transformation route (three blocks of up to three model calls and 20 minutes each), transformations solve more problems than continued feedback retries for DeepSeek-V4.1-Flash (83.2% vs. 76.5% of all 1,445 problems), GPT-4.1 (28.7% vs. 19.3%) and Mistral Large (26.9% vs. 24.2%); for DeepSeek V4, both reach 84.8%. The setup and model details are in our post under the general response, and the revised PDF adds the comparison as Section 4.6.

Correction on call budgets. For DeepSeek-V4.1-Flash, both strategies share a cap of three calls per block, transformation proposals included, and use about the same number of calls (5.6 against 5.4 per problem). For DeepSeek V4, GPT-4.1 and Mistral Large, the transformation results come from the original runs' routes, which use 3.7–5.0 calls on average, against at most three per retry block; these comparisons match blocks and time, not calls. The revised PDF states this in Section 4.6 and Appendix C.5.

Reviewer_zu79OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes VASIC, a validation-guided framework for improving LLM-based symbolic integration. Instead of trusting an LLM's output directly, VASIC uses a computer algebra system (CAS) to check candidate solutions, feeds validation failures back to the LLM for correction, and, when necessary, transforms the integral into an alternative representation before attempting to solve it again.

The evaluation covers 1,445 integration problems and 12 LLM systems. The reported results show improvements of 3.5–10.0 percentage points for several frontier models, with larger gains for weaker models. The paper also investigates the reliability of its validation procedure and the contributions of feedback and transformation strategies.

Critical strengths
  1. A practical and well-motivated approach to improving mathematical reliability. The paper addresses a concrete limitation of LLM-based symbolic integration: producing a plausible expression is not the same as producing a correct one. By separating answer generation from correctness checking, VASIC reduces reliance on the LLM's own confidence or self-assessment. The use of CAS-based verification is a natural and principled choice for this task.

  2. A systematic evaluation across models of different capabilities. The evaluation includes 1,445 integration problems and 12 LLM systems, providing a broader assessment than experiments limited to a single model. The reported improvements are particularly substantial for weaker models. For example, DeepSeek Reasoner V3 improves from 47.8% to 84.0% validated success, while GPT-4.1 improves from 9.0% to 32.2%. These results suggest that external validation and feedback can substantially improve the effective performance of models that are less reliable at direct symbolic integration.

  3. Useful analysis of the contributions of feedback and problem transformation. The paper goes beyond reporting end-to-end success rates by examining the roles of different components. Its controlled comparison reports 41.6% success for feedback-guided retries versus 21.9% for retries without feedback on failed problems under the same retry budget. The finding that feedback contributes substantially for stronger models, while transformations are particularly useful for weaker models, also provides insight into how different forms of assistance interact with model capability.

Critical weaknesses
  1. The methodological novelty appears limited relative to the empirical gains. The central idea—generate a candidate solution, verify it with an external tool, feed errors back to the model, and reformulate the problem when necessary—is intuitive and closely related to established tool-assisted reasoning and verification-guided generation paradigms. The paper's contribution therefore appears to lie primarily in its integration of these components for symbolic integration and its empirical evaluation, rather than in a fundamentally new algorithmic principle. The paper would benefit from a sharper explanation of what is technically novel beyond the particular orchestration of existing techniques. 2 The reported success metric conflates mathematical correctness with validator coverage. A solution may be mathematically correct yet fail to be accepted because the CAS cannot simplify or verify it within the available procedure. This is especially relevant for difficult definite integrals and expressions involving numerical approximations or branch-sensitive complex functions, limitations acknowledged by the authors. Consequently, validated success rate measures not only the LLM's ability to solve an integral but also the validator's ability to recognize the solution. The paper would benefit from a clearer separation between incorrect answers, correct but unverified answers, and verified correct answers.
  2. The generality of the empirical conclusions remains uncertain. The benchmark contains problems from multiple sources, which is a strength, but the results do not by themselves establish that VASIC generalizes equally well across different types of integrals or difficulty levels. The relative benefits of feedback and transformation may depend heavily on the distribution of problem types. More granular results would help establish whether the method provides broad improvements or is particularly effective on certain subsets.
Questions for authors
  1. What is the main methodological contribution beyond combining existing techniques? How does VASIC differ substantively from a straightforward pipeline that performs CAS verification, error-feedback retries, and a fixed collection of symbolic transformations? Can the authors identify a design principle or component that is responsible for improvements beyond what simpler tool-assisted baselines achieve?

  2. How much of the measured failure rate comes from the LLM and how much from the validator? Can the authors distinguish incorrect candidate solutions from correct solutions that the CAS fails to verify? Such a breakdown would help determine whether the next improvement should target the generation process or the validation procedure.

  3. Do the benefits persist across different categories of integration problems? Could the authors report performance broken down by problem type or difficulty, particularly for definite integrals, transformations, and expressions involving numerical or complex-valued functions? This would clarify where VASIC is most effective and where its limitations dominate.

2 comments
AuthorsOpenReview ↗

Response to Reviewer zu79

Thank you for the careful review and the constructive questions.

Q1: Contribution beyond combining techniques. We agree that the individual components are known; the contribution is where the checker sits. The design principle is that an untrusted generator proposes and an independent checker accepts, applied both to answers and to the problem itself. VASIC lets the model reformulate the integral, and each reformulation must pass a CAS equivalence check before it is solved. A pipeline with a fixed set of symbolic transformations needs each transformation to be correct by construction. VASIC lets the LLM propose the concrete rewrite within five transformation types, and relies on validation for acceptance: every final answer is checked against the original integral. This is what lets transformations account for most of the gain for weaker models (Table 2). No non-equivalent transformation passed among 100 labelled cases (Table 15).

Q2: Failures from the LLM vs. the validator. Table 3 separates the outcomes on the original integral:

  • Fail is a detected mismatch: the answer is wrong. In our 400-case test, the validator never failed a correct answer (Table 15).
  • Inconclusive is the outcome that can hide a correct answer the CAS could not confirm. It covers 2.1–3.5% of problems for the three strongest models and up to 22.0% for the other models.
  • Unresolved means the model returned no answer or the integral unevaluated.
  • Timeout or error covers runs that hit the job limit, which includes both generation and validation (Appendix B.3).

So Fail and Unresolved lie with the generator, and the Inconclusive and Timeout-or-error shares together bound how far validator coverage can lower measured success. Table 4 splits these outcomes by integral kind. For the three strongest models, Inconclusive stays at 1.2–6.7% for both kinds. For several weaker models it concentrates in definite integrals (up to 60.0%), and it stays at or below 12.2% for indefinite integrals. Validation against the original integral is also more reliable than reference matching: 284 of the 1,445 stored references receive no CAS pass (Table 8).

Q3: Benefits by category. Table 5 reports success on the original integral (direct solving and feedback) by source family. The three strongest models solve 94–100% of CUMath and the other sources, and ASyMOB is the hardest family for all three. Table 4 gives the same stage by integral kind: the three strongest models pass 76–90% of definite and 84–89% of indefinite problems. Table 17 reports how passes on definite and indefinite integrals were accepted. The transformation stage's contribution is reported per model (Table 2). We will add this breakdown in the next revision.

We hope this addresses your concerns and are happy to clarify any point further.

As promised, the revised PDF adds Appendix C.11: Table 19 breaks down the transformation stage by type, and Table 20 reports validated success after full VASIC by source family, with the gain from transformations. Tables 5 and 6 give the earlier stages by integral kind and by source family.

Reviewer_vHKvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VASIC, a validation-guided framework combining large language models with computer algebra systems (CAS) for symbolic integration. The framework validates candidate solutions through symbolic and numerical checks, uses feedback to refine unsuccessful attempts, and explores equivalent integral transformations.

Experiments across 1,445 integration problems and 12 LLMs demonstrate improved validated success rates, with gains ranging from 3.5 to 36.2 percentage points. The authors additionally evaluate validator reliability, feedback effectiveness, and estimated inference costs.

Critical strengths
  1. The paper evaluates a diverse set of LLMs across multiple integration benchmarks, demonstrating the potential benefits of combining language models with symbolic computation. The reported improvements, particularly for smaller models, suggest practical opportunities to improve mathematical problem-solving efficiency without additional model training.

  2. The paper explicitly distinguishes symbolic verification from numerical evidence and conducts separate validator reliability experiments. Importantly, the matched-retry evaluation demonstrates that validator feedback improves performance beyond simply generating additional attempts. This provides meaningful evidence supporting the feedback-guided component.

Critical weaknesses
  1. VASIC combines several relatively established techniques, including external validation, iterative refinement, and symbolic transformations. Its overall effectiveness is therefore an important part of its contribution.

However, full VASIC allows substantially more computation than direct solving, with up to 33 model calls per problem. Although the paper provides matched-retry comparisons for validator feedback, it does not establish whether the full framework outperforms alternative strategies under equivalent computational budgets.

This is particularly important for DeepSeek Reasoner V3, where transformation attempts account for 30.0 of the reported 36.2 percentage-point improvement. Without matched-budget comparisons against repeated direct solving or simpler transformation strategies, the distinctive contribution of VASIC remains unclear.

Questions for authors
  1. How does full VASIC compare against repeated direct solving or alternative transformation strategies under identical model-call, token, and wall-clock budgets?

  2. What proportion of final accepted solutions rely exclusively on numerical validation? Can the authors estimate false-positive rates using independent mathematical checks, particularly for difficult definite integrals?

  3. Can the authors report end-to-end latency and total computational costs, including CAS execution and unsuccessful validation attempts?

  4. How do the gains vary across different benchmark sources and between definite and indefinite integrals? Are improvements concentrated in particular classes of integrals or transformation types?

3 comments
AuthorsOpenReview ↗

Response to Reviewer vHKv

Thank you for the review and for recognising the matched-retry evidence.

Q1: Equal call, token and wall-clock budgets. 33 calls is the cap rather than typical use: actual use is 2.4–4.1 calls per problem for the three strongest models and 13–18 for the others (Table 13). The feedback comparison gives both branches the same retry cap and time limits (41.6% vs. 21.9% on 512 non-passes from three models; Section 4.4). Realized tokens can differ, and so does wall-clock time: the feedback branch has the longer median time for Gemini 3.1 Pro and DeepSeek V4, and a slightly shorter one for GPT-5.5 Pro (Appendix C.6). You are right that for DeepSeek Reasoner V3 transformations provide most of the gain: 30.0 of the 36.2 points (Table 2). The whole pipeline costs USD 5.68 per 100 problems and matches GPT-5.5 Pro used directly at USD 56.30 (Figure 3b). We are running the equal-budget comparison against continued retries on the original integral and will post the results in this thread before the discussion period ends (see the general response).

Q2: Numerical-only acceptance and false positives. On the original integral, numerical-only acceptance accounts for 32–75% of each model's accepted definite answers, and about three-quarters for the three strongest models. These are definite integrals the CAS cannot evaluate in closed form. For indefinite integrals, 98.8–100% of accepted answers pass a symbolic check (Table 17). Over all accepted solutions, more than three-quarters of each model's pass a symbolic check (Section 4.5). For an independent check, we rechecked a stratified sample of 1,037 accepted records with the CAS that had not accepted them. It confirmed 794, was inconclusive on 240 and contradicted 3, and manual review found one incorrect acceptance, a numerically accepted answer (Section 4.5, Table 16). Because the sample is stratified, these counts describe the sample rather than an overall rate. No answer passed among 100 constructed incorrect answers (Table 15).

Q3: End-to-end latency and cost. Model costs include every request, including failed attempts and transformation proposals (Table 13); CAS execution runs locally and is not priced. Table 14 gives median active time per solved problem with CAS validation included: under 2.5 minutes for a direct pass, 0.2–14.7 minutes after feedback, and 2.3–18.1 minutes for a successful transformation route. These are per-route times for solved problems; the PDF does not report total wall-clock time for a full run over all five transformation types. Standalone CAS medians are 1.5 s (SymPy) and 0.9 s (Wolfram) (Appendix C.6).

Q4: Gains by source, kind and transformation type. Table 5 gives success on the original integral by source family, Table 4 by integral kind, and Table 17 gives the acceptance evidence for definite and indefinite integrals. The transformation stage's total contribution per model is in Table 2. We will add this breakdown in the next revision.

We hope this addresses your concerns, and we would be glad to run any further check you find useful.

Matched-compute results. Under the budget of one VASIC transformation route (three blocks of up to three model calls and 20 minutes each), transformations solve more problems than continued feedback retries for DeepSeek-V4.1-Flash (83.2% vs. 76.5% of all 1,445 problems), GPT-4.1 (28.7% vs. 19.3%) and Mistral Large (26.9% vs. 24.2%); for DeepSeek V4, both reach 84.8%. The setup and model details are in our post under the general response, and the revised PDF adds the comparison as Section 4.6.

Correction on call budgets. For DeepSeek-V4.1-Flash, both strategies share a cap of three calls per block, transformation proposals included, and use about the same number of calls (5.6 against 5.4 per problem). For DeepSeek V4, GPT-4.1 and Mistral Large, the transformation results come from the original runs' routes, which use 3.7–5.0 calls on average, against at most three per retry block; these comparisons match blocks and time, not calls. The revised PDF states this in Section 4.6 and Appendix C.5.

As promised, the revised PDF adds Appendix C.11: Table 19 breaks down the transformation stage by type, and Table 20 reports validated success after full VASIC by source family, with the gain from transformations. Tables 5 and 6 give the earlier stages by integral kind and by source family.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for their careful and constructive reading. Three concerns are shared: the compute used by full VASIC, the role of numerical acceptance, and results by problem category. The revised PDF adds analyses of the existing runs:

  • Table 3: outcomes on the original integral (Pass / Fail / Inconclusive / Timeout or error / Unresolved) for every model.
  • Table 4: the same outcomes split into definite and indefinite integrals.
  • Table 5: success on the original integral by source family.
  • Tables 8–9: an audit of the benchmarks' stored reference answers.
  • Table 13: mean model calls per problem under full VASIC.
  • Table 14: median solve time per solved problem, CAS validation included.
  • Table 17: symbolic vs. numerical-only acceptance, for definite and indefinite integrals separately.

Compute. 33 calls is the cap, not typical use. The three strongest models use 2.4–4.1 calls per problem on average, and the other models 13–18 (Table 13). The feedback comparison uses the same retry cap and time limits in both branches: on 512 direct non-passes from three models, feedback-guided retries solve 41.6% against 21.9% without feedback (Section 4.4). Realized tokens and time can differ between the branches (Appendix C.6). Full-pipeline gains include the extra calls, and we price them: DeepSeek Reasoner V3 with VASIC matches GPT-5.5 Pro used directly (84.0% vs. 84.1%) at USD 5.68 instead of USD 56.30 per 100 problems (Figure 3b).

Numerical acceptance. On the original integral, numerical-only acceptance is concentrated in definite integrals: it accounts for 32–75% of each model's accepted definite answers, and about three-quarters for the three strongest models, because the CAS often cannot evaluate these integrals in closed form. For indefinite integrals, 98.8–100% of accepted answers pass a symbolic check (Table 17). Over all accepted solutions, more than three-quarters of each model's pass a symbolic check (Section 4.5). Two audits test the validator:

  • Labelled cases. In 400 cases with labels fixed in advance, including constructed incorrect answers (sign flips, factor-of-two changes, dropped terms), no incorrect answer or non-equivalent transformation passed, and no correct answer failed (Table 15).
  • Cross-CAS recheck. We rechecked a stratified sample of 1,037 accepted records with the CAS that had not accepted them. It confirmed 794, was inconclusive on 240 and contradicted 3. Manual review of the three found one incorrect acceptance, a numerically accepted answer (Section 4.5, Table 16).

Matched compute for transformations. We are running the comparison requested by Reviewers LwYX and vHKv. On the problems that reach the transformation stage, the baseline continues feedback-guided retries on the original integral with the same number of calls and the same 20-minute limit per route as VASIC's transformation routes. The first models are DeepSeek Reasoner V3 and DeepSeek V4. We thank both reviewers for suggesting this control. The runs are in progress, and we will post the results in this thread before the discussion period ends.

We respond to each reviewer's specific questions in the individual replies.

Matched-compute results for transformations. As promised, we compared VASIC's transformations with continued feedback-guided retries under the same compute budget. For each model, both strategies run on the problems it left unsolved after feedback, with the budget of one VASIC transformation route: three blocks, each a context window with up to three model calls and a 20-minute limit, and each problem stops at its first pass. One strategy continues feedback-guided retries on the original integral; the other applies one transformation type per block.

Validated success on all 1,445 problems after three blocks:

ModelAfter feedback+ retries+ transformations
DeepSeek-V4.1-Flash, on DeepSeek Reasoner V3's unsolved problems54.0%76.5%83.2%
GPT-4.115.0%19.3%28.7%
Mistral Large17.0%24.2%26.9%
DeepSeek V481.9%84.8%84.8%
  • Transformations solve more problems than retries for DeepSeek-V4.1-Flash, GPT-4.1 and Mistral Large; for DeepSeek V4, both strategies end at the same success, with transformations ahead after the first two blocks. For DeepSeek-V4.1-Flash, both strategies use about the same number of calls (5.6 against 5.4 per problem).
  • Weaker direct solvers gain most: for GPT-4.1, transformations add three times as much as retries.
  • The deepseek-reasoner endpoint used in the paper now serves DeepSeek-V4.1-Flash, so we ran both strategies with it. For the other models, the retry runs use the same model versions as the paper, and the transformation results come from the original runs; for DeepSeek V4, they use its three strongest transformation types.

We have added this comparison to the revised PDF as Section 4.6 and Figure 5, with details in Appendix C.5.

Table numbers. The revised PDF moves the reference-answer audit into Section 2, which renumbers the tables. In our earlier replies, Table 8 is now Table 1, Tables 2–5 are now Tables 3–6, Tables 13–17 are now Tables 14–18, and Appendix C.6 is now Appendix C.7. Table 9 and all section and figure numbers are unchanged.

Correction on call budgets. For DeepSeek-V4.1-Flash, both strategies share a cap of three calls per block, transformation proposals included, and use about the same number of calls (5.6 against 5.4 per problem). For DeepSeek V4, GPT-4.1 and Mistral Large, the transformation results come from the original runs' routes, which use 3.7–5.0 calls on average, against at most three per retry block; these comparisons match blocks and time, not calls. The revised PDF states this in Section 4.6 and Appendix C.5.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.