- Rating
- 2(−1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents VASIC, a symbolic integration framework that combines large language models with computer algebra systems (CAS). The framework uses CAS to check candidate answers and proposed integral transformations, provides validation feedback to guide retries, and attempts transformations such as integration by parts and substitution when direct solving fails. The authors evaluate 12 LLMs on a benchmark of 1,445 integration problems and report improvements in validated success rates over direct solving. A matched-retry experiment supports the effectiveness of validation feedback. The paper also evaluates validator reliability through constructed test cases and cross-CAS rechecks. Validation combines symbolic and numerical checks, with numerical agreement alone sufficient to accept some definite-integral results; it therefore does not constitute formal proof.
Critical strengths
-
The authors evaluate 12 LLMs on 1,445 integration problems, demonstrating the framework’s benefits across different models. Under the same retry budgets and time limits, retries with validation feedback outperform retries without feedback, supporting the usefulness of the feedback.
-
Beyond evaluating problem-solving performance, the authors test the validator using constructed cases and cross-check results with another CAS, reporting both errors and inconclusive outcomes.
Critical weaknesses
-
The methodological novelty appears limited. Prior work such as CRITIC, CSV, and AlphaIntegrator has explored tool-based feedback, verification and correction, or neural–symbolic integration. Although VASIC differs in its implementation, its contribution appears primarily to be the integration and adaptation of existing ideas. The paper does not yet clearly establish a distinctive methodological contribution.
-
The matched-retry experiment supports the usefulness of feedback, but it does not isolate the contribution of integral transformations. Full VASIC allows up to 33 model calls per problem. Comparisons with continued feedback-guided retries under matched compute budgets are needed to distinguish the benefits of transformations from those of additional attempts.
-
The reported success rate combines symbolic validation with numerical-only acceptance for some definite integrals. Although the authors acknowledge this distinction and audit the validator, the overall risk of accepting incorrect results remains unclear. Reporting success rates and gains separately by validation type would help clarify the reliability of the results.
-
The cost analysis does not capture the complete end-to-end overhead. The paper reports model inference costs and some timing measurements, but a fuller assessment should include CAS validation, transformation attempts, and retries.
Questions for authors
-
What is VASIC’s main methodological contribution beyond existing tool-based verification and feedback-driven refinement approaches?
-
Under matched compute budgets, how does full VASIC compare with continued feedback-guided retries on the original integral? How much of the improvement comes specifically from integral transformations?
-
Could the authors report success rates separately for symbolic validation and numerical-only acceptance, and further assess the risk of accepting incorrect results?
3 comments
Matched-compute results. Under the budget of one VASIC transformation route (three blocks of up to three model calls and 20 minutes each), transformations solve more problems than continued feedback retries for DeepSeek-V4.1-Flash (83.2% vs. 76.5% of all 1,445 problems), GPT-4.1 (28.7% vs. 19.3%) and Mistral Large (26.9% vs. 24.2%); for DeepSeek V4, both reach 84.8%. The setup and model details are in our post under the general response, and the revised PDF adds the comparison as Section 4.6.
Correction on call budgets. For DeepSeek-V4.1-Flash, both strategies share a cap of three calls per block, transformation proposals included, and use about the same number of calls (5.6 against 5.4 per problem). For DeepSeek V4, GPT-4.1 and Mistral Large, the transformation results come from the original runs' routes, which use 3.7–5.0 calls on average, against at most three per retry block; these comparisons match blocks and time, not calls. The revised PDF states this in Section 4.6 and Appendix C.5.
Response to Reviewer LwYX
Thank you for the positive assessment and the precise questions.
Q1: Methodological contribution. VASIC differs from CRITIC, CSV and AlphaIntegrator in three ways (Sections 1–3):
Evaluation follows the same principle: every answer is checked against the original integral, not against a stored reference. This matters in practice: 284 of the 1,445 stored references receive no CAS pass, and both backends reject 23 of them (Tables 8–9).
Q2: Matched compute; share from transformations. The share is in Table 2. Transformations add most for weaker direct solvers, for example 30.0 of DeepSeek Reasoner V3's 36.2 points, while feedback dominates for the three strongest models. The feedback comparison uses the same retry cap and time limits in both branches (41.6% vs. 21.9% on 512 non-passes from three models; Section 4.4). Full VASIC uses 2.4–4.1 calls per problem for the strongest models and 13–18 for the others, against the cap of 33 (Table 13). We are running the matched-compute comparison against continued feedback retries that you suggest, and will post the results in this thread before the discussion period ends (see the general response).
Q3: Success by validation type; risk of incorrect acceptance. Table 17 splits each model's passes into symbolic and numerical-only, for definite and indefinite integrals. Numerical-only acceptance accounts for 32–75% of accepted definite answers, about three-quarters for the three strongest models, and for at most 1.2% of accepted indefinite answers. It arises mainly for definite integrals without a closed-form CAS value, under the tolerance and sampling criteria in Appendix B.6. On risk, both audits from the general response apply: among 400 labelled cases, none of the 100 incorrect answers and none of the 100 non-equivalent transformations passed (Table 15). The cross-CAS recheck of 1,037 accepted records confirmed 794, was inconclusive on 240 and contradicted 3, of which one was an incorrect, numerically accepted answer (Section 4.5, Table 16).
End-to-end overhead. Model costs cover every request, including transformation proposals, retries and failed attempts (Table 13); CAS execution runs locally and is not priced. Table 14 gives median solve times per solved problem with CAS validation included: a direct pass takes under 2.5 minutes for every model, and a successful transformation route 2.3–18.1 minutes.
We hope this addresses your concerns and are happy to clarify any point further.