- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a selective reasoning framework to reduce over-solving, where models continue reasoning after the requested answer is already determined. The method formalizes answer determination under incomplete information and combines task-specific stopping rules with a learned answer-relevance energy to prioritize useful computations. Experiments across five benchmarks show that it eliminates 33.95%–99.98% of task-specific computation while maintaining or improving accuracy, with gains of up to 30.99 percentage points. The key contribution is shifting reasoning efficiency from faster computation to computing only what is necessary to determine the answer.
Critical strengths
-
The paper presents a general and practical framework that combines a task-specific completion rule for sound stopping with a learned answer-relevance energy function that prioritizes the remaining computations.
-
The notion of “over-solving” is clearly motivated through intuitive examples, such as the blicket machine, and connected to established concepts, including certain answers and partial identification.
-
The framework yields substantial computational savings across diverse settings, suggesting that its applicability extends beyond stopping LLM chain-of-thought reasoning.
Critical weaknesses
-
The framework relies heavily on task-specific completion rules. Deriving exact rules may require substantial domain expertise, and this process is not automated. The generality of the approach therefore depends on how readily such rules can be formulated for new tasks.
-
Some completion rules, such as the one used for MNS, are operational rather than exact and may trade stopping safety for computational efficiency. The paper would benefit from a more thorough quantification of the frequency and consequences of premature termination.
-
In RAVEN-FAIR and ACRE, much of the wall-clock speedup comes from reformulating the baseline computation through decomposition or set-inclusion checks, rather than from selective execution itself. Although the authors acknowledge this distinction, it complicates the interpretation of the system-level comparisons and the attribution of the reported gains.
-
The evaluation is relatively small in some settings, such as ACRE with 384 pooled queries. Statistical significance tests or confidence intervals would help establish whether the reported accuracy improvements are reliable.
-
The sensitivity of the results to the capacity and input features of the learned energy function remains unclear. Please report the model size, input features, and training overhead for each task, and include a small ablation study examining the contribution of different feature subsets.
Questions for authors
-
For tasks using approximate or operational completion rules, such as MNS, how often does premature termination occur, and how significantly does it affect answer accuracy? Can the authors quantify the trade-off between computational savings and stopping reliability?
-
In RAVEN-FAIR and ACRE, how much of the reported wall-clock speedup comes from selective execution itself, as opposed to computational reformulations such as decomposition and set-inclusion checks? Please provide controlled comparisons to isolate the contribution of each component if possible.
6 comments
Response to Reviewer fKz5 (Part 2/2)
How much of the ACRE and RAVEN-FAIR speedup comes from computational reformulation, and how much from selective execution?
ACRE reformulation eliminates complete causal model fitting; RAVEN-FAIR reformulation replaces full neural scoring with decomposed scoring that can be evaluated term by term. Across the 2,000 problems corresponding to 8,000 ACRE queries, model fitting decreases from 2,000 calls to 0, replaced by direct observation checks. RAVEN-FAIR neural multiply-accumulate operations decrease from approximately 7.296×10¹² to 1.688×10¹¹. Fixed and Learning then use completion rules to omit remaining work in these reformulated computations. The table below compares work on the five complete evaluation cohorts. Within each task, the three versions use the same counting unit and return identical answers on every input.
| Task | Counting unit | Reformulated | Fixed | Learning | Learning reduction relative to complete execution |
|---|---|---|---|---|---|
| ACRE | Observation checks | 48,000 | 12,233 | 12,012 | 74.975% |
| CLUTRR | Relation pairs and splits | 46,570 | 14,280 | 13,550 | 70.904% |
| RAVEN-FAIR | Local score evaluations | 131,072 | 61,786 | 61,690 | 52.934% |
| AlphaGeometry | Matching calls | 26,934 | 24,795 | 23,108 | 14.205% |
| MNS | Candidate-check rows | 507,548,943 | 38,918,682 | 36,748,107 | 92.760% |
The corresponding measured runtimes are shown below, in seconds, including control overhead:
| Task | Baseline | Reformulated | Fixed | Learning |
|---|---|---|---|---|
| ACRE | 2,909.778 | 0.569 | 8.825 | 0.259 |
| CLUTRR | 20,396.397 | 3.428 | 3.439 | 2.509 |
| RAVEN-FAIR | 7,521.611 | 12.665 | 11.667 | 8.205 |
| AlphaGeometry | 599.684 | 421.371 | 398.340 | 270.313 |
| MNS | 1,021,553.839 | 8,236.636 | 5,967.635 | 5,962.081 |
Complete execution after reformulation takes 0.569 seconds on ACRE, compared with 0.259 seconds for Learning, a 2.195× net speedup. The corresponding RAVEN-FAIR times are 12.665 and 8.205 seconds, a 1.544× net speedup. Native operations are further reduced by 74.975% and 52.934%, with accuracy maintained at 100.00% and 99.71%, respectively. Complete and selective execution after reformulation use the same answer criterion; the RAVEN-FAIR completion rule applies to the decomposed scorer. Table 1 in Section 4 of the revised paper describes the reformulations; Tables 2 and 3 in Section 5 report runtime across the three stages and work across the two transitions.
ACRE has only 384 queries. Is the accuracy improvement reliable?
We have expanded the evaluation to 2,000 problems and 8,000 queries. Baseline accuracy is 68.59%, while Learning achieves 100.00%, an improvement of 31.41 percentage points. Resampling by problem gives a 95% confidence interval of [30.25, 32.59] percentage points. Table 2 in Section 5 of the revised paper has been updated with accuracy and runtime at this scale; Appendix C.3 gives the evaluation settings.
Please report model size, input features, and training cost, and provide feature ablations.
All five tasks share the same policy. The graph policy and shared cost term have 21,219 parameters in total. The graph representation has 9 node types, 26 edge types, and 112-dimensional node fields. Inputs describe answer requirements, acquired evidence, candidate computations, and their dependencies. Object information describes what a computation involves, while evidence state indicates which work remains necessary. Execution uses a lightweight program extracted and refined from the graph policy.
The recorded training times are 1,400.64 seconds for the graph network and 22.09 seconds for the shared head. In the final round, cost collection at root and dynamic states took 36.96 seconds in total, and complete execution and selection of candidate policies took 10.82 seconds.
We fix the final model and separately mask two types of information on precomputed scoring inputs for 2,048 RAVEN-FAIR problems:
| Setting | Local score evaluations | Increase | Answer changes |
|---|---|---|---|
| Complete model | 61,707 | — | 0 |
| Without computation-object information | 85,034 | 37.80% | 0 |
| Without answer and evidence-progress information | 69,824 | 13.15% | 0 |
Masking computation-object information increases score evaluations by 37.80%; masking answer and evidence-progress information increases them by 13.15%. All three settings return identical answers on every input. Appendices C.1–C.2 of the revised paper describe the model configuration and training objective.
Thank you again for your review. We clarified the stopping criteria for all five tasks in Table 1 and updated Table 2 to report the expanded ACRE evaluation on 8,000 queries. Please let us know if you have any further questions.
Thanks for the author's detailed response, which have addressed some of my concerns and helped improve the clarity of the paper. Additionally, the proposed method relies on task-specific completion rules, its applicability to other tasks still requires further validation. I will maintain my rating.
Response to Reviewer fKz5 (Part 1/2)
Thank you for recognizing the motivation and practical value of the paper, and for your suggestions on stopping reliability and experimental design. We have added the following results.
Can the domain knowledge needed to design stopping rules by hand be reduced?
We concentrate the required domain knowledge in task rules, basic operations, and answer requirements. The program uses these to generate or maintain answer constraints and check whether the acquired evidence determines the answer. Tasks with native proof verification reuse their verifiers. The shared learner determines computation order. The tasks are handled as follows:
Table 1 in Section 4 of the revised paper summarizes the completion criteria; Appendix B gives their construction and conditions of validity.
How often does incorrect early stopping occur in MNS, and what is its effect on accuracy?
The current MNS certificates return 0 incorrect answers, and selective execution maintains the accuracy of complete execution. We corrected the original implementation and use checks for rule coverage and answer agreement. The program verifies that the input satisfies the coverage conditions of the declared rule space and checks the query answers given by compatible rules. It returns when all compatible cases agree, and invokes the native query solver when answers are ambiguous or coverage is insufficient. The outcomes across 56,000 inputs are as follows:
Certificates directly answer 17,100 inputs, or 30.54% of the complete cohort, and all of these answers agree with the ground truth. The remaining 38,900 inputs proceed to native query solving. Exhaustive rule enumeration and selective execution produce identical stopping classifications. Learning and Reformulated return identical answers on all 56,000 inputs, both achieving 79.11% accuracy, compared with 75.97% for the original Baseline. Candidate checks decrease from 507,548,943 to 36,748,107 rows, a 92.760% reduction. Appendix B.5 of the revised paper describes the current completion checks and continuation procedure.