- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper trains a lightweight classifier on initial CoT features to decide whether to accept the answer or invoke Dynamic Voting. Each backbone uses a MedQA-trained classifier with task-specific threshold calibration. Experiments across five LLMs and four benchmarks show reduced token usage and average inference time, with some accuracy losses.
Critical strengths
-
The framework addresses a practical inference-cost problem through a modular design. Assessing an initial completed trajectory before invoking multi-path reasoning can avoid unnecessary sampling, and the approach integrates with an existing training-free aggregation method without modifying the underlying LLM.
-
The evaluation covers five backbones and four datasets and includes adaptive baselines, wall-clock measurements, and component and feature ablations. Training each decision model on MedQA and transferring it to other tasks with threshold calibration provides useful evidence of applicability beyond the training dataset. These experiments substantiate the practical motivation, although they do not fully establish the value of the proposed modeling choices.
Critical weaknesses
-
The methodological contribution appears limited. The framework combines a correctness classifier, a calibrated threshold, and an existing DV fallback. The experiments do not establish whether GRU-based temporal modeling outperforms simple confidence or length thresholds or pooled-feature classifiers. This omission is particularly consequential because Table 4 identifies reasoning length as the strongest individual signal.
-
The accuracy-latency advantage remains insufficiently established. Table 3 does not clarify whether timing includes the additional sentence-prefix answer scoring described in Appendix A and all routing overhead, which matters where speedups are marginal. Moreover, nonsignificant accuracy differences do not establish preservation within a specified tolerance, and some appendix settings show significant losses.
Questions for authors
-
Additional experimental evidence — incremental value of temporal modeling: Can the proposed decision network outperform simple gates based on final-answer confidence or trajectory length, and a classifier using pooled trajectory features? Please compare these approaches using the same validation calibration budget and DV fallback, include a single-CoT test reference, and report their accuracy–cost trade-offs. Given Table 4, this comparison would materially affect my assessment of the paper’s methodological contribution.
-
Clarification — end-to-end timing: Does Table 3 include the sentence-prefix answer-option scoring described in Appendix A, feature extraction, and decision-model inference? Please specify inference batching, KV-cache reuse, and whether multi-path generation is sequential or parallel. Do the baseline timings include only the computations required by each method?
-
Clarification — sampling and baseline configuration: Appendix E describes the initial trajectory as greedy, whereas the multi-path experiments use temperature 1.0. Is the initial greedy trajectory reused in the DV fallback and included in the ten-trajectory budget and voting pool? What stopping criteria and hyperparameters were used for DV and ReASC, and were they selected with a comparable validation budget?
-
Clarification and additional evidence — accuracy preservation: Could the authors report numerical paired accuracy differences with confidence intervals for each model–dataset pair and clarify the meaning and aggregation of the ± values in the abstract? Given the significant losses in some appendix settings, please distinguish the absence of a statistically significant difference from evidence that degradation remains within an acceptable tolerance.