- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Entropy Dynamics-based Reasoning Manifold, a framework that decides when an LLM should use CoT reasoning instead of always applying it. It probes a short initial decode and summarizes the token-level entropy trajectory into three signals — cumulative entropy, Spearman trend, and von Neumann ratio — then uses a training-free rule to route each query among Direct, Standard, or CoT decoding. Across 15 benchmarks and 7 LLMs, it reduces token consumption by ~16–55% versus always-on CoT, with model-dependent accuracy trade-offs.
Critical strengths
- The heuristic method requires no training, applies broadly, and incurs low cost.
- The experimental scope is comprehensive, covering a large number of datasets and models.
- The research question is important, with strong practical needs and applications.
Critical weaknesses
-
There are inconsistencies in the table data. In Table 3, the accuracy improvement of Inst-C over Inst-E on every individual dataset is no more than 0.68 percentage points, yet the Overall accuracy rises from 63.63% to 65.84%, an increase of 2.21 percentage points. In the same table, Inst-C's token cost is not lower than Inst-E's on any single dataset, yet the overall cost drops from 179.4 to 178.6, which likewise cannot be explained by identical weighting. Table 19 has a similar issue: for Qwen3.5, Inst-C's accuracy is no higher than Inst-E's on every dataset, yet the overall accuracy rises from 85.25% to 86.81%. There are conflicts in the model configuration and cost reporting that need verification. Section B.1.1 claims a maximum generation length of 4096 tokens, whereas in Table 13, on GPQA, Qwen3.5's average Standard and CoT lengths are 6410.3 and 6294.9 tokens, respectively. Since the paper defines cost in terms of generated tokens, the two are inconsistent. The paper also has not measured end-to-end latency. The regenerated prefill, full-vocabulary entropy computation, and scheduling costs mean that output-token savings cannot be directly equated with actual speedup.
-
The most significant accuracy gains in the main experiments come from the Oracle, and the advantage of the actual routing itself is not yet sufficient. The novel contributions and advantages relative to the most closely related work have not been fully validated. The Token Signature paper already studied probability trends of early generated tokens and used Spearman correlation and a learned router to decide whether to apply CoT; its instance-level Dynamic CoT trains a logistic regression on a small number of labeled samples. In contrast, the comparison version in Appendix B.13 of this paper replaces this with entropy trends and a sign-based rule.
Questions for authors
See Weaknesses
Dear reviewer i8su,
As the discussion period closes tomorrow evening, we wanted to quickly check if you have any remaining questions or concerns regarding our responses. We are fully available to provide further clarifications or additional results if needed. Thank you again for your time and effort!
Best regards,
Authors