acceptodds
Under review as a conference paper at ICLR 2027

When Should LLMs Reason? Adaptive Inference Routing via Entropy Dynamics

Abstract

Chain-of-thought (CoT) reasoning has become the default strategy for enhancing LLM capabilities, yet its application raises a fundamental question: when is explicit reasoning actually beneficial? Empirical evidence reveals a striking paradox: CoT often provides marginal or even negative gains on factual and open-ended tasks while multiplying token consumption. In this work, we show that LLM reasoning is not a static property of tasks or models, but a dynamic decoding state that emerges during generation. Through systematic analysis, we find early-stage entropy dynamics provide a reliable signal of this state: tasks benefiting from CoT exhibit consistent entropy reduction, while others display unstable or increasing patterns. This behavior can be interpreted as a regime shift from a high-entropy exploratory regime to a low-entropy structured reasoning regime. Based on these insights, we propose EDRM (Entropy Dynamics-based Reasoning Manifold), a lightweight and training-free routing framework that leverages early decoding entropy to adaptively select inference strategies. EDRM embeds entropy trajectories into a compact and interpretable feature embedding, enabling both zero-shot deployment and fine-grained instance-level adaptation. Across 15 benchmarks and 7 LLMs of varying scales and architectures, fully deployable variants of EDRM achieve consistent token reduction (16–55%, all ) while maintaining comparable accuracy, and the Oracle fallback upper bound further provides up to 4.9% accuracy improvement. These results suggest that reasoning should be invoked selectively rather than by default, and demonstrate the effectiveness of entropy-driven decoding control for efficient and adaptive LLM inference.

Primary area:
foundation or frontier models, including LLMs
Keywords:
efficient reasoning, adaptive inference, entropy dynamics, LLM routing, test-time compute

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_i8suOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Entropy Dynamics-based Reasoning Manifold, a framework that decides when an LLM should use CoT reasoning instead of always applying it. It probes a short initial decode and summarizes the token-level entropy trajectory into three signals — cumulative entropy, Spearman trend, and von Neumann ratio — then uses a training-free rule to route each query among Direct, Standard, or CoT decoding. Across 15 benchmarks and 7 LLMs, it reduces token consumption by ~16–55% versus always-on CoT, with model-dependent accuracy trade-offs.

Critical strengths
  1. The heuristic method requires no training, applies broadly, and incurs low cost.
  2. The experimental scope is comprehensive, covering a large number of datasets and models.
  3. The research question is important, with strong practical needs and applications.
Critical weaknesses
  1. There are inconsistencies in the table data. In Table 3, the accuracy improvement of Inst-C over Inst-E on every individual dataset is no more than 0.68 percentage points, yet the Overall accuracy rises from 63.63% to 65.84%, an increase of 2.21 percentage points. In the same table, Inst-C's token cost is not lower than Inst-E's on any single dataset, yet the overall cost drops from 179.4 to 178.6, which likewise cannot be explained by identical weighting. Table 19 has a similar issue: for Qwen3.5, Inst-C's accuracy is no higher than Inst-E's on every dataset, yet the overall accuracy rises from 85.25% to 86.81%. There are conflicts in the model configuration and cost reporting that need verification. Section B.1.1 claims a maximum generation length of 4096 tokens, whereas in Table 13, on GPQA, Qwen3.5's average Standard and CoT lengths are 6410.3 and 6294.9 tokens, respectively. Since the paper defines cost in terms of generated tokens, the two are inconsistent. The paper also has not measured end-to-end latency. The regenerated prefill, full-vocabulary entropy computation, and scheduling costs mean that output-token savings cannot be directly equated with actual speedup.

  2. The most significant accuracy gains in the main experiments come from the Oracle, and the advantage of the actual routing itself is not yet sufficient. The novel contributions and advantages relative to the most closely related work have not been fully validated. The Token Signature paper already studied probability trends of early generated tokens and used Spearman correlation and a learned router to decide whether to apply CoT; its instance-level Dynamic CoT trains a logistic regression on a small number of labeled samples. In contrast, the comparison version in Appendix B.13 of this paper replaces this with entropy trends and a sign-based rule.

Questions for authors

See Weaknesses

1 comment
AuthorsOpenReview ↗

Dear reviewer i8su,

As the discussion period closes tomorrow evening, we wanted to quickly check if you have any remaining questions or concerns regarding our responses. We are fully available to provide further clarifications or additional results if needed. Thank you again for your time and effort!

Best regards,

Authors

Reviewer_Ycy8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper asks a quite interesting question: when (whether) an LLM should spend extra tokens on reasoning.

The core idea of this paper is pipeline-wise: first to let the model generate a short response first, and then look at how its token entropy changes, and final choose between Direct, Standard, and CoT decoding.

They call their method EDRM. Their method uses three signals for decision of strategy: i) cumulative entropy, (ii) its overall trend, and (iii) local fluctuations to decide its strategies. Authors evaluate EDRM on 15 benchmarks and 7 models family. Through their experimental results, authors see clear token savings. However, the reasoning accuracy is very depdenent on the experimental setting. THis may harm the general effectiveness of their paper in my opinion.

Critical strengths

Please note that I am not expert in this domain, please weight-down my evaluation for this paper. Thanks.

  1. I think what might interest the community most in this paper is that the research problem is quite worth studying. Sometimes, we do not always need a long reasoning chain, and extra reasoning does not always improve the answer. I like the idea that that this paper considers both accuracy and token cost, and checks how this trade-off changes across tasks and models.

  2. Besides, use how uncertainty changes in the generation process is a reasonable idea. The heuristic only needs three original-existing features and does not require training another model to select. Which is an plug-and-play method.

  3. What I am less convinced about is whether instance-level routing really pays off compared with just sticking to Standard decoding. The paper focuses largely on how much it saves over always using CoT, but Standard already performs well with fewer tokens. For Llama-3.1-8B, Standard gets 66.08% accuracy using 127.1 tokens, whereas instance-level routing without fallback gets 65.11% using 149.8 tokens. The practical selector also gives lower accuracy while using more tokens than Standard. There are useful gains from dataset-level routing, but I would like a clearer explanation of when making the decision separately for each instance is worth the extra complexity.

  4. The choosen backbone is also a bit of outdated. Why not use Qwen's most up-to-date model?

Critical weaknesses
  1. The biggest concern of mine is that the strongest accuracy results rely on knowing the correct answer. In Table 1, the instance-level methods use Full (Oracle), which checks whether either the routed answer or the Direct answer is correct. This cannot be used in practice. Maybe I misunderstand something.
  2. The experimental comparison is quite weak. The only external adaptive baseline is an implementation inspired by Token-Signature, and and this baseline makes decisions using a simple entropy-trend rule. That is a reasonable starting point, but comparing against this simplified version does not tell us whether the proposed method improves on Token-Signature itself or how it compares with other routing approaches. So, I want to see more comparison if possible.

Again, Please note that I am not expert in this domain, please weight-down my evaluation for this paper. Thanks.

Questions for authors

There might be some errors in the experimental restults table. In Table 3, in almost every setting, Inst-C improves over Inst-E by at most less than 1 percentage points on any dataset. However, in the last rows, wh the reported overall improvement is 2.21 points, from 63.63% to 65.84%. I think it is impossible. Please check it.

6 comments
AuthorsOpenReview ↗

Response 1: Full (Oracle) and practical routing benefits

Thank you for the careful assessment and concrete questions. We first clarify answer selection and substantiate the practical benefits of routing.

EDRM aims to reduce generated tokens while retaining accuracy close to CoT. Its router selects Direct, Standard, or CoT from the early generation trajectory, without accessing the reference answer at inference.

1.1 What Full measures. When routing selects Standard or CoT, Full also generates a Direct candidate and charges its output cost, regardless of whether the routed answer is correct. When the router already selects Direct, its output is reused. Full (Oracle) measures whether at least one candidate is correct; correctness information is used for final selection, not to decide whether to generate the fallback.

This candidate-generation design is applicable when an external verifier is available: programming tasks with automated tests, mathematical/logical tasks with formal checks, or human-reviewed problem solving. The reported oracle accuracy represents ideal selection; practical accuracy and additional judging cost depend on the verifier. Without an external judge, w/o-FB returns the routed answer, while Selected-FB selects candidates using calibrated model signals.

These distinctions and experiments were already included in the submission: §3.3 defines the configurations; §4.7, Table 4 compares Full, Selected-FB, and w/o-FB; Appendix B.6, Tables 20–21 reports MLP with/without fallback; Appendix B.10, Table 25 reports the practical selector. We make the distinction explicit here and support deployment benefits with results that do not use oracle selection.

1.2 Global routing: all 15 tasks. Each model below covers 23,061 questions, aggregated by sample count. Entries are accuracy (%) / mean output tokens; savings are relative to CoT. It's noted that Global-EDRM never adds fallback.

ModelCoTGlobal-EDelta acc. (pp)Savings (%)
Llama-3.2-3B-Instruct60.20 / 251.7059.87 / 113.20-0.3355.03
Qwen2.5-7B-Instruct72.44 / 330.3071.66 / 191.80-0.7841.93
Llama-3.1-8B-Instruct68.11 / 277.8067.27 / 164.40-0.8440.82
Qwen2.5-32B-Instruct79.92 / 306.7079.21 / 194.60-0.7136.55

Global-E saves 36.55%–55.03% of output tokens with accuracy differences of -0.33 to -0.84 pp. The table follows the revised main-table convention, with one-time calibration probing accounted for separately. Including its amortized cost, Llama-3.2-3B still saves approximately 54.24%.

1.3 Instance-level routing: representative tasks. All routers below use w/o-FB. Inst-E uses each task's full evaluation set; MLP uses its held-out test90 split. Each row compares identical questions, and routing costs include the actual probe.

ModelDatasetRouterCoTEDRMSavings (%)
Llama-3.1-8B-InstructPIQAMLP83.15 / 215.9282.25 / 36.8182.95
Llama-3.1-8B-InstructSIQAInst-E70.68 / 214.4270.16 / 43.0479.93
Qwen2.5-7B-InstructMuSRInst-E52.78 / 384.7554.37 / 100.5273.87
Llama-3.1-8B-InstructLSATMLP49.89 / 469.8550.33 / 150.2368.03
Llama-3.2-3B-InstructSIQAInst-E63.36 / 191.0963.61 / 66.5165.20
Qwen2.5-7B-InstructCommonsenseQAInst-E80.84 / 252.3080.84 / 92.4063.38
Qwen2.5-7B-InstructARC-EasyInst-E96.21 / 270.4995.96 / 103.9361.58
Qwen3.5-35B-A3BCommonsenseQAMLP85.65 / 1138.2185.38 / 459.0759.67
Qwen2.5-7B-InstructARC-ChallengeInst-E91.72 / 289.7191.72 / 129.8455.18

For example, Inst-E on Qwen2.5-7B saves 73.87% on MuSR while accuracy rises from 52.78% to 54.37%; on CommonsenseQA, accuracy remains 80.84% with 63.38% fewer tokens. These benefits require neither fallback nor an external judge. MLP is an optional supervised router; it likewise does not access reference answers at inference.

AuthorsOpenReview ↗

Response 2: When instance-level routing is worthwhile relative to Standard

Standard is an explicit action in EDRM. Dataset-level routing suits relatively consistent generation needs; instance-level routing lets questions within a task use different modes. Continuing Standard reuses the probe prefix. The practical criterion is whether branch selection saves more output than probing adds, at an acceptable accuracy cost.

The Llama-3.1-8B heuristic configuration you cite does not outperform Standard. EDRM also provides a learned MLP router that adapts mode selection to the backbone's early entropy trajectory. The following representative configurations come from our completed MLP ablations.

For each model, Standard, CoT, and MLP are evaluated on identical held-out questions across all 15 tasks. The common pool is the intersection of the archived test90 split and the heuristic evaluation set (20,084–20,112 questions per model), aggregated by sample count. All MLP results use w/o-FB, include probe tokens, and require no reference answers at inference. Entries are accuracy (%) / mean output tokens.

ModelMLP configurationStandardCoTEDRM-MLPSavings vs. Standard (%)
Llama-3.2-3B-InstructML-3D59.88 / 194.3060.14 / 251.5958.94 / 167.6413.72
Llama-3.1-8B-InstructML-3D66.16 / 126.6668.17 / 277.3565.71 / 118.756.24
Qwen2.5-7B-InstructSL-3D71.36 / 239.9272.38 / 329.6970.42 / 161.4132.72
Qwen2.5-32B-InstructML-64D79.26 / 238.3079.80 / 306.0379.23 / 224.395.84
Qwen3-4B-Instruct-2507ML-64D79.88 / 514.7780.72 / 639.1677.03 / 414.9619.39
DeepSeek-R1-Distill-Qwen-7BML-64D, β=0.370.66 / 1382.9171.30 / 1306.5369.31 / 1097.8720.61
Qwen3.5-35B-A3BML-64D85.72 / 1895.8985.56 / 1701.0083.85 / 1513.5920.16

ML/SL denote multi-label/single-label training; 3D uses the three entropy statistics, and 64D uses the early entropy trajectory. DeepSeek retains its archived β=0.3 setting. Differences and savings are calculated before rounding.

The first four models save 5.84%–32.72% relative to Standard, with accuracy lower by 0.02–0.94 percentage points. Qwen2.5-7B reduces output from 239.92 to 161.41 tokens, saving 32.72% at a 0.94 pp accuracy cost. For Llama-3.1-8B, MLP saves 6.24% at a 0.45 pp cost, providing a useful instance-level operating point beyond the cited heuristic.

For a tighter accuracy budget, Qwen2.5-7B's existing ML-67D configuration achieves 71.24% / 189.49 tokens: 21.02% fewer tokens than the same Standard baseline, with accuracy lower by only 0.12 pp. The 67D representation concatenates the trajectory and three statistics.

The remaining three models save 19.39%–20.61%, with accuracy costs of 1.35–2.85 pp. These are configuration-dependent trade-offs from existing ablations. Deployment can select among them using separate calibration data and a specified accuracy budget; Standard remains available within the framework. The additional Gemma result saves 12.23% versus Standard at a 0.95 pp cost using a separately reported supervised-calibration rule (author_response_5).

AuthorsOpenReview ↗

Response 3: Token-Signature's contribution, reproduction, and cross-model evidence

3.1 Our contribution: three-way control of reasoning intensity.

Our starting point is how much explicit reasoning a model–question pair needs. This motivates three distinct choices: Direct requests a concise answer, Standard preserves the model's natural generation, and CoT explicitly elicits step-by-step reasoning. We introduce EDRM as a Direct–Standard–CoT (DSC) routing framework, using early entropy dynamics to guide this choice at both dataset and instance levels.

The strong performance of Standard, which you highlight, is precisely why the third action matters. When natural generation already offers a good accuracy–cost balance, the router should be able to preserve it. EDRM can retain Standard and reuse its prefix, choose Direct for concise answering, or invoke explicit CoT according to the observed trajectory.

TS addresses whether to use CoT through a Direct/CoT decision; Standard supplies its probe but is not an available output action.[1] Our contribution makes Direct, natural Standard continuation, and explicit CoT jointly available to entropy-guided reasoning control.

3.2 Original TS mechanism. The submission's TS-inspired adaptation used a shared entropy probe to compare single- and multivariate routing signals. We also evaluate the original mechanism: greedy Standard top-1 probabilities → Spearman correlation with token position → per-task logistic regression → Direct/CoT.[1]

For the seven-model experiment, we fit on up to 50 correctness-disagreement examples per task (seed 42; CoT probability threshold 0.5). Both N=50/64 use the same backbones, prompts, and candidate-generation records as EDRM. TS cost includes the actual probe, min⁡(N,TStandard)\min(N,T_{\mathrm{Standard}}), plus the selected branch's output.

3.3 All seven models and 15 tasks. Each row covers 23,061 questions, weighted by sample count. Entries are accuracy (%) / mean output tokens. Global-E and Inst-E use no oracle answer selection.

ModelTS (N=50)TS (N=64)Global-EInst-E (w/o-FB)
Llama-3.2-3B-Instruct60.83 / 233.1260.78 / 251.0059.87 / 113.2058.87 / 177.06
Llama-3.1-8B-Instruct68.18 / 255.2668.26 / 264.2567.27 / 164.4065.11 / 149.80
Qwen2.5-7B-Instruct72.91 / 331.1372.78 / 356.6971.66 / 191.8071.63 / 240.69
Qwen2.5-32B-Instruct80.12 / 270.2180.32 / 287.4579.21 / 194.6078.99 / 245.62
Qwen3-4B-Instruct-250780.70 / 676.5480.74 / 682.3476.35 / 335.1076.81 / 397.93
DeepSeek-R1-Distill-Qwen-7B71.27 / 1284.3571.27 / 1298.3570.49 / 1342.9068.70 / 1136.57
Qwen3.5-35B-A3B85.63 / 1753.4385.63 / 1767.4385.38 / 1711.3083.48 / 1271.68

TS and Inst-E include instance-level probes; Global-E accounts for one-time calibration separately. This full-suite comparison includes TS fitting examples. Excluding them changes accuracy by <0.25 pp; the full clean table and its full-pool candidate-screening protocol will be included in the revised appendix.

Relative to TS50, Global-E saves 27.98%–51.44% on the first four models at an accuracy cost of 0.91–1.25 pp. On Qwen3.5, Inst-E saves 27.47% at 2.15 pp. These are useful cross-model trade-offs against the original TS signal and learner.

3.4 Independent training-source check. On Gemma 4 E4B-it, TS fits on 50 disagreements per task from the official ARC-Challenge/GSM8K training splits. All methods share 1,587 held-out questions (470 ARC-Challenge, 1,117 GSM8K), separate from those training splits.

MethodAcc. (%)Mean output tokens
Token-Signature (N=50)93.82403.04
Token-Signature (N=64)93.82417.04
Inst (SupCal, w/o-FB)92.75279.50

EDRM saves 30.65% versus TS50 at a 1.07 pp accuracy cost. All methods use Direct capped at 128 tokens and charge actual probes. TS screens 1,393 training questions for its 100 fitting disagreements; EDRM uses a frozen rule with supervised parameter calibration (SupCal) on 150 separate labeled questions. Acquisition budgets are reported separately. MLP results appear in author_response_2; additional all-task baselines in author_response_4.

Reference

[1] Liu, P., Xu, F., & Li, Y. (2025). Token Signature: Predicting Chain-of-Thought Gains with Token Decoding Feature in Large Language Models. ICML, PMLR 267, 39947–39967. Official implementation.

Reviewer_M6VGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • ERDM is an adaptive inference framework to determine when LLMs should use CoT reasoning instead of direct decoding.
  • It works by analyzing token-level entropy trajectories for the first 64 tokens.
  • This can operate as a training-free heuristic to decide different modes: direct, standard, and reasoning. There's also an optional learned MLP-based classifier.
  • Results show reduced token consumption with reasonable accuracy tradeoffs.
Critical strengths
  • The approach is inexpensive because it can calculate statistics (S_H, V_sp, and a_vnr) directly from the decoding steps.
  • The authors evaluated on different model architectures (dense, MoE, distilled) and performed extensive ablations.
Critical weaknesses
  • Next-token entropy is likely very sensitive to system instructions (prompting, few shot examples) and post-training data mixtures. It makes me wonder how generalizable this approach is across different prompting strategies.
  • Llama and Qwen have similar model architectures. It'd be good to check if this approach generalizes to other model families.
Questions for authors
  • In agentic workflows, I believe high entropy might indicate there's missing external knowledge and the high uncertainty can be resolved through tool usage (e.g., agentic search). How does this fit within your framework?
7 comments
AuthorsOpenReview ↗

Response 1: Generalization across prompting strategies (Part 1/2)

We thank the reviewer for recognizing EDRM's low computational overhead, diverse model evaluation, and extensive ablations. Your questions motivated additional validation of prompt transfer, model-family coverage, and tool-based workflows. We address your concerns one by one.

1. Existing coverage: diverse domains, input structures, and answer formats.

We agree that prompting and post-training can affect next-token entropy. Our original evaluation spans 15 datasets with varied domains, input structures, and answer formats:

DatasetInput and prompt characteristics
GSM8K, MultiArithArithmetic word problems with numerical answers.
BBHVaried instructions, including tables with explanatory examples.
CommonsenseQA, StrategyQA, PIQA, SIQACommonsense, yes/no, physical, and social reasoning.
MuSRLong narratives with distributed clues.
ARC-Easy, ARC-Challenge, GPQAMultiple-choice science questions.
FOLIO, ContextHub-Abductive/Deductive, LSATPremises, conditional rules, and logical constraints.

For example, BBH's penguins-in-a-table explains age, weight, and height entries before asking about an updated table.

To directly test prompt sensitivity, we then varied system instructions, final-answer demonstrations, and worked reasoning demonstrations while holding questions and routing parameters fixed.

2. System instructions and answer demonstrations: 20,733 paired questions.

On Gemma 4 E4B-it, we evaluated the same 20,733 held-out questions across all 15 datasets under P0 (original prompt), P1 (an added system instruction emphasizing careful, accurate answers and format compliance), and P2 (question-final-answer demonstrations). All methods share the questions and prompt condition.

We froze two rules after P0 calibration: Inst-C, calibrated without labels on 750 separate examples (50 per dataset; k=0.07k=0.07, θ=26\theta=26), and Inst (SupCal), a supplementary supervised calibration of the same routing rule using 150 separate ARC/GSM8K/FOLIO examples (k=1k=1, θ≈21.626\theta\approx21.626). Neither uses P1/P2 evaluation labels for adaptation.

Each cell reports accuracy (%) / mean generated tokens (Acc. / Tok.), aggregated over questions and including consumed probes. Gemma uses greedy decoding with native thinking disabled; explicit CoT remains available.

MethodP0: OriginalP1: System instructionP2: QA demonstrations
Direct59.06 / 22.9660.96 / 9.8161.53 / 14.23
Standard80.88 / 477.7480.69 / 454.2682.67 / 412.96
CoT80.89 / 544.6380.90 / 533.8183.65 / 515.66
Inst-C (w/o-FB)79.28 / 461.8179.17 / 461.9182.07 / 440.58
Inst (SupCal, w/o-FB)79.92 / 419.3079.73 / 405.9481.86 / 376.76
Inst-C (Full, oracle)81.53 / 482.2182.01 / 469.6584.41 / 453.77
Inst (SupCal, Full, oracle)82.31 / 440.3282.46 / 413.9284.53 / 390.14

As in the paper, w/o-FB evaluates routing alone. Full adds a Direct candidate on non-Direct routes and uses oracle answer selection; all additional Direct tokens are charged. The deployment comparisons below use w/o-FB.

Without prompt-specific retuning, Inst-C saves 13.47%-15.21% of CoT tokens across all three conditions, with accuracy differences of -1.58 to -1.74 percentage points (pp). Inst (SupCal) saves 12.23%, 10.64%, and 8.77% relative to Standard, with respective accuracy differences of -0.95, -0.95, and -0.81 pp. Thus, a rule calibrated once can retain useful cost-accuracy trade-offs under these prompt changes.

The complementary experiment with worked reasoning demonstrations across three model families is reported in author_response_2 (Part 2/2).

AuthorsOpenReview ↗

Response 2: Generalization across prompting strategies (Part 2/2)

Continuing our response to prompt sensitivity (author_response_1), we report the experiment with demonstrations containing worked reasoning.

3. Few-shot demonstrations containing reasoning steps.

We additionally tested demonstrations with worked reasoning on 100 fixed ARC-Challenge and 100 fixed GSM8K questions per model, using each model's frozen P0 rule. These extend P2's final-answer demonstrations to explicit reasoning demonstrations. All methods within each model use the same 200 questions; cells again report Acc. (%) / Tok.

ModelDirectStandardCoTInst-C (w/o-FB)
Gemma 4 E4B-it62.00 / 13.3194.00 / 201.5994.50 / 338.3089.50 / 209.90
Qwen2.5-7B-Instruct61.00 / 10.0989.50 / 121.6693.00 / 216.4090.50 / 154.65
Llama-3.1-8B-Instruct78.50 / 48.0284.00 / 163.4285.00 / 256.6587.00 / 166.30

Inst-C saves 28.54%-37.95% of CoT tokens across these models. Llama improves accuracy by 2.00 pp over CoT while saving 35.20% of tokens; Gemma and Qwen show accuracy differences of -5.00 and -2.50 pp. Gemma's same frozen Inst (SupCal) reaches 92.50% / 161.24 tokens, saving 20.01%/52.34% versus Standard/CoT, with accuracy differences of -1.50/-2.00 pp.

These experiments support EDRM's utility across the tested prompting strategies, with the trade-off depending on the model and calibration. EDRM responds to the entropy trajectory induced by the current prompt, so its routing decisions can change accordingly. For post-training mixtures, our cross-model evaluation provides empirical coverage of different training recipes; their individual causal effects are not isolated here.

AuthorsOpenReview ↗

Response 3: Generalization across model families (Part 1/2)

You suggest evaluation beyond Llama and Qwen. We have added Gemma 4 E4B-it across all 15 benchmarks. The new results extend the evidence for EDRM to another model family: its supervised calibration achieves 12.23%/23.01% fewer output tokens than Standard/CoT, with accuracy differences below one percentage point.

1. Original coverage across families, generations, scales, and model types.

Our original evaluation covers seven models on 15 reasoning benchmarks, spanning multiple generations of Llama (3.1/3.2) and Qwen (2.5/3/3.5) at scales from 3B to 35B. It includes dense instruction-tuned models, the reasoning-distilled DeepSeek-R1-Distill-Qwen-7B, and the MoE model Qwen3.5-35B-A3B. The Qwen2.5-7B/32B pair also provides a comparison across scales within the same generation.

We also compare nine generation or routing strategies: Direct, Standard, CoT, Token-Signature, and EDRM's Global-E/C, Inst-E/C, and MLP variants. This evaluates both task-level and instance-level routing across models with different generation behavior.

2. New Gemma evaluation and overall results.

We evaluate Gemma on the same 20,733 held-out questions across all 15 tasks for every method. Inst-C uses 750 separate unlabeled calibration examples (50 per task); Inst (SupCal) is the supplementary supervised calibration of the same rule using 150 separate labeled ARC/GSM8K/FOLIO examples. Both are frozen before evaluation (author_response_1).

The table uses the original prompt P0 and reports accuracy (%) / mean output tokens (Acc. / Tok.), weighted by sample count.

MethodOverall Acc. (%) / Tok.
Direct59.06 / 22.96
Standard80.88 / 477.74
CoT80.89 / 544.63
Inst-C (w/o-FB)79.28 / 461.81
Inst (SupCal, w/o-FB)79.92 / 419.30
Inst-C (Full, oracle)81.53 / 482.21
Inst (SupCal, Full, oracle)82.31 / 440.32

As in the paper, w/o-FB evaluates routing alone. Full adds a Direct candidate on non-Direct routes and uses oracle answer selection, with every extra Direct token charged. The following deployment comparisons use w/o-FB.

The label-free Inst-C saves 15.21% of CoT tokens with a 1.61 percentage-point (pp) accuracy reduction. Inst (SupCal) saves 12.23%/23.01% versus Standard/CoT, with respective reductions of 0.95/0.97 pp. These results demonstrate useful accuracy-cost trade-offs on the additional model family under both calibration protocols.

Configuration. We use text-only inputs, BF16, and greedy decoding, with native thinking disabled; explicit CoT is induced by "Let's think step by step." Output limits are 128 tokens for Direct and 8,192 for Standard/CoT. Probes use up to 64 tokens; prefixes shorter than 32 retain Standard. Continuing Standard reuses the probe, and switching branches includes the consumed probe tokens.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions of PDF

Dear AC and Reviewers,

Thank you for your constructive feedback. We have revised the manuscript to improve clarity and incorporate the additional evidence discussed in our responses.

Main text. We reorganized the method and experiment sections around EDRM's Direct-Standard-CoT framework, clarified Standard's role and prefix reuse, and distinguished routing without fallback, oracle Full, and the deployable selector. The main results now include Gemma alongside the original seven models, with stronger analysis of when dataset- and instance-level routing improve the accuracy-cost trade-off (Sections 3-4).

Additional experiments and analyses.

  • Generalization and tools (M6VG): Gemma evaluation on all 15 benchmarks; paired prompt changes with frozen routing parameters; MBPP-based code and TextWorld Cooking tasks; and controlled tests of how tool observations affect entropy and subsequent routing (Appendix C.1-C.2, C.6).
  • Practical value and baselines (Ycy8, i8su): Matched held-out comparisons against Standard and CoT; reproduction of Token-Signature's probability-trend logistic regression with 50/64-token probes, plus evaluation using independent training data; and a CAR-style routing comparison on a fixed backbone. Detailed protocols and results are provided in Appendices B.10, B.13, and C.3-C.4.
  • Runtime (i8su): End-to-end latency measurements on two models, including probing and routing overhead, alongside output-token costs (Appendix C.5).

Tables and settings (Ycy8, i8su). We corrected table summaries and clarified aggregation, evaluation splits, model-specific generation limits, and probe/fallback token accounting. The revised tables and settings provide consistent reporting.

These revisions strengthen the evidence for EDRM's core contribution: selectively allocating explicit reasoning to reduce token consumption, with comparable or higher accuracy on selected tasks. We appreciate your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.