acceptodds
Under review as a conference paper at ICLR 2027

How Deep is Your Thinking? Evaluating the Depth and Efficiency of Long Horizon Reasoning

Abstract

Large language models (LLMs) with chain-of-thought reasoning capabilities have driven a step change in the classes of problems that are solvable with foundation models. In particular, the vanilla Transformer architecture is limited in the number of sequential computations it can perform within a fixed number of layers, whereas chain-of-thought reasoning greatly expands this capability in a manner that adapts to the problem complexity. However, the precise extent of this increase in capability, as well as the efficiency with which it is achieved, has not yet been systematically explored. To address this gap, we propose a new evaluation suite, DepthBench. DepthBench aims to measure the effective reasoning depth of models i.e. the number of sequential computations they are reliably capable of performing during inference. We focus on the domains of function composition for arithmetic and state tracking for planning, where we have direct control over problem complexity in terms of the required sequential computations necessary for success, allowing us to study how LLMs with and without chain-of-thought reasoning perform as a function of this complexity. We find that prominent open-source LLMs exhibit increases of one to three orders of magnitude in the achievable sequential computation depth, depending on the domain. We also find that models in a medium parameter range are the most efficient in their use of layers to increase depth, both with and without chain-of-thought. DepthBench enables the systematic evaluation of the effective depth of LLMs that employ adaptive computation and helps foster well-grounded innovation on the limits and efficiency of these models.

Primary area:
datasets and benchmarks
Keywords:
Chain-of-Thought, Deep Thinking, Long Horizon Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WWnbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces DepthBench, a benchmark designed to evaluate the effective reasoning depth and computational efficiency of LLMs. It uses tasks such as function composition and state tracking to assess how well models handle problems requiring increasingly long sequences of dependent operations. The authors find that CoT models achieve substantially greater estimated depth than direct-answer models, but at lower efficiency per layer traversed.

Critical strengths

It addresses an important and timely question, namely, how to measure the ability of LLMs to sustain long-horizon, sequential reasoning beyond conventional benchmark accuracy. DepthBench’s use of controllable tasks with adjustable depth is a promising approach, enabling systematic evaluation as sequential demands increase. The study considers a relatively broad range of models and compares direct-answer and CoT settings, while also attempting to quantify uncertainty and relate its metrics to established benchmarks.

Critical weaknesses

The main weakness is that it is unclear whether DepthBench truly measures reasoning depth or whether models can use shortcuts. The reported 10–300× gains rely on extrapolations that may be unreliable, and the results depend on an arbitrary 5% accuracy threshold. Computational efficiency estimates and comparisons between CoT and direct-answer models may not be fully fair. The paper does not clearly show that its metrics measure reasoning ability beyond general model performance.

Questions for authors

How robust are the reported 10-300x effective-depth gains to the choice of curve-fitting method and accuracy threshold? What is the evidence that DepthBench measures genuine sequential reasoning rather than task-specific shortcuts or general model capability?

1 comment
AuthorsOpenReview ↗

Re: Official Review of Submission36181 by Reviewer WWnb

Thank you for taking the time to review our paper. We really appreciated you highlighting the “important and timely question” considered by our paper, the “promising approach” for “systematic evaluation” we propose, and the “relatively broad range of models” involved in our experiments. We believe that your main concerns may have actually been addressed within our submission, but were not displayed prominently enough. We have also conducted deeper analysis to make sure each aspect of your questions are fully addressed. We hope that you will consider changing your recommendation for our paper with these updates in mind.

Main Weakness: Unclear if DepthBench truly measures reasoning depth or whether models can use task-specific shortcuts

Thank you for highlighting that this is your main concern. We were wondering if you had caught the line in our submission at the end of Section 3.2 saying that “We discuss the analysis that we performed to assure that sequential computations are performed without shortcuts skipping steps in Appendix C.” Indeed, Appendix C (unaltered from the submitted version) is entirely devoted to looking deeper into this question. For example, we demonstrate through inspection of the CoT itself that correct answers actually include each intermediate step within the CoT. We consider how the number of passes through the intermediate steps varies across model families and we discuss if problem specific shortcuts can exist for the arithmetic task. Does this inspection of the intermediate steps within the CoT itself address your concern? If so, we would be happy to revise the paper to showcase this fact more prominently.

Robustness of effective depth gains to curve-fitting method and accuracy threshold.

Thank you for bringing up this concern. We should note that we did show robustness to thresholds of 5%, 10%, and 15% in our submitted draft and we did provide an analysis of the robustness of the BNSL in Appendix D of the submission. However, we have used some of the extra allotted space in our revised draft to address your concern in further detail. The robustness paragraph at the end of Section 5.3 has been extended to discuss our new Appendix E, which looks at these specific questions. In Table 12, we consider four other alternatives to the BNSL curve-fitting method and report on the amount of variation they can account for. Then we report on the minimum and maximum depth ratios derived for each performant curve-fitting method at each accuracy threshold in Table 13. We find that the gains are still over 200x even with a 50% accuracy threshold. Moreover, the other curve fitting methods actually make the ratios even a little bit higher. For example, they are sometimes showing even an over 400x gain at a 5% accuracy threshold. There are no extrapolated values for accuracy threshold over 10%, so extrapolation is also not a significant factor in the results.

”Computational efficiency estimates and comparisons between CoT and direct-answer models may not be fully fair.”

We want to make sure we address this point, but your review did not provide us much context regarding, which comparisons may not be “fully fair” and why. If your concern is the post-training procedure inflating the multiplier, please see our response to reviewer RvET21 on this matter. We believe our rebuttal should have addressed each weakness mentioned in your review and we would really appreciate more clarity regarding the remaining weaknesses if your opinion of the paper remains unchanged.

Reviewer_49f4OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

DepthBench evaluates LLMs on arithmetic composition and puzzle state tracking with increasing operation counts. It compares direct answering with CoT using fitted accuracy thresholds and compute-normalized depth metrics. Experiments across several model families show large CoT gains, supported by threshold checks, bootstrap intervals, and external benchmark correlations.

Critical strengths
  1. Controlled tasks. Automatically generated problems with exact answers allow systematic evaluation as operation counts increase, reducing reliance on subjective scoring.

  2. Within-model comparisons. Comparing direct answering and CoT with the same model provides stronger evidence of the inference procedure’s effect than comparisons between unrelated models.

  3. Efficiency analysis. Measuring both capability and resource use highlights that the model reaching the greatest depth may not offer the best performance per unit of compute.

  4. Validation efforts. Bootstrap intervals, threshold sensitivity checks, and intermediate-trace analysis strengthen the empirical evidence beyond the headline multipliers.

Critical weaknesses
  1. Depth interpretation: Prescribing (d) operations does not establish that (d) sequential computations are necessary. Appendix C acknowledges algebraic shortcuts; observing step-by-step CoT does not prove a computational lower bound.

  2. Metric validity: A fitted 5% success threshold does not characterize reliable execution, and some estimates extrapolate beyond tested depths. The expected-longest-correct-answer metric also needs justification when guessing leaves nonzero success probabilities.

  3. Compute accounting: Table 1’s direct-answer efficiencies appear to use (k/L), despite Section 4.2 including answer tokens in the denominator. The stronger efficiency claims also need comparisons at matched task depth and accuracy.

  4. Reporting gaps: Qwen3-1.7B’s direct Hanoi threshold changes from 1.18 in Table 1 to 2.6 in Table 8. Varying parsed-item counts and unclear treatment of malformed outputs further complicate interpretation.

Questions for authors
  1. Depth definition: Is (d) prescribed execution length or a minimum computational depth? What justifies the latter interpretation and the assumption of zero direct-answer success beyond (L)?

  2. Threshold robustness: Do the rankings and CoT gains persist at 50% and 90% accuracy and under simpler monotone fits? Please distinguish measured threshold brackets from extrapolated estimates.

  3. Efficiency calculation: How are answer tokens and reasoning-token averages included in Tables 1–3? Does the efficiency conclusion hold when comparing the same task depth and target accuracy?

  4. Scoring and consistency: Can you reconcile the conflicting tables and depth grids, and report attempted, parsed, and correct counts? Are malformed and truncated responses counted as failures?

2 comments
AuthorsOpenReview ↗

Re: Official Review of Submission36181 by Reviewer 49f4 (Part 1)

Thank you for taking the time to review our paper. We really appreciate that your review highlights our systematic evaluation with reduced reliance on subjective scoring, the strong evidence provided by our within-model comparisons, and our efforts to validate the key empirical findings of our experiments. We are grateful for your thorough analysis of our results and the insightful low-level questions that you brought up. We have attempted to address each of the concerns outlined in your review, and hope that you will consider updating your evaluation of our paper if we have resolved them. Most of the issues raised resulted in straightforward edits that we believe were fairly easy for us to address without changing the core substance of the paper.

What justifies considering d the minimum computational depth and what justifies the zero direct-answer success beyond L?

Thank you for raising your confusion on this point as this is a critical aspect of the motivation of our paper. The problems we consider are algorithmic problems that can only be solved reliably with d sequential computations one after another. In principle sequences could be memorized within the weights of the network for particular examples / domains, which creates a shortcut highly specific to the particular prompt / problem. This is why we validate (as you mentioned) that the CoT includes the actual intermediate solutions in its traces. This also becomes less likely as a use of resources in the neural network as you look at problems with a wider scope of variation. Just to be clear, we don’t expect zero success beyond L, which is why we subtract off the expected performance of a random guess when computing our expected depth estimates. In our experiments, we never saw a model get above chance performance at a problem depth of greater than L layers, which is consistent with this assumption. A feed-forward network can by definition do a maximum of L computations on its input. While it cannot be rigorously proven for a particular problem, the only reason is because of the possibility that some of the sequential computations are done in weights of the network that do not actually causally depend on variations in the input. This is a very severe form of memorization that is not conducive to solving algorithmic problems with diverse inputs and outputs. We believe it is an extremely mild assumption that this generally holds, as is evidenced by our experiments.

”The expected-longest-correct-answer metric also needs justification when guessing leaves nonzero success probabilities.”

In footnote 1 of the submission, we mentioned that we subtract off the baseline of uniform chance guessing. We only did it in Towers of Hanoi because of how small the probability is in the other domains. Would it help if we also addressed this in the discourse around Definitions 1 and 2?

Fit threshold and functional form robustness analysis

Thank you for bringing up this concern. We should note that we did show robustness to thresholds of 5%, 10%, and 15% in our submitted draft and we did provide an analysis of the robustness of the BNSL in Appendix D of the submission. However, we have used some of the extra allotted space in our revised draft to address your concern in further detail. The robustness paragraph at the end of Section 5.3 has been extended to discuss our new Appendix E, which looks at these specific questions. In Table 12, we consider four other alternatives to the BNSL curve-fitting method and report on the amount of variation they can account for. Then we report on the minimum and maximum depth ratios derived for each performant curve-fitting method at each accuracy threshold in Table 13. We find that the gains are still over 200x even with a 50% accuracy threshold. Moreover, the other curve fitting methods actually make the ratios even a little bit higher. For example, they are sometimes showing even an over 400x gain at a 5% accuracy threshold. There are no extrapolated values for accuracy threshold over 10%, so extrapolation is also not a significant factor in the results. We have made sure to denote extrapolated confidence intervals with *s in our update to Appendix D. We stopped the threshold at 50% rather than 90% because models must be excluded when they don’t even meet the threshold at a depth of 1 and we can only robustly compare across thresholds when the full model suite is considered.

AuthorsOpenReview ↗

Re: Official Review of Submission36181 by Reviewer 49f4 (Part 2)

Table 1 / Section 4.2: Should Direct Answer Efficiency Be k/L or include the answer?

Thank you for this note about Section 4.2. Yes. It makes sense to exclude the answer in our analysis of efficiency because generally speaking you cannot still be solving the problem during the answer generation process. While in some cases, it may be possible to fill in the first few tokens before discovering the rest, it is not a robust solution that relies on token-perfect execution that must mean the problem has been essentially solved if we generally see this capability. We have updated the discourse in Section 4.2 accordingly to make sure this is clear. In Tables 1, 2, and 3 the divisor in total layers traversed is consistently L x (N + 1) where N is the number of CoT tokens (e.g. N=0 for direct answers).

Unclear treatment of malformed outputs and reporting gaps between main text and Appendix D

Thank you for mentioning this. As you pointed out, the submitted version of Appendix D included inconsistent treatment of parsed items. This issue has been addressed now so that everything corresponds with the main text e.g. 1.18 in Table 1 is 1.2 in the updated Table 10. We do not consider malformed outputs or those that can’t be parsed at all. They are not right or wrong, the model must regenerate until it can be parsed. We made the mistake of including these items in our original analysis in Appendix D. We have resolved this issue now and parsed items is no longer a consideration in the analysis given our simple rule for only including those that do parse correctly.

Reviewer_RvETOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DepthBench, an evaluation framework designed to measure the effective reasoning depth (i.e., the number of reliable sequential computations) of Large Language Models (LLMs) with and without Chain-of-Thought (CoT). The authors construct synthetic tasks in two domains: function composition (arithmetic) and state tracking (Tower of Hanoi and 15 Puzzle), where problem complexity is strictly parameterized by the required sequential steps d. The paper proposes rigorous metrics for effective depth and computational efficiency (layers traversed and FLOPs). Through extensive experiments on Qwen, Gemma, and GPT-OSS model families, the paper demonstrates that CoT expands effective reasoning depth by 1 to 3 orders of magnitude, but at a cost of being 30x to 100x less efficient per layer traversed compared to direct answering.

Critical strengths
  1. Measurement of an open question: The paper turns the theoretical concept of "sequential computation depth" into a measurable benchmark. Metrics like the "expected depth of the longest correct answer" and "depth at ϵ-precision" can handle the randomness in LLM outputs. They are cleverly designed and fairly robust.

  2. Findings on efficiency: The paper clearly shows that while CoT increases the upper limit of reasoning depth a lot (up to about 300x), its efficiency per layer traversed is very low (30 to 100 times less efficient). This is important for the community: it shows that the current approach of scaling test-time compute via CoT is brute-force (relying on stacking compute) and mathematically inefficient. It will also push people to study other architecture alternatives, like latent reasoning and looped transformers.

Critical weaknesses
  1. Limited task coverage: The benchmark only uses arithmetic function composition and two grid-world planning tasks. While this allows for precise control over the number of steps, it is unclear whether the measured "effective depth" can transfer to natural language tasks (e.g., multi-hop QA, legal reasoning). In these tasks, sequential dependencies rely on complex semantics rather than rigid symbolic rules.

  2. Measuring "Direct Answer" depth for CoT-native models may be inaccurate: The authors note that for some models (like Gemma 4 and GPT-OSS), it is difficult to make them give direct answers, and they often still reason internally. If a model's post-training (RLHF/RL) strongly biases it toward CoT, forcing it to answer directly might just measure its ability to suppress its learned reasoning habits, rather than its true architectural depth limit. This could inflate the apparent multiplier of CoT over direct answers.

Questions for authors
  1. Natural Language Generalization: Given the correlation analysis in Table 6 with benchmarks like GPQA and MMLU-Pro, do you believe the effective depth measured by DepthBench is a causal factor for performance on natural language reasoning? Could a model have high depth on DepthBench but still fail at deep multi-hop QA due to semantic bottlenecks?

  2. Direct Answer Artifacts: For models like GPT-OSS where CoT cannot be turned off, how did you ensure that the measured "direct" depth is a true reflection of the model's layer limit rather than a failure to follow instructions? Have you considered prompting with a few-shot direct answer exemplar to mitigate this?

  3. Efficiency Metrics: You use FLOPs as a proxy for computation. How do you account for the quadratic attention costs and KV-cache memory overheads that grow with context length during CoT, which are typically the actual bottlenecks in long-horizon reasoning?

1 comment
AuthorsOpenReview ↗

Re: Official Review of Submission36181 by Reviewer RvET

Thank you for your thoughtful review of our paper. We really appreciate that you highlighted the quality of our presentation and the fact that you learned something new from our paper. We were also pleased to see your praise for the clever design of our metrics and the importance of our empirical findings to the community. We have attempted to address the key weaknesses and questions you raised below.

Weakness / Question 1: unclear whether the measured "effective depth" can transfer to natural language tasks such as deep multi-hop QA

Thank you for bringing up this distinction. It motivated us to add experiments explicitly looking at the correlation between our measurements of effective depth and performance on common benchmarks for multi-hop QA. This is very relevant for the aims of DepthBench given how common discussions of reasoning depth are in this area of the literature.

We have now performed comprehensive experiments on the MuSiQue, HotpotQA, 2WikiMultihopQA, and FRAMES benchmarks. In our update to the paper, Table 4 has been extended with a second part to highlight these experiments. This was Table 6 in the original submission because we moved it to the main text with the extra space allotted. We also added further detail explaining these results in the new Section 5.4. We find that the correlation between multi-hop QA performance with effective depth measurements is always highly statistically significant with correlation strengths ranging from 0.86 to 0.96.

While DepthBench does correlate with deep multi-hop QA performance in practice, we agree with the premise of your question. We have added a note at the end of Section 5.4 to further highlight this “dependence on training.” The effectiveness of a model’s reasoning will indeed naturally be domain dependent. We know that CoT models have the capacity for much greater depths of reasoning and our results highlight that they do deliver on this promise in practice for multiple domains (although inefficiently). We believe DepthBench serves as a useful benchmark for comparing the capacity for reasoning that different architecture’s display. On the other hand, the relative competencies of models by domain will naturally be heavily tied to the training data.

Weakness / Question 2: How were direct answers enforced and did you try few shot prompting?

We did try few shot prompting in our initial experiments and did not find that it made a substantial difference in the number of direct answers. We decided not to include these results as it requires a significant change to the prompt structure, which makes comparisons more difficult with the CoT models that do not need it. In our submission we included detail about the specific prompts in Appendix A. Our basic approach was to modify the prompt and system prompt to highly encourage direct answers. Then for our analysis, only actually direct answers (determined by the number of the tokens and the content of the tokens in the answer) were considered. As a result, our direct answer results are guaranteed to only include actual direct answers.

If a model’s post-training biases it toward CoT this could inflate the apparent multiplier of CoT over direct answers.

Thank you for raising this insightful point. This is part of the reason that we thought it was important to include the Qwen2.5 results in Table 1. These models were instruction tuned without CoT training in a similar overall style to the Qwen3 models that were trained with CoT (but have CoT turned off in Table 1). It can be seen that results are very similar in terms of the measured effective depths between the two model families in the direct answer setting. If the Qwen2.5 results were used as the denominator in the multiplier calculations, it would change the conclusions very little. These results help us build confidence against your concern, which is about as confident as we can be without being able to intervene on the post-training procedure itself. We had hoped that the OLMO-2 models would give us more insight into the roles of each training phase, as intermediate checkpoints are available. However, as mentioned at the start of the Section 5, these models did not perform competitively enough on these tasks for us to include them.

Question 3: How do you account for efficiency with respect to KV-cache memory overheads?

Thank you for bringing up this important point as well. As we agree that this is also an important aspect of consistency to consider, we have added a new efficiency metric based on KV-cache memory to the revised paper. We discuss the new metric alongside the other efficiency metrics in Section 4.2 and added Tables 7 and 8 in Appendix B to showcase these results, which we discuss in Sections 5.1 and 5.2. Overall, memory-based depth efficiency shows a slight bias towards smaller models, but not quite as severely as the compute-based efficiency metric.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.