- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces DepthBench, a benchmark designed to evaluate the effective reasoning depth and computational efficiency of LLMs. It uses tasks such as function composition and state tracking to assess how well models handle problems requiring increasingly long sequences of dependent operations. The authors find that CoT models achieve substantially greater estimated depth than direct-answer models, but at lower efficiency per layer traversed.
Critical strengths
It addresses an important and timely question, namely, how to measure the ability of LLMs to sustain long-horizon, sequential reasoning beyond conventional benchmark accuracy. DepthBench’s use of controllable tasks with adjustable depth is a promising approach, enabling systematic evaluation as sequential demands increase. The study considers a relatively broad range of models and compares direct-answer and CoT settings, while also attempting to quantify uncertainty and relate its metrics to established benchmarks.
Critical weaknesses
The main weakness is that it is unclear whether DepthBench truly measures reasoning depth or whether models can use shortcuts. The reported 10–300× gains rely on extrapolations that may be unreliable, and the results depend on an arbitrary 5% accuracy threshold. Computational efficiency estimates and comparisons between CoT and direct-answer models may not be fully fair. The paper does not clearly show that its metrics measure reasoning ability beyond general model performance.
Questions for authors
How robust are the reported 10-300x effective-depth gains to the choice of curve-fitting method and accuracy threshold? What is the evidence that DepthBench measures genuine sequential reasoning rather than task-specific shortcuts or general model capability?
Re: Official Review of Submission36181 by Reviewer WWnb
Thank you for taking the time to review our paper. We really appreciated you highlighting the “important and timely question” considered by our paper, the “promising approach” for “systematic evaluation” we propose, and the “relatively broad range of models” involved in our experiments. We believe that your main concerns may have actually been addressed within our submission, but were not displayed prominently enough. We have also conducted deeper analysis to make sure each aspect of your questions are fully addressed. We hope that you will consider changing your recommendation for our paper with these updates in mind.
Main Weakness: Unclear if DepthBench truly measures reasoning depth or whether models can use task-specific shortcuts
Thank you for highlighting that this is your main concern. We were wondering if you had caught the line in our submission at the end of Section 3.2 saying that “We discuss the analysis that we performed to assure that sequential computations are performed without shortcuts skipping steps in Appendix C.” Indeed, Appendix C (unaltered from the submitted version) is entirely devoted to looking deeper into this question. For example, we demonstrate through inspection of the CoT itself that correct answers actually include each intermediate step within the CoT. We consider how the number of passes through the intermediate steps varies across model families and we discuss if problem specific shortcuts can exist for the arithmetic task. Does this inspection of the intermediate steps within the CoT itself address your concern? If so, we would be happy to revise the paper to showcase this fact more prominently.
Robustness of effective depth gains to curve-fitting method and accuracy threshold.
Thank you for bringing up this concern. We should note that we did show robustness to thresholds of 5%, 10%, and 15% in our submitted draft and we did provide an analysis of the robustness of the BNSL in Appendix D of the submission. However, we have used some of the extra allotted space in our revised draft to address your concern in further detail. The robustness paragraph at the end of Section 5.3 has been extended to discuss our new Appendix E, which looks at these specific questions. In Table 12, we consider four other alternatives to the BNSL curve-fitting method and report on the amount of variation they can account for. Then we report on the minimum and maximum depth ratios derived for each performant curve-fitting method at each accuracy threshold in Table 13. We find that the gains are still over 200x even with a 50% accuracy threshold. Moreover, the other curve fitting methods actually make the ratios even a little bit higher. For example, they are sometimes showing even an over 400x gain at a 5% accuracy threshold. There are no extrapolated values for accuracy threshold over 10%, so extrapolation is also not a significant factor in the results.
”Computational efficiency estimates and comparisons between CoT and direct-answer models may not be fully fair.”
We want to make sure we address this point, but your review did not provide us much context regarding, which comparisons may not be “fully fair” and why. If your concern is the post-training procedure inflating the multiplier, please see our response to reviewer RvET21 on this matter. We believe our rebuttal should have addressed each weakness mentioned in your review and we would really appreciate more clarity regarding the remaining weaknesses if your opinion of the paper remains unchanged.