- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies prompt-induced resource exhaustion in black-box reasoning models. The authors construct Reasoning-Bench, a benchmark covering different task types, complexity levels, and task constraints, and analyze the model behaviors elicited by its prompts. Based on this analysis, they propose ReasoningSink, a framework for generating resource-exhaustion prompts. The authors report higher completion-token consumption than the comparison baselines across the evaluated models.
Critical strengths
- This paper studies an interesting and useful security question: how attackers can exploit normal reasoning tasks to exhaust computational resources, which may serve as a reference for future work on defenses against such attacks.
Critical weaknesses
- My main criticism is that the paper provides limited insight and it is unclear how its findings could guide future work.
- The analysis of Reasoning-Bench is somewhat disappointing. Many findings are intuitive and unsurprising, while others appear specific to the constructed benchmark. The paper does not show whether these observations reflect general model behavior that can be validated on other existing datasets. I discuss these concerns in more detail in the Questions section.
- ReasoningSink also appears closely tied to the authors' own benchmark, with insufficient evidence of generalizability beyond this setting. Moreover, adding more task constraints to prolong model reasoning appears to be an intuitive strategy. This does not seem to offer a compelling methodological contribution.
Questions for authors
- Regarding Main Results on Reasoning-Bench: The prompts in the benchmark are generated directly by LLMs, and each investigated condition does not appear to be rigorously controlled. This makes many of the experimental conclusions less convincing for me.
- Finding 1: How do you define task complexity? Before testing the models, how do you determine whether a prompt, mathematics problem, or coding task is easy or complex for a model?
- Finding 2: First, the benchmark covers a limited range of task categories. Second, since the questions are LLM-generated, their quality and complexity are difficult to control consistently across categories. How can you directly attribute the observed differences in resource consumption to the task domain?
- Finding 3: "When a task is declared solvable, reasoning models tend to 'execute first and judge later.'" I find this claim confusing. How can a model be expected to assess solvability without performing any reasoning? Can you provide some representative cases whose unsolvability is apparent without reasoning, but that still lead all evaluated models to perform extensive reasoning?
- Finding 4: Why does round-by-round iteration constitute "continuous decomposition," and why is repeated computation described as "local exploration"?
- Overall, these analyses share a common problem: the claimed rules are derived by changing factors that already seem likely to affect reasoning effort, followed by behavioral analysis of only a few cases and models to explain the observations. I do not find this approach convincing, and there is a risk of overinterpreting the results.
- A stronger approach would be to use existing benchmarks instead of relying on an LLM-generated dataset. The relevant variables could be controlled using existing annotations, such as difficulty labels, or through a rigorous method whose validity is clearly justified. The experiments and analysis would then be more convincing.
- Regarding ReasoningSink:
- My main concern is that much of the methodology appears specifically tailored to Reasoning-Bench. Does the method also work on other established datasets to demonstrate its generalizability?
- For truly black-box model services, such as those offered by OpenAI and Anthropic, where the chain-of-thought length is not observable, how would Execution-Feedback Optimization be applied?
- Other issues and questions:
- Figures 1, 2, and 4 are too small, making the text difficult to read.
- Is the prompt shown in Figure 2(b) the actual template used to generate the prompts in Reasoning-Bench?
- What is the intended title of Figure 3(c)? It appears to be rendered incorrectly.
- In Section 4.3, line 355, what exactly is the input ? What does it look like in practice? Could you provide a concrete example?
- Why does performance decline after Round 3 in Table 8?