- Rating
- 2
- Confidence
- 1(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates an important, but under-explored problem: the allocation of compute resources in traditional scenarios may not be aligned with real deployments. The authors further formulate the problem as an optimization problem and propose consequence-aware test-time compute allocation. Specifically, the authors try to predict how serious it will be if the task fails, and then allocate compute based on this. Experiments suggest good performance of their proposed allocation.
Critical strengths
The core idea of this paper is intuitive and interesting. In fact, it is quite normal to treat allocation equally. Therefore, few previous works have explored this topic. The authors start with a "common" phenomenon and then present interesting assumptions, explorations, and findings on compute allocation. These contributions are just as important as works aiming at the "state-of-the-art".
The evaluation consists of baselines of three levels, human evaluation and extensive studies such as boundary conditions, etc. These experiments support the authors' claims.
Critical weaknesses
-
I am afraid that the core weakness is that the technical contribution of the methodology seems to be weak. The authors mainly redefine the objective and apply the simple predicted consequence as the ranking method. This design is helpful in proving the concepts, but the technical contribution may be weak. Additionally, there is no new/unique evaluation data proposed under the authors' settings. After reading the paper, I feel the idea is interesting, but the methodology is not attractive enough.
Maybe the style of this paper is a software-engineering style (auditing), and it may also be a good choice to submit to some conferences in software engineering. Otherwise, you may redesign the methods section (like new training methods based on the observations, even on a toy set). Moreover, there exists a gap between the theory and the real deployment. -
The consequence label, which is a core design in the paper, is indeed a judge-defined proxy. There also exists a gap between the real deployment cost and this design. In sum, I feel that the authors try to solve an important problem; however, they may limit themselves to a small space for evaluation/trying.
Questions for authors
Besides the weaknesses, here are some questions (currently I give a slightly negative recommendation, and after rebuttal I may reconsider whether this paper should be accepted):
The evaluations are mainly conducted on SWE-bench. Will the task designs in this benchmark be "more suitable" for your consequence-aware design? (I see that, in SWE-bench, there are some tasks quite hard, and even large compute is allocated, the task may still fail) In other words, can your design be generalized to tasks in other benchmarks?
Are the consequence labels reliable enough? That is to say, is the "consequence" aligned with real software behaviors? This should be carefully confirmed. What is the background of the human annotators? You mention that they are CS Ph.D. students; how to make sure that their annotations really reflect the risks? (Self-consistency is not the validity of the label construction, and moderate agreement between annotators can be seen) Is this study IRB-approved?
A small suggestion: you may add figures (like the main pipeline) in your paper to help readers better understand the work.
9 comments
Response to Reviewer N5ba (Part 2 of 3)
W2 and Q2. Label reliability and the human annotations
Indeed, the consequence label is not the directly observable deployment loss, and we do not treat the LLM judge's output as real deployment risk. The key question is whether the labels agree with real software maintenance behavior that does not involve the LLM judge. To further investigate this issue, we explore (1) maintainer behavior that involves neither an LLM nor our annotators, (2) the annotation protocol and agreement, and (3) whether the conclusions hold under human labels.
(1) Maintainer behavior
The people best placed to judge real risk in these projects are the maintainers, and their revealed behavior on SWE-bench Lite follows our classes (Appendix D). Backporting means that maintainers port a fix to older maintained release branches, a real decision that does not involve our LLM judge. The backport rate rises monotonically with our consequence class, from 8.3% (class 0) to 20.6% (class 1) and 41.9% (class 2), with trend . Higher-consequence issues are thus the ones that maintainers more often treat as important enough to fix in older releases as well.
(2) Annotation protocol and agreement
The three annotators were one author, one non-author CS PhD student and one non-author CS graduate student. They saw no gold patches, diffs or each other's labels, completed a nine-task calibration, and applied a rubric defined by concrete failure types (for example, documentation errors for class 0, wrong return values for class 1, silent data corruption or permission bypass for class 2), which needs no knowledge of a specific deployment. We agree that self-consistency is not validity. Agreement between annotators is limited (Fleiss' ), so we use human labels only as a supplementary reference. Still, the human majority agrees with our predictor (), and consequence routing beats difficulty and random routing when human labels define the weights (55.2 vs. 75.8 and 67.5, Table 8).
(3) Conclusions under human labels
| Analysis on human-labeled tasks | Result |
|---|---|
| Consequence vs. difficulty, Spearman (75 Lite tasks) | |
| Issue-only predictor vs. human majority | |
| With-patch judge vs. human majority | |
| Loss under human weights, difficulty / random / consequence oracle (Table 8) | 75.8 / 67.5 / 55.2 |
The issue-only predictor agrees with humans as well as the with-patch judge does, consequence stays orthogonal to difficulty, and consequence routing beats difficulty and random routing when humans define the weights. This indicates that the main conclusions do not depend on the LLM judge.
IRB. The annotation involved only severity judgments on public software issue descriptions, and no sensitive personal information about the annotators was collected. We will disclose the ethics-review status accurately in the revision, in line with our institution's requirements and records.
Q1. Is SWE-bench especially favorable, and does the approach generalize?
We test the proposed framework on SWE-bench because software engineering is a well-studied application of AI, where inference-time resource allocation is important.
The point of the paper is therefore not that consequence always beats difficulty, but whether consequence is a useful deployable signal when the marginal gain cannot be estimated in advance. To further examine the proposed framework's generalization, we examine it from two angles, (1) comparisons on SWE-bench that do not depend on this handicap, and (2) boundary conditions across five pools.
To answer this, we evaluate on SWE-bench Lite and Verified, and test the generalization ability of the method on BIRD, FinQA and MATH. We report all results, including those where our method does not win.
(1) When does consequence routing apply? (Appendix O)
We compare five pools on three conditions, namely whether error costs differ across tasks, whether extra compute helps, and whether difficulty already identifies the tasks that gain from it.
Table R2. Applicability conditions across five pools (a top-25% weight share of 0.25 means equal costs, and is the correlation between difficulty and the gain from extra compute).
| Pool | Tasks | Top-25% weight share | Premium headroom | Better deployable signal | |
|---|---|---|---|---|---|
| SWE-bench Lite | 300 | 0.42 | −0.78 | Yes | Consequence |
| SWE-bench Verified DB/ORM | 140 | 0.39 | −0.38 | Yes | Consequence trends positive, not significant vs. predicted difficulty |
| BIRD | 150 | 0.35 | +0.06 | Limited | No clear advantage, near random |
| FinQA | 150 | 0.36 | +0.01 | Almost none | No clear advantage |
| MATH | 150 | 0.50 | +0.26 | Large | Difficulty |
Response to Reviewer N5ba (Part 3 of 3)
On SWE-bench, error costs differ and difficulty points away from the tasks that extra compute can save, so consequence gives a better ranking. This hurts difficulty routing rather than favoring ours, since our method also beats random on Lite (9.3% lower loss). The gain is significant on Lite and positive but not significant on Verified.
(2) BIRD, where consequence differences are limited
Only 3 of 150 BIRD tasks are high-consequence, so there is little to prioritize and our method is close to random under the main weights. When GPT-4o assigns higher stakes to some queries, our method beats difficulty routing by 3.1%, so the result depends on how costs are defined. These are point estimates.
Table R3. BIRD allocation results (weighted loss, lower is better).
| Consequence weights | Strategy | Weighted loss | vs. difficulty-aware |
|---|---|---|---|
| Qwen | Difficulty-aware | 66.67 | 0% |
| Qwen | Random | 68.27 | −2.4% |
| Qwen | Consequence-aware (question-only) | 67.33 | −1.0% |
| Qwen | Consequence-aware (with-gold oracle) | 65.67 | +1.5% |
| GPT-4o | Difficulty-aware | 75.00 | 0% |
| GPT-4o | Random | 75.80 | −1.1% |
| GPT-4o | Consequence-aware (question-only) | 72.67 | +3.1% |
| GPT-4o | Consequence-aware (with-gold oracle) | 66.67 | +11.1% |
(3) FinQA, where extra compute does not help
The stronger model is no better than the cheap one on FinQA (64.2% vs. 66.2%), so no routing method can help, even though our predictor identifies high-consequence tasks well (, no class-2 task predicted as class 0).
(4) MATH, where difficulty already points to the gain
On MATH, harder problems are exactly where the stronger model helps (), so difficulty routing is the better choice under the original costs.
(5) Larger cost differences reverse the result on MATH
When stakes are assigned independently of difficulty and the gap between high and low stakes grows, consequence routing overtakes difficulty routing on MATH between 10× and 30×, and it stays ahead on FinQA at every tested gap (point estimates). Positive values in Table R4 mean lower loss than difficulty routing.
Table R4. Declared-stakes sweep on MATH and FinQA.
| Cost gap between high and low stakes | 1× | 3× | 10× | 30× | 100× | 10³× | 10⁶× |
|---|---|---|---|---|---|---|---|
| MATH | −28.3% | −18.4% | −6.5% | +3.2% | +11.5% | +20.2% | +24.9% |
| FinQA | +2.9% | +3.6% | +4.4% | +5.1% | +5.7% | +6.3% | +6.6% |
(6) Live run (Appendix Q)
In a live run on 300 FinQA and MATH tasks with real model calls and a shared budget, the system works as intended. Difficulty routing has the lowest loss, as expected from MATH, but its difference from our method is not significant (95% CI [−9,493, +2,301]) and the two used different numbers of premium calls (76 vs. 58). Our method still lowers loss by 18.8% relative to random.
Table R5. Live streaming run on FinQA and MATH.
| Policy | Weighted loss | Reduction vs. random | Premium calls | Accuracy |
|---|---|---|---|---|
| Random | 24,426 | 0% | 23.0% | 70.0% |
| Difficulty | 16,432 | 32.7% | 25.3% | 70.7% |
| Consequence | 19,831 | 18.8% | 19.3% | 68.7% |
| Verifier-gated sequential | 21,627 | 11.5% | 5.0% | 70.0% |
(7) Summary
Our conclusion is therefore not that consequence routing replaces difficulty routing on all tasks, but that the consequence of failure is an independent dimension for allocating compute. Its value depends on whether error costs differ, whether extra compute helps, and whether difficulty already points to the gain. In the revision we will move Table R2 into the main text.
Gap between the formulation and real deployment. The labels are a proxy for production cost by design, so we test whether the conclusions depend on the proxy. They hold across judges, five monotone cost mappings, continuous severity and human labels, and the dependence on cost heterogeneity can be checked before deployment by comparing the top-quartile share of predicted weight with the uniform 25%.
Figures. We will add a pipeline figure accordingly.
Response to Reviewer N5ba (Part 1 of 3)
We thank the reviewer for the detailed review and for the willingness to reconsider after the rebuttal. The newly added experiments include (1) ten deployable baselines on SWE-bench Lite, including learned difficulty, learned cheap-tier failure, learned Δx and learned consequence routers, and (2) inter-annotator agreement for the human study. We also summarize the maintainer-behavior anchors (Appendix D) and the boundary conditions across five pools and a live streaming run (Appendices O and Q). All of them consistently support the paper's central claim that consequence of failure is a distinct allocation dimension that can be predicted before solving and whose value can be anticipated from measurable conditions.
General clarification. We would like to first clarify the positioning of this work. Existing adaptive test-time compute methods decide where extra compute goes by how likely a task is to fail or how much extra compute is expected to help (difficulty, uncertainty or predicted gain), and implicitly treat every failure as equally costly. This paper instead focuses on the cost of failure and addresses a different and underexplored question, namely which failures are worth preventing under a fixed budget. Consequence and difficulty or gain signals therefore address different factors of the same priority, and they are complementary rather than competing. Our goal is not a more complex scheduler that is optimal on every task, but a new allocation principle, namely that the consequence of failure is a decision dimension in its own right.
W1. Technical contribution
Our contribution does not rest on proposing a new large network or a complex classifier. We focus on a factor that test-time compute allocation has not adequately considered, the cost of failure, and on how to use it under the information available in real deployment. Hence, our contribution is not the scheduler itself but three connected components, (1) a formulation with testable predictions, (2) an empirical predictability asymmetry that determines the design, and (3) an evaluation resource for this setting. The simple scheduler is what these findings call for.
(1) A formulation with testable predictions
We recast test-time compute allocation as a cost-sensitive decision problem. Its exact priority rule, wxΔx (wx the consequence weight of a failure, Δx the success gain from premium compute), contains accuracy-oriented allocation as the special case of equal costs. The rule also makes falsifiable predictions. Consequence routing should help only when costs differ, premium compute has headroom, and difficulty is not already aligned with Δx. The out-of-domain results confirm this, including MATH, where difficulty routing wins until declared stakes differ by 10× to 30× (Q1).
(2) A training-based method does not do better
The key empirical finding is which signal can be learned from pre-solution inputs. Δx is nearly unpredictable, while consequence is learnable. Following the reviewer's suggestion, we trained routers on issue-text embeddings for four different targets under the same budget.
Table R1. Learned routers on SWE-bench Lite under the same budget (loss, lower is better).
Only routers that predict consequence reach the lowest loss, and the learned consequence router matches but does not beat the prompted one. The bottleneck is therefore the choice of signal, not the learner, which is why we keep the scheduler training-free. As a result, the scheduler can be added to existing reasoning systems as a plug-in layer, without modifying or retraining the underlying model.
(3) A new, validated evaluation resource
We will release consequence labels for 1,290 tasks across six pools (SWE-bench Lite, Multi-SWE-bench mini, a SWE-bench Verified DB/ORM pool, BIRD, FinQA and MATH), the 150 human-labeled tasks, and the maintainer-behavior data (backports and Django Trac triage), so that later work can compare consequence definitions, predictors and allocation policies.