acceptodds
Under review as a conference paper at ICLR 2027

Not All Errors Are Equal: Consequence-Aware Reasoning Compute Allocation

Abstract

Test-time compute has emerged as an effective paradigm for improving large language model capability at inference time. Existing allocation strategies primarily prioritize tasks according to difficulty, uncertainty, or expected performance gain, implicitly treating prediction errors as equally costly. This assumption is often misaligned with real deployment, where failures can differ substantially in their downstream tasks. To address this limitation, this paper introduces consequence-aware test-time compute allocation by formulating a cost-weighted scheduling problem where the priority of a task is its failure consequence with the marginal gain of additional compute. In practice, however, marginal gain is difficult to predict before execution, so we propose a deployable scheduler that uses consequence as the routing signal. The scheduler predicts task consequence from pre-solution inputs and allocates the available premium compute to the corresponding top-ranked tasks. Experiments on the SWE bench Lite show that consequence provides information beyond task difficulty and can be predicted before solving. Under a fixed compute budget, consequence-aware routing achieves the best high-consequence task success, while overall accuracy remains competitive. A controlled within-model experiment further confirms the same advantage when only inference attempts are reallocated.

Primary area:
foundation or frontier models, including LLMs
Keywords:
test-time compute, adaptive inference, reasoning models, compute allocation, cost-sensitive learning, consequence-aware reasoning, SWE-bench

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_N5baOpenReview ↗
Rating
2
Confidence
1(−1)
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates an important, but under-explored problem: the allocation of compute resources in traditional scenarios may not be aligned with real deployments. The authors further formulate the problem as an optimization problem and propose consequence-aware test-time compute allocation. Specifically, the authors try to predict how serious it will be if the task fails, and then allocate compute based on this. Experiments suggest good performance of their proposed allocation.

Critical strengths

The core idea of this paper is intuitive and interesting. In fact, it is quite normal to treat allocation equally. Therefore, few previous works have explored this topic. The authors start with a "common" phenomenon and then present interesting assumptions, explorations, and findings on compute allocation. These contributions are just as important as works aiming at the "state-of-the-art".

The evaluation consists of baselines of three levels, human evaluation and extensive studies such as boundary conditions, etc. These experiments support the authors' claims.

Critical weaknesses
  1. I am afraid that the core weakness is that the technical contribution of the methodology seems to be weak. The authors mainly redefine the objective and apply the simple predicted consequence as the ranking method. This design is helpful in proving the concepts, but the technical contribution may be weak. Additionally, there is no new/unique evaluation data proposed under the authors' settings. After reading the paper, I feel the idea is interesting, but the methodology is not attractive enough.
    Maybe the style of this paper is a software-engineering style (auditing), and it may also be a good choice to submit to some conferences in software engineering. Otherwise, you may redesign the methods section (like new training methods based on the observations, even on a toy set). Moreover, there exists a gap between the theory and the real deployment.

  2. The consequence label, which is a core design in the paper, is indeed a judge-defined proxy. There also exists a gap between the real deployment cost and this design. In sum, I feel that the authors try to solve an important problem; however, they may limit themselves to a small space for evaluation/trying.

Questions for authors

Besides the weaknesses, here are some questions (currently I give a slightly negative recommendation, and after rebuttal I may reconsider whether this paper should be accepted):

The evaluations are mainly conducted on SWE-bench. Will the task designs in this benchmark be "more suitable" for your consequence-aware design? (I see that, in SWE-bench, there are some tasks quite hard, and even large compute is allocated, the task may still fail) In other words, can your design be generalized to tasks in other benchmarks?

Are the consequence labels reliable enough? That is to say, is the "consequence" aligned with real software behaviors? This should be carefully confirmed. What is the background of the human annotators? You mention that they are CS Ph.D. students; how to make sure that their annotations really reflect the risks? (Self-consistency is not the validity of the label construction, and moderate agreement between annotators can be seen) Is this study IRB-approved?

A small suggestion: you may add figures (like the main pipeline) in your paper to help readers better understand the work.

9 comments
AuthorsOpenReview ↗

Response to Reviewer N5ba (Part 1 of 3)

We thank the reviewer for the detailed review and for the willingness to reconsider after the rebuttal. The newly added experiments include (1) ten deployable baselines on SWE-bench Lite, including learned difficulty, learned cheap-tier failure, learned Δx\Delta_x and learned consequence routers, and (2) inter-annotator agreement for the human study. We also summarize the maintainer-behavior anchors (Appendix D) and the boundary conditions across five pools and a live streaming run (Appendices O and Q). All of them consistently support the paper's central claim that consequence of failure is a distinct allocation dimension that can be predicted before solving and whose value can be anticipated from measurable conditions.

General clarification. We would like to first clarify the positioning of this work. Existing adaptive test-time compute methods decide where extra compute goes by how likely a task is to fail or how much extra compute is expected to help (difficulty, uncertainty or predicted gain), and implicitly treat every failure as equally costly. This paper instead focuses on the cost of failure and addresses a different and underexplored question, namely which failures are worth preventing under a fixed budget. Consequence and difficulty or gain signals therefore address different factors of the same priority, and they are complementary rather than competing. Our goal is not a more complex scheduler that is optimal on every task, but a new allocation principle, namely that the consequence of failure is a decision dimension in its own right.

W1. Technical contribution

Our contribution does not rest on proposing a new large network or a complex classifier. We focus on a factor that test-time compute allocation has not adequately considered, the cost of failure, and on how to use it under the information available in real deployment. Hence, our contribution is not the scheduler itself but three connected components, (1) a formulation with testable predictions, (2) an empirical predictability asymmetry that determines the design, and (3) an evaluation resource for this setting. The simple scheduler is what these findings call for.

(1) A formulation with testable predictions

We recast test-time compute allocation as a cost-sensitive decision problem. Its exact priority rule, wxΔxw_x\Delta_x (wxw_x the consequence weight of a failure, Δx\Delta_x the success gain from premium compute), contains accuracy-oriented allocation as the special case of equal costs. The rule also makes falsifiable predictions. Consequence routing should help only when costs differ, premium compute has headroom, and difficulty is not already aligned with Δx\Delta_x. The out-of-domain results confirm this, including MATH, where difficulty routing wins until declared stakes differ by 10× to 30× (Q1).

(2) A training-based method does not do better

The key empirical finding is which signal can be learned from pre-solution inputs. Δx\Delta_x is nearly unpredictable, while consequence is learnable. Following the reviewer's suggestion, we trained routers on issue-text embeddings for four different targets under the same budget.

Table R1. Learned routers on SWE-bench Lite under the same budget (loss, lower is better).

RouterTarget predicted from issue textLoss
Learned difficulty (embedding ridge)Difficulty241.25
Learned cheap-tier failure (embedding logistic)Whether the cheap model fails237.25
Learned Δx\Delta_x (embedding ridge)Marginal gain219.50
Learned consequence (embedding ridge)Consequence209.00
Ours (prompted Qwen2.5-7B)Consequence209.75

Only routers that predict consequence reach the lowest loss, and the learned consequence router matches but does not beat the prompted one. The bottleneck is therefore the choice of signal, not the learner, which is why we keep the scheduler training-free. As a result, the scheduler can be added to existing reasoning systems as a plug-in layer, without modifying or retraining the underlying model.

(3) A new, validated evaluation resource

We will release consequence labels for 1,290 tasks across six pools (SWE-bench Lite, Multi-SWE-bench mini, a SWE-bench Verified DB/ORM pool, BIRD, FinQA and MATH), the 150 human-labeled tasks, and the maintainer-behavior data (backports and Django Trac triage), so that later work can compare consequence definitions, predictors and allocation policies.

AuthorsOpenReview ↗

Response to Reviewer N5ba (Part 2 of 3)

W2 and Q2. Label reliability and the human annotations

Indeed, the consequence label is not the directly observable deployment loss, and we do not treat the LLM judge's output as real deployment risk. The key question is whether the labels agree with real software maintenance behavior that does not involve the LLM judge. To further investigate this issue, we explore (1) maintainer behavior that involves neither an LLM nor our annotators, (2) the annotation protocol and agreement, and (3) whether the conclusions hold under human labels.

(1) Maintainer behavior

The people best placed to judge real risk in these projects are the maintainers, and their revealed behavior on SWE-bench Lite follows our classes (Appendix D). Backporting means that maintainers port a fix to older maintained release branches, a real decision that does not involve our LLM judge. The backport rate rises monotonically with our consequence class, from 8.3% (class 0) to 20.6% (class 1) and 41.9% (class 2), with trend p=10−4p = 10^{-4}. Higher-consequence issues are thus the ones that maintainers more often treat as important enough to fix in older releases as well.

(2) Annotation protocol and agreement

The three annotators were one author, one non-author CS PhD student and one non-author CS graduate student. They saw no gold patches, diffs or each other's labels, completed a nine-task calibration, and applied a rubric defined by concrete failure types (for example, documentation errors for class 0, wrong return values for class 1, silent data corruption or permission bypass for class 2), which needs no knowledge of a specific deployment. We agree that self-consistency is not validity. Agreement between annotators is limited (Fleiss' κ=0.233\kappa = 0.233), so we use human labels only as a supplementary reference. Still, the human majority agrees with our predictor (κ=0.517\kappa = 0.517), and consequence routing beats difficulty and random routing when human labels define the weights (55.2 vs. 75.8 and 67.5, Table 8).

(3) Conclusions under human labels

Analysis on human-labeled tasksResult
Consequence vs. difficulty, Spearman (75 Lite tasks)ρ=−0.066\rho = -0.066
Issue-only predictor vs. human majorityκ=0.517\kappa = 0.517
With-patch judge vs. human majorityκ=0.50\kappa = 0.50
Loss under human weights, difficulty / random / consequence oracle (Table 8)75.8 / 67.5 / 55.2

The issue-only predictor agrees with humans as well as the with-patch judge does, consequence stays orthogonal to difficulty, and consequence routing beats difficulty and random routing when humans define the weights. This indicates that the main conclusions do not depend on the LLM judge.

IRB. The annotation involved only severity judgments on public software issue descriptions, and no sensitive personal information about the annotators was collected. We will disclose the ethics-review status accurately in the revision, in line with our institution's requirements and records.

Q1. Is SWE-bench especially favorable, and does the approach generalize?

We test the proposed framework on SWE-bench because software engineering is a well-studied application of AI, where inference-time resource allocation is important.

The point of the paper is therefore not that consequence always beats difficulty, but whether consequence is a useful deployable signal when the marginal gain cannot be estimated in advance. To further examine the proposed framework's generalization, we examine it from two angles, (1) comparisons on SWE-bench that do not depend on this handicap, and (2) boundary conditions across five pools.

To answer this, we evaluate on SWE-bench Lite and Verified, and test the generalization ability of the method on BIRD, FinQA and MATH. We report all results, including those where our method does not win.

(1) When does consequence routing apply? (Appendix O)

We compare five pools on three conditions, namely whether error costs differ across tasks, whether extra compute helps, and whether difficulty already identifies the tasks that gain from it.

Table R2. Applicability conditions across five pools (a top-25% weight share of 0.25 means equal costs, and ρ\rho is the correlation between difficulty and the gain from extra compute).

PoolTasksTop-25% weight shareρ(difficulty,Δx)\rho(\mathrm{difficulty},\Delta_x)Premium headroomBetter deployable signal
SWE-bench Lite3000.42−0.78YesConsequence
SWE-bench Verified DB/ORM1400.39−0.38YesConsequence trends positive, not significant vs. predicted difficulty
BIRD1500.35+0.06LimitedNo clear advantage, near random
FinQA1500.36+0.01Almost noneNo clear advantage
MATH1500.50+0.26LargeDifficulty
AuthorsOpenReview ↗

Response to Reviewer N5ba (Part 3 of 3)

On SWE-bench, error costs differ and difficulty points away from the tasks that extra compute can save, so consequence gives a better ranking. This hurts difficulty routing rather than favoring ours, since our method also beats random on Lite (9.3% lower loss). The gain is significant on Lite and positive but not significant on Verified.

(2) BIRD, where consequence differences are limited

Only 3 of 150 BIRD tasks are high-consequence, so there is little to prioritize and our method is close to random under the main weights. When GPT-4o assigns higher stakes to some queries, our method beats difficulty routing by 3.1%, so the result depends on how costs are defined. These are point estimates.

Table R3. BIRD allocation results (weighted loss, lower is better).

Consequence weightsStrategyWeighted lossvs. difficulty-aware
QwenDifficulty-aware66.670%
QwenRandom68.27−2.4%
QwenConsequence-aware (question-only)67.33−1.0%
QwenConsequence-aware (with-gold oracle)65.67+1.5%
GPT-4oDifficulty-aware75.000%
GPT-4oRandom75.80−1.1%
GPT-4oConsequence-aware (question-only)72.67+3.1%
GPT-4oConsequence-aware (with-gold oracle)66.67+11.1%

(3) FinQA, where extra compute does not help

The stronger model is no better than the cheap one on FinQA (64.2% vs. 66.2%), so no routing method can help, even though our predictor identifies high-consequence tasks well (κ=0.661\kappa = 0.661, no class-2 task predicted as class 0).

(4) MATH, where difficulty already points to the gain

On MATH, harder problems are exactly where the stronger model helps (ρ=+0.26\rho = +0.26), so difficulty routing is the better choice under the original costs.

(5) Larger cost differences reverse the result on MATH

When stakes are assigned independently of difficulty and the gap between high and low stakes grows, consequence routing overtakes difficulty routing on MATH between 10× and 30×, and it stays ahead on FinQA at every tested gap (point estimates). Positive values in Table R4 mean lower loss than difficulty routing.

Table R4. Declared-stakes sweep on MATH and FinQA.

Cost gap between high and low stakes1×3×10×30×100×10³×10⁶×
MATH−28.3%−18.4%−6.5%+3.2%+11.5%+20.2%+24.9%
FinQA+2.9%+3.6%+4.4%+5.1%+5.7%+6.3%+6.6%

(6) Live run (Appendix Q)

In a live run on 300 FinQA and MATH tasks with real model calls and a shared budget, the system works as intended. Difficulty routing has the lowest loss, as expected from MATH, but its difference from our method is not significant (95% CI [−9,493, +2,301]) and the two used different numbers of premium calls (76 vs. 58). Our method still lowers loss by 18.8% relative to random.

Table R5. Live streaming run on FinQA and MATH.

PolicyWeighted lossReduction vs. randomPremium callsAccuracy
Random24,4260%23.0%70.0%
Difficulty16,43232.7%25.3%70.7%
Consequence19,83118.8%19.3%68.7%
Verifier-gated sequential21,62711.5%5.0%70.0%

(7) Summary

Our conclusion is therefore not that consequence routing replaces difficulty routing on all tasks, but that the consequence of failure is an independent dimension for allocating compute. Its value depends on whether error costs differ, whether extra compute helps, and whether difficulty already points to the gain. In the revision we will move Table R2 into the main text.

Gap between the formulation and real deployment. The labels are a proxy for production cost by design, so we test whether the conclusions depend on the proxy. They hold across judges, five monotone cost mappings, continuous severity and human labels, and the dependence on cost heterogeneity can be checked before deployment by comparing the top-quartile share of predicted weight with the uniform 25%.

Figures. We will add a pipeline figure accordingly.

Reviewer_WojmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to allocate limited test-time compute when errors have unequal deployment consequences. It formulates allocation as minimizing consequence-weighted error, where the ideal priority combines consequence and marginal compute gain. Since gain is hard to predict before solving, the method predicts consequence from issue text and routes premium compute to high-ranked tasks. Experiments on SWE-bench Lite evaluate this strategy under matched budgets.

Critical strengths

The empirical analysis is thorough. It includes label ablations, human checks, judge swaps, bootstrap intervals, a second SWE-bench pool, and explicit boundary conditions.

The paper is well-written, reflecting considerable effort on the part of the authors. Such attention to presentation is particularly commendable in the current academic landscape.

Critical weaknesses
  1. This paper studies how to allocate limited extra reasoning resources: should they go to tasks that may cause more serious consequences, or to harder tasks? In theory, the best allocation needs a future value function. However, the authors admit that even if an LLM predicts the success rates of a normal model and a strong model from the issue text and takes the difference, the prediction is almost useless. Therefore, the paper does not really learn a full future value function. In practice, it only predicts ωx\omega_x: it uses Qwen or Claude to read only the GitHub issue text and the file paths inside it, and divides tasks into low, medium, and high consequence levels. This actually goes against the original goal of the paper, and this kind of operation is already quite common.\textbf{This actually goes against the original goal of the paper, and this kind of operation is already quite common.}

  2. The headline 21.8% reduction is measured against outcome-derived difficulty, which relies on pass rates observed from the same 16 systems and is unavailable before deployment. Appendix E further shows that all 1,000 random allocations outperform this baseline, indicating that it is weak. Against the deployable predicted-difficulty baseline, consequence routing reduces loss from 224.00 to 209.75 (6.4%), but the paper does not report a direct confidence interval for this gap or compare against stronger deployable routing baselines.

Questions for authors
  1. Why is Qwen2.5-7B-Instruct used as the primary consequence predictor when Qwen3 models are already available in the native-thinking audit? Please evaluate Qwen3 (Qwen3.5 or Qwen 3.8) under the same issue-only protocol and report consequence-prediction calibration and matched-budget routing results.

  2. The paper does not reliably estimate the marginal gain Δx\Delta_x (at least in my view), which is central to the optimal priority rule. The deployable method instead asks Qwen2.5 to classify issue text by consequence and allocates the same fixed compute budget accordingly. While the matched-budget results demonstrate a redistribution effect, it remains unclear why this coarse classifier should yield such a large practical benefit beyond alignment with the Qwen2.5 with-patch labels used to define the evaluation loss. Could the authors isolate the source of the gain by evaluating against independent consequence labels and reporting its stability across predictor models, and prompts?

5 comments
AuthorsOpenReview ↗

Response to Reviewer Wojm (Part 1 of 3)

We thank the reviewer for the detailed review (Weakness: W; Question: Q). The newly added experiments include (1) paired-bootstrap comparisons against deployable predicted difficulty and random in four settings, (2) ten stronger deployable baselines, including learned difficulty, learned cheap-tier failure and learned Δx\Delta_x routers, (3) Qwen3-8B and Qwen3.5-9B predictors under the identical protocol, and (4) evaluation under independent judges' labels, together with eight predictor variants across prompts and model families. All of them consistently support the paper's central claim that consequence is a distinct and ex-ante predictable allocation signal, and that its gain is not an artifact of a single predictor, prompt or judge.

General clarification. We would like to first clarify the positioning of this work. Existing adaptive test-time compute methods decide where extra compute goes by how likely a task is to fail or how much extra compute is expected to help (difficulty, uncertainty or predicted gain), and implicitly treat every failure as equally costly. This paper instead focuses on the cost of failure and addresses a different and underexplored question, namely which failures are worth preventing under a fixed budget. Consequence and difficulty or gain signals therefore address different factors of the same priority, and they are complementary rather than competing.

Our goal is not a more complex scheduler that is optimal on every task, but a new allocation principle, namely that the consequence of failure is a decision dimension in its own right. It is therefore reasonable to assess the contribution based on whether the paper provides a theoretical foundation for this principle, demonstrates that consequence is more predictable than marginal gain before execution, and shows through matched-budget experiments that consequence-aware allocation yields measurable gains.

W1. Why the method predicts only wxw_x

The theoretical rule and the deployable scheduler serve different purposes. Under the two-tier setting, the optimal priority is wxΔxw_x\Delta_x, where wxw_x is the consequence of failure and Δx\Delta_x is the expected success gain from premium compute. We do not claim that predicting wxw_x alone is optimal. We use it because our experiments show that consequence is more reliably estimated from information available before execution than marginal gain.

We verify this design choice by testing whether each factor of the wxΔxw_x\Delta_x priority can be predicted from pre-solution inputs on SWE-bench Lite.

Table R1. Predicting each factor of wxΔxw_x\Delta_x from pre-solution inputs on SWE-bench Lite.

TargetPre-solution estimatorAgreement with the true valueRouting loss
Δx\Delta_x (marginal gain)Prompted forecasts, two model families (Appendix I)ρ\rho between −0.07 and +0.05-
Δx\Delta_x (marginal gain)Ridge probe on issue embeddingsρ=0.11\rho = 0.11219.50
wxw_x (consequence)Ridge probe on the same embeddingsout-of-fold ρ=0.597\rho = 0.597209.00
wxw_x (consequence)Prompted Qwen2.5-7B (paper)κ=0.572\kappa = 0.572209.75

The learned consequence predictor achieves nearly the same loss as our prompted router, whereas the tested gain predictors provide much weaker pre-execution estimates. This supports consequence-only routing as a useful practical approximation, not as a replacement for wxΔxw_x\Delta_x. A high-consequence task may still have negligible gain from extra compute; estimating that gain reliably remains an important direction for improvement. Appendix E also shows that post-hoc priority-aware results are optimistic because observed outcomes are used for both selection and evaluation.

W2. Deployable baselines and confidence intervals

We first clarify that we include outcome-derived difficulty because difficulty is a widely studied signal for inference-time compute allocation. However, this baseline relies on observed pass rates and can be weak (it loses even to random because, on SWE-bench, the hardest tasks are often ones that extra compute cannot save), motivating the exploration of other allocation signals.

To establish the practical benefit of our method, we provide two additional evaluations: (1) paired comparisons against deployment-time predicted difficulty and random routing across different settings, and (2) comparisons with ten additional baselines, including learned routing methods.

For clarity, we report two statistics separately. The direct reduction is calculated from the full-dataset losses as (Lbaseline−Lours)/Lbaseline(L_{\mathrm{baseline}}-L_{\mathrm{ours}})/L_{\mathrm{baseline}}. The paired-bootstrap mean is the average relative reduction over 5,000 paired task-level resamples, together with its 95% confidence interval (CI).

AuthorsOpenReview ↗

Response to Reviewer Wojm (Part 2 of 3)

(1) Paired comparisons against predicted difficulty and random We first examine whether the improvement persists against deployable baselines under matched compute budgets. We also include controlled within-model experiments to isolate the effect of compute allocation and an independent SWE-bench pool to assess robustness.

Table R2. Paired comparisons against predicted difficulty and random (loss, lower is better).

SettingLoss (ours / predicted difficulty / random)Direct reduction vs. predicted difficulty / randomPaired-bootstrap mean vs. predicted difficulty [95% CI]P(ours better than predicted difficulty)Paired-bootstrap mean vs. random [95% CI]
SWE-bench Lite (300 tasks, top 25%)209.75 / 224.00 / 231.206.4% / 9.3%6.9% [+2.2, +11.8]99.8%9.0% [+5.1, +13.0]
Within-model, Claude judge (127 tasks, top 50%)83.00 / 100.00 / 91.9217.0% / 9.7%19.4% [+10.6, +29.2]100%11.7% [+5.0, +19.3]
Within-model, DeepSeek judge (126 tasks)95.00 / 107.00 / 99.4811.2% / 4.5%11.8% [+4.3, +20.6]99.9%5.9% [+0.8, +11.7]
Verified DB/ORM (140 tasks, top 25%)131.75 / 135.00 / 135.612.4% / 2.8%5.2% [−1.5, +11.6]93.4%5.6% [+0.1, +10.7]

Table R2 shows that our router keeps its advantage over the deployable baselines, with lower loss in all four settings, although the margin is smaller than the 21.8% headline. On Lite, the direct reduction against predicted difficulty is 6.4%, as the reviewer computed, and the paired-bootstrap mean is 6.9% with a 95% CI of [+2.2, +11.8], so the gain is statistically significant. Both within-model settings show significant gains against predicted difficulty and random.

(2) Comparison with ten additional baselines

To further examine whether the improvement comes from comparing against weak baselines, we evaluate ten additional routing strategies on SWE-bench Lite. These include simple issue-based heuristics, repository-level priors, and supervised predictors of difficulty, cheap-tier failure, marginal gain, and consequence. All strategies allocate premium compute to 25% of tasks under the same budget. The direct reduction and bootstrap results below compare each baseline against our method (loss 209.75).

Table R3. Ten deployable baselines on SWE-bench Lite under the same budget, each compared with our router.

Deployable baselineLossClass-2 successDirect reductionPaired-bootstrap mean [95% CI]P(ours better)
Longest issue first226.5024.4%7.4%8.3% [+3.6, +13.0]100%
Shortest issue first236.0017.0%11.1%11.0% [+6.2, +16.1]100%
Number of code blocks239.7510.8%12.5%12.2% [+7.4, +16.9]100%
Number of referenced .py paths229.0015.9%8.4%8.2% [+3.3, +12.9]100%
Repository difficulty prior (leave-one-out)225.0013.6%6.8%6.6% [+1.2, +12.0]99.1%
Repository consequence prior (leave-one-out)233.756.8%10.3%10.0% [+4.9, +15.3]100%
Learned difficulty (embedding ridge, 10-fold out-of-fold, ρ=0.167\rho = 0.167)241.2511.4%13.1%12.8% [+7.9, +17.9]100%
Learned cheap-tier failure (embedding logistic, out-of-fold)237.2515.3%11.6%11.7% [+6.5, +17.0]100%
Learned Δx\Delta_x (embedding ridge, out-of-fold, ρ=0.115\rho = 0.115)219.5035.2%4.4%4.4% [−0.3, +9.0]96.8%
Learned consequence (embedding ridge, out-of-fold, ρ=0.597\rho = 0.597)209.0043.8%−0.4%−0.4% [−4.7, +3.6]41.6%

As shown in Table R3, our router has lower loss than nine of the ten baselines and achieves the highest class-2 success (51.1%). The cheap-tier-failure baseline corresponds to accuracy-oriented learned routers, which predict whether the cheap model suffices. The only baseline that matches our router is the learned consequence predictor (209.00 vs. 209.75), itself a consequence router. This indicates that the gain does not come from beating a weak difficulty baseline, and that the consequence signal does not depend on one prompt.

Q1. Qwen3 predictors

We would like to note that the Qwen2.5-7B predictor and prompt were frozen before the Qwen3 audit, and we kept them fixed to avoid tuning the predictor on evaluation outcomes. To address the reviewer's concern, we evaluated Qwen3-8B and Qwen3.5-9B under the identical protocol (issue-only input, same prompt, greedy decoding, no gold patch, and top-25% budget).

AuthorsOpenReview ↗

Response to Reviewer Wojm (Part 3 of 3)

Table R4. Qwen3 and Qwen3.5 predictors under the identical issue-only protocol.

Predictorκ\kappa (Lite / Verified)Lite predicted class-2 shareBootstrap mean reduction vs. predicted difficulty [95% CI]
Qwen2.5-7B (paper)0.57 / 0.5828%6.9% [2.2%, 11.8%]
Qwen3-8B, non-thinking0.08 / 0.0682%0.0% [−5.2%, 4.7%]
Qwen3-8B, thinking0.08 / not evaluated66%0.1% [−5.2%, 5.1%]
Qwen3.5-9B, non-thinking0.40 / 0.2238%4.1% [−1.0%, 9.1%]
Qwen3.5-9B, thinking0.40 / not evaluated34%5.7% [0.9%, 10.7%]

Under this fixed protocol, the newer models do not consistently improve routing. In particular, Qwen3-8B labels 82% of Lite tasks as class 2, leaving too many tied candidates for a 25% premium quota. Over 1,000 random tie-breaks, its issue-only non-thinking loss averages 225.79 ± 3.78, compared with 210.2 ± 1.8 for the original Qwen2.5 predictor. Qwen3.5-9B thinking improves on predicted difficulty, and its difference from Qwen2.5 is inconclusive. The evidence shows that useful ranking resolution matters more than simply replacing the predictor with a newer checkpoint.

Q2. Isolating the source of the gain

We isolate the source of the gain from three angles. (1) Does the gain hold when the loss is defined by labels that do not come from Qwen? Yes. (2) Does it hold for predictors from other model families? Yes, although weaker predictors give smaller gains. (3) Is it stable across prompts? Yes.

(1) Independent reference labels. We keep our Qwen issue-only router fixed and evaluate it using DeepSeek's consequence weights. Our router achieves a loss of 362.75, compared with 378.31 for random routing (4.1% reduction), matching the result of routing directly by DeepSeek's reference labels. The improvement also persists under continuous severity weights from DeepSeek and GPT-4o (Appendix G). Furthermore, our predictor agrees with human-majority labels (κ=0.517\kappa = 0.517), and maintainer backport rates increase with consequence severity (8.3%, 20.6%, 41.9%; Appendix D). These results show that the benefit is not limited to Qwen-defined labels.

(2) Different predictor families. Under the original evaluation weights, GPT-4o achieves a routing loss of 214.00, with a paired-bootstrap mean reduction of 5.4% [0.7%, 10.1%] relative to predicted difficulty. A learned consequence predictor achieves a loss of 209.00, comparable to our 209.75. DeepSeek performs less strongly, indicating that predictor quality affects routing performance. Nevertheless, these results show that useful consequence predictions are not restricted to Qwen2.5.

(3) Stability across prompts and predictors We further evaluate Qwen2.5 with different prompts. Using issue text only or including file references gives losses of 211.25 and 208.00, respectively, close to the original 209.75, with class-2 success rates of 50.0%–53.4%. Across all eight predictor and prompt variants, seven achieve paired-bootstrap confidence intervals excluding zero against random routing, and six against predicted difficulty. This supports the stability of the consequence signal across different implementations.

Table R5. Eight predictor and prompt variants on SWE-bench Lite, scored with the paper labels.

PredictorLossClass-2 successDirect reduction vs. predicted difficulty / randomPaired-bootstrap mean vs. predicted difficulty [95% CI]Paired-bootstrap mean vs. random [95% CI]
Qwen2.5-7B issue-only (paper)209.7551.1%6.4% / 9.3%6.9% [+2.2, +11.8]9.0% [+5.1, +13.0]
Qwen2.5-7B rerun, issue text only211.2550.0%5.7% / 8.6%6.0% [+1.0, +11.2]8.2% [+4.2, +12.4]
Qwen2.5-7B rerun, issue text with file paths208.0053.4%7.1% / 10.0%7.6% [+2.7, +12.6]9.7% [+5.8, +13.8]
Claude Sonnet 4.5, three classes (paper)220.5036.9%1.6% / 4.6%1.9% [−3.1, +6.9]4.2% [+0.5, +8.2]
DeepSeek continuous severity220.5033.0%1.6% / 4.6%0.9% [−3.9, +5.5]3.2% [−0.2, +6.7]
GPT-4o continuous severity214.0043.8%4.5% / 7.4%5.4% [+0.7, +10.1]7.6% [+3.9, +11.5]
Mean of DeepSeek and GPT-4o severity215.5043.2%3.8% / 6.8%4.9% [+0.1, +9.6]7.1% [+3.2, +11.0]
Embedding ridge on consequence (out-of-fold)209.0043.8%6.7% / 9.6%7.3% [+2.4, +12.2]9.4% [+5.7, +13.4]

Together, these results indicate that several types of predictor extract consequence information from pre-solution inputs, and that the gain is not specific to one prompt or to the judge that defines the loss. We will add this table to the appendix.

Reviewer_TAi8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

While prior work on adaptive test-time compute allocation often focuses on maximizing overall task accuracy, this paper introduces a new perspective: the real-world consequence (or cost) of task failure.

Instead of treating all errors equally and maximizing naive average accuracy, the authors formulate compute allocation as minimizing a consequence-weighted failure objective under a fixed budget. Under a two-tier compute setting, this formulation yields an exact optimal priority rule determined by the product of failure consequence and marginal compute gain (wxΔxw_x \Delta_x). Because marginal gain Δx\Delta_x is difficult to predict prior to execution, the authors propose a lightweight, training-free scheduler that routes premium compute based solely on the consequence predicted from pre-solution inputs (e.g., GitHub issue descriptions). Experiments on SWE-bench Lite and Verified demonstrate that consequence is largely orthogonal to task difficulty and that consequence-aware routing substantially improves success rates on high-stakes tasks without degrading overall accuracy.

Critical strengths
  • Proposed a practical and novel problem, where the cost-weighted deployment risk objective is well-motivated and clearly formulated.

  • The empirical evaluation and ablations are comprehensive. Beyond the main 16-system routing setup on SWE-bench Lite, the authors include a controlled within-model experiment, cross-pool transfer to SWE-bench Verified DB/ORM tasks, human annotation checks, external validation using maintainer backport behavior, and clear boundary-condition analyses on out-of-domain datasets (BIRD, FinQA, and MATH).

Critical weaknesses
  • In most simulated and benchmark settings, there is no objective "oracle" consequence value. The paper relies on an LLM-with-patch judge (mainly Qwen2.5-7B) to define the reference consequence labels, which is still a heuristic proxy rather than realized deployment harm. More importantly, Appendix shows that changing the reference judge from Qwen2.5-7B to DeepSeek-Chat drops inter-judge agreement to κ=0.182\kappa = 0.182 on SWE-bench Lite and κ=0.140\kappa = 0.140 on Verified, because DeepSeek labels over half of the tasks as high-consequence (Class 2). When the judge marks most tasks as high-stakes, the cost distribution flattens and the advantage of consequence-aware routing over random allocation largely disappears.
  • Estimating consequence before solving the task (offline setting) is often unrealistic in practice: the deployable router assumes that failure consequence can be reliably inferred from the pre-solution issue description alone. But in real world, consequence may only be estimated after a certain amount of compute is allocated.
  • While the optimal allocation shows wxΔxw_x \Delta_x as the exact priority rule, the deployable method drops Δx\Delta_x entirely because pre-execution marginal gain prediction fails (Appendix I). As shown in Table 3, there remains a noticeable gap between consequence-only routing (209.75 loss) and the priority-aware upper bound (186.00 loss). Allocating expensive compute to a high-consequence task that has Δx=0\Delta_x = 0 (i.e., either the cheap model can already solve it, or even the premium model cannot solve it) still results in wasted budget.
Questions for authors
  • Given LLM judges have huge influence on consequence evaluation (the low agreement between Qwen and DeepSeek judges in Appendix N), how should practitioners calibrate the consequence predictor in a new domain without gold patches so that it does not over-predict Class 2 and degenerate into random routing?
  • Typo: Line 262, should add a space after "We compare"
4 comments
AuthorsOpenReview ↗

Response to Reviewer TAi8 (Part 1 of 3)

We thank the reviewer for the detailed review. Corresponding to the reviewer's three weaknesses (W) and one question (Q), this response provides (1) a judge-swap analysis (W1), (2) a six-pool safety check of the pre-solution predictor (W2), (3) a split-half estimate of the gap to wxΔxw_x\Delta_x and an analysis of Δx≈0\Delta_x \approx 0 tasks (W3), and (4) a calibration recipe without gold patches (Q1). Together, these results show that the routing gain does not hinge on a particular judge, can be obtained before solving, and depends on cost heterogeneity in a way that can be checked before deployment.

General clarification. Existing adaptive test-time compute methods allocate compute by how likely a task is to fail or how much extra compute would help, and implicitly treat all failures as equally costly. This paper instead focuses on the cost of failure and addresses a different and underexplored question, namely which failures are worth preventing under a fixed budget. Consequence thus complements difficulty and gain signals rather than competing with them.

Our goal is not a more complex scheduler that is optimal on every task, but a new allocation principle, namely that the consequence of failure is a decision dimension in its own right. It is therefore reasonable to assess the contribution based on whether the paper provides a theoretical foundation for this principle, demonstrates that consequence is more predictable than marginal gain before execution, and shows through matched-budget experiments that consequence-aware allocation yields measurable gains.

W1. Dependence on the LLM judge

The reviewer notes that agreement between LLM judges is limited, especially as DeepSeek labels many tasks as class 2. We would like to clarify that this does not by itself mean that the labels lack validity or that the gain is tied to one judge. We address this concern by answering two questions. (1) Do the labels reflect real consequences beyond the judge? Yes, they agree with maintainer behavior. (2) Why is agreement with DeepSeek low, and why does the advantage over random shrink under its labels? DeepSeek places 54.7% of Lite tasks in class 2 against 14.7% for Qwen. This forces disagreement and flattens the weights, so any ranking can gain less over random, while our ranking still captures all of the available gain. More details are given below.

(1) The labels agree with real maintainer behavior

Our labels are produced by an LLM judge, but they are not just that judge's opinion. They match how maintainers treated the same SWE-bench Lite issues, which involves no LLM (Appendix D). In other words, the issues our labels rate as more severe are the ones maintainers more often patch in older releases and more often confirm as real bugs. Django's “easy pickings” tag, which marks tickets that are easy to fix, is unrelated to our labels (ρ=−0.12\rho = -0.12), so our labels capture severity rather than difficulty. Human annotators also agree with our labels (Cohen's κ=0.50\kappa = 0.50 with the reference labels and κ=0.517\kappa = 0.517 with our predictor).

(2) Why agreement is low and the advantage shrinks under DeepSeek

DeepSeek labels 54.7% of Lite tasks and 77.1% of Verified tasks as class 2, against 14.7% and 33.6% for Qwen. This forces disagreement between the two judges. It also gives most tasks the same high weight, so the weights become nearly flat and a consequence ranking has little left to separate.

Table R1. Share of total weight held by the top quarter of tasks under each judge's labels (0.25 means uniform weights).

PoolQwen weightsDeepSeek weightsUniform weights
Lite0.4190.3290.250
Verified0.3910.2830.250

These results support three conclusions. First, our ranking is not tied to the Qwen judge. Under DeepSeek's weights on Lite, our predictor's loss (362.75) equals that of the DeepSeek oracle, 4.1% below random (378.31). Second, the smaller margin comes from DeepSeek's coarse and flat labels, not from our ranking. Adding finer information, by breaking ties within classes with an issue-only continuous severity score, lowers the loss to 353.25 (6.6% below random), better than the three-class oracle. Third, when costs are almost flat, no consequence ranking can help. On Verified, where DeepSeek puts 77.1% of tasks in class 2, all signals stay within about 1% of random, and even the oracle is 0.8% worse than random.

AuthorsOpenReview ↗

Response to Reviewer TAi8 (Part 2 of 3)

When the reference weights are nearly uniform, a consequence ranking has little to exploit, and even the oracle ranking cannot gain much. The Verified result under DeepSeek's weights therefore does not mean that our predictor is too weak. It means that consequence routing needs sufficiently heterogeneous costs. The same holds at deployment, where a predictor that pushes most tasks into class 2 flattens the weights in the same way, and the procedure in our answer to W2 and Q1 checks for this.

We therefore do not claim that routing results are identical across judges. Our claim is that the consequence labels track independent maintainer behavior, and that the routing gain depends on how well cost differences are preserved. When the weights become flat, the gain shrinks. This marks the method's range of applicability, and it does not contradict the claim that consequence should be a dimension of compute allocation. We will state this condition in the main text.

W2 and Q1. Estimating consequence before solving and in a new domain

W2 asks whether consequence can be estimated before solving, and Q1 asks how to keep that estimate reliable in a new domain without gold patches, so that it does not over-predict class 2. Both concern the pre-solution estimate, so we answer them together in three parts. (1) Is a pre-solution estimate realistic and accurate enough for routing? Yes. The issue text exists before any compute is allocated, and the estimate never sends a high-consequence task to the lowest class. (2) How can it stay reliable in a new domain? A three-step procedure that needs no gold patches keeps the ranking intact even when a predictor over-predicts class 2. (3) What if a consequence appears only during solving? The framework can update its estimate after a first attempt.

(1) A pre-solution estimate is realistic and accurate enough

The predictor uses only the issue text and the file paths the issue mentions, which exist before any compute is allocated. This matches how software teams already work, since issue trackers classify tickets before anyone fixes them, and our response to W1 shows that such maintainer classifications track our labels. Routing needs that high-consequence tasks are not sent to the cheapest tier, and the predictor meets this in all six pools as shown in Table R2.

Table R2. High-consequence (class-2) tasks recovered by the pre-solution predictor in six pools.

PoolClass-2 tasksFound by the predictorPredicted as class 0
SWE-bench Lite4488.6%0
SWE-bench Verified DB/ORM4795.7%0
Multi-SWE-bench mini3100%0
BIRD3100%0
FinQA1060%0
MATH30%0

The predictor did not make the error that matters most for routing. Across all 110 class-2 tasks in the six pools, none is predicted as class 0 (95% upper bound 2.7%), so no high-consequence task is pushed to the bottom of the ranking. When the predictor misses a class-2 task, it always labels it as class 1 (medium), never class 0 (low), so these tasks still rank above all low-consequence tasks. On Lite, this is accurate enough that routing with the predictor keeps 93.9% of the gain of routing with the reference labels (Appendix E). A pre-solution estimate is therefore sufficient for routing.

(2) Keeping the estimate reliable in a new domain

We validated, on frozen data, a standard calibration procedure that needs no gold patches from the new domain. Its aim is not to recover true deployment costs without any reference, but to keep tasks in a useful order when the predictor puts too many of them in one class, such as class 2. The procedure has three steps.

  1. Rank by continuous severity, not by absolute class. Use a continuous issue-only severity score (0 to 100), or use it to break ties within each class.
  2. Match the budget by quantile. Send the top qq% of tasks to the premium tier according to the budget, instead of requiring the predictor in every domain to output the same class-2 rate.
  3. Check weight concentration before deployment. Compute the share of predicted weight held by the top quarter of tasks. A value near 0.25 means that costs in this domain may be nearly uniform, and consequence routing should then not be expected to beat random.
AuthorsOpenReview ↗

Response to Reviewer TAi8 (Part 3 of 3)

To test the procedure, we apply it on Lite to a deliberately over-severe signal that puts 54.7% of tasks in class 2, and the results are summarized in Table R3.

Table R3. Calibration on SWE-bench Lite under the paper weights (loss, lower is better).

Predictor or calibrationLossReduction vs. random
Over-severe signal (DeepSeek labels, 54.7% class 2), random tie-breaks224.253.0%
Same labels, continuous-severity tie-breaks215.007.0%
GPT-4o continuous severity, used directly214.007.4%
Paper predictor (Qwen2.5-7B, issue-only)209.759.3%
With-patch oracle205.7511.0%

Continuous tie-breaks raise the over-severe signal's gain from 3.0% to 7.0%, recovering most of the gap to our predictor (9.3%). They also help Qwen3-8B, whose three classes over-predict class 2, lowering its loss with the file-path prompt from 228.04 to 221.75 on Lite and from 135.04 to 125.25 on Verified. Continuous ranking thus repairs the damage caused by too many tasks in one class, although it cannot repair a weak severity signal.

The value of the procedure is a check that can be run before deployment. Without gold patches from the new domain, continuous ranking, budget quantiles and the concentration check together tell whether the predictor still separates tasks by consequence. We do not present it as an automatic calibration that works in every domain. We will add it to the appendix.

(3) Consequences that appear only during solving

In that case, the allocation can be updated after a first cheap attempt, and Appendix P tests this with a verifier that needs no gold patch. Its verifier is still weak (AUC 0.65), its gains currently rely on oracle-level success detection, and it does not yet beat our static router at an equal attempt budget. We therefore present it as an extension.

W3. Gap to the wxΔxw_x\Delta_x priority and premium compute spent on Δx≈0\Delta_x \approx 0 tasks

We agree that consequence-only routing does not achieve the theoretical optimum, which depends on both failure consequence (wxw_x) and marginal compute gain (Δx\Delta_x). Our method does not discard Δx\Delta_x from the theoretical formulation; instead, it uses consequence as a practical approximation because Δx\Delta_x is much harder to predict before execution. Therefore, we do not claim that consequence-only routing is theoretically optimal or eliminates wasted computation. Our finding is more specific: when marginal gain cannot be reliably predicted before execution, consequence remains a useful allocation signal that reduces consequence-weighted loss under matched compute budgets. This demonstrates the practical value of treating failure consequence as an independent decision dimension.

We analyze this concern from three angles, (1) how large the gap really is, (2) whether consequence routing favors Δx≈0\Delta_x \approx 0 tasks, and (3) whether Δx\Delta_x can be estimated before execution.

(1) The in-sample gap overstates the attainable gain

The priority-aware rows in Table 3 compute Δx\Delta_x from the same outcomes used for evaluation. In split-half evaluation (selecting on half of each tier and evaluating on the other half, over 200 splits), the oracle priority-aware gain over difficulty falls from +33.8% to +27.2% (Appendix E). This leaves priority-aware routing roughly 4 to 5 points ahead of consequence-only routing (+23.3% for the oracle, +21.8% for our deployable router), rather than the in-sample gap between 186.00 and 209.75.

(2) Consequence routing does not favor Δx≈0\Delta_x \approx 0 tasks

Difficulty routing concentrates premium compute in the tail where no system succeeds (Table 15), whereas consequence routing meets Δx≈0\Delta_x \approx 0 tasks at about their base rate. This indicates that spending on Δx≈0\Delta_x \approx 0 tasks is a cost of not knowing Δx\Delta_x, and that this cost is far larger for difficulty routing than for consequence routing.

(3) Δx\Delta_x is not available before execution

Prompted forecasts of Δx\Delta_x reach ρ\rho between −0.07 and +0.05 with the true gain, and a learned probe on issue embeddings reaches ρ=0.11\rho = 0.11, with routing loss 219.50, worse than our 209.75. Routing by wxw_x is therefore the best rule available before execution, and wxΔxw_x\Delta_x becomes usable once in-flight signals exist. We will report the corrected gap in Section 4.2.

Q2. Typo. Fixed at line 262.

AuthorsOpenReview ↗

Response to Reviewer TAi8

Dear Reviewer TAi8,

Thank you for your detailed comments on the reference consequence labels, pre-solution prediction, and marginal compute gains. We have incorporated the additional analyses from our earlier rebuttal into the revised manuscript, with all changes highlighted in red. The main text now reports further prediction evidence and applicability conditions, while the appendices provide the judge-swap analysis, a gold-patch-free ranking procedure, and a more careful evaluation of priority-aware allocation.

W1. Sensitivity to the reference judge. We now assess the consequence labels using both human annotations and LLM-independent maintainer behavior (Sections 4.1 and Appendices B–D), and report the alternative-judge results in Appendix N. Under DeepSeek-derived evaluation weights on SWE-bench Lite, our fixed issue-only router achieves a weighted loss of 362.75, versus 378.31 for random allocation. On Verified DB/ORM, DeepSeek labels 77.1% of tasks as class 2; under these nearly flattened weights, the evaluated policies remain within approximately 1% of random allocation. The size of the gain tracks how strongly the evaluation weights differentiate tasks, as our objective predicts. When a judge labels most tasks as high consequence, the weights approach uniform, and any ranking, including the oracle, gains little over random allocation. Section 4.4 (Table 5) in the revised manuscript states this condition, and Appendix N reports the full judge swap.

W2 and Q1. Pre-solution prediction and calibration without gold patches. As reported earlier, none of the 110 class-2 tasks across six pools is predicted as class 0, and on Lite the predictor retains 93.9% of the reference-label gain (Section 4.1, Table 2, Appendix E). Appendix H now describes the calibration procedure without gold patches, which on Lite raises the over-severe signal's reduction relative to random from 3.0% to 7.0% (Table 19). Appendix P reports the sequential extension with a gold-free verifier (AUC 0.65), and the static issue-only router remains the stronger choice at the same attempt budget. Pre-solution information is therefore sufficient for routing, and its reliability in a new domain can be checked before deployment without gold patches.

W3. Gap to the ideal wxΔxw_x\Delta_x priority. Since Δx\Delta_x cannot be predicted before execution with any estimator we tested (Appendix I), consequence is the best routing signal available at deployment. It also wastes less compute on tasks with Δx≈0\Delta_x \approx 0 than difficulty does. Consequence routing selects never-solved tasks at about their pool frequency, while difficulty routing concentrates premium compute on them (Appendix J). The split-half protocol in Appendix E separates the outcomes used for selection from those used for evaluation, and lowers the oracle-priority reduction relative to outcome-derived difficulty from 33.8% to 27.2% (Section 4.2). Measured this way, the ideal priority leads our deployable router by roughly 4 to 5 points, not by the larger in-sample gap in Table 3. Consequence-only routing thus captures most of the attainable gain before execution, and reliable estimates of Δx\Delta_x are the way to close the rest.

These additions are presented in Sections 4.1, 4.2 and 4.4 and in Appendices E, H, J, N and P. We would be grateful if the reviewer would consider these points when finalizing the rating, and we remain happy to answer any further questions during the discussion period.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of the Discussion(Part 2 of 2)

  • Independent validation of failure consequence. We strengthened the evaluation with 150 human-annotated tasks, alternative LLM judges, and real software-maintainer behavior. Maintainer backport rates increase from 8.3% to 20.6% to 41.9% across the three consequence classes. The issue-only predictor achieves Cohen's κ=0.517\kappa=0.517 agreement with human-majority labels. Under human-defined weights, the consequence oracle achieves a loss of 55.2, compared with 67.5 for random and 75.8 for difficulty-based allocation. Under independent DeepSeek-derived weights on Lite, our fixed router also reduces loss from 378.31 to 362.75 relative to random allocation. We have moved the human-validation results into the main text. Together, these analyses provide evidence for consequence as a meaningful allocation signal beyond the primary LLM judge.

  • Prediction overhead and practical deployment. In response to Reviewer Wojm's follow-up, we conducted an additional analysis that charges consequence prediction against the same total monetary budget as execution, using recorded execution tokens and explicit pricing assumptions. Under Claude-equivalent predictor pricing, prediction accounts for 1.96% of the budget, while consequence-aware allocation achieves 10.3% lower weighted loss than random allocation (95% CI: [+4.2%, +17.5%]). Even under the conservative 4.60% prediction-cost scenario, the reduction remains 9.9% (95% CI: [+3.8%, +17.1%]). The comparison with uniform allocation remains statistically inconclusive. These results demonstrate that prediction overhead does not eliminate the observed allocation benefit under equal total spending in the evaluated setting.

  • Generalization, applicability, and evaluation resource. We extended the evaluation beyond SWE-bench to BIRD, FinQA, and MATH, and further examined a SWE-bench Verified DB/ORM pool and an online task stream. The results identify three conditions governing the value of consequence-aware allocation: heterogeneous failure costs, useful additional compute, and the relationship between difficulty and marginal gain. On MATH, difficulty-based routing is preferable under the original cost assumptions, while larger declared cost differences can reverse this ordering. We also constructed a consequence-annotated evaluation suite covering 1,290 tasks across six benchmark pools, including independent human annotations and maintainer-behavior associations, which we plan to publicly release. The revised manuscript now presents the cross-domain applicability conditions more prominently (Table 5). These results establish both the practical value and the operating boundaries of consequence-aware allocation.

We believe the core contribution should be evaluated as a complete test-time compute allocation methodology: a new consequence-weighted objective, its theoretical priority rule, a practical pre-solution allocation algorithm, and the empirical evidence supporting its design. The extensive validation experiments establish the motivation for the method, test its assumptions, and demonstrate its effectiveness under matched compute budgets.

Taken together, the original allocation results, stronger deployable baselines, independent human and maintainer-based validation, cross-predictor and cross-judge evaluations, new prediction-cost analysis, and cross-domain experiments provide a comprehensive empirical foundation for our proposed framework. They demonstrate the value of incorporating failure consequence into reasoning compute allocation, establish that the resulting benefits persist under several independent comparisons, and clarify the conditions under which this allocation principle is most useful.

We sincerely thank the Area Chair and all reviewers again for their time, thoughtful feedback, and consideration. We respectfully ask the Area Chair to consider the complete rebuttal discussion, additional experimental evidence, and revised manuscript in the final assessment.

Best regards,
The Authors

AuthorsOpenReview ↗

Summary of the Discussion(Part 1 of 2)

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair and all three reviewers for their time, careful evaluations, and constructive feedback. We especially appreciate Reviewer N5ba's continued engagement, acknowledgment of our useful findings, and suggestions for clarifying the methodological contributions. We also thank Reviewer Wojm for the follow-up questions on allocation optimality and prediction overhead, which motivated further analyses and an additional cost-accounting experiment. As the discussion period comes to a close, we regret that we have not had the opportunity to engage in further discussion with Reviewer TAi8 following our point-by-point rebuttal and subsequent follow-up. We respectfully ask the Area Chair to consider the additional evidence and revised manuscript alongside the original submission. We have incorporated the additional experiments, analyses, and clarifications into the revised manuscript, with all changes highlighted in red for ease of review.

Our core contribution is a consequence-aware test-time compute allocation framework for LLM reasoning that explicitly accounts for the unequal costs of failures. We formulate a consequence-weighted allocation objective, derive the optimal two-tier priority rule wxΔxw_x\Delta_x, and develop a practical pre-solution scheduler that allocates limited reasoning compute according to predicted failure consequence. The central methodological challenge is making allocation decisions before execution, when marginal compute gains are difficult to predict. Our work establishes a new allocation principle, provides a corresponding deployable method, and systematically evaluates its effectiveness and applicability.

We summarize the main evidence and clarifications from our rebuttal below.

  • Methodological contribution and routing signal. We distinguish task difficulty, failure probability, marginal compute gain, and failure consequence, and clarify their different roles in allocation. Under the same issue embeddings, predicting marginal gain achieves a Spearman correlation of only 0.115, while predicting consequence achieves 0.597. This predictability asymmetry motivates our consequence-based scheduler. Following Reviewer N5ba's suggestions, we strengthened the methodological presentation, added a pipeline figure (Figure 2), and clarified the differences from existing difficulty-based allocation, model routing, and related approaches. The method provides a practical allocation strategy derived from a consequence-weighted objective under the information available before execution.

  • Stronger baselines and matched-budget comparisons. We supplemented the original evaluation with ten additional deployable routing baselines, including learned difficulty, cheap-tier failure, marginal gain, and consequence predictors. On SWE-bench Lite, our method achieves a weighted loss of 209.75, compared with 224.00 for deployment-time predicted difficulty and 231.20 for random allocation. The direct loss reductions are 6.4% and 9.3%, respectively. The paired-bootstrap mean improvement over predicted difficulty is 6.9% (95% CI: [+2.2%, +11.8%]). In the controlled within-model Claude experiment, the corresponding paired-bootstrap improvement over predicted difficulty reaches 19.4% (95% CI: [+10.6%, +29.2%]). A learned consequence predictor achieves comparable performance (209.00), and six of eight predictor/prompt variants significantly outperform predicted difficulty. These results establish that the benefit extends beyond the original outcome-derived difficulty baseline and a particular predictor implementation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.