acceptodds
Desk rejected at ICLR 2027

When Local Choices Matter in Language Model Generation

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Language models can reach the same answer through different plans, and produce different answers from the same plan. To explain their errors and decide where to intervene, we need to know which choices influence the outcome. Differences between sampled answers mix the effect of a choice with randomness in later generation. We introduce outcome-resolution covariance (ORC), a matrix describing how future outcome probabilities vary across choices under a fixed continuation policy. Independent continuations separate choice effects from continuation noise and yield unbiased ORC estimates. We use one continuation to define intervention weights and another to predict their gain, separating effects of the shared choice from dependence within a continuation. Across three Qwen text-to-SQL panels, these forecasts more accurately predict gains on fresh continuations, including new questions and plans. On a 128-question panel, they reduce mean squared error by 80% relative to uncalibrated same-continuation forecasts; using them to allocate a fixed intervention budget offline improves correctness by 1.05 percentage points over equal-budget random selection. Recording outcome categories at different checkpoints reveals plan-specific failures that correctness scores hide and state differences that later computation removes. A covariance identity connects reweighting gains to probability shifts, branch variation, and target alignment. Ledger experiments show why alignment matters: on tasks with incorrect consensus targets, reweighting increases agreement while reducing correctness on average. Variation between branches provides the room for steering; alignment with the task determines the benefit.

Primary area:
interpretability and explainable AI
Keywords:
large language models, reasoning, process reward models, test-time compute, resampling, sensitivity analysis, variance decomposition, interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HNRoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how local choices in language model generation affect downstream outcomes. It introduces outcome-resolution covariance (ORC) to measure how outcome probabilities vary across choices and estimates it using independent continuations. SQL experiments show improved gain prediction on Qwen, while synthetic ledger experiments illustrate how target alignment affects intervention benefits.

Critical strengths
  • The distinction between branch effects and continuation noise is useful and statistically well motivated.

  • The categorical outcomes and checkpoint analyses provide useful diagnostic information beyond final correctness.

Critical weaknesses
  • The SQL interventions rely on oracle correctness feedback to construct reweighting policies and score potential interventions. It remains unclear how the method would operate without reference answers or reliable verifiers.
  • The computational overhead of the proposed method is not clearly accounted. Table 2 fixes the intervention budget at 32 questions, but selecting these questions involves generating 16 plans and two discovery continuations per plan for all 128 questions. This amounts to 2,048 plans and 4,096 discovery continuations, plus verification, whose costs are not captured by the reported intervention budget.
  • The main forecasting and allocation results concentrate on Spider and Qwen (Tables 1–2). Replication on Gemma is limited, and its improvement in forecast accuracy is not statistically significant in Table 1.
Questions for authors
  • How would the method obtain feedback without reference answers, and how robust are its predictions and intervention gains to verifier errors?
  • Could the authors examine how question rankings and selected interventions change across discovery seeds and different numbers of continuations per plan?
  • Could the authors compare against direct sampling and verification under the same total budget, including plan generation, discovery, and verification costs?
  • Could the main forecasting experiments be repeated on additional benchmarks and model families to assess the generality of the findings?
5 comments
AuthorsOpenReview ↗

Response to Reviewer HNRo: feedback without reference answers and verifier errors (W1, Q1)

Thank you for the detailed and constructive review. The forecasting procedure needs only binary outcome labels on the sampled continuations, which can come from the model's own consensus or from a verifier. The forecast then targets the gain in whatever outcome those labels measure. We use agreement with the model's consensus to obtain feedback without reference answers, and an imperfect execution verifier to test robustness to labeling errors. In three new studies the analysis plans and success criteria were fixed before any data were collected: E1 repeats the 128-question allocation panel with Mistral-Small-24B, and E2 and E3 apply the same design to 128 level-5 MATH-500 problems with Qwen and Gemma. The Qwen results on Spider below come from analyses that we added after submission, using the saved outputs.

To do without reference answers, we executed every saved query on the original databases and took as each plan's target the plurality execution result among the discovery answers of the other fifteen plans. Directions, forecasts and selections then use only agreement with this target. For the gain in agreement, the independent forecast (CROSS) has a bias of −0.35 [−0.88, 0.11] pp, against 1.65 [1.08, 2.16] pp for the same-continuation forecast (SAME). Reweighting under these labels raises true correctness by 1.14 [0.38, 1.88] pp when it is applied to every question, and by 0.70 [0.07, 1.23] pp when it is applied to the 32 questions that CROSS selects. How much of the agreement gain reaches correctness depends on whether the target is right:

Qwen, 128 questionsCountAgreement changeCorrectness change
Consensus correct96+2.8 pp+1.9 pp
Consensus wrong32+5.4 pp−1.1 pp

Each change is the average over all questions in the row; a question whose reweighting is uniform counts as zero.

With every question's direction applied, the wrong-consensus questions contribute −0.27 [−0.52, −0.01] pp to panel correctness, which reproduces on real SQL the target-alignment result of Section 5.4.

In the new studies CROSS again shows no detectable bias for the agreement gain, and SAME overestimates it. In E1, the biases are 0.48 [−0.14, 1.15] pp for CROSS and 1.58 [0.97, 2.23] pp for SAME. With the majority answer as the label in E2, they are 0.03 [−0.35, 0.41] and 1.81 [1.22, 2.43] pp. Because the majority is right on 124 of 128 problems, allocation under majority labels raises true correctness by 0.40 [0.16, 0.65] pp, close to the 0.46 pp obtained with reference answers.

Execution match on the original database, without the test suite, is a real verifier that makes errors: on the saved Qwen outputs, 29% to 30% of the queries it accepts fail the suite. For the gain this verifier measures, the bias of CROSS is −0.06 [−0.43, 0.32] pp, against 1.45 [0.85, 2.07] pp for SAME. Allocation with these labels raises test-suite correctness by 1.80 [0.96, 2.67] pp, 77% of the 2.33 pp obtained with test-suite labels, and exceeds random selection by 0.92 [0.40, 1.45] pp. In E1, where this check was pre-specified, 41% of the queries this verifier accepts fail the suite, and the biases of CROSS and SAME are 0.16 [−0.56, 0.85] and 1.34 [0.95, 1.73] pp.

When every label is flipped independently with probability ε, the expected measured gain is scaled by 1 − 2ε, because the direction weights sum to zero. In a simulation on the saved Qwen outputs, the mean bias of CROSS relative to the gain that the noisy verifier itself measures stays between 0.04 and 0.23 pp for ε between 0.05 and 0.3, against 0.27 to 1.64 pp for SAME.

AuthorsOpenReview ↗

Response to Reviewer HNRo: compute and the equal-budget comparison (W2, Q3)

On W2, the selection stage of the 128-question panel consists of 2,048 plans and 4,096 discovery continuations, which generated 0.67M and 0.87M tokens in 559 s and 414 s on one 4×H100 node. Test-suite verification ran concurrently on a local CPU and finished scoring the discovery continuations 40 s after generation ended. The 16,384 evaluation continuations (3.46M tokens, 1,526 s) are not part of the method: they directly measure the gains that the forecasts predict from the discovery continuations, and here they only validate the forecasts. Every selector in Table 2 and Appendix M.2 is computed from the same discovery data, so the selectors differ only in how they use it.

The alternatives raised in Q3 draw on the same pool of 16 plans and 32 discovery answers per question, and therefore incur the same plan-generation and discovery costs as above. The table gives their correctness on the paper's panel and on three new studies whose analysis plans were fixed before any data were collected. E1 runs Mistral-Small-24B on the same Spider questions, and E2 and E3 run Qwen and Gemma on 128 level-5 MATH-500 problems. The first three rows choose one answer per question among the 32 discovery answers. The last two rows give the expected correctness of a single new sample, which is the quantity that a reweighting of the plans changes:

Qwen, SpiderMistral, Spider (E1)Qwen, MATH-500 (E2)Gemma, MATH-500 (E3)
One answer chosen among the 32 discovery answers
Best of 32 with the reference answer (upper bound)73.4%71.9%97.7%96.1%
Best of 32 with the original-database verifier59.5%51.9%n/an/a
Majority vote of 3254.6%37.7%96.9%96.1%
Expected correctness of one new sample
Uniform plans49.1%16.9%87.1%81.2%
Plans reweighted on 32 questions by the independent forecast (CROSS)51.5%17.3%87.6%81.4%

In the Qwen Spider column, the two single-sample rows are from Appendix M.2 and the three selection rows come from an analysis that we added after submission, using the saved outputs. In E3, 14% of the evaluation continuations end without the closing line that the parser requires and count as incorrect, which mainly lowers the two single-sample rows.

The two groups of rows answer different questions, and the same discovery pool serves both. The first group returns one answer to a question for which 32 answers have already been sampled. The second group concerns samples that have not been drawn yet: the forecast says in advance how much a change to the plan distribution will change their correctness, which is what a decision to steer generation depends on. The evaluation continuations confirm the forecast: on all four panels the bias interval of CROSS contains zero.

AuthorsOpenReview ↗

Response to Reviewer HNRo: stability, generality and Gemma (Q2, Q4, W3)

On Q2 (stability), analyses of the saved outputs of the 128-question panel, added after submission, show that allocation gains change little and most selections are shared across the continuations that were drawn and across numbers of continuations per plan. Each of the eight evaluation continuations gives an independent single-continuation forecast (CROSS) for the frozen directions. On the remaining continuations, the allocation that each of these forecasts selects realizes 2.24 to 2.41 pp, against 2.09 to 2.13 pp for the allocation of the same-continuation forecast (SAME). Any two of the nine single-continuation CROSS forecasts, these eight and the original discovery forecast, share 23 to 30 of their 32 selections. Allocations based on forecasts from one, two or four continuations per plan realize 2.33, 2.37 and 2.40 pp, with Spearman correlations of 0.86, 0.91 and 0.93 with the held-out gains. Rebuilding the directions from the other discovery continuation retains 27 of the original 32 CROSS selections and reproduces the forecast advantage: the MSE of SAME is 13.8 pp² against 4.1 pp² for CROSS, a difference of 9.67 [5.70, 15.50] pp².

On the original 32-question Qwen panel, an analysis of the saved outputs, also added after submission, finds the loss difference L of Table 1 positive for all 45 ways of choosing the two discovery continuations among the ten per plan, with a mean of 11.2 [5.6, 18.3] pp².

On Q4 (generality), three new studies repeat the main forecasting experiment on a third model family and a second benchmark, with analysis plans and success criteria fixed before any data were collected and directions frozen before any evaluation continuation was generated. E1 repeats the 128-question Spider panel with Mistral-Small-24B, and E2 and E3 use 128 level-5 MATH-500 problems with Qwen and Gemma. Under the pre-specified scoring rules, all three studies show the same pattern as the paper's Qwen panel: SAME overestimates the gain, every CROSS bias interval contains zero, and CROSS has the lower squared error.

PanelDirectionsSAME bias, ppCROSS bias, ppSAME − CROSS sq. error, pp²
Qwen, Spider (paper)802.08 [1.56, 2.72]0.30 [−0.07, 0.73]12.46 [7.67, 18.92]
Mistral, Spider (E1)680.79 [0.24, 1.42]0.07 [−0.46, 0.62]5.12 [1.31, 8.60]
Qwen, MATH-500 (E2)901.79 [1.21, 2.41]0.10 [−0.28, 0.48]10.55 [6.25, 15.48]
Gemma, MATH-500 (E3)†870.94 [0.50, 1.43]−0.04 [−0.32, 0.23]5.52 [1.41, 10.85]

Directions: questions whose reweighting is not uniform. Brackets give 95% bootstrap intervals from resampling databases on Spider and problems on MATH-500. †On answer format in E3, see the last paragraph.

On W3 (Gemma), the new Gemma panel (E3) has 87 nonuniform directions, and under the pre-specified scoring CROSS lowers the squared error by 5.52 [1.41, 10.85] pp² relative to SAME. The original Gemma panel left little to compare: 55 of its 64 reweighting directions are uniform, with zero predicted and realized gain, which leaves nine for the forecast comparison. An analysis of the saved outputs, added after submission, shows that even with all ten continuations per plan, Gemma's draft correctness varies across plans on only 10 of the 32 questions. In E3 most of the variation in final correctness across plans comes from answer format: Gemma often ends a correct solution without the closing line that the parser requires. When such answers are accepted, in a check made after the data were collected, Gemma is correct on 95.0% of continuations and the reweightings change correctness by less than 0.1 pp on average, so little is left to forecast, and the difference in squared error is −0.22 [−0.77, 0.22] pp². The overstatement from reusing a continuation is still present in that check: in the pre-specified analysis of E3 that reweights the plans by the final answer of one continuation (Appendix N.4 of the revised PDF), the gain read from that same continuation is too high by 0.73 [0.34, 1.20] pp, against −0.05 [−0.19, 0.08] pp when it is read from an independent continuation.

Reviewer_HiUhOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper asks how a local choice during generation affects the final outcome. It defines a covariance matrix describing how outcomes vary across such choices, and estimates it by generating several independent continuations from each fixed choice. It also argues that, when reweighting choices to improve outcomes, the expected gain should be predicted using a continuation independent of the one used to set the weights. Experiments on text-to-SQL and synthetic inventory-ledger tasks show that independent forecasts are less biased than reused ones.

Critical strengths
  1. The experimental reporting is careful: weights and selections are fixed before evaluation data are generated, confidence intervals are reported throughout.

  2. The target-alignment result gives empirical evidence for caution when steering toward the model's own consensus. This is relevant to methods that use majority vote or other proxy rewards. The conclusions may be potentially useful to the community.

Critical weaknesses
  1. The paper's question is clear: "when do local choices matter in language model generation?" But it tries to answer several things at once, like independent forecasts, outcome resolution and target alignment, spread across math, SQL and ledger experiments. The SQL forecast result seems to be the main result, but it is not clear how much the other parts add to it. Many new terms and the heavy reliance on the appendix also make the paper hard to follow.
Questions for authors
  1. CROSS reduces forecast MSE by about 80% relative to SAME, but the two allocations differ by only 0.22 pp (Section 5.1), and Appendix M.4 notes they share 25 of 32 selections. Why does the large gain in forecast accuracy translate into so little difference in selection?

  2. The paper motivate ORC by pointing to rollout-based methods for judging step importance (for example, Math-Shepherd, Thought Anchors, Critical Tokens). However, the experiments do not apply the correction to any of these settings. Does the noise correction or independent forecasting change the conclusions or downstream performance of such methods?

5 comments
AuthorsOpenReview ↗

Response to Reviewer HiUh: forecast accuracy and allocation (Q1)

Thank you for the careful reading and for the clear question in Q1. Forecast accuracy and fixed-budget allocation measure different things: the squared error depends on how accurately the forecasts estimate the size of each gain, whereas the allocation depends only on their order. Analyses of the saved outputs of the 128-question panel, added after submission, show why a large gain in forecast accuracy changes the selection so little.

First, part of the difference in squared error concerns calibration rather than ranking. The squared biases are 4.34 pp² for SAME and 0.09 pp² for CROSS, and their difference accounts for 34% of the difference in mean squared error. Subtracting a common offset from the forecasts changes their squared error but leaves the allocation unchanged.

Second, both forecasts identify the questions with the largest realized gains. The gains are concentrated: the eight largest gains average 13.2 pp and the next sixteen average 9.5 pp. Both forecasts select all of the largest eight; among the largest 24, SAME selects 21 and CROSS selects 23. The two allocations share 25 of their 32 questions (Appendix M.4). SAME predicts about 11 to 15 pp for its seven other picks, which realize 3.92 pp on average, against 7.92 pp for the seven that CROSS chooses instead. The two allocations therefore differ by 7 × (7.92 − 3.92)/128, or 0.22 pp.

Third, CROSS is already close to a split-sample reference that sees more data. Ranking with four of the eight evaluation continuations per plan and scoring on the other four, averaged over all 70 such splits, gives 2.40 pp, against 2.33 pp for CROSS and 2.11 pp for SAME.

Forecast accuracy does matter when a decision depends on the size of the gains. In the same analyses, averaged over all 128 questions, SAME predicts a gain of 3.29 pp for its selected allocation and realizes 2.11 pp, while CROSS predicts 2.60 pp and realizes 2.33 pp. Consider a rule that intervenes only when the forecast gain exceeds a cost of 5 pp, with net value defined as the realized gain minus the cost of the interventions. Deciding with CROSS instead of SAME raises its panel-average net value from 0.65 to 1.13 pp, a difference of 0.48 [0.33, 0.62] pp.

AuthorsOpenReview ↗

Response to Reviewer HiUh: the structure of the paper (W1) and rollout-based importance methods (Q2)

On the structure of the paper (W1), the parts answer one question in three steps: ORC measures how much a local choice matters for each outcome category and checkpoint, the independent forecast says how much a given reweighting will change the outcome, and target alignment says whether that change helps the task. An analysis of the saved Qwen outputs, added after submission, shows the third step on real SQL. We set each plan's target to the plurality execution result among the model's own answers from the other fifteen plans. Reweighting toward this consensus raises agreement by 5.4 pp and lowers correctness by 1.1 pp on the 32 of 128 questions where the consensus is wrong, which reproduces the synthetic-ledger result of Section 5.4.

The revised PDF follows this sequence. The introduction poses the three steps as questions on a running SQL example (page 2). During the discussion period we ran three new studies whose analysis plans were fixed before any data were collected. Their main results and those of the SQL consensus analysis are in the main text: the forecasts and the decisions with an intervention cost are in Table 3 and Section 5.1, and target alignment on SQL is in Section 5.4. The main text keeps ORC, capacity, temporal alignment and target alignment, and replaces readout, branch, replica, suffix and future law with plain words.

On Q2, two of the new studies show that a gain read from the rollouts that selected it overstates what independent rollouts show. When the rollouts that select a step also report its value, the reported value is biased by that selection, and the value from an independent rollout is not. For the noise correction, the setting in the paper closest to the methods named in Q2 is the token-level analysis of Section 5.1 and Appendix H.2. On 60 prefixes from 30 math problems, with the top 20 candidate tokens at each prefix and four continuations per candidate, the correction lowers the mean squared error of the capacity estimate against an independent reference by 62%.

One of the three new studies, E2, includes a pre-specified analysis that labels the model's plans by rollout outcomes, as Math-Shepherd labels steps. Qwen samples 16 plans for each of 128 level-5 MATH-500 problems, each plan is labeled by the outcome of one rollout, and the plans are reweighted by these labels. Reported from the same rollouts, the gain of this reweighting is too high by 4.04 [3.22, 4.90] pp. Reported from an independent rollout, it shows no detectable bias (0.27 [−0.11, 0.65] pp), and the squared error falls by 35.2 [25.8, 45.5] pp². With Gemma on the same problems (E3), the same-rollout report is too high by 4.18 [3.25, 5.14] pp, against −0.05 [−0.41, 0.30] pp for an independent rollout. Gemma often omits the closing line that the parser requires. When answers without this line are accepted, in a check made after the data were collected, the same-rollout report is still too high by 0.73 [0.34, 1.20] pp, against −0.05 [−0.19, 0.08] pp. This analysis is in Appendix N.4 of the revised PDF, and its E2 result is also in Section 6.

On downstream use, the effect depends on how the estimates are used. A selection that needs only their order changes little, as in Q1. A decision that depends on their size does change: in an analysis added after submission, at a cost of 5 pp per intervention, deciding with the independent forecast instead of the same-rollout forecast raises the gain net of the cost from 0.65 to 1.13 pp on the paper's 128-question panel.

AuthorsOpenReview ↗

Response to Reviewer HiUh: independent rollouts applied to Thought Anchors (Q2)

On Q2, we have now applied independent rollouts to one of the methods that the question names, Thought Anchors, on the rollouts that its authors released. The importance read for the top-ranked sentence of a trace is overstated, averages over sentence categories and the sentence of its case study hold up, and independent rollouts change which way of spending a fixed budget of rollouts looks better.

In this post hoc analysis, added during the discussion, we used the release on Hugging Face (uzaymacar/math-rollouts): reasoning traces of DeepSeek R1-Distill-Qwen-14B and R1-Distill-Llama-8B on 20 MATH problems each, with about 100 rollouts resampled from the start of every sentence. We use the correct trace of each problem and the accuracy-based resampling importance of a sentence, the accuracy of rollouts that keep the sentence minus the accuracy of rollouts that resample it. The release gives this importance for every sentence, and we reproduce the released values exactly on all 39 complete traces. Thought Anchors shows these accuracies in its case study, and its category analyses use measures based on the KL divergence between answer distributions, which we leave aside.

For each trace, we split the rollouts of every sentence into two halves 200 times. We ranked the sentences with 10, 20 or 50 rollouts per sentence from the first half, and read the importance of the sentence ranked first both from those rollouts and from the second half, which played no part in the ranking. This mirrors the paper's comparison of SAME, read from the continuation that set the weights, and CROSS, read from an independent continuation; here a ranking of sentences takes the place of a reweighting of plans.

The importance of the top-ranked sentence is overstated when it is read from the rollouts that ranked it, even at 50 rollouts per sentence, the most that a half holds.

Rollouts per sentence102050
Importance on the ranking rollouts (pp)44.634.327.0
Importance on independent rollouts (pp)7.110.916.1
Share that holds up16% [11%, 22%]32% [22%, 42%]60% [47%, 71%]

The share is the importance on independent rollouts divided by that on the ranking rollouts, and brackets are 95% intervals from resampling the traces.

On the conclusions, independent rollouts change the importance of a sentence chosen because it ranks first, as the table shows, while averages over many sentences hold up. In this data the categories with the largest mean importance are uncertainty management for R1-Distill-Qwen-14B and plan generation for R1-Distill-Llama-8B, the two categories that Thought Anchors singles out. With 10 to 50 rollouts per sentence, their mean importance is 0.72 to 0.80 pp and 0.69 to 0.71 pp on the rollouts that rank them, and 0.76 and 0.63 pp on independent rollouts. A sentence with a large effect also holds up. Sentence 13 of problem 4682, at the center of the case study of Thought Anchors, has an importance of 42 pp, against at most 11 pp for every other sentence of its trace. With 50 rollouts per sentence, it is ranked first on both halves in all 200 splits.

On downstream performance, independent rollouts change which way of spending a fixed budget looks better. When the goal is to find the most important sentence of a trace, a two-stage rule spends half of the budget evenly over the sentences and the other half on the sentences that lead after this first stage. We compared it with uniform allocation of the same total, 10 or 20 rollouts per sentence, on a second set of 200 splits. On independent rollouts, the two-stage rule raises the importance of the selected sentence by 3.1 [2.1, 4.1] pp, to 10.4 pp, and by 3.8 [2.7, 4.9] pp, to 14.8 pp, with an interval above zero for each model. Read from the rollouts that selected them, the comparison points the other way: uniform allocation reports 44.8 and 34.4 pp, and the two-stage rule 24.6 and 26.2 pp. Uniform allocation needs about 18 and 39 rollouts per sentence, almost twice as many, to select sentences this important.

Appendix R of the revised PDF gives the details.

Reviewer_cm3DOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates how local generation choices influence final outcomes and how their effects can be separated from subsequent sampling noise. Experiments cover two language models and several hundred evaluation instances.

Critical strengths

The research question is interesting. The paper examines how local choices during language model generation influence final outcomes, a worthwhile question for analyzing model behavior.

Critical weaknesses
  1. Insufficiently demonstrated practical value. Improved forecasting accuracy is not accompanied by convincing evidence of superior task performance over simpler alternatives, leaving the framework’s practical benefit unclear.

  2. Fragmented presentation. The unclear organization and dispersed quantitative reporting disrupt the progression from methodology to empirical validation, making the central contributions difficult to assess.

Questions for authors

Please find the weakness above.

4 comments
AuthorsOpenReview ↗

Response to Reviewer cm3D: practical value

Thank you for the review. An accurate forecast has practical value when a decision depends on the size of a gain and not only on a ranking, as when deciding whether an intervention is worth its cost in compute or review. With such a cost, the independent forecast (CROSS) recommends fewer interventions than the same-continuation forecast (SAME) in every row of the table below, and the correctness gain net of the cost is higher with CROSS in all five rows, with a 95% interval for the difference that excludes zero in four. The first two rows use the paper's own panels, in analyses that we added after submission, and the last three are new studies in which the cost was fixed in advance.

We ran the three new studies during the discussion period, with analysis plans and success criteria fixed before any data were collected. E1 repeats the 128-question allocation panel of Appendix M with Mistral-Small-24B, a third model family, and E2 and E3 apply the same design to 128 level-5 MATH-500 problems with the paper's two models, Qwen and Gemma. Every question has 16 frozen plans, two discovery continuations per plan from which both forecasts are computed, and eight independent evaluation continuations per plan on which the gains are measured.

Both forecasts estimate how much a specified reweighting of the plans will change correctness on fresh continuations. The reweighting is built from one discovery continuation of each plan. SAME predicts its gain from the outcomes of that same continuation, and CROSS predicts it from the outcomes of each plan's other discovery continuation. The fixed-budget allocation of Table 2, which E1 to E3 repeat, depends only on the ranking of the forecasts and not on their size, so it distinguishes the two forecasts less than a decision with a cost does. In this allocation, under the pre-specified scoring, CROSS exceeds SAME in E2 by 0.21 [0.01, 0.44] pp.

For a decision with a cost, we use a rule that intervenes on a question only when its forecast gain exceeds a cost c per intervention. Its net value is the realized gain minus c, summed over the questions where the rule intervenes and divided by the number of questions in the panel, which makes it a correctness gain net of the cost. The analysis plans of the new studies fixed the cost at 5 pp. The table gives the number of interventions each forecast recommends and the difference in net value:

PanelInterventions, SAMEInterventions, CROSSCROSS − SAME net value, pp
Qwen, Spider, 128 questions (paper)*64430.48 [0.33, 0.62]
Qwen, Spider, two 32-question panels (paper)*62340.34 [0.12, 0.60]
Mistral, Spider (E1)22120.30 [−0.03, 0.71]
Qwen, MATH-500 (E2)2580.65 [0.37, 0.96]
Gemma, MATH-500 (E3)†1340.28 [0.10, 0.50]

*Computed after submission on the saved outputs. On the two 32-question panels each question has two directions, one set by each discovery continuation, as in the paper's analysis of these panels, so their counts are out of 128 directions. The reweighting strength is that of the 128-question panel. †In E3 most of the variation in final correctness across plans comes from answer format (Appendix N.3 of the revised PDF). All intervals are 95% bootstrap intervals from resampling databases on Spider and problems on MATH-500.

On the paper's Qwen panels, in the analyses added after submission, deciding with CROSS instead of SAME raises the net value from 0.65 to 1.13 pp on the 128-question panel and from 0.44 to 0.78 pp on the two 32-question panels. On the 128-question panel, 6 of CROSS's 43 interventions gain less than the cost on the evaluation continuations, against 24 of SAME's 64.

In E1, E2 and E3† few interventions are worth 5 pp: averaged over all questions, the reweightings gain 0.65, 0.63 and 0.18 pp. SAME nevertheless forecasts 1.44, 2.42 and 1.12 pp and recommends 22, 25 and 13 interventions, while CROSS forecasts 0.72, 0.73 and 0.15 pp and recommends 12, 8 and 4. The pre-specified difference in net value favors CROSS in all three studies, and its interval excludes zero in E2 and E3.

AuthorsOpenReview ↗

Response to Reviewer cm3D: presentation and soundness

On presentation, the revised PDF is built around one running SQL example and three questions: how much of the outcome variation is due to the plan, how much a reweighting of the plans will gain, and whether that gain is worth its cost and, when plans are judged by a proxy, carries over to correctness. The introduction poses the three questions (page 2), Sections 2 to 4 define the quantity behind each one and how to estimate it, and Section 5 answers them. The theory beyond Eqs. (1) to (6) is now in the appendix, and a new Eq. (7) sets the same-continuation forecast and the independent forecast side by side. The main text keeps ORC, capacity, temporal alignment and target alignment, and replaces readout, branch, replica, suffix and future law with plain words. The results across models and tasks are collected in Table 3, with forecast accuracy in part A and decisions at an intervention cost in part B.

On soundness, the statistical basis is the noise correction of Eq. (6) and the conditional-expectation identity of Appendix I.3, Eq. (66). Put simply, the continuation that sets the weights also carries its own luck, and a gain read from that continuation counts this luck as an effect of the plan. A second continuation of the same plan does not share it. When the continuations of a plan are independent draws from the same distribution, the independent forecast (CROSS) is unbiased for the expected gain of the frozen reweighting on fresh continuations, while the same-continuation forecast (SAME) carries the additional dependence term of Eq. (67). Four panels test these predictions in finite samples: the paper's 128-question Qwen panel on Spider and three studies we ran during the discussion period, with analysis plans and success criteria fixed before any data were collected. E1 repeats the Spider panel with Mistral-Small-24B, and E2 and E3 run Qwen and Gemma on 128 level-5 MATH-500 problems. In all four, reweighting directions were frozen before any evaluation continuation was generated, and the intervals are 95% bootstrap intervals from resampling databases on Spider and problems on MATH-500:

PanelSAME bias, ppCROSS bias, ppSAME − CROSS sq. error, pp²
Qwen, Spider (paper)2.08 [1.56, 2.72]0.30 [−0.07, 0.73]12.46 [7.67, 18.92]
Mistral, Spider (E1)0.79 [0.24, 1.42]0.07 [−0.46, 0.62]5.12 [1.31, 8.60]
Qwen, MATH-500 (E2)1.79 [1.21, 2.41]0.10 [−0.28, 0.48]10.55 [6.25, 15.48]
Gemma, MATH-500 (E3)†0.94 [0.50, 1.43]−0.04 [−0.32, 0.23]5.52 [1.41, 10.85]

The first row is from Appendix M.3. †In E3 most of the variation in final correctness across plans comes from answer format (Appendix N.3 of the revised PDF).

Under the pre-specified scoring rules, SAME is biased upward on every panel, and all four CROSS bias intervals contain zero. Each 95% interval for the bias of SAME excludes zero, and the estimated bias of CROSS ranges from −0.04 to 0.30 pp. SAME's overestimate is large relative to the gains themselves: the mean realized gains in E1, E2 and E3 are 0.65, 0.63 and 0.18 pp. The paired difference in squared error, SAME minus CROSS, is positive on every panel, and each of its 95% intervals excludes zero. If the soundness score rests on a point that this response does not cover, we would be glad to address it.

Thanks for the clarification. I will raise my assesment to 2.

AuthorsOpenReview ↗

Thank you for the updated assessment

Thank you for reading our responses and for raising your rating. In response to your point on dispersed quantitative reporting, the revised PDF now collects the forecast results and the cost-based decisions for all models and tasks in a single table (Table 3), and we have edited the text throughout for clarity.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response: new studies and what they show

We thank the reviewers and the AC. We have added evidence on a third model family, a mathematics benchmark, feedback without reference answers and decisions with intervention costs. During the discussion period we ran three new studies whose analysis plans and success criteria were fixed before any data were collected. E1 repeats the 128-question Spider allocation panel of Appendix M with Mistral-Small-24B, and E2 and E3 use the paper's two models, Qwen3.8-27B and Gemma-4-31B-it, on 128 level-5 MATH-500 problems. Each question has 16 plans, each with two discovery continuations and eight independent evaluation continuations. The reweighting directions and selections were frozen before any evaluation continuation was generated.

Under the pre-specified scoring rules, the independent forecast (CROSS) has lower squared error than the same-continuation forecast (SAME) on all four panels. SAME overestimates the gain on every panel, while all four CROSS bias intervals contain zero. In E1, E2 and E3†, SAME forecasts gains of 1.44, 2.42 and 1.12 pp and CROSS forecasts 0.72, 0.73 and 0.15 pp, while the evaluation continuations deliver 0.65, 0.63 and 0.18 pp.

PanelSAME bias, ppCROSS bias, ppSAME − CROSS sq. error, pp²
Qwen, Spider (paper)2.08 [1.56, 2.72]0.30 [−0.07, 0.73]12.46 [7.67, 18.92]
Mistral, Spider (E1)0.79 [0.24, 1.42]0.07 [−0.46, 0.62]5.12 [1.31, 8.60]
Qwen, MATH-500 (E2)1.79 [1.21, 2.41]0.10 [−0.28, 0.48]10.55 [6.25, 15.48]
Gemma, MATH-500 (E3)†0.94 [0.50, 1.43]−0.04 [−0.32, 0.23]5.52 [1.41, 10.85]

Brackets give 95% bootstrap intervals from resampling databases on Spider and problems on MATH-500. †In E3 most of the variation in final correctness across plans comes from answer format (Appendix N.3 of the revised PDF).

Without the noise correction, capacity is overstated by a factor of four to six on the new panels. Capacity is the variance across plans of each plan's probability of a correct answer. In E1, E2 and E3 its uncorrected estimate from the two discovery continuations per plan is 6.3, 3.9 and 4.9 times a reference computed from the eight evaluation continuations, and the interval of the noise-corrected estimate contains the reference in all three.

Forecast size matters when interventions have a cost. We test this with a rule that intervenes on a question only when its forecast gain exceeds a cost of 5 pp. Its net value is the realized correctness gain minus the cost of the interventions, averaged over all questions of a panel. In analyses of the paper's saved Qwen outputs on Spider, added after submission, the rule reaches a net value of 1.13 pp with CROSS and 0.65 pp with SAME on the 128-question panel, a difference of 0.48 [0.33, 0.62] pp, and it intervenes on 43 questions with CROSS and on 64 with SAME. The paper's two 32-question Qwen panels give 0.78 pp against 0.44 pp, a difference of 0.34 [0.12, 0.60] pp. The new studies tested the same rule with the cost fixed in advance. The CROSS − SAME differences in net value are 0.30 [−0.03, 0.71], 0.65 [0.37, 0.96] and 0.28 [0.10, 0.50] pp in E1, E2 and E3†. On these panels few interventions are worth 5 pp, and CROSS recommends 12, 8 and 4 where SAME recommends 22, 25 and 13.

With feedback from the model's own consensus, the forecast and the intervention both work without reference answers. In an analysis of the saved Qwen outputs, added after submission, we used agreement with the consensus as the label for directions, forecasts and selections. For the gain in agreement, CROSS then has a bias of −0.35 [−0.88, 0.11] pp, against 1.65 pp for SAME. Reweighting every question under these labels raises true correctness by 1.14 [0.38, 1.88] pp, and the CROSS allocation on 32 questions raises it by 0.70 [0.07, 1.23] pp. With the majority answer as the label in E2, the biases of CROSS and SAME are 0.03 [−0.35, 0.41] and 1.81 pp, and the allocation raises true correctness by 0.40 [0.16, 0.65] pp. The SQL analysis also shows why target alignment matters: averaged over the 32 of 128 questions where the consensus is wrong, reweighting toward it raises agreement by 5.4 pp and lowers correctness by 1.1 pp, which reproduces on real SQL the synthetic-ledger result of Section 5.4.

The revised PDF follows one argument in three steps: measuring how much a local choice matters, forecasting the gain of a given reweighting, and checking whether that gain helps the task. The introduction poses these steps as three questions on a running SQL example. The results above are in the main text (Table 3, Sections 5.1 and 5.4) and in Appendices N to P, and the main text uses fewer coined terms.

AuthorsOpenReview ↗

Revised PDF: what changed and where to find it

We have uploaded a revised PDF. The text now follows one argument, as described in our responses. Changes are marked in blue, and the main text stays within ten pages.

The revised paper follows one running example and three questions. The abstract has been rewritten, and the introduction is now built around a text-to-SQL example in which the model writes a plan before its query (page 2). The introduction asks how much of the outcome variation is due to the plan, how much a reweighting of the plans will gain, and whether that gain is worth its cost and, when plans are judged by a proxy, carries over to correctness. Sections 2 to 4 define the quantity behind each question and how to estimate it, with the definitions illustrated on this example, and Section 5 answers the three questions. A new Eq. (7) sets the two forecasts side by side, and the text explains why the same-continuation forecast (SAME) overestimates the gain and the independent forecast (CROSS) does not. The main text keeps ORC, capacity, temporal alignment and target alignment, and replaces readout, branch, replica, suffix and future law with plain words.

The main results of the three new studies (E1 to E3, called the replication studies in the PDF) are in the main text. The results across models and tasks are collected in Table 3. Part A gives forecast accuracy on the four 128-question panels (Qwen and Mistral on Spider, Qwen and Gemma on MATH-500): under the pre-specified scoring rules, SAME overestimates the gain on every panel, while the bias interval of CROSS contains zero on each. Part B reports decisions at an intervention cost of 5 pp, scored by net value (the realized gain minus the cost of the interventions), and CROSS recommends fewer interventions in every row. Two new paragraphs of Section 5.1 present these results, and Appendices N to Q give the designs, the pre-specified criteria and further tables.

For Reviewer cm3D, the practical value of an accurate forecast (W1) has its own paragraph in Section 5.1, "Forecast size matters when interventions have a cost", with Table 3B. On the 128-question Qwen panel, in an analysis added after submission, deciding with CROSS instead of SAME raises the net value from 0.65 to 1.13 pp, with 43 interventions instead of 64. The progression from method to validation (W2) now follows the three questions in order, with the results across panels in one table. On soundness, Section 4 states the two forecasts in Eq. (7) and explains why reusing a continuation biases SAME, and Table 3A tests this on four panels.

For Reviewer HiUh, Section 5.1 explains why an 80% lower squared forecast error changes the fixed-budget selection so little (Q1). A fixed budget uses only the order of the forecasts, whereas their size matters once interventions have a cost, and Appendix O.2 gives the analysis. On Q2, the rollout-label result is in Section 6 and Appendix N.4: when plans are reweighted by the final-answer label of one rollout, the gain read from the same rollout is overstated by 4.0 pp on the Qwen mathematics panel, and an independent rollout shows no detectable bias. Appendix R, added later in the discussion, applies the same check to the released rollouts of Thought Anchors. On W1, the parts of the paper answer one question in three steps, and the introduction poses each step as a question. Section 5.1 covers the first two steps and the cost of an intervention, Sections 5.2 and 5.3 follow the plan's effect across checkpoints and outcome categories, and Section 5.4 tests target alignment on the ledger task and on SQL.

For Reviewer HNRo, feedback without reference answers (W1, Q1) is in Section 5.4, where the model's own consensus replaces the test suite, and Appendix P adds an imperfect verifier and random label errors. Appendix Q.1 gives the stability of forecasts and selections across the continuations that were drawn and across numbers of continuations per plan (Q2), and Appendix Q.2 gives the sampling cost and the comparison with selecting among the sampled answers (W2, Q3). The third model family, the mathematics benchmark and a 128-question Gemma panel (W3, Q4) are in Section 5.1, Table 3 and Appendix N.

The revision adds one reference, Hendrycks et al. (2021), for the MATH dataset. The title and the section numbers are unchanged, and so are the equation, table and appendix numbers that the reviews and our responses cite from the submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.