- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how local choices in language model generation affect downstream outcomes. It introduces outcome-resolution covariance (ORC) to measure how outcome probabilities vary across choices and estimates it using independent continuations. SQL experiments show improved gain prediction on Qwen, while synthetic ledger experiments illustrate how target alignment affects intervention benefits.
Critical strengths
-
The distinction between branch effects and continuation noise is useful and statistically well motivated.
-
The categorical outcomes and checkpoint analyses provide useful diagnostic information beyond final correctness.
Critical weaknesses
- The SQL interventions rely on oracle correctness feedback to construct reweighting policies and score potential interventions. It remains unclear how the method would operate without reference answers or reliable verifiers.
- The computational overhead of the proposed method is not clearly accounted. Table 2 fixes the intervention budget at 32 questions, but selecting these questions involves generating 16 plans and two discovery continuations per plan for all 128 questions. This amounts to 2,048 plans and 4,096 discovery continuations, plus verification, whose costs are not captured by the reported intervention budget.
- The main forecasting and allocation results concentrate on Spider and Qwen (Tables 1–2). Replication on Gemma is limited, and its improvement in forecast accuracy is not statistically significant in Table 1.
Questions for authors
- How would the method obtain feedback without reference answers, and how robust are its predictions and intervention gains to verifier errors?
- Could the authors examine how question rankings and selected interventions change across discovery seeds and different numbers of continuations per plan?
- Could the authors compare against direct sampling and verification under the same total budget, including plan generation, discovery, and verification costs?
- Could the main forecasting experiments be repeated on additional benchmarks and model families to assess the generality of the findings?
5 comments
Response to Reviewer HNRo: compute and the equal-budget comparison (W2, Q3)
On W2, the selection stage of the 128-question panel consists of 2,048 plans and 4,096 discovery continuations, which generated 0.67M and 0.87M tokens in 559 s and 414 s on one 4×H100 node. Test-suite verification ran concurrently on a local CPU and finished scoring the discovery continuations 40 s after generation ended. The 16,384 evaluation continuations (3.46M tokens, 1,526 s) are not part of the method: they directly measure the gains that the forecasts predict from the discovery continuations, and here they only validate the forecasts. Every selector in Table 2 and Appendix M.2 is computed from the same discovery data, so the selectors differ only in how they use it.
The alternatives raised in Q3 draw on the same pool of 16 plans and 32 discovery answers per question, and therefore incur the same plan-generation and discovery costs as above. The table gives their correctness on the paper's panel and on three new studies whose analysis plans were fixed before any data were collected. E1 runs Mistral-Small-24B on the same Spider questions, and E2 and E3 run Qwen and Gemma on 128 level-5 MATH-500 problems. The first three rows choose one answer per question among the 32 discovery answers. The last two rows give the expected correctness of a single new sample, which is the quantity that a reweighting of the plans changes:
| Qwen, Spider | Mistral, Spider (E1) | Qwen, MATH-500 (E2) | Gemma, MATH-500 (E3) | |
|---|---|---|---|---|
| One answer chosen among the 32 discovery answers | ||||
| Best of 32 with the reference answer (upper bound) | 73.4% | 71.9% | 97.7% | 96.1% |
| Best of 32 with the original-database verifier | 59.5% | 51.9% | n/a | n/a |
| Majority vote of 32 | 54.6% | 37.7% | 96.9% | 96.1% |
| Expected correctness of one new sample | ||||
| Uniform plans | 49.1% | 16.9% | 87.1% | 81.2% |
| Plans reweighted on 32 questions by the independent forecast (CROSS) | 51.5% | 17.3% | 87.6% | 81.4% |
In the Qwen Spider column, the two single-sample rows are from Appendix M.2 and the three selection rows come from an analysis that we added after submission, using the saved outputs. In E3, 14% of the evaluation continuations end without the closing line that the parser requires and count as incorrect, which mainly lowers the two single-sample rows.
The two groups of rows answer different questions, and the same discovery pool serves both. The first group returns one answer to a question for which 32 answers have already been sampled. The second group concerns samples that have not been drawn yet: the forecast says in advance how much a change to the plan distribution will change their correctness, which is what a decision to steer generation depends on. The evaluation continuations confirm the forecast: on all four panels the bias interval of CROSS contains zero.
Response to Reviewer HNRo: stability, generality and Gemma (Q2, Q4, W3)
On Q2 (stability), analyses of the saved outputs of the 128-question panel, added after submission, show that allocation gains change little and most selections are shared across the continuations that were drawn and across numbers of continuations per plan. Each of the eight evaluation continuations gives an independent single-continuation forecast (CROSS) for the frozen directions. On the remaining continuations, the allocation that each of these forecasts selects realizes 2.24 to 2.41 pp, against 2.09 to 2.13 pp for the allocation of the same-continuation forecast (SAME). Any two of the nine single-continuation CROSS forecasts, these eight and the original discovery forecast, share 23 to 30 of their 32 selections. Allocations based on forecasts from one, two or four continuations per plan realize 2.33, 2.37 and 2.40 pp, with Spearman correlations of 0.86, 0.91 and 0.93 with the held-out gains. Rebuilding the directions from the other discovery continuation retains 27 of the original 32 CROSS selections and reproduces the forecast advantage: the MSE of SAME is 13.8 pp² against 4.1 pp² for CROSS, a difference of 9.67 [5.70, 15.50] pp².
On the original 32-question Qwen panel, an analysis of the saved outputs, also added after submission, finds the loss difference L of Table 1 positive for all 45 ways of choosing the two discovery continuations among the ten per plan, with a mean of 11.2 [5.6, 18.3] pp².
On Q4 (generality), three new studies repeat the main forecasting experiment on a third model family and a second benchmark, with analysis plans and success criteria fixed before any data were collected and directions frozen before any evaluation continuation was generated. E1 repeats the 128-question Spider panel with Mistral-Small-24B, and E2 and E3 use 128 level-5 MATH-500 problems with Qwen and Gemma. Under the pre-specified scoring rules, all three studies show the same pattern as the paper's Qwen panel: SAME overestimates the gain, every CROSS bias interval contains zero, and CROSS has the lower squared error.
| Panel | Directions | SAME bias, pp | CROSS bias, pp | SAME − CROSS sq. error, pp² |
|---|---|---|---|---|
| Qwen, Spider (paper) | 80 | 2.08 [1.56, 2.72] | 0.30 [−0.07, 0.73] | 12.46 [7.67, 18.92] |
| Mistral, Spider (E1) | 68 | 0.79 [0.24, 1.42] | 0.07 [−0.46, 0.62] | 5.12 [1.31, 8.60] |
| Qwen, MATH-500 (E2) | 90 | 1.79 [1.21, 2.41] | 0.10 [−0.28, 0.48] | 10.55 [6.25, 15.48] |
| Gemma, MATH-500 (E3)† | 87 | 0.94 [0.50, 1.43] | −0.04 [−0.32, 0.23] | 5.52 [1.41, 10.85] |
Directions: questions whose reweighting is not uniform. Brackets give 95% bootstrap intervals from resampling databases on Spider and problems on MATH-500. †On answer format in E3, see the last paragraph.
On W3 (Gemma), the new Gemma panel (E3) has 87 nonuniform directions, and under the pre-specified scoring CROSS lowers the squared error by 5.52 [1.41, 10.85] pp² relative to SAME. The original Gemma panel left little to compare: 55 of its 64 reweighting directions are uniform, with zero predicted and realized gain, which leaves nine for the forecast comparison. An analysis of the saved outputs, added after submission, shows that even with all ten continuations per plan, Gemma's draft correctness varies across plans on only 10 of the 32 questions. In E3 most of the variation in final correctness across plans comes from answer format: Gemma often ends a correct solution without the closing line that the parser requires. When such answers are accepted, in a check made after the data were collected, Gemma is correct on 95.0% of continuations and the reweightings change correctness by less than 0.1 pp on average, so little is left to forecast, and the difference in squared error is −0.22 [−0.77, 0.22] pp². The overstatement from reusing a continuation is still present in that check: in the pre-specified analysis of E3 that reweights the plans by the final answer of one continuation (Appendix N.4 of the revised PDF), the gain read from that same continuation is too high by 0.73 [0.34, 1.20] pp, against −0.05 [−0.19, 0.08] pp when it is read from an independent continuation.
Response to Reviewer HNRo: feedback without reference answers and verifier errors (W1, Q1)
Thank you for the detailed and constructive review. The forecasting procedure needs only binary outcome labels on the sampled continuations, which can come from the model's own consensus or from a verifier. The forecast then targets the gain in whatever outcome those labels measure. We use agreement with the model's consensus to obtain feedback without reference answers, and an imperfect execution verifier to test robustness to labeling errors. In three new studies the analysis plans and success criteria were fixed before any data were collected: E1 repeats the 128-question allocation panel with Mistral-Small-24B, and E2 and E3 apply the same design to 128 level-5 MATH-500 problems with Qwen and Gemma. The Qwen results on Spider below come from analyses that we added after submission, using the saved outputs.
To do without reference answers, we executed every saved query on the original databases and took as each plan's target the plurality execution result among the discovery answers of the other fifteen plans. Directions, forecasts and selections then use only agreement with this target. For the gain in agreement, the independent forecast (CROSS) has a bias of −0.35 [−0.88, 0.11] pp, against 1.65 [1.08, 2.16] pp for the same-continuation forecast (SAME). Reweighting under these labels raises true correctness by 1.14 [0.38, 1.88] pp when it is applied to every question, and by 0.70 [0.07, 1.23] pp when it is applied to the 32 questions that CROSS selects. How much of the agreement gain reaches correctness depends on whether the target is right:
Each change is the average over all questions in the row; a question whose reweighting is uniform counts as zero.
With every question's direction applied, the wrong-consensus questions contribute −0.27 [−0.52, −0.01] pp to panel correctness, which reproduces on real SQL the target-alignment result of Section 5.4.
In the new studies CROSS again shows no detectable bias for the agreement gain, and SAME overestimates it. In E1, the biases are 0.48 [−0.14, 1.15] pp for CROSS and 1.58 [0.97, 2.23] pp for SAME. With the majority answer as the label in E2, they are 0.03 [−0.35, 0.41] and 1.81 [1.22, 2.43] pp. Because the majority is right on 124 of 128 problems, allocation under majority labels raises true correctness by 0.40 [0.16, 0.65] pp, close to the 0.46 pp obtained with reference answers.
Execution match on the original database, without the test suite, is a real verifier that makes errors: on the saved Qwen outputs, 29% to 30% of the queries it accepts fail the suite. For the gain this verifier measures, the bias of CROSS is −0.06 [−0.43, 0.32] pp, against 1.45 [0.85, 2.07] pp for SAME. Allocation with these labels raises test-suite correctness by 1.80 [0.96, 2.67] pp, 77% of the 2.33 pp obtained with test-suite labels, and exceeds random selection by 0.92 [0.40, 1.45] pp. In E1, where this check was pre-specified, 41% of the queries this verifier accepts fail the suite, and the biases of CROSS and SAME are 0.16 [−0.56, 0.85] and 1.34 [0.95, 1.73] pp.
When every label is flipped independently with probability ε, the expected measured gain is scaled by 1 − 2ε, because the direction weights sum to zero. In a simulation on the saved Qwen outputs, the mean bias of CROSS relative to the gain that the noisy verifier itself measures stays between 0.04 and 0.23 pp for ε between 0.05 and 0.3, against 0.27 to 1.64 pp for SAME.