acceptodds
Under review as a conference paper at ICLR 2027

Scaling Changes the Direction of Correctness Churn More Than Its Amount

Abstract

Scaling improves aggregate language model performance, but individual predictions do not improve monotonically. Larger models correct some errors made by smaller models and also turn some previously correct predictions into errors. We characterize these changes using two quantities. Correctness churn, C, measures the fraction of positions whose correctness changes. Directional imbalance, D, measures whether those changes favor the larger model. Across three released model families, their base pretrained checkpoints, three corpora, and six data controlled Pythia transitions from 160M to 12B parameters, 3.1–5.2% of correct next token predictions become incorrect after scaling, with no consistent decrease as model size grows. A similar pattern appears in log likelihood. Positive and negative shifts of 0.289 and 0.275 nats yield a net improvement of only 0.014 nats. PolyPythias provides nine independent runs at each of five model sizes, allowing scaling transitions to be compared with retraining at a fixed size. Reinitialization and data reshuffling reproduce most of the observed churn but produce little directional imbalance. At the 160M→410M transition, correctness churn is 11.3% under scaling and 9.5% under retraining, with a bootstrap 95% confidence interval of [+1.45, +2.13] percentage points for the difference. Directional imbalance is D = +0.273 under scaling and |D| = 0.024 under retraining. The directional gap remains positive at every model size after Holm–Bonferroni correction, while larger models increasingly make the same errors. These directional differences also support selective routing. We derive a total variation bound on the maximum gain obtainable from a candidate routing statistic. Peer model disagreement provides an effective signal for separating rescued and broken predictions. A consensus based cascade that escalates positions with peer disagreement improves over universal large model escalation across all evaluated transitions while using less compute, and approaches the empirical ceiling for the routing rules considered.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language model scaling, correctness churn, training stochasticity, selective routing, model cascades

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gPsAOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes an analysis of the next token prediction ability as models size grows. They show that larger models have a higher probability to generate the correct next token compare to smaller models, but the gain is not uniform across all instances: on a small but significant amount of instances, they have a higher probability than smaller models to generate a wrong next token.

Critical strengths

The paper proposes an interesting angle that goes further the standard story that "larger models are better", and rather investigate where they are better and where they are not.

Up to section 6 I find the paper easy to read.

Results are consistent and convincingly presented.

Critical weaknesses
  • My first remark is on the chosen task. Overall, results are on the "next token prediction task". While an interesting diagnoses, I am unsure the conclusions can be extended to more operational settings.
  • Also, the paper implies that a model predicting a "wrong next token" is "wrong", but without further analysis, we cannot infer the model's overall ability beyond this simple task (churns tokens can correspond to noisy data, position with several valid continuations etc).
  • So I think the overall scope of the paper is much more narrow that what the narrative seems to imply.
  • Section 6 current positioning is unclear. It comes after the main results, with a list of theoretical results, general discussions and idea suggestions with no experiments in the main paper. From what I understand, it seems to converge to Speculative Decoding idea without really naming it. In the current form, it is unclear what this section brings.

Overall, I think this paper main contribution is to count over several datasets the positions where models disagrees on the next token prediction task. Without further analysis and practical implication I am unsure this clears the bar for ICLR.

Questions for authors

See weaknesses.

9 comments
Reviewer_gPsAOpenReview ↗

Very dense and confusing rebuttal

Dear authors, I thank you for your answer. However, I find the current answer extremely long and confusing (9 successive 5000 characters long comments). It features lots of new results, with lots of technical jargon and unclear interpretation.

As it stands, I cannot assess if my concerns are solved. I keep my original scores. Maybe if answering my concerns necessitates that much new textual content, a re submission might more relevant.

Thank you for your response, and we apologize for the length of our previous replies.

We tried to address your concerns in detail, but may have made our response difficult to follow.

To briefly summarize:

  • Beyond next-token prediction: We evaluated seven full-answer benchmarks across 161 comparisons.
  • Correctness: We conducted a blind audit of 2,800 continuations to distinguish token mismatches from actual errors.
  • Main contribution: We showed that retraining explains much of the prediction churn, while scaling mainly changes its direction.
  • Practical relevance: We clarified how our approach differs from speculative decoding and tested routing on 145 configurations with measured costs.

We would appreciate it if you could point out any specific concerns that remain, particularly regarding soundness, novelty, or practical relevance.

We are happy to conduct further experiments or analyses during the discussion period and look forward to your feedback.

Thank you again for your time and consideration.

AuthorsOpenReview ↗

Summary for our previous replies

We also wanted to briefly summarize the key results from our additional experiments.

  • Task-level correctness: Across seven full-answer benchmarks and 161 comparisons, larger models break 1.0–11.4% of task instances, despite overall improvements in 157/161 comparisons. We also audited 2,800 continuations to examine valid alternative tokens.
  • Beyond counting disagreements: Independent pretraining runs show that retraining accounts for 73–85% of token-level churn, while scaling primarily changes its direction. This distinction is the main finding of our analysis.
  • Practical relevance: Unlike speculative decoding, our routing method can retain the smaller model's correct answer. It outperforms confidence-based routing in 52/58 math configurations. In 27/145 configurations, it matches or exceeds the larger model's accuracy with lower GPU cost.

These results and their experimental details are provided in W1–W4.

We hope these additional results help clarify the paper's contributions and address your concerns.

Reviewer_GGnYOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how individual next-token predictions change when language models are scaled. It is decomposed into a rescue rate R and a break date B. The proposed metric is evaluated on several open-weight models and replicated PolyPythias pretraining runs.

Critical strengths
  • The proposed metric is simple and interpretable. It is also proven to be useful through experiments.
  • The replicated PolyPythias comparison between scaling and retraining is sound and interesting.
Critical weaknesses
  • The experiments on open-weight models have limited family/provider diversity. The paper uses Qwen3, Qwen2.5, and Gemma-3 and states that these are "three open families from two providers". If Qwen2.5 and 3 are treated as part of the same broader Qwen series, these experiments are based on only two model families. Then, the claim of "scaling changes the direction of correctness churn" is less strongly supported. Although the paper partially acknowledges this issue in the limitations section, the title and abstract still sound more general than the conducted experiments.
  • Although the replicated PolyPythias experiments are sound, the five model sizes, from 14M to 410M are too small and limited. This issue also affects the above generalization issue (title, abstract, etc.) of the main claim.
Questions for authors
  • Can the position/context analysis be expanded? The Table 5 in the appendix compares 512-token truncation with 1024 tokens. From this table, there are no significant differences with the values of C, B, and D among the four position bins. However, we could expect that D values would decrease for later positions because they can use richer context. It would be better to add discussion about positional effects.
4 comments
AuthorsOpenReview ↗

W1. Model family diversity (1/3)

Thank you for highlighting the value of the replicated experiments and for identifying the need for broader model coverage and better contextual controls. We substantially expanded the evaluation to 20 scaling ladders from nine developers, added independent pretraining replications up to 1B and supporting OLMo-2 variants through 32B, and evaluated seven generated-answer benchmarks. We also ran a fixed-target experiment on 19,072 identical prediction targets over eight context lengths. We respond to each concern below.

We agree that Qwen2.5 and Qwen3 belong to the same broader lineage. We expanded the released-model analysis to 20 scaling ladders from nine developers, rather than describing the original checkpoints as three independent families.

DeveloperEvaluated laddersLaddersSize transitions
AlibabaQwen2.5 base/instruct, Qwen3 base/post-trained419
GoogleGemma-2 and Gemma-3 base/instruct48
MetaLlama-3.x base/instruct, OPT39
AI2OLMo-2 base/instruct24
Hugging FaceSmolLM2 base/instruct24
TIIFalcon3 base/instruct26
EleutherAIPythia17
BigScienceBLOOM14
OpenAIGPT-213
Total2064

Each adjacent transition was evaluated on our corpus, WikiText-103, and C4, giving 64 transitions and 192 transition–corpus comparisons.

MeasurementResult across the expanded evaluation
Break rate, as a share of all evaluated token positions2.6–6.9%, median 3.9%
Conditional regression, among smaller-model correct predictions5.0–13.9%
Breaks along a size ladderNo consistent decrease with increasing size
Directional imbalanceD > 0 in 190/192 comparisons, with both exceptions showing declining accuracy
Matched base/instruct comparisonsInstruct breaks more in 61/66, by 0.85 percentage points on average

Positive DD follows from positive accuracy gain. The separate observation is that breaks remain substantial across the ladders. Pythia breaks change from 3.75% to 3.50% over an approximately 170-fold size range, and OPT from 3.51% to 3.56% over an approximately 100-fold range. Pythia, OPT, BLOOM, and GPT-2 use a fixed training dataset within each ladder.

These comparisons include model and training differences beyond parameter count. We will state that the observed bidirectional flow spans the tested models through 32B. The independent-pretraining decomposition has stricter evidence through 1B, addressed in W2.

AuthorsOpenReview ↗

W2. Independent pretraining at larger scales (2/3)

We extended the 410M PolyPythias replication with independently pretrained CRFM GPT-2 and DataDecide runs. The within-size share is (CSS+CLL)/(2CSL)(C_{SS}+C_{LL})/(2C_{SL}).

Independent-pretraining setSize transition(s)Runs at S / LRetraining share (ours / WikiText-103)
PolyPythias14M→…→410M9 / 973–85% / 76–86%
CRFM GPT-2, OpenWebText124M→355M5 / 575% / 79%
DataDecide, Dolma 1.7300M→1B2 / 374% / 74%
DataDecide, C4300M→1B2 / 373% / 75%
DataDecide, FineWeb-Edu300M→1B3 / 374% / 75%

Across independent-pretraining comparisons, the within-size component is 73–85% of adjacent cross-size churn on our corpus. CRFM GPT-2 and DataDecide independently reproduce the pattern through 1B. DataDecide has only two or three runs per size.

Larger-scale tests use different controls.

Supporting comparisonWithin-size component (ours / WikiText-103)
OLMo-2 stage-2 variants, 1B→7B29% / 28%
OLMo-2 stage-2 variants, 7B→13B43% / 42%
OLMo-2 stage-2 variants, 13B→32B49% / 50%
Pythia / deduplicated-Pile variants, 410M→…→12B85–98% / 84–98%

OLMo-2 runs share earlier training and vary the order of the final 50B–100B tokens. Pythia and deduplicated-Pile variants change the dataset. Neither is equivalent to fully independent pretraining.

Task-defined correctness gives similar evidence on eight benchmarks.

Independent-pretraining stepWithin-size share of task-level churn
PolyPythias 70M→160M70–90%
PolyPythias 160M→410M77–94%
DataDecide 300M→1B, Dolma 1.743–87%

For complete generated answers, OLMo-2 stage-2 runs account for 42–76% of task-level churn at 7B→13B and 45–87% at 13B→32B. These figures measure late-stage variability.

AuthorsOpenReview ↗

Q1. Position and context effects (3/3)

Thank you for this suggestion. The original Table 5 (revised Table 6) compared position bins, but those bins contain different target tokens and later bins select longer documents. We therefore added a same-target context intervention (new App. A.12) that varies the available context while holding targets fixed.

From 298 long LoCoMo, HotpotQA and LongMemEval documents, we fixed 19,072 target tokens in 210 source clusters. Every model predicts the same targets with 16, 32, 64, 128, 256, 512, 1,024, or 1,536 preceding tokens. Accuracy increases by approximately 10 points over this range, verifying that the context manipulation matters.

We tested six Pythia transitions.

Accuracy and correctness flow across context lengths

Pythia stepAcc. 16/1536D rangeB (%)
160M→410M43.8%, 54.0%+0.38 to +0.413.45–3.68
410M→1B45.3%, 56.0%+0.17 to +0.223.56–3.89
1B→1.4B46.7%, 57.1%+0.10 to +0.183.52–3.88
1.4B→2.8B47.8%, 58.5%+0.12 to +0.173.55–3.68
2.8B→6.9B49.0%, 59.5%+0.09 to +0.163.38–3.60
6.9B→12B49.6%, 60.2%+0.08 to +0.143.30–3.61

Directional-imbalance slopes

Pythia stepD slope [95% CI]
160M→410M+0.000 [−0.007, +0.007]
410M→1B+0.005 [−0.004, +0.014]
1B→1.4B−0.003 [−0.012, +0.006]
1.4B→2.8B+0.004 [−0.004, +0.012]
2.8B→6.9B−0.003 [−0.011, +0.006]
6.9B→12B+0.000 [−0.010, +0.010]

The confidence intervals for the change of DD per context doubling include zero in all six Pythia steps. Break rates also remain stable. We replicated the intervention on Qwen2.5 and Llama-3.x.

StepD rangeD slope [95% CI]
Qwen2.5 0.5B→1.5B+0.28 to +0.31+0.001 [−0.005, +0.007]
Qwen2.5 1.5B→3B+0.14 to +0.18+0.005 [−0.002, +0.013]
Qwen2.5 3B→7B+0.09 to +0.15+0.009 [+0.001, +0.017]
Llama-3.2 1B→3B+0.21 to +0.29+0.009 [+0.001, +0.017]
Llama-3 3B→8B+0.23 to +0.27+0.000 [−0.008, +0.009]

No transition exhibits a statistically supported decrease of DD with context. Two show small positive slopes (+0.009 per doubling). Most importantly, in replicated PolyPythias 160M→410M, the systematic share of correctness churn stays at 14–15% at all eight context lengths, despite ~10-point accuracy gains. Richer context improves both model sizes without materially changing the scaling-versus-retraining decomposition in this controlled test. We added these results and discussion to App. A.12 and Sec. 4.

Dear Reviewer,

We have worked carefully to address all your concerns and added substantial new experiments and results.

We would really appreciate it if you could review our responses and let us know if they address your concerns. If anything is still unclear or if you have any further questions, we would be happy to provide more details.

Thank you very much for your time and consideration.

Reviewer_6WfjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Scaling improves language model accuracy, but larger models do not always make better predictions. They can fix some mistakes while introducing new ones. This paper investigates how these prediction changes relate to model scaling and training randomness.

The authors introduce two metrics: correctness churn (C), measuring how many predictions change correctness, and directional imbalance (D), measuring whether these changes are mostly beneficial. They also propose a peer-disagreement-based routing strategy to decide when to use a larger model.

Experiments on Qwen, Gemma, Pythia, and PolyPythias show that scaling and retraining produce similar amounts of churn, but scaling leads to more beneficial changes. The proposed routing strategy also achieves modest accuracy improvements over always using the larger model in next-token prediction.

Critical strengths

S1. Interesting and relevant problem.

The paper looks beyond average accuracy and studies how individual predictions change with scaling. This provides a useful perspective on model reliability that standard scaling curves often miss.

S2. Interesting empirical finding.

The paper provides several interesting empirical findings:

  • Scaling vs. retraining: Training randomness produces almost as much correctness churn as scaling, despite much smaller directional improvements.

  • Persistent regressions: Break rates remain substantial as models grow, even as rescue rates decrease.

  • Shared errors: Larger models increasingly make similar errors, suggesting reduced error diversity.

  • Effective routing signals: Peer disagreement is more effective than model confidence for identifying predictions that benefit from escalation.

S3. Solid experimental coverage.

The authors evaluate multiple model families, datasets, and model sizes. The controlled Pythia experiments and repeated PolyPythias runs make the findings more convincing.

S4. Useful for model routing.

The paper connects its empirical observations to selective inference. The finding that peer disagreement can be more useful than model confidence for routing is interesting and potentially practical.

Critical weaknesses

W1. Limited conceptual novelty.

The observation that larger models can introduce new errors is already well established. Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Level (ACL 2021) studies instance-level regressions under scaling, while Launch and Iterate: Reducing Prediction Churn (NeurIPS 2016) investigates prediction changes across model updates. Although the controlled scaling-versus-retraining comparison is valuable, the proposed rescue/break decomposition is straightforward, and the additional insight provided by the new metrics remains somewhat limited.

W2. The directional difference is partly expected.

Since D = ΔAccuracy/C, scaling naturally produces positive D when accuracy improves, whereas same-size retraining tends to yield similar accuracies. Although the authors analyze some mathematical effects on D in Appendix B.3, it remains unclear how much additional insight the scaling-versus-retraining directional gap provides beyond the expected accuracy difference. The similarity in churn magnitude seems more surprising and deserves greater emphasis.

W3. The underlying cause of similar churn remains unclear.

The authors show that scaling and retraining produce similar amounts of correctness churn. However, this does not necessarily mean they affect the same predictions or share similar underlying mechanisms. A direct comparison of the token-level overlap between scaling-induced and retraining-induced changes would help explain this similarity and strengthen the interpretation of the results.

W4. Limited generalizability and practical validation.

The strongest replicated scaling-versus-retraining experiments only cover models up to 410M parameters, leaving their generalizability to larger LLMs uncertain. Moreover, the routing improvements are mainly demonstrated through teacher-forced next-token prediction, without end-to-end generation experiments or measured inference latency. It remains unclear whether these improvements translate into practical benefits.

Questions for authors

Q1. Appendix B.3 discusses the effect of changing accuracy pools on D. However, since D = ΔAccuracy/C, isn't the larger directional imbalance under scaling largely expected, given that scaling improves accuracy while same-size retraining typically does not? What additional insight does this directional gap provide beyond the difference in accuracy gains?

Q2. The paper shows that scaling and retraining produce similar amounts of correctness churn, but does not directly establish whether they affect the same predictions. Could the authors compare the token-level overlap of correctness changes under scaling and retraining? This would help clarify whether training stochasticity explains scaling-induced churn or merely produces a similar amount of prediction instability.

4 comments
AuthorsOpenReview ↗

W1, W2, Q1. Novelty, directionality, and churn (1/4)

Thank you for your careful and constructive review. Thanks to your comments, we have strengthened both the explanation of correctness churn and its empirical validation. We added an exact stochastic decomposition, direct scaling-versus-retraining flip-overlap experiments, independent pretraining replications, evaluations of 20 scaling ladders from nine developers, and end-to-end routing experiments covering 145 configurations. We address the individual concerns below, with the additional results incorporated into the revised manuscript.

We agree that instance-level regressions were reported by Zhong et al. (2021) and Milani Fard et al. (2016). We will clarify that Zhong et al. used multiple training seeds. We also agree that positive D=ΔAcc/CD=\Delta\mathrm{Acc}/C follows when accuracy improves. Our main result concerns the amount of cross-size churn attributable to within-size training variation.

For correctness probabilities qS(x)q_S(x) and qL(x)q_L(x) over independent runs, expected churn satisfies

CSL=12(CSS+CLL)+Ex[(qL(x)−qS(x))2].C_{SL}=\tfrac12(C_{SS}+C_{LL})+\mathbb E_x[(q_L(x)-q_S(x))^2].

The first term is within-size churn. The second is the size-dependent component. The mean accuracy gain Ex[qL−qS]\mathbb E_x[q_L-q_S] does not determine either term. We estimated both terms using nine independent PolyPythias runs per size.

TransitionCross-size CC (%)Within-size term (%)Size-dependent term (%)Within-size share (ours / WT103)
14M→31M11.299.112.1881% / 82%
31M→70M11.449.581.8784% / 86%
70M→160M13.069.483.5873% / 76%
160M→410M11.329.681.6585% / 86%

Values are percentages of evaluated positions, except the within-size shares. The within-size term accounts for 73–85% of adjacent cross-size churn on our corpus and 76–86% on WikiText-103. Scaling breaks 3.46–4.10% of positions, compared with 4.39–4.86% under smaller-model retraining. We do not claim that the break-rate difference is statistically significant.

The within-size share decreases as the size gap widens.

Parameter-size ratioWithin-size share (ours)WikiText-103
2.2–2.6×, adjacent steps73–85%76–86%
Approximately 5×56–61%59–64%
11–13×44–50%45–50%
29×, 14M→410M40%40%

The majority-within-size finding holds for the tested adjacent transitions, not every size gap. We will emphasize the similarity in churn magnitude rather than the expected positive sign of DD. The decomposition measures expected run-to-run variability, not the cause of any individual flip.

AuthorsOpenReview ↗

W3, Q2. Scaling and retraining flip overlap (2/4)

We compared scaling flips from Si→LjS_i\to L_j with retraining flips from Sk↔SlS_k\leftrightarrow S_l. All four runs are distinct. No model is shared between the two comparisons. The table covers four adjacent PolyPythias transitions.

Overlap measureOur corpusWikiText-103
P(retraining flip∣scaling flip)P(\text{retraining flip}\mid\text{scaling flip})0.29–0.310.29–0.32
P(retraining flip)P(\text{retraining flip}), random-position baseline0.09–0.100.07–0.10
Conditional overlap / baseline3.0–3.4×3.1–4.0×
Jaccard overlap of the two flip sets0.20–0.210.19–0.21
Jaccard overlap under an independent-position null0.05–0.060.04–0.06

Scaling and retraining flips overlap 3.0–4.0× more than expected from their marginal rates. The absolute conditional overlap is 29–32%. We report both values to avoid implying that the flip sets are identical.

We separately identified unstable positions using held-out same-size runs.

Held-out resultRange across transitions
Positions identified as unstable under same-size retraining16–18% of all positions
Scaling breaks falling within this unstable subset59–61% of all scaling breaks
Scaling rescues falling within this unstable subset51–55% of all scaling rescues
Positions stable within both sizes but different across sizes0.5–1.4% of all positions

Only 16–18% of positions are classified as unstable, yet they contain 59–61% of scaling breaks and 51–55% of rescues. At PolyPythias 160M, data-order variation produces 8.9–9.2% churn, initialization variation 8.7–9.0%, and varying both 9.2–9.3%. These controls do not isolate additive causal effects.

We also tested a retrained same-size model as a routing peer.

Routing signalPositions escalatedFraction of always-large accuracy gain recovered
Smaller peer35–45%58–67%
Retrained same-size peer25–39%65–80%
Entropy, matched to retrained-peer escalation25–39%37–42%

The retrained peer recovers 65–80% of the always-large accuracy gain, compared with 37–42% for matched-rate entropy. The smaller-peer row uses a different escalation rate. These experiments support shared positional susceptibility and a useful routing signal. They do not establish identical mechanisms for scaling and retraining.

AuthorsOpenReview ↗

W4. Replication and model coverage (3/4)

The original independent PolyPythias experiments ended at 410M. We added fully independent CRFM GPT-2 and DataDecide runs through 1B, and separately evaluated OLMo-2 stage-2 variants through 32B.

Training setTransitionRuns per size, S/LWithin-size share (ours / WT103)
CRFM GPT-2, OpenWebText124M→355M5/575% / 79%
DataDecide, Dolma 1.7300M→1B2/374% / 74%
DataDecide, C4300M→1B2/373% / 75%
DataDecide, FineWeb-Edu300M→1B3/374% / 75%
OLMo-2, stage-2 variants1B→7B3/329% / 28%
OLMo-2, stage-2 variants7B→13B3/343% / 42%
OLMo-2, stage-2 variants13B→32B3/349% / 50%

The first four rows use fully independent pretraining. Their within-size shares are 73–79% across our corpus and WikiText-103. The OLMo-2 variants share all but the last 1–2% of training. They measure late-stage variability, not independent pretraining variability. The 13B→32B result uses bfloat16. Other rows use float32.

We also expanded the released-model evaluation to 20 ladders from nine developers, with base and instruction-tuned ladders counted separately.

DeveloperIncluded seriesLaddersAdjacent transitions
AlibabaQwen2.5, Qwen3419
GoogleGemma-2, Gemma-348
MetaLlama-3.x, OPT39
AI2OLMo-224
Hugging FaceSmolLM224
TIIFalcon326
EleutherAIPythia17
BigScienceBLOOM14
OpenAIGPT-213
Total2064

The 64 transitions × three corpora give 192 comparisons over models from 70M to 32B. Break rates are 2.6–6.9% (median 3.9%) of token positions, without a consistent decline along a ladder. Pythia, OPT, BLOOM, and GPT-2 use the same dataset across sizes. Complete-answer tests add 23 transitions × seven tasks = 161 comparisons. Independent task-level tests give within-size shares of 70–94% for PolyPythias and 43–87% for DataDecide 300M→1B.

Fully independent replication remains limited to 1B. We will separate that evidence from the shared-training 32B results in the abstract and limitations.

AuthorsOpenReview ↗

W4. Generated-answer routing and GPU cost (4/4)

We evaluated 145 request-level configurations on GSM8K, MATH-500, TriviaQA, NQ-Open, and MMLU. The smaller model SS and peer PP generate complete answers. Disagreement triggers escalation to LL. Random and confidence baselines use matched escalation rates, with cross-fitted confidence thresholds. GPU time includes every SS, PP, and escalated LL call, including two-GPU execution where applicable.

Evaluation outcomeConfigurations
Peer routing exceeds matched-rate random accuracy141/145
Peer routing exceeds confidence accuracy on GSM8K/MATH-50052/58
Cascade reaches or exceeds observed always-LL accuracy59/145
Cascade significantly exceeds always-LL accuracy13/145
At least observed always-LL accuracy with lower GPU time27/145

The 59/145 and 27/145 counts use observed accuracy, not statistical noninferiority. Six of the 27 lower-cost configurations significantly improve accuracy.

Representative accuracy and GPU-cost results follow. Model sizes are in billions and GPU/LL is measured relative to always using LL.

TaskModel configurationSSLLCascadeGPU / LL
TriviaQAQwen2.5 14/7→3264.967.868.40.70×
GSM8KQwen2.5 14/7→3295.195.795.80.32×
GSM8KGemma-3 12/1→2793.695.895.90.68×
GSM8KQwen3 14/4→3295.194.895.80.29×
MATH-500Qwen3 8/4→1480.884.886.61.04×
MATH-500Llama 3 / Qwen2.5 1.5 → Llama 844.447.853.61.36×

The TriviaQA gain is +0.6 points [95% CI +0.3, +0.9]. The two MATH-500 configurations gain accuracy at higher GPU cost. The second and third GSM8K rows do not show individually significant accuracy gains.

We also tested free-running token-level routing on generated reasoning traces with matched escalation rates.

S/P→LS/P\to LTaskPeer accuracyConfidenceRandom
Llama 3/1→8GSM8K81.883.480.4
Llama 3/1→8MATH-50046.250.243.8
Qwen2.5 1.5/0.5→3GSM8K80.082.575.4
Qwen2.5 1.5/0.5→3MATH-50059.062.654.6
Qwen2.5 3/0.5→7GSM8K89.290.287.6
Qwen2.5 3/0.5→7MATH-50070.069.667.2
Qwen3 4/0.6→8GSM8K93.792.692.3
Qwen3 4/0.6→8MATH-50071.271.070.6

Peer routing beats random in all eight settings, but confidence is better in five. Peer disagreement is not uniformly superior. The request-level figures are measured GPU costs, while our separate token-cascade latency figures are modeled from component timings. We will label them accordingly.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Additional Experiments and Revisions

Dear Area Chair and Reviewers,

We sincerely thank you for your thoughtful reviews and constructive feedback. We also appreciate the Area Chair's guidance regarding the length of our earlier responses.

We have substantially revised the manuscript and conducted additional experiments to address the concerns regarding novelty, generalizability, correctness definitions, and practical relevance. Below, we summarize the most important additions and findings.

1. Generalizability and evaluation beyond next-token prediction

  • Broader model coverage: We expanded our evaluation to 20 scaling ladders from nine developers, covering 64 size transitions across three corpora (192 comparisons, up to 32B parameters). Break rates remain substantial (2.6–6.9%, median 3.9%) without consistently decreasing as models grow.
  • Complete-answer evaluation: Across seven benchmarks spanning reasoning, knowledge, coding, and multiple-choice tasks, we evaluated 23 transitions and 161 task-level comparisons. Larger models break 1.0–11.4% of previously correct answers (median 4.3%), despite overall accuracy improvements in 157/161 comparisons.

2. Core contribution: distinguishing scaling from training stochasticity

  • We acknowledge that positive directional imbalance follows naturally from positive accuracy gains. Our key contribution is instead an exact decomposition of cross-size correctness churn into within-size training variability and a size-dependent component.
  • Across independently pretrained PolyPythias, CRFM GPT-2, and DataDecide runs, extending to 1B parameters, within-size variability accounts for 73–85% of expected churn across the tested adjacent scaling steps.
  • Direct flip-overlap experiments show that scaling and retraining affect the same positions 3.0–4.0× more frequently than expected by chance. This supports shared positional instability, without assuming identical causal mechanisms.

3. Correctness validity and contextual confounds

  • We clarified that token-level correctness means reference-token exact match, not semantic correctness. A blinded audit of 2,800 continuations further examines this distinction, while the complete-answer benchmarks provide task-defined correctness evaluations.
  • A controlled experiment holding 19,072 prediction targets fixed across eight context lengths (16–1,536 tokens) shows approximately 10-point accuracy improvements with longer contexts, but no systematic decrease in break rates or directional imbalance.

4. Practical validation of selective routing

  • We added 145 end-to-end request-level routing configurations across five benchmarks, including measured GPU computation costs.
  • Peer-disagreement routing outperforms matched-rate random routing in 141/145 configurations and confidence-based routing in 52/58 mathematical reasoning configurations.
  • It significantly outperforms always using the larger model in 13/145 configurations. In 27/145, it matches or exceeds the larger model's observed accuracy with lower GPU time; six of these also show statistically significant accuracy gains.
  • For example, on GSM8K, a Qwen3 cascade improves accuracy from 94.8% to 95.8% using 0.29× the GPU time of always using the 32B model. Additional free-running generation experiments support the routing application, although peer disagreement is not uniformly better than confidence.

In summary, our revised paper goes beyond documenting prediction regressions under scaling. It shows that much of the churn between adjacent model sizes is already present as within-size training variability, while scaling systematically shifts the balance toward beneficial changes. The additional experiments establish the positional overlap underlying this observation, broaden its empirical scope, and demonstrate its practical relevance through end-to-end routing.

We hope this concise summary makes the strengthened contributions and remaining limitations easier to assess. We sincerely appreciate your time, feedback, and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.