acceptodds
Under review as a conference paper at ICLR 2027

PriorBPB: Reducing the Target Length Confound of Bits-per-Byte While Improving Accuracy Prediction

18 views

Abstract

Bits-per-byte (BPB) normalises log-likelihood (in bits) by bytes rather than tokens to stay tokeniser-independent, and is widely used to compare language models across heterogeneous tasks. It also depends on how long the answer strings are. The bits a model spends on a gold string can be written as , with a fixed overhead, a per-byte rate, the least-squares fit in length, and the rest. Since the residuals sum to zero, total bits over total bytes is exactly , with the task's mean gold length. Because costs that do not scale with length are spread over the answer, tasks with long answers score better, as do long items within a task. Averaging per-item BPB has the same dependence. Across a 19-task suite on Qwen3 models, BPB correlates with mean gold length at Spearman to . OLMo-3-7B behaves the same way (Spearman ). Among the Qwen3 models we tested, the smallest model appears to score best (an apparent inverse scaling behaviour) due primarily to differences in the overhead. We propose a less length-biased metric, PriorBPB, which estimates a per-(model,task) marginal per-byte cost curve from per-token surprisal and averages it under one shared length prior instead of under each task's own length distribution. This cuts the mean absolute Spearman correlation with answer length from to across 60 base checkpoints from five model families, while correlating with accuracy as well or better on base and instruct checkpoints (mean Spearman ).

Primary area:
foundation or frontier models, including LLMs
Keywords:
bits-per-byte, language model evaluation, length bias

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_jYBtOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper investigates answer-length dependence in bits-per-byte (BPB) evaluation. Using an affine decomposition of total answer surprisal, it shows that corpus BPB contains a fixed-cost term divided by mean answer length, while averaging per-item BPB introduces additional length-dependent terms. It indicates how these effects can produce misleading comparisons, including apparent inverse scaling among Qwen3 models. The proposed metric, PriorBPB, estimates a marginal cost curve over answer-byte positions and aggregates it using a shared length prior instead of each task’s empirical length distribution. Validation includes multiple model families, dense Pythia training checkpoints, prior-sensitivity analyses, held-out tasks, and comparisons with downstream accuracy. The reported results show substantially reduced cross-task length correlation and modestly improved accuracy tracking, with larger gains for instruction-tuned models.

Critical strengths

The strongest aspect of this paper, in my view, is its diagnosis of length bias in a metric that the community uses widely and largely without question. The core identity is algebraically simple, and the authors acknowledge this, so I do not consider the identity itself the contribution. What I found more useful is the empirical evidence around it. Simply rephrasing the same answer changes BPB by a factor of four to five. Per-item BPB on SQuAD2 turns out to mostly reflect answer length, and the matched-position analysis shows that the per-byte cost at a given position does not depend on how long the answer eventually becomes. Together, these make a persuasive case that a fixed first-token cost, rather than genuine difficulty, drives much of the reported score. I also found the observation about mean-of-ratios BPB particularly useful, since this is the form reported by lm-eval-harness, and the authors show it can inflate scores by up to 145%.

I also appreciated the inverse-scaling case study as a cautionary example. The results suggest that the smallest Qwen3 model performs best, but the decomposition shows that this trend is primarily due to the fixed cost growing with model size, while the per-byte rate shows no such pattern. The base-checkpoint control makes the authors’ explanation credible to me: once the chat template is removed, the fixed-cost growth largely disappears. For example, on GSM8K it stays at around 12/6/8 bits for the base models compared with 23/94/154 bits for the instruct models. I think this is a valuable demonstration of how a metric artifact can look like a real capability trend, and it supports the paper’s broader message that BPB comparisons should not be taken at face value without examining what drives them.

Critical weaknesses

My main concern is that PriorBPB may not fully remove the length bias. Under an affine cost, Prop. 9 shows the fixed cost is still divided by a term that depends on each task’s longest answer, so part of the bias remains. The late positions of R(k) are also estimated from a few long answers, but they still receive substantial weight. App. H shows that the remaining correlation comes from exactly these positions. Within tasks, the gain over corpus BPB is small (0.944 to 0.888, App. L), so calling PriorBPB "length-invariant" in Fig. 3 seems too strong.

I am also not fully convinced by the evaluation. Low correlation with length is not enough on its own, since some tasks may genuinely be harder per byte. On held-out tasks, the correlation even flips sign (−0.73 to +0.37). For accuracy tracking, PriorBPB is compared only to mean-of-ratios BPB, not directly to corpus BPB, and the gains are small for base models (+0.021) with an expansive interval for instruct models.

Questions for authors
  • In Eq. 3, is the position budget each task’s longest gold (n_max), or a common budget across tasks? If it is per-task, how sensitive is PriorBPB to the single longest item? A quick check would help: recompute the cross-task correlation after removing the longest gold in each task or with one common budget for all tasks.

  • Table 1 compares PriorBPB only against mean-of-ratios BPB. Since the paper recommends corpus BPB, could you report the same paired fidelity delta against corpus BPB on the same base and instruct models? This should be possible with the scores you already have.

  • Since the length mix is shared by all models within a task, where does the within-ladder accuracy gain come from? Is it concentrated in tasks where alpha changes a lot along the ladder? Splitting delta by tasks with large versus small alpha change would answer this.

  • On the held-out suite, the correlation flips from −0.73 to +0.37. Could you report the per-task values for the four held-out tasks so we can see whether this is driven by a single task or reflects over-correction?

  • Which estimator does the abstract’s −0.82 to −0.86 range refer to? Also, how does the claim that cross-lingual neutrality improves fit with Fig. 10, which shows PriorBPB with the largest DE/EN deviation (0.18 vs. 0.13 for corpus BPB)?

Happy to discuss these questions during the rebuttal period, and I am open to reconsidering my scores if my concerns get addressed.

3 comments
AuthorsOpenReview ↗

Response to Reviewer jYBt (1/2)

We thank Reviewer jYBt for the detailed and generous review, and especially for the five questions, each of which we answer below. We are glad they find the paper's strongest aspect to be “its diagnosis of length bias in a metric that the community uses widely and largely without question”, and that they find the mean-of-ratios BPB observation and the inverse-scaling case study persuasive. The central concerns, that PriorBPB may not fully remove the length bias and that the evaluation was not yet be convincingly meaningful in practice, have shaped our revision.

In response to the critical weaknesses listed, we have revised the paper and provide the following response (corresponding to the two paragraphs):

  1. Length dependency claims. We agree on each point, and the revision no longer overstates the within-task result. §5 now states that the sums in the PriorBPB definition run only to a task's longest answer and that the prior is renormalised on that support, so by Prop. 9 a short-answer task still divides its fixed cost by about its longest answer rather than by μ\mu. We have also replaced phrases like “length-invariant” and “substantially length-neutral” throughout with weaker but more precise claims like “less length-dependent”. The caption of Fig. 3 and Appendix L include the within-task results: over the 168 (model, task) conditions, mean ∣ρ∣|\rho| falls only from 0.939 (corpus BPB) and 0.936 (mean-of-ratios BPB) to 0.879 (PriorBPB), a paired reduction of +0.060 [0.015, 0.133]. As you note, the residual sits in the sparsely-supported late positions of R(k)R(k) (also see Appendix H). We instead claim PriorBPB offers a reduction of a known confound rather than being entirely length-neutral, and in Appendix M.1 we provide some alternative forms (minimum support, a terminal-rate tail, and a pooled tail) that trade this residual off against other evaluation goals.

  2. Correlation with accuracy. We agree that a low length correlation cannot by itself show that a score is better, since some tasks may genuinely be harder per byte, and the revision now presents the length result as the removal of a known artefact and reports improved accuracy prediction against both mean-of-ratios and corpus BPB in what we think are practically-meaningful settings (also see Q2 below). Measured on the smallest models used in a pre-training data mixing experiment ladder, PriorBPB predicts accuracy at 1B parameters better than mean-of-ratios or corpus BPB (see Figure 4 and Table 2 in §6). On the held-out sign flip: you were right to be suspicious, as this was an over-correction on a single task (DROP); we no longer draw any conclusion from that three-task suite (Q4). We have removed the instruct checkpoints accuracy claim given the weak effect size.

AuthorsOpenReview ↗

Response to Reviewer jYBt (2/2)

We also address the five questions raised (numbered by the order of dot points):

  1. Position budget and sensitivity to the longest item. The budget is per task, the task's longest gold nmaxn_{max} (Appendix M). Removing the single longest gold answer can change a task's PriorBPB by 0.4% at the median, by more than 5% in 11.6% of the 198 (model, task) cells, and by 14% to 39% on NaturalQsOpen for 10 of the 14 models (there the longest gold is 78 bytes and the next 56). The cross-task length correlation barely changes, however, with mean ∣ρ∣|\rho| going from 0.163 to 0.141. Replacing the per-task budget with one common budget of 8, 16, 32, or 64 bytes gives mean ∣ρ∣|\rho| between 0.142 and 0.163, compared to 0.496 for corpus BPB. So the cross-task result does not rely only on the longest benchmark items, even though it can change individual task's value.

  2. Paired fidelity against corpus BPB. Table 2 now reports this for the DataDecide suite and the base ladders (the instruct ladders are in Appendix K). Using 4M and 20M parameter models, PriorBPB predicts accuracy at 1B better than corpus and mean-of-ratios BPB (see Figure 4 and Table 2 in §6). Along the 52 dense base ladders, the three BPB variants scores are similar at a mean fidelity of 0.50. However, again for the smallest model size, Pythia-160M, PriorBPB appears to demonstrate an advantage over both baselines, e.g., +0.098 [+0.035, +0.166] over corpus BPB.

  3. Where the within-ladder gain comes from. By more systematically answering this question we have identified something we are quite excited about, which is that the largest gains are on the smallest models (Q2), and it is concentrated where the fixed cost is a large share of the score, e.g., CommonsenseQA (+0.10 over corpus BPB; 10-byte answers) and SocialIQA (+0.07; 20-byte answers), where the first answer token carries 0.73 and 0.41 of an answer's bits (Appendix N.4).

  4. Held-out per-task values. These are now in a table (Appendix J, Table 4). The sign flip is mostly DROP. Its PriorBPB is 0.23 to 0.66 compared to a corpus BPB of 1.14 to 3.85, and it falls to 0.19 of its corpus BPB because only 43 of its 250 golds exceed 10 bytes while byte positions from 6 onward contribute 92% of the prior‘s weight, so a few cheap copied spans influence a large fraction of the score. This indicates over-correction on one task, as you suspected. The held-out suite has only three tasks, so each model's correlation can take just four values (it is −0.50 under corpus BPB and +0.50 under PriorBPB for all six models), and we no longer draw any conclusion from it. For a systematic test we instead enumerate held-out splits within the suites (Appendix J.1): over all 258,077 splits of the 18 tasks into a μ\mu-fitting set and a held-out set, the held-out length correlation falls from 0.67 to 0.31 on average, and PriorBPB has the lowest length correlation in >92% of the splits compared to corpus and mean-of-ratios BPB.

  5. The abstract's range and cross-lingual neutrality. The submission's −0.82 to −0.86 was mean-of-ratios BPB on the three Qwen3 models; re-computed on the 18 length-varying tasks these are −0.79 to −0.84, with corpus BPB at −0.67 to −0.74 (OLMo-3-7B: −0.77 and −0.62). The abstract now reports the mean-of-ratios values and states that they are the average of per-item BPB, and §3 gives the corpus values. Regarding the cross-lingual neutrality, you are right and we have removed the neutrality claim.

We are very grateful for these questions, which materially improved the paper, in particular by making us report against both mean-of-ratios and corpus BPB throughout, withdraw the weak or incorrect instruct checkpoint and cross-lingual claims, and more clearly present the within-task results. We hope the revision now shows clearly both what PriorBPB removes and what it does not, and that this makes the remaining practical case (a length-robust early signal on small models, e.g., for data-mixture ablations) easier to weigh. We wish to again thank Reviewer jYBt for their careful and constructive assessment, and will be happy to make additional changes as requested and engage in discussion.

Thank you for the detailed response and the additional analyses. You have addressed my main concerns. While the accuracy-prediction gains appear concentrated in smaller-model settings, I find the revised contribution useful and the paper’s claims better calibrated. I am therefore updating my rating to "Weak Accept", and I ask that the final version fully reflect these revisions.

Reviewer_VaqrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper shows algebraically and empirically that bits-per-byte (BPB) is heavily confounded by answer length: under a linear fit of item surprisal to byte length (Li=α+βni+εiL_i = \alpha + \beta n_i + \varepsilon_i), corpus BPB equals β+α/nˉ\beta + \alpha/\bar{n}, while the standard lm-eval-harness mean-of-ratios BPB adds a strictly positive Jensen penalty that inflates scores by up to 99%–145%.

The authors demonstrate that this fixed-cost overhead α\alpha (dominated by the first answer token and chat-template mismatch) produces an apparent inverse-scaling artefact on Qwen3 (0.6B beating 1.7B and 4B on 15–18 of 19 tasks). To debias cross-task comparisons while still using every answer byte, they propose PriorBPB, which estimates a per-(model, task) marginal byte-position cost curve R(k)R(k) and averages it under a frozen geometric length-survival prior (μ=111\mu = 111 bytes), recovering corpus BPB when the prior equals the task's empirical length distribution.

Across 60 base checkpoints from five model families and 22 tasks (plus 4 held-out tasks), PriorBPB reduces mean cross-task length rank correlation ∣ρ∣|\rho| from 0.68–0.76 to 0.18 and improves downstream accuracy tracking over mean-of-ratios BPB (Δ=+0.021\Delta = +0.021 on base, +0.122+0.122 on instruct).

Critical strengths
  1. The exact β+α/nˉ\beta + \alpha/\bar{n} corpus identity, the Jensen decomposition of lm-eval-harness's mean-of-ratios "BitsPerByte", and the Qwen3 false inverse-scaling case study (verified via base-vs-instruct chat-template controls) cleanly expose how first-token overhead and answer-length distributions distort cross-task leaderboards and scaling claims.

  2. Recasting corpus BPB via the layer-cake identity as survival-weighted marginal byte cost As⋆[R]\mathcal{A}_{s^\star}[R] and swapping s⋆s^\star for a shared prior s(k)s(k) requires no extra forward passes, preserves monotonicity, and substantially cuts cross-task length correlation (0.68 -> 0.18) across 60+ checkpoints, 5+ model families, and held-out datasets with thorough task-clustered statistical checks.

Critical weaknesses
  1. Because R(k)R(k) is only defined up to a task's longest gold answer nmax⁡n_{\max}, Prop. 9 shows PriorBPB's fixed-cost denominator is μ(1−(1−1/μ)nmax⁡)≈min⁡(μ,nmax⁡)\mu(1 - (1 - 1/\mu)^{n_{\max}}) \approx \min(\mu, n_{\max}). Tasks or strata with nmax⁡≪111n_{\max} \ll 111 therefore still divide α\alpha by ≈nmax⁡\approx n_{\max} rather than μ\mu—explaining why actual within-task stratum length correlation barely drops (0.944 -> 0.888 in Appendix L, unlike Section 6's 0.98 ratio which restricts quintiles to shared byte support) and why sparse single-item tails leave residual bias (-0.34, which disappears in Appendix H only when capping k≤64k \le 64).

  2. Despite featuring accuracy prediction in the title, the gain on base models (the main target of BPB) is very small (+0.021), while the larger instruct gain (+0.122, wide 95% CrI [+0.005, +0.571]) rests on only n=19n=19 conditions (including 3-rung ladders prone to Spearman ±1\pm 1 saturation), compares against flawed mean-of-ratios BPB rather than corpus BPB (which already recovers +0.031), and omits generative math/code/QA tasks (GSM8K, MATH, HumanEval, MBPP, SQuAD2). Similarly, Section 6 claims cross-lingual neutrality for PriorBPB, whereas Fig. 10's caption notes PriorBPB actually has slightly larger DE/EN deviation (0.18) than corpus BPB (0.13).

Questions for authors
  1. By Prop. 9, whenever a task's maximum answer length satisfies nmax⁡≪μ=111n_{\max} \ll \mu = 111, the denominator μ(1−(1−1/μ)nmax⁡)≈nmax⁡\mu(1 - (1 - 1/\mu)^{n_{\max}}) \approx n_{\max}, so short-answer tasks still divide the fixed overhead α\alpha by ≈nmax⁡\approx n_{\max} rather than μ\mu, while longer tasks weight noisy single-survivor tail positions near nmax⁡n_{\max} (which Appendix H shows causes the -0.34 residual that vanishes when capping k≤64k \le 64). Why not combine a minimum-support/cap rule on R(k)R(k) or explicitly separate entry cost R(1…k0)R(1\dots k_0) from continuation rate, and how does PriorBPB behave across tasks whose nmax⁡n_{\max} spans 2 to 50 bytes?

  2. Table 1 benchmarks accuracy fidelity only against mean-of-ratios BPB on cloze/multiple-choice tasks. What is the paired accuracy-tracking posterior (∆ and 95% CrI) of PriorBPB directly against corpus BPB (on base and instruct), and does PriorBPB improve accuracy tracking on the suite's generative math/code/QA tasks (GSM8K, MATH, MBPP, HumanEval, SQuAD2)?

  3. Section 6 states that cross-lingual neutrality improves from 0.100 to 0.027 for PriorBPB, whereas Fig. 10's caption reports median ∣DE/EN−1∣|\text{DE/EN} - 1| is 0.18 for PriorBPB vs. 0.13 for corpus BPB -- can you reconcile these two numbers in the main text? Also, given Prop. 11's ~4% leading-space numerator/denominator mismatch, why not attribute leading-space bytes consistently so the self-prior reduction to corpus BPB holds without an offset?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Vaqr (1/2)

We thank Reviewer Vaqr for the precise reading of Prop. 9 and of the accuracy results. Both criticisms were well founded, and acting on them improved the paper. We are glad they find the corpus identity, the Jensen decomposition of mean-of-ratios BPB, and the Qwen3 case study to “cleanly expose how first-token overhead and answer-length distributions distort cross-task leaderboards and scaling claims”. We have considered your two central concerns (the residual fixed cost on short-answer and sparse-tail tasks and the weakness of the accuracy evidence) when completing our revision.

In response to the critical weaknesses listed, we have revised the paper and provide the following response (corresponding to the provided numbered points):

  1. Residual length dependence of PriorBPB. You are right that R(k)R(k) is defined only up to a task's longest gold nmaxn_max, so by Prop. 9 the fixed-cost denominator is about min(μ,nmax)\text{min}(\mu, n_{max}) and a short-answer task still divides α\alpha by roughly nmaxn_max. We now state this in §5, and the within-task stratum test shows that for over 168 (model, task) conditions mean ∣ρ∣|\rho| falls only from 0.939 (corpus BPB) and 0.936 (mean-of-ratios BPB) to 0.879 (PriorBPB), because each stratum is scored on its own support (Appendix L). We have removed “length-invariant” and “substantially length-neutral” and now write “less length-dependent”. The −0.34 residual is the value on Qwen3-0.6B, and comes from the sparsely-supported deep tail (see Appendix H), and a common cap of 64 bytes brings it to ρ=∼0\rho = \sim 0 on that model, while over the 14 stored models the mean ∣ρ∣|\rho| is 0.142 with the cap and 0.163 without. A cap therefore does not remove the residual in the general case, which is why we offer some additional alternative forms below rather than claiming to eliminate the length dependence entirely.

  2. Accuracy prediction with PriorBPB. We agree the accuracy gains in the submission were weak, especially for the instruct model checkpoints. The instruct claim came from 16 two- or three-rung ladders prone to Spearman ±1\pm 1 saturation and has been withdrawn (we keep one such ladder only as an illustration that the choice of score can reverse a comparison; Appendix K). The revision now reports against corpus BPB throughout (Table 2), and claims a gain mainly for small base models, e.g., for DataDecide, when using 4M and 20M models to predict a 1B model’s accuracy, PriorBPB is more regularly predictive than corpus BPB (+0.023 [−0.005, +0.048]) and mean-of-ratios BPB (+0.058 [+0.024, +0.090]). We also added the generative math, code, and QA tasks by scoring public checkpoints on them directly (Appendix R). On cross-lingual neutrality point: you are right, the sentence was wrong and has been removed.

AuthorsOpenReview ↗

Response to Reviewer Vaqr (2/2)

We also address the three questions raised (corresponding to the provided numbered points):

  1. A minimum-support or cap rule, an explicit entry/continuation split, and behaviour across nmaxn_{max} from 2 to 50 bytes. We implemented the minimum-support rule, in two forms (Appendix M.1). In the first, R(k)R(k) is used only where at least 20 items and 5% of the task reach position kk, with the prior renormalised on those positions. This changes a task's value by 1.7% at the median and 62% at the 95th percentile, keeps the order of the tasks (Spearman 0.98 with the shipped score), agrees with accuracy on 53 of 74 model pairs, brings the largest effect of removing a task's longest gold from 39% down to 5%, and has mean ∣ρ∣|\rho| 0.20 for correlation with answer lengths. In the second, the mean rate of the last few supported positions is carried out under the full prior, so that every task is still scored under the same prior. We see these as trade-offs and present them all side by side in Table 9. One could use minimum support for a task scored on its own, and the pooled tail for comparing or averaging a fixed suite. The explicit entry-versus-continuation split is what the (α,β)(\alpha, \beta) decomposition and PrefixBPB provide, and in §7 we recommend reporting that decomposition alongside corpus BPB. On the range of nmaxn_{max}, the share of the prior's weight that a task's own answers cover is 2% for letter-label tasks and about one half to two thirds for short-answer tasks, which is the Prop. 9 regime where the fixed cost is divided by roughly nmaxn_{max} rather than μ\mu (Appendix M).

  2. The paired accuracy posterior against corpus BPB and the generative tasks. Table 2 now gives the paired posterior against corpus BPB on base and instruct models (see also the reply to jYBt Q2). From the smallest DataDecide models it is +0.023 [−0.005, +0.048], level along the 52 dense base ladders (+0.001 [−0.012, +0.016]), and withdrawn for instruct. On the generative tasks, the stored runs had no generation-based accuracy, so we scored public checkpoints on GSM8K, MATH, HumanEval, MBPP, and a gold-continuation form of SQuAD directly (Appendix R). Across 26 instruct checkpoints of seven families PriorBPB is not ahead of corpus BPB over the 14 tasks (−0.021 [−0.054, +0.011]) and is behind on the four generated-answer benchmarks; across 28 base checkpoints of ten families it is behind corpus BPB by −0.040 [−0.061, −0.020], a loss we trace to the task with the shortest answers (ARC) and to where the scored string is cut off, and which costs corpus BPB too.

  3. Reconciling the cross-lingual numbers and attributing the leading space consistently. The 0.100 and 0.027 figures were wrong, computed for one model and for PrefixBPB at a fixed prefix, and the sentence is removed; the caption of Fig. 12 is the correct comparison (0.18 for PriorBPB against 0.13 for corpus BPB). On the leading space, we now attribute it consistently so that the self-prior reduction to corpus BPB is exact (Prop. 11). Task values rise by 3% at the median and 24% at the 95th percentile over 218 (model, task) cells, the rank agreement with the shipped score is Spearman 0.99, the mean ∣ρ∣|\rho| with length moves from 0.16 to 0.18, and the accuracy comparison against corpus BPB is unchanged (Appendix F).

We are very grateful for these corrections, which strengthened the paper, in particular by prompting us to offer the minimum-support and pooled-tail forms, to report against corpus BPB throughout, and to withdraw the instruct and cross-lingual claims that the data did not support. We hope the revision now makes clear where each form of the score is appropriate, and that it leaves a precise practical case (a length-robust early signal on small models, e.g., for data-mixture ablations) rather than an over-broad one. We wish to again thank Reviewer Vaqr for their precise and constructive assessment, and will be happy to make additional changes as requested and engage in discussion.

Reviewer_P9ikOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper focuses on limitations of the popular bits-per-byte metric, specifically that it's often times strongly correlated with task length which can confound results and potentially reverse rankings across models. They propose a new metric which they show to be less length-biased, and provide some recommendations about how and when to use different metrics. Some highlight results include showing that length can be a very substantial portion of metric performance, and large reductions in length correlation with their debiasing.

Critical strengths
  • Overall I think that the problem statement here is really interesting and important. Bits-per-byte and negative log likelihood in general are heavily used in pre-training evaluations, and understanding their potential biases, as well as their implications for evaluation and optimization is really important
  • The PriorBPB correction is pretty interesting, and the downstream metrics reported with it are reasonably convincing to me at least in showing that it's effective in handling debiasing
Critical weaknesses
  • I think the experiment setting is slightly disconnected from how I could see this being valuable. BPB is typically used as a pretraining loss (language modeling) metric, and not directly on downstream task answers, which seem to consist the entire focus of this paper's evaluation. The key result which I would've liked to see is a case where BPB (as used in an actual technical report or existing research paper) disagrees or produces incorrect model rankings compared to the corrected metric (w.r.t. predicting downstream performance). I think it's possible to motivate applying BPB directly on downstream tasks in the "smooth" or "early signal" case where you may not have a lot of test examples, or have very low downstream performance and want to derive scaling laws in spite of this, but the paper doesn't really explore that motivation either in its experiments.
  • For checking the debiasing metric's effectiveness, it would've been helpful to see correlation numbers with downstream metrics, and what those plots look like. I couldn't get a good sense from the results and the metrics used in Table 1 are really confusing. Figure 1(c) is better but still doesn't give a full sense of things
  • The writing is really difficult to parse at different points (there's various passages, for example Section 6, which seem potentially generated by AI in the main paper and the prose is very difficult to get through). I had a really hard time figuring out what the method is, how the results are set up, and I felt like much of the writing leaned towards being much too verbose.

Overall I think this paper has a lot of potential to be really awesome, but I think would require substantial rewriting and re-orienting of experiments with respect to a clearer and stronger motivation grounded in potential practice.

Questions for authors
  • Why does the paper not report and focus on BPB on large held out language modeling settings (as is standard practice)?
  • Why not apply your debiasing to perplexity as well?
  • (Future work question) Have you considered if length bias issues may play a role in optimization (e.g. pre-training)?
3 comments
AuthorsOpenReview ↗

Response to Reviewer P9ik (1/2)

We thank Reviewer P9ik for their frank and constructive review. We are glad they find the problem statement “really interesting and important” and the PriorBPB correction "reasonably convincing ... in showing that it's effective in handling debiasing". The review's central concern, that PriorBPB’s practical usefulness over other BPB variants is not clear, has shaped our revision. We have conducted several additional experiments to address this concern.

In response to the critical weaknesses listed (in order of the provided dot points):

  1. Practical application of PriorBPB in pre-training evaluation. We agree that (a) the interesting regime for BPB is early pre-training signal and (b) that we hadn’t sufficiently exemplified that rankings between different pre-training checkpoints ranking can be dominated by the reported length bias, and that PriorBPB changes the rankings. We now emphasise exactly this setting on DataDecide (Magnusson, 2025), an existing study that uses per-byte likelihood on downstream answers from 4M-1B models to choose between 25 pre-training data recipes (§6, Table 2, Fig. 4; Appendix N.4). It also answers your request for BPB “as used in an existing research paper”: the BPB column DataDecide released is the mean-of-ratios form, and we find it makes measurably worse data decisions.
  • From models of 4M and 20M parameters, PriorBPB orders 0.77 of the recipe pairs in the same way as accuracy does at the 1B scale, compared 0.74 for corpus BPB (+0.023 [−0.005, +0.048]), and 0.71 for mean-of-ratios BPB (+0.058 [+0.024, +0.090]). All three BPB variants are well ahead of using the small models' own accuracy to predict the 1B scale accuracy (0.60). This demonstrates a real-world, highly practical and (we think) important case of the kind of consequential disagreement or reranking you requested in your review.

  • As a specific example, on CommonsenseQA at 4M parameters mean-of-ratios and corpus BPB prefer a recipe that reaches accuracy 0.58 at 1B over one that reaches 0.67, whereas PriorBPB prefers the better 1B recipe. Of the 539 pairs that corpus BPB and PriorBPB order differently at 4M and 20M, PriorBPB agrees with accuracy at 1B on 331 (compared to 207 for corpus BPB). As we report in the new Table 2, we also tested fidelity (the negative Spearman correlation between a score and accuracy along a ladder of checkpoints) for the Pythia model series, finding that at the smallest scale (160M) PriorBPB has markedly high fidelity than mean-of-ratios (+0.090 [+0.022, +0.184) or corpus (+0.098 [+0.035, +0.166]) BPB.

  1. Presentation of accuracy result. You are right that the original Table 1 was hard to read, and we have revised how these accuracy results are presented. §6 now carries a single accuracy table (Table 2) that gives the level of each score next to the paired differences, and a new figure (Fig. 4) showing decision accuracy at each evaluated model size for DataDecide, the difference between PriorBPB and each baseline with its interval, and the same differences broken down by task. We added plain Spearman correlations between accuracy and each score along each family's ladders, including scatter plots for the Pythia checkpoints (Appendix N.2), and across-family correlations in the larger sweep appendix (Appendix R). In general, we find that PriorBPB is particularly useful as a proxy signal on smaller models, e.g., ablation settings. If you believe any of these results could be presented more clearly in another or more detailed form, we would be glad to adjust them.

  2. Readability. We have revised large parts of the text to improve readability. §5 now opens with a description of the method in one sentence, and §6 opens with its three main questions and then answers them in that order. The (in our view) most interesting accuracy correlation results are now condensed into one table and figure (Table 2 and Figure 4). We have also moved less important content to the appendix, e.g., PrefixBPB. We added a per-token trace that follows a single prompt--response pair (the example from Figure 1a) token by token to make the fixed cost more intuitive by example (Appendix P), and added a full evaluations settings table (Appendix Q). We made a language pass over the whole paper. We would welcome pointers to any passage that still reads poorly.

AuthorsOpenReview ↗

Response to Reviewer P9ik (2/2)

We also address the three questions raised:

  1. Held-out language-modelling text. We have added this (Appendix O.1). On Paloma's C4 test split (99 web domains, median document length 2,237 bytes), scored with the three Qwen3 base checkpoints, the fixed cost divided by document length is ~3% of corpus BPB for the median domain, compared to ~33% to ~100% on the short task answers of Fig. 2. The checkpoints are ordered by size on all 99 domains under corpus and mean-of-ratios BPB and on 98 under PriorBPB. On long documents from one source, any length confound is therefore negligible and corpus BPB essentially needs no correction. We have also stated this in introduction. However, the same analysis shows the length confound returning as the scored text span shortens, since a 32-byte span costs about 2.1x to 2.3x the per-byte rate of the first 8 kB. Our current work concerns this second common use of BPB, on short task answers, e.g., where a mean answer of roughly 13 to 37 bytes is typical for six of our length-varying tasks.

  2. Debiasing perplexity. Bits-per-token and perplexity on task answers has the same length confound as BPB, and the same correction works (Appendix O.2). Across tasks, the mean absolute Spearman correlation with mean answer length is 0.48 for corpus bits-per-token (compared to 0.50 for corpus BPB on the same checkpoints), whereas averaging the cost of the t-th answer token under a shared prior over token positions lowers it to 0.18 (a reduction of 0.30 [0.21, 0.38]), compared to the byte form which reaches 0.16. We keep bytes as the unit in the paper because a prior over token positions is tied to one tokeniser, so the token form can compare tasks for a single model but cannot compare models with different vocabularies.

  3. Length bias in optimisation. We ran a very limited pilot of this idea at a small dense model scale (Appendix T). We trained a 412M-parameter model on 8.2 billion tokens of DCLM-Baseline under three losses with weights normalised to a mean of one: the ordinary token mean, a flat prior over document positions, and a short prior on the start of a document. The short prior lowers the held-out loss on the first 32 tokens of a document but raises it later, and is worse both overall and on its own weighted objective. We note, however, that this is a single model size and corpus, and there are still other possibilities to try. We would be interested in exploring this further in future work.

Thank you again for pushing us towards concentrating more on the practical use-cases where PriorBPB helps the most: In small ablation models, where model builders need to make consequential decisions. Here, PriorBPB is most useful, as other metrics like accuracy or other BPB variants are less predictive. We are happy to discuss any of these points further and incorporate additional feedback.

Reviewer_P9ikOpenReview ↗

Response to Updates from Authors

Firstly, I want to say I think the updated Section 6 is a big gradient step in the right direction (specifically the new table / Figure for the datadecide setting), so thanks for taking the time to run those and add the updates I think they're great and exactly what I had in mind! I think with those results the paper feels a lot more compelling and promising. I went ahead and updated my score to narrow accept based on that and the other responses / updated (though I still feel like overall writing and framing could be improved a lot).

Separately from my conference recommendation (I think this version is ok to accept), I do still feel like this paper has a lot of potential that I don't think is yet fulfilled by the current version (specifically in terms of focusing the writing / framing on the DataDecide results which I think are the most exciting, but also currently feel hidden, understandably as they were added last-minute). I think if the authors were to do a larger-scale rewrite of the paper, focusing on DataDecide-style results, connecting better to other work in the area, as well as potentially 1-2 other well-motivated settings (e.g. some ideas could be-implications for optimization in multi-task SFT mixes, literature review showing that a lot of existing work / decisions relying on perplexity are incorrect, other ideas that can be connected to well-motivated issues with BPB and NLL), this could go from being a currently ok paper to a really really awesome paper (e.g. top-10 NLP papers of the year)! Given this, regardless of whether the paper gets accepted, I strongly encourage the authors to give the paper another 1-2 months to try to see if they can get this better version!

Reviewer_4F2uOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies bits-per-byte (BPB), a metric widely used to compare language models across tasks. The authors show that BPB depends on answer length: under a least-squares fit, corpus BPB equals β + α/n̄, where α is a fixed per-item cost and n̄ is the task's mean gold length. Tasks with long answers therefore score better, and the effect is strong (Spearman about −0.8 with length across 19 tasks).

Critical strengths

•The core identity (corpus BPB = β + α/n̄ under an OLS fit) is simple, correct, and easy to check. It explains the length bias clearly. •The paper is unusually careful with statistics: Bayesian bootstraps, Bayes factors, prior sweeps, family-clustered resampling, and a within-task stratification test. •The method reduces exactly to corpus BPB under a self-prior (Proposition 3), so the relationship to the existing metric is clear. •The paper flags the mean-of-ratios BPB issue (up to 99–145% inflation), which is a real and practical bug in common eval frameworks.

Critical weaknesses

•Overclaims in the abstract. The abstract says the smallest Qwen3 model's apparent inverse scaling is "due primarily to differences in the overhead." The appendix admits this is "supported but not proven," because base and instruct checkpoints differ in more than the chat template. •Mixed numbers. The abstract cites OLMo's −0.79 correlation, which is the mean-of-ratios number. The corpus number is −0.67. For Qwen3, the abstract says −0.82 to −0.86, but Section 3 says −0.77 for corpus BPB. These should be stated consistently. •Weak accuracy gains. The instruct result is +0.122 with a 95% CrI of [+0.005, +0.571]. That interval is very wide. The base result is +0.021 with CrI [+0.004, +0.079]. The "better accuracy proxy" claim is real but small. •The accuracy gold is partly mis-specified for base models. The authors admit this. So the base-model accuracy claim is weaker than the abstract suggests. •Residual length bias is not zero. PriorBPB's cross-task correlation is −0.34 with CrI [−0.67, +0.04]. Changing the prior mean μ moves it from −0.01 to −0.42. The held-out check flips sign to +0.37 on four tasks. Calling it "substantially length-neutral" is generous. •Within-task improvement is modest. Mean |ρ| goes from 0.944 to 0.888. The authors acknowledge this, but it weakens the "length-neutral" story.

Questions for authors

as in weakness.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 4F2u

We thank Reviewer 4F2u for their kind review of our paper. We are glad they appreciate our contributions, both empirical and theoretical, and that they note this has practical significance insofar as it is a "real and practical bug in common eval frameworks".

In response to the critical weaknesses listed, we have revised the paper and provide the following response (in order of the provided dot points):

  1. The claim in the abstract can be decomposed into two aspects. The revised version now separates them.
  • The observed scoring gap comes from the overhead. This is an exact decomposition by Eq. (1): of the −0.32 BPB by which Qwen3-0.6B-Instruct leads 4B. Out of that, −0.30 comes from the fixed cost and only −0.01 from the rate (Appendix S, Table 22). So this claim is factual.

  • The overhead grows with scale. To address the confound you raise, we added an experiment that scores the same base and instruct weights with and without the chat template (§4, Table 1 and also Table 22 in Appendix S). For the instruct checkpoint, removing the template removes the effect (0.6B beats 4B on 15/17 tasks with it, 2/17 without), whereas the effect is absent for the base checkpoint (0/17 in both settings). Why this growing overhead occurs only for instruct models with chat template remains open and we state this in the text.

  1. Inconsistent numbers. The abstract now uses one estimator consistently (corpus BPB) and names it explicitly. In the main text, we report both estimators (§3). Note, some values moved slightly after extending the evaluations suite; no conclusion changes.

  2. PriorBPB as an accuracy proxy. We agree that across the models we tested in our original submission, our claim that PriorBPB is a "better accuracy proxy" had a small effect size, especially on base models. In our revision we narrowed our claim and emphasise more the regime where PriorBPB provides a larger benefit: very small base models. For example, across five training seeds of Pythia-160M, PriorBPB more closely predicts accuracy of future training steps than mean-of-ratios BPB (+0.090 with CrI [+0.022, +0.184]) and corpus BPB (+0.098 with CrI [+0.035, +0.166]).

  3. Base mode accuracy. We agree that this is a limitation and state it explicitly in the paper (Appendix K). However, the main contribution, reducing length bias, never uses accuracy and is unaffected by this limitation. Additionally, our new DataDecide experiment (see §6) asks “which small-model metric best predicts the ranking at 1B?”. At 1B scale, accuracy is a commonly-used metric for recipe decisions. We have revised the abstract to avoid suggesting more than what our results actually support and hopefully improved the clarity. Please let us know if you would like us to make any further specific changes.

  4. Length dependence. Agreed, we now describe PriorBPB as less length-dependent, not length-neutral. We have also added results in §6 which show that PriorBPB has a lower length correlation in the Qwen3 and OLMo-3-7B models than corpus and mean-of-ratios BPB in 92% and 96% of possible test/held-out eval splits, respectively.

  5. Within-task improvement. We clarified this point in the Figure 3 caption. Following the previous point, the paper now states that PriorBPB is less length-dependent, but not length-neutral.

Thank you again for the constructive review which helped us improve the manuscript. We are happy to discuss any of these points further.

Thanks for rebuttal, I'd maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.