- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper investigates answer-length dependence in bits-per-byte (BPB) evaluation. Using an affine decomposition of total answer surprisal, it shows that corpus BPB contains a fixed-cost term divided by mean answer length, while averaging per-item BPB introduces additional length-dependent terms. It indicates how these effects can produce misleading comparisons, including apparent inverse scaling among Qwen3 models. The proposed metric, PriorBPB, estimates a marginal cost curve over answer-byte positions and aggregates it using a shared length prior instead of each task’s empirical length distribution. Validation includes multiple model families, dense Pythia training checkpoints, prior-sensitivity analyses, held-out tasks, and comparisons with downstream accuracy. The reported results show substantially reduced cross-task length correlation and modestly improved accuracy tracking, with larger gains for instruction-tuned models.
Critical strengths
The strongest aspect of this paper, in my view, is its diagnosis of length bias in a metric that the community uses widely and largely without question. The core identity is algebraically simple, and the authors acknowledge this, so I do not consider the identity itself the contribution. What I found more useful is the empirical evidence around it. Simply rephrasing the same answer changes BPB by a factor of four to five. Per-item BPB on SQuAD2 turns out to mostly reflect answer length, and the matched-position analysis shows that the per-byte cost at a given position does not depend on how long the answer eventually becomes. Together, these make a persuasive case that a fixed first-token cost, rather than genuine difficulty, drives much of the reported score. I also found the observation about mean-of-ratios BPB particularly useful, since this is the form reported by lm-eval-harness, and the authors show it can inflate scores by up to 145%.
I also appreciated the inverse-scaling case study as a cautionary example. The results suggest that the smallest Qwen3 model performs best, but the decomposition shows that this trend is primarily due to the fixed cost growing with model size, while the per-byte rate shows no such pattern. The base-checkpoint control makes the authors’ explanation credible to me: once the chat template is removed, the fixed-cost growth largely disappears. For example, on GSM8K it stays at around 12/6/8 bits for the base models compared with 23/94/154 bits for the instruct models. I think this is a valuable demonstration of how a metric artifact can look like a real capability trend, and it supports the paper’s broader message that BPB comparisons should not be taken at face value without examining what drives them.
Critical weaknesses
My main concern is that PriorBPB may not fully remove the length bias. Under an affine cost, Prop. 9 shows the fixed cost is still divided by a term that depends on each task’s longest answer, so part of the bias remains. The late positions of R(k) are also estimated from a few long answers, but they still receive substantial weight. App. H shows that the remaining correlation comes from exactly these positions. Within tasks, the gain over corpus BPB is small (0.944 to 0.888, App. L), so calling PriorBPB "length-invariant" in Fig. 3 seems too strong.
I am also not fully convinced by the evaluation. Low correlation with length is not enough on its own, since some tasks may genuinely be harder per byte. On held-out tasks, the correlation even flips sign (−0.73 to +0.37). For accuracy tracking, PriorBPB is compared only to mean-of-ratios BPB, not directly to corpus BPB, and the gains are small for base models (+0.021) with an expansive interval for instruct models.
Questions for authors
-
In Eq. 3, is the position budget each task’s longest gold (n_max), or a common budget across tasks? If it is per-task, how sensitive is PriorBPB to the single longest item? A quick check would help: recompute the cross-task correlation after removing the longest gold in each task or with one common budget for all tasks.
-
Table 1 compares PriorBPB only against mean-of-ratios BPB. Since the paper recommends corpus BPB, could you report the same paired fidelity delta against corpus BPB on the same base and instruct models? This should be possible with the scores you already have.
-
Since the length mix is shared by all models within a task, where does the within-ladder accuracy gain come from? Is it concentrated in tasks where alpha changes a lot along the ladder? Splitting delta by tasks with large versus small alpha change would answer this.
-
On the held-out suite, the correlation flips from −0.73 to +0.37. Could you report the per-task values for the four held-out tasks so we can see whether this is driven by a single task or reflects over-correction?
-
Which estimator does the abstract’s −0.82 to −0.86 range refer to? Also, how does the claim that cross-lingual neutrality improves fit with Fig. 10, which shows PriorBPB with the largest DE/EN deviation (0.18 vs. 0.13 for corpus BPB)?
Happy to discuss these questions during the rebuttal period, and I am open to reconsidering my scores if my concerns get addressed.
3 comments
Response to Reviewer jYBt (2/2)
We also address the five questions raised (numbered by the order of dot points):
-
Position budget and sensitivity to the longest item. The budget is per task, the task's longest gold (Appendix M). Removing the single longest gold answer can change a task's PriorBPB by 0.4% at the median, by more than 5% in 11.6% of the 198 (model, task) cells, and by 14% to 39% on NaturalQsOpen for 10 of the 14 models (there the longest gold is 78 bytes and the next 56). The cross-task length correlation barely changes, however, with mean going from 0.163 to 0.141. Replacing the per-task budget with one common budget of 8, 16, 32, or 64 bytes gives mean between 0.142 and 0.163, compared to 0.496 for corpus BPB. So the cross-task result does not rely only on the longest benchmark items, even though it can change individual task's value.
-
Paired fidelity against corpus BPB. Table 2 now reports this for the DataDecide suite and the base ladders (the instruct ladders are in Appendix K). Using 4M and 20M parameter models, PriorBPB predicts accuracy at 1B better than corpus and mean-of-ratios BPB (see Figure 4 and Table 2 in §6). Along the 52 dense base ladders, the three BPB variants scores are similar at a mean fidelity of 0.50. However, again for the smallest model size, Pythia-160M, PriorBPB appears to demonstrate an advantage over both baselines, e.g., +0.098 [+0.035, +0.166] over corpus BPB.
-
Where the within-ladder gain comes from. By more systematically answering this question we have identified something we are quite excited about, which is that the largest gains are on the smallest models (Q2), and it is concentrated where the fixed cost is a large share of the score, e.g., CommonsenseQA (+0.10 over corpus BPB; 10-byte answers) and SocialIQA (+0.07; 20-byte answers), where the first answer token carries 0.73 and 0.41 of an answer's bits (Appendix N.4).
-
Held-out per-task values. These are now in a table (Appendix J, Table 4). The sign flip is mostly DROP. Its PriorBPB is 0.23 to 0.66 compared to a corpus BPB of 1.14 to 3.85, and it falls to 0.19 of its corpus BPB because only 43 of its 250 golds exceed 10 bytes while byte positions from 6 onward contribute 92% of the prior‘s weight, so a few cheap copied spans influence a large fraction of the score. This indicates over-correction on one task, as you suspected. The held-out suite has only three tasks, so each model's correlation can take just four values (it is −0.50 under corpus BPB and +0.50 under PriorBPB for all six models), and we no longer draw any conclusion from it. For a systematic test we instead enumerate held-out splits within the suites (Appendix J.1): over all 258,077 splits of the 18 tasks into a -fitting set and a held-out set, the held-out length correlation falls from 0.67 to 0.31 on average, and PriorBPB has the lowest length correlation in >92% of the splits compared to corpus and mean-of-ratios BPB.
-
The abstract's range and cross-lingual neutrality. The submission's −0.82 to −0.86 was mean-of-ratios BPB on the three Qwen3 models; re-computed on the 18 length-varying tasks these are −0.79 to −0.84, with corpus BPB at −0.67 to −0.74 (OLMo-3-7B: −0.77 and −0.62). The abstract now reports the mean-of-ratios values and states that they are the average of per-item BPB, and §3 gives the corpus values. Regarding the cross-lingual neutrality, you are right and we have removed the neutrality claim.
We are very grateful for these questions, which materially improved the paper, in particular by making us report against both mean-of-ratios and corpus BPB throughout, withdraw the weak or incorrect instruct checkpoint and cross-lingual claims, and more clearly present the within-task results. We hope the revision now shows clearly both what PriorBPB removes and what it does not, and that this makes the remaining practical case (a length-robust early signal on small models, e.g., for data-mixture ablations) easier to weigh. We wish to again thank Reviewer jYBt for their careful and constructive assessment, and will be happy to make additional changes as requested and engage in discussion.
Thank you for the detailed response and the additional analyses. You have addressed my main concerns. While the accuracy-prediction gains appear concentrated in smaller-model settings, I find the revised contribution useful and the paper’s claims better calibrated. I am therefore updating my rating to "Weak Accept", and I ask that the final version fully reflect these revisions.
Response to Reviewer jYBt (1/2)
We thank Reviewer jYBt for the detailed and generous review, and especially for the five questions, each of which we answer below. We are glad they find the paper's strongest aspect to be “its diagnosis of length bias in a metric that the community uses widely and largely without question”, and that they find the mean-of-ratios BPB observation and the inverse-scaling case study persuasive. The central concerns, that PriorBPB may not fully remove the length bias and that the evaluation was not yet be convincingly meaningful in practice, have shaped our revision.
In response to the critical weaknesses listed, we have revised the paper and provide the following response (corresponding to the two paragraphs):
Length dependency claims. We agree on each point, and the revision no longer overstates the within-task result. §5 now states that the sums in the PriorBPB definition run only to a task's longest answer and that the prior is renormalised on that support, so by Prop. 9 a short-answer task still divides its fixed cost by about its longest answer rather than by μ. We have also replaced phrases like “length-invariant” and “substantially length-neutral” throughout with weaker but more precise claims like “less length-dependent”. The caption of Fig. 3 and Appendix L include the within-task results: over the 168 (model, task) conditions, mean ∣ρ∣ falls only from 0.939 (corpus BPB) and 0.936 (mean-of-ratios BPB) to 0.879 (PriorBPB), a paired reduction of +0.060 [0.015, 0.133]. As you note, the residual sits in the sparsely-supported late positions of R(k) (also see Appendix H). We instead claim PriorBPB offers a reduction of a known confound rather than being entirely length-neutral, and in Appendix M.1 we provide some alternative forms (minimum support, a terminal-rate tail, and a pooled tail) that trade this residual off against other evaluation goals.
Correlation with accuracy. We agree that a low length correlation cannot by itself show that a score is better, since some tasks may genuinely be harder per byte, and the revision now presents the length result as the removal of a known artefact and reports improved accuracy prediction against both mean-of-ratios and corpus BPB in what we think are practically-meaningful settings (also see Q2 below). Measured on the smallest models used in a pre-training data mixing experiment ladder, PriorBPB predicts accuracy at 1B parameters better than mean-of-ratios or corpus BPB (see Figure 4 and Table 2 in §6). On the held-out sign flip: you were right to be suspicious, as this was an over-correction on a single task (DROP); we no longer draw any conclusion from that three-task suite (Q4). We have removed the instruct checkpoints accuracy claim given the weak effect size.