acceptodds
Under review as a conference paper at ICLR 2027

Measuring Order Learnability via Prequential Codelength

Abstract

Does sequential data favor left-to-right prediction? Although entropy is invariant to factorization order, a finite learner may acquire some factorizations more efficiently than others. We measure order-learnability using prequential codelength, which accumulates the log-loss incurred when a model predicts each batch before training on it. After synthetic calibration, we compare 21 fixed orders across 13 corpora under a controlled Transformer training protocol. Our experiments reveal four findings. First, left-to-right is rarely uniquely favored: it is significantly better than every tested alternative only on mathematical chain-of-thought traces. Second, text is substantially more sensitive to local shuffling than to full reversal, suggesting that preserving local adjacency matters more than preserving forward direction for this learner. Third, the differences between orders change with training: both additional data and repeated passes narrow the relative gap on text but widen it on two protein sources. Fourth, frequency-sorted orders achieve substantially lower prediction loss on English Wikipedia while supplying additional information about the content. Our framework turns intuitions about a corpus's preferred prediction order into quantitative comparisons of learning efficiency under a fixed learner.

Primary area:
interpretability and explainable AI
Keywords:
Data ordering, Order-learnability, Prequential coding, Minimum Description Length, Autoregressive modeling, Diffusion language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_AxUMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes order-learnability, measured through prequential codelength, to quantify how efficiently a fixed learner acquires the same corpus under different token prediction orders. Since the true joint entropy is invariant to factorization order, differences in prequential codelength are interpreted as differences in the finite learner's redundancy while learning. Authors compare 21 fixed orders across 13 corpora spanning natural language, code, mathematical chain-of-thought, biological sequences, VQ-tokenized images, and synthetic tasks. Interesting findings are that left-to-right is uniquely favored only for mathematical chain-of-thought; reversal has surprisingly little cost on text compared with local shuffling; order effects change with additional training; and content-dependent frequency sorting can dramatically reduce prediction loss.

Critical strengths

The experimental question is well motivated. The reversal/local-shuffle comparison is interesting. For the seven text corpora, reversal accounts for at most 5.5% of the global-shuffle excess while four-token local shuffling incurs 50–62%. This is the paper's most interesting empirical observation. Prequential codelength is a sensible operational measure. The coding interpretation is clear. An order that allows the learner to make accurate predictions earlier pays less redundancy while learning. The synthetic calibration is strong. On IID data, the instrument correctly detects no order preference; when the authors plant a Markov dependency along a particular order, the metric identifies that order as most learnable. The statistical protocol is thoughtful. The authors use eight paired runs, shared data ordering across orders, separate learning-rate tuning data, and multiplicity correction when selecting the best alternative.

Critical weaknesses

My primary concern is that the central result is based on only one learner. This is especially important because prior work cited by the authors indicates that Transformer positional and locality biases themselves affect performance under permuted orders. The paper explicitly treats those biases as part of the learner which could be inconclusive.

Scale is particularly problematic because the paper shows that the answer changes with training. The paper itself demonstrates that order rankings and gaps change as learning progresses. Figure 3 further shows that additional training narrows order differences for text but widens them for protein.

The interpretation of reversal versus local shuffling is not fully identified. This concern is especially relevant because the learner has explicit positional encodings and Transformer positional biases are known to matter under permutation.

Frequency-sorted-order experiment is interesting but not comparable to the main experiments. The finding that reversal performs similarly to left-to-right for a 17M-parameter learner does not imply that reverse generation is preferable: generation quality, inference caching, streaming, prompting conventions, downstream use, and architecture all matter. The paper explicitly does not compare downstream performance.

Questions for authors

I would consider at least one additional architecture or meaningful model-size scaling experiment important. Otherwise it is difficult to know whether they characterizes the corpora or this specific learner.

The paper could temper the causal interpretation or provide more controlled permutations matching additional structural statistics.

I think Frequency-sorted-order experiment should be presented more clearly as a conceptual upper-bound rather than one of the four main empirical findings. As implemented, it does not demonstrate a usable generation policy because the policy needs information about tokens that have not yet been revealed.

Can the authors more precisely justify the conclusion that the reversal/local-shuffle result specifically identifies local adjacency, rather than other positional structures preserved by reversal?

What practical model-design decision should the proposed order-learnability measurement inform? In particular, how should its conclusions translate to variable-order or masked-diffusion models rather than fixed permutations?

4 comments
AuthorsOpenReview ↗

Model size and architecture

Thank you for raising this single learner concern. We agree: order-learnability is by definition relative to a learner, so how far the conclusions extend across learners needs to be checked. Of the two options you name, we have added a model-size experiment (new appendix "Model size"). Model size. We repeated the single pass with a smaller and a larger learner of the same architecture: 5M and 60M parameters, beside the 17M of the main experiments, a 12-fold range; one corpus per data type (seven in all) and eight orders, with the learning rate tuned at each size.

  • The contrast between reversal and local shuffling holds at all three sizes: on the collected corpora reversal costs at most 6% of the excess of a global shuffle, against 24–73% for shuffling within blocks of four.
  • The verdicts on order preference largely agree, and no corpus receives opposite verdicts at different sizes. Relative to the main experiment, there are three exceptions: at 5M, chain-of-thought becomes unresolved, and on RNA an alternative is better than left-to-right by 0.17%; at 17.1M, the shared-learning-rate comparison over the reduced set of orders favors tail reversal on arithmetic, whereas the main experiment is unresolved. On the positional biases of Transformers. The calibration tests this directly. On iid data, which have no structure, the model shows no preference among orders. On data constructed along a target order, the model identifies that order as the most learnable one every time, even when following it means jumping between positions. The cost of shuffling on text therefore comes from the data itself, not from a bias of the learner toward a particular order of positions. On the answer changing with training. This is why we compare orders by the codelength averaged over the whole course of training rather than at a single point, and why the convergence experiment follows the gaps as training continues. Architecture. We did not add a second architecture. The Transformer is the mainstream architecture now shared by models of text, images and biological sequences, which is why we use it as the common learner. A different architecture would also require redesigning how the order is given to the model and recalibrating it, which we could not complete in this revision. The Limitations section states that our results are relative to one architecture
AuthorsOpenReview ↗

Local adjacency and positional biases

Thank you for raising these concerns about local adjacency and positional biases. We have followed both of your suggestions: we added a controlled permutation, and we tempered the causal wording in the main text. We explain both below.

First, on reversal versus local shuffling. The two orders do differ in two respects at once. Local shuffling (for example ls4) still proceeds left to right overall and only permutes the positions inside each block: every token is predicted within one block of its original position, with the prefix outside its block complete, but its local relation to its neighbors is scrambled. Reversal keeps every token next to its neighbors, but predicts it far from its original position, and from a suffix instead of a prefix, so the global context is entirely different. For people who already know a language, a scrambled local order is fairly easy to adapt to, whereas reversed text is very hard (Mirault et al., 2018; Saberi & Perrott, 1999); our results show that for a model learning from scratch it is the other way round.

To separate the two, we have added block reversal (appendix, Table 4): the blocks are read in order, and each block from its last position to its first. Inside a block it is the same as reversal, with every adjacency kept; outside the block it is the same as local shuffling, with all earlier blocks already read in full. Adjacency is broken only once, at the head of each block. On the seven text corpora, block reversal costs 37–47%, 17–29% and 4–15% of the excess of a global shuffle at block widths 4, 16 and 64, against −2.5% to 5.5% for full reversal. A regular order can therefore be expensive too, and the wider the blocks and the fewer the cuts, the lower the cost.

Split by position inside a block (six text corpora), the token read immediately after a cut costs 2.7–6.1 bits more than under left-to-right, whereas the other tokens of the block cost only 0–0.4 bits more on average; at width 4, this single token accounts for 69–95% of the total cost.

These results show that the cost depends on how much adjacency is broken, not on the reading direction, nor on whether the order is regular. We have also softened the wording in the main text (Sections 1, 3.2 and 4.2): it now says that the results suggest, rather than establish.

Second, on whether this is only a bias of the model's positional encodings. The calibration answers this directly. On synthetic data we lay a chain of adjacent dependencies into the window along a target order and let the model read in that order. Under all nine target orders its codelength lies between 0.5839 and 0.5848 bits/token, close to the entropy of 0.5773; it is 0.5846 when the chain is laid along a global random order and 0.5845 when it is laid along the reversed order. The model itself is therefore not averse to a scrambled order of positions: as long as the dependencies follow the reading order, it learns equally well under any order. The cost of shuffling on text thus comes from how the dependencies in the data are arranged, not from a bias of the positional encodings.

AuthorsOpenReview ↗

Clarifying frequency-sorted orders

We agree that frequency order should be set apart from the main experiment more clearly. The submission already stated that these losses exclude the cost of communicating the order, that they are not standalone codelengths, and that the runs are separate from the main experiment. The revision now explicitly calls them assisted orders (abstract, introduction, Sections 3.2 and 4.4): the order is computed from tokens that have not yet been predicted, so it is not a usable generation policy, and the reported losses cannot be compared with the codelengths of the main experiment.

We did not use the term “upper bound” because the experiment does not establish a formal bound. Instead, we call these “assisted orders” to emphasize that the ordering itself supplies information derived from tokens that have not yet been predicted.

AuthorsOpenReview ↗

Limitations of downstream evaluation and implications for variable-order and masked-diffusion models

We agree: a codelength for reversal close to that of left-to-right does not make backward generation preferable. The codelength measures only how efficiently the data are learned under a given order. Because each document is seen once and scored before the update, the single-pass codelength is also a held-out loss, but it is not an evaluation of generation. The paper does not compare sample quality or downstream performance, with the single exception of exact-match accuracy on arithmetic, and it does not address practical aspects of generating in an order other than left-to-right, such as the loss of streaming output. We now state this in the Limitations section and in the appendix. We should note that every order in the paper is a fixed permutation and that we do not measure the objective of variable-order training or of masked diffusion itself, so the implication for such models is indirect. Within this scope, the results show one thing: the data themselves do not require left-to-right. Except on chain-of-thought, left-to-right has no significant advantage over its best alternative on any corpus. As far as the data are concerned, there is thus no built-in obstacle to models that are not tied to the left-to-right order, which encourages the further exploration of non-autoregressive models. We have added this point to the Conclusion of the revised paper.

Reviewer_5W3COpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

By the chain rule, every factorization order of a joint distribution should be equal to the true joint; however, under a fixed model, different orders can affect the rate at which a particular model actually learns the distribution. This paper measures this effect using prequential code length, which is the accumulated log loss of each batch before updating the model, as a large-scale empirical study.

They compared 21 fixed orders across 13 corpus categories, including text, biological sequences, images, arithmetic, and path-star records. Models were trained from scratch, with matched initialization and data presentation across orders, but separately tuned learning rates. Tokens retained their original positional information even when their prediction order changed.

They find that 1) there are relatively few statistical differences between left-right orderings and the most efficient ordering for most datasets, with a few exceptions, or, to put it another way, left-right is rarely privileged. 2) For natural language corpora, local token shuffling incurred much greater loss than full reversal. 3) Differences in learning efficiency are not constant across training. 4) Positive content- or data-dependent orders learn efficiently.

Critical strengths
  1. The empirical experimentation is broad.

  2. Learnability given different orderings is an intresting topic with a large reach. I believe their results (as is) may be often cited simply for the claim that 'despite any factorization order being valid under the chain rule, the chosen order matters for actual learnability' (see question/issue about prior work, though).

  3. The sanity check experiments were welcome.

Critical weaknesses
  1. The paper presented a broad empirical comparison, but its explanatory depth was limited. It quantified order-dependent prediction loss without sufficiently establishing the mechanisms behind the differences or their relationship to task learnability. Basically, it reports numbers and differences without sufficiently explaining those differences.
  2. I'm concerned that the results may not be robust to learning rates. Each order for each experiment is tuned with its own learning rate. Each experiment controls for training by matching steps and tokens, but not learning rate. I think the authors made the correct choice in tuning the learning rate for the main experiments, but I would really like to see an ablation to establish how much this actually matters.
  3. The synthetic task experiments (path-star, multiplication) lack a clear scientific role. As presented, they are confusing, unexplained, and misleading. This is a major issue for me. Both these tasks are conditional tasks that are generally modelled as p(target | source); however, they are treated as p(source, target), with different orderings applied to the entire sequence. I'm going to focus on the path-star experiments, as I know this task very well.

Because path-star has known differences in learnability based on the order in which the target arm is written, I initially assumed you were considering different orderings of the target arm. In the standard path-star task, the model is conditioned on the random edge list and query, and only trained to predict the target arm. Learning the task with the forward arm is much harder than with the reversed arm. But your main comparison concerns the likelihood of the entire sequence. You also consider target-only reversal, which makes it clear that you recognized this distinction, but the relationship between these experiments and the known learning properties of the task is not adequately explained.

A) The edge-list order and node IDs are random. There is therefore substantial unavoidable uncertainty in predicting the source. However, the source is not completely random: the path-star structure constrains which edges can occur. For example, observing most of an arm can constrain its remaining connections. The important distinction is between this unavoidable uncertainty and the model's ability to learn and exploit the structural constraints.

B) p(target | source) is hard to learn in the forward direction, and you do not show whether your model actually learns to solve the task. A model that simply chooses one of the graph's five arms can achieve 1/5 accuracy without learning to select the arm leading to the queried goal. Based on my knowledge of prior work, I would not expect standard next-token training to reliably solve this 56-node setting as to my knowledge, no one has done this. So I will assume that it did not converge, which would otherwise change the explainability of the results.

Predicting the reversed target given the source is much easier algorithmically because it only requires successive predecessor lookups. However, under teacher forcing, edge retrieval also allows most of the forward target to be predicted without learning the full task. Once the correct first branch is supplied, the remaining arm can be followed through unique-successor lookups. The main computational difference can therefore be concentrated in a single target token. Averaging the loss over the entire source and target can make this important difference in task learnability appear very small.

C) Why would target-first be easier to learn than source-first? You provide no explanation for the large result in Figure 1. Knowing a full arm before predicting the source supplies 11 edges that must appear in the edge list, leaving the other four arms unknown, although the positions of all the edges remain shuffled. A model could exploit this information through edge retrieval without learning to solve the path-star task.

However, this is only a partial explanation, and looking at the numbers, I do not think it can account for the difference. Predicting the target without the source also incurs a cost, while seeing the source can help predict target tokens even if the model cannot solve the full task. The entropy of the complete record is unchanged, so the reduction in source uncertainty alone does not establish a net advantage. What needs explaining is why your model exploits the dependencies more efficiently in one order. Without separate source, query, and target losses, I cannot determine what accounts for the observed values.

Deep in the appendix you state:

“This backward reading is not what makes an order cheap: tail reversal (tr44), which reads the edge list first and then the answer from the goal back, is no cheaper than left-to-right (excess codelength −0.0004, [−0.0164, +0.0156]), whereas the two orders that are, full reversal (−0.3534) and gf44 (−0.3462), both place the answer before the edge list (Table 3).” [Lines 1007–1010]

But this explains what was done, not why these results occurred or what they establish. Interpreting the implications is left to the reader. In particular, the absence of a significant whole-sequence difference for target-only reversal does not establish that the two target directions were equally easy to learn as path-solving tasks.

I completely fail to understand the motivation for this experiment as presented. POTENTIALLY, it could be framed as a sanity check, which you already have, but that would need to be clearly explained. Right now, it is offered as just another domain alongside the others, without adequately distinguishing its role. Path-star has a known task-specific asymmetry: forward path generation requires selecting a branch through multi-hop reasoning, whereas backward generation can use local predecessor lookups. It is unclear how the reported metric captures this asymmetry, or how the large answer-first effect relates to it. Presenting these results together in Figure 1 without explaining that distinction is misleading.

To be clear, I understand that you can treat this as a joint sequence of source and target tokens and mathematically factorize it in any order. My issue is not the mathematical viability of this, but the reasoning for doing these experiments. Reordering changes which task information is available during prediction, but there is insufficient explanation of the observed behaviour or what scientific question this experiment answers.

These synthetic experiments feel like something included to preemptively satisfy reviewers, but this is a case where adding more to the paper was unnecessary and made it worse, to the point of being misleading. I think the paper would be better without these experiments, and I recommend their removal.

Questions for authors

Q1) Beyond the established observation that finite models perform differently under different valid factorizations, what new scientific conclusions did the prequential measurement enable, particularly relative to Luo et al. (2025) and Papadopoulos et al. (2024)? My concern is that the broad takeaway I identified in strength #2 was already supported by those papers. What additional understanding did this work provide beyond extending the empirical comparison to more orders and corpora?

Q2) I have serious concerns about the path-star results and the multiplication (labelled as arithmetic?) results, in that they are conditional sequences presented as being equal to the other domains, which are unconditional sequences, when I do not believe they are equal in terms of their reorderability. Are the other domains equal in that they are unconditional domains where reordering the input does not leak supervision? My major worry is that this also affects the CoT experiments as well as these should also have a conditional structure. Indeed, the fact that Figure 1 shows that left-to-right order dominates the other orders, is probably a consequence of the CoT being left-to-right conditionally generated in the first place. I do not think you address this in the paper, nor do you give an explanation of the data.

Q3). The path-star experiment exposed an issue in methodology. The entire point of this work is to show that order matters under a learned system. The path-star task is famously difficult to learn with the forward arm direction and incredibly easy for the reverse direction. The forward model will almost certainly not learn the task and predict at 1/5 accuracy, whereas the backward model should easily get 100% accuracy (you do not report convergence accuracy, which makes interpreting results very hard). You run this exact experiment and show that there is hardly any difference between the two, despite this being a known thing. The reason for this is that your metric is the likelihood of the entire sequence. Not only is generating the source edge-list very information-low, and hence dominates the overall likelihood, but by conditioning on the true sequence, the difference between the forward and reverse comes down to the entropy of a single token in the 268ish token sequence, and hence is nearly invisible in terms of the reported metric. But this means that the exact phenomenon you are studying is nearly invisible because of how you reported the experiment. So the question is: Figure 1 shows that the best alternative is almost never significantly different from the left-right order. Is this an artifact of not carefully considering the value/metric you reported?

Q4) For natural language, you found that they are more sensitive to local reshuffling than reverse. This makes complete intuitive sense for language, and reverse still preserves relative word order. As far as I can tell, this is only briefly discussed all the way in Appx F on page 17 [lines 914–916] and it does not discuss tokenization issues at all.

Q5) This work is listed under interpretability and explainability; however, it primarily characterized empirical differences between prediction orders, with limited explanation of their causes. Which findings provided explanations beyond identifying which orders achieved lower prediction loss?

Comments: Section 3.2 needs more explanation, given more paper space.

7 comments
AuthorsOpenReview ↗

Revisiting and removing the path-star experiment

Thank you for pointing out this issue. Your analysis has been very helpful to us. Path-star is a conditional task, and a codelength of the joint p(edge list, query, answer) cannot be compared with the codelengths of the collected corpora; p(edge list | answer) is indeed not worth learning. We re-examined the training runs, and your concerns are confirmed.

  1. The model does not learn the task. In the paper's setting (5 arms × 11 edges), no order solves it. We scored the final models teacher-forced on 1,024 held-out graphs (8 seeds, common learning rate 1.5e-4). Left-to-right predicts an interior answer node correctly 2.3% of the time (7.3 bits) and the first node 2.0% of the time (7.5 bits). Tail reversal predicts the first node of each answer edge correctly 11.0% of the time (6.7 bits).

  2. Where the codelength gap comes from. You noted that left-to-right and tail reversal have a longer codelength than full reversal and gf44. We explain this by splitting the prequential codelength by segment (bits per record):

order                  edge list   query   answer    total
left-to-right (id)          983.3    8.87    99.63   1091.8
tail reversal (tr)          986.8    9.35   101.27   1097.4
full reversal (rev)         910.6    1.81   126.55   1039.0
answer-first (gf44)         914.7    1.72   126.59   1043.0

What the models learn is formatting and copying, not search. Under left-to-right, four classes of token reach 100% accuracy: the separators; the first node of each answer edge, a copy of the end of the preceding edge; the start node in the query, the one node that begins five edges; and the last answer node, a copy of the goal in the query. Each order can make one segment cheaper, but the copy it needs differs in difficulty:

  • Full reversal and gf44 make the edge list cheaper. They read the answer first, so the path edges in the edge list can be copied from the answer. The answer is a small pool at fixed positions, and this copy is easy to learn: under full reversal these tokens end at 0.28 bits and 93.5% accuracy, against 8.2 bits at the same positions under left-to-right.
  • Left-to-right and tail reversal make the answer cheaper. They read the edge list first, so each answer node has to be found in the edge list. The edge list is a large pool, and final accuracy is only 2.3% (left-to-right) and 11.0% (tail reversal). The reported advantage of the answer-first orders therefore shows only that copying from a small pool is easier to learn than retrieving from a large one; it has nothing to do with planning or search direction.
  1. Restricting the codelength to the answer does not rescue the comparison. Left-to-right and tail reversal differ in principle only on the choice of arm, which is worth at most log₂ 5 = 2.32 bits, or 0.2% of the record. Under teacher forcing every other answer token is a one-hop lookup from the gold prefix. Those lookups fail in both orders, so the arm choice never becomes the bottleneck. On the answer segment alone the two orders are indistinguishable: 99.6 against 101.3 bits over the pass, and 73.4 against 73.2 bits at the end of training.

For these reasons we have removed path-star from the paper. For multiplication, please see our other response for detailed explanation.

AuthorsOpenReview ↗

Why we retain CoT and arithmetic

Thank you for also raising the concern about supervision leakage in CoT and multiplication. Path-star. The leak you describe arises when an order reads the answer first, so that the model predicts the problem given the answer, P(problem | answer). We agree that on path-star this conditional is of no interest: given the path, predicting the edge list amounts to finding 11 known edges among random ones. This is why we have removed path-star, as you suggested. Chain-of-thought. We have redefined the corpus: every window is drawn from inside the reasoning, so it contains neither the problem nor the answer, and no order reads an answer before predicting a problem. On the new corpus left-to-right is still best (+0.63%). We also agree with your reading: these traces were generated left to right by language models, and the result describes text of this kind. The revision says in the description of the corpus (Section 3.1) that it was generated by autoregressive models, and says so again where the result is reported (Section 4.1). We keep the corpus because large reasoning corpora are at present produced in this way, and models of other kinds that learn to reason are trained on such data too. Multiplication. We now use two-digit by eight-digit multiplication, one problem per window, and keep only problems whose product has a unique factorization. P(problem | answer) is then factoring, a task that we are intrested in rather than an answer leak. We also report the accuracy of each order (appendix): 0.92 when the product is written from its least significant digit, 0.58 from its most significant digit, and 0.05–0.07 for factoring. The other corpora are unconditional sequences. Section 3.1 and the appendix of the revision now describe both corpora.

AuthorsOpenReview ↗

LR ablation

We thank the reviewer for raising this point. We first ran the requested ablation: on all 12 corpora, every order shares one learning rate. Define lr of identity as ll, we have 3 alternatives for lr sweeping: 0.5l0.5l,lland$2l$(8 seeds each). It shows that not all conclusions of the submission were stable. On protein, the submission (each order at its own rate) reported an alternative significantly better than left-to-right; with a shared rate the sign reverses and left-to-right is slightly better, though only by about 0.01%. This led us to re-examine how the learning rates were selected. The selection used a single seed and 20M tokens, on a grid spaced by a factor of two. On protein, some orders were assigned twice the identity rate although, in tuning, that rate was better than the identity rate by only 0.0002%–0.05%. In the 100M-token runs, these two rates change the codelength by 0.12%–0.14%, more than the differences between orders reported in the submission (−0.10% and −0.12%). The protein result of the submission therefore came from the difference in learning rate, not from the order. We have revised our conclusion accordingly. We have added one rule, applied uniformly to every corpus: an order uses the identity learning rate unless its own optimum is better in tuning by more than 0.1%, and the same 0.1% tolerance is used when judging differences between orders. Applied to the existing tuning results, the rule gives all 21 orders on both protein corpora the same rate, 1.5e-4. The value 0.1% is set with reference to the effect of one grid step of the learning rate on protein (it changes the codelength of identity by 0.13%–0.17%): this is the corpus with the smallest differences between orders and the one most exposed to the learning rate, and a difference below this size cannot be separated from the effect of the rate. The rule was adopted after the ablation; it was not specified in advance. Under this rule, no corpus receives opposite verdicts across the three shared rates. The verdicts that change do so only between resolved and unresolved: chain-of-thought and RNA become unresolved at 2×, and arithmetic, unresolved in the main experiment and at 0.5×, has an alternative better at 1× and 2×. The full ablation is in the new appendix "One learning rate for all orders".

Reviewer_jFXqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates how different token prediction orders affect the learning efficiency of a fixed model. The authors propose using prequential codelength to quantify order-learnability and systematically compare 21 fixed prediction orders across 13 corpora. The experiments suggest that left-to-right prediction is not universally optimal, local adjacency is particularly important for text, and the differences between prediction orders evolve throughout training.

Critical strengths

1. Meaningful research question. The paper investigates an important yet insufficiently quantified question: how factorization order affects learning efficiency. Unlike comparisons between different generative model families, this work attempts to isolate the effect of prediction order under a controlled learning protocol.

2. Principled quantitative framework and informative experiments. Applying prequential codelength provides an interpretable way to measure learning efficiency. The synthetic calibration and experiments across diverse corpora support the proposed measurement framework and reveal interesting patterns in order sensitivity.

Critical weaknesses

1. Poor organization and insufficient exposition. The paper frequently introduces specialized terminology and experimental settings before providing sufficient intuition or explanation, making the research difficult to follow. For example, path-star is discussed as a major finding before the task is properly introduced, while the visualization of different prediction orders (Figure 6) is relegated to the appendix. A clearer presentation of the motivation, key concepts, and experimental design would substantially improve readability.

2. Limited robustness of some empirical findings. Most differences between prediction orders are relatively small, and the learning rates are tuned separately for each order. Consequently, some observed differences may partly reflect learning-rate selection rather than factorization order itself, potentially affecting the relative ranking of prediction orders and weakening the reliability of the paper's central empirical conclusions.

Questions for authors

Please check the weakness section.

2 comments
AuthorsOpenReview ↗

Improving readability

Thank you. We agree and have revised the paper accordingly (changes are marked in blue):

  1. Definitions before use. We went through the main text and made sure every term is explained where it first appears. In particular, the abstract and the introduction no longer rely on terms that are defined later (relative gap, excess codelength, global shuffle, collected corpora, and the verdicts), and the criteria behind every verdict are stated in Section 2.3 and at the start of Section 4.1, before any result.
  2. Visualization of the orders. The former Figure 6 has been moved from the appendix into Section 3.2, so that each order can be seen where it is first introduced.
  3. Codelength and learnability. Section 2 now states the full chain of the argument: the learner's codelength is the ideal codelength plus a surcharge; the difference between two orders' codelengths is exactly the difference between their surcharges; and the surcharge is the area between the learning curve and the ideal codelength, which can be smaller only if the learner gets close to the true distribution earlier or ends closer to it. This is what we call learnability.
AuthorsOpenReview ↗

Learning-rate Ablation

We thank the reviewer for raising this point. We first ran the requested ablation: on all 12 corpora, every order shares one learning rate. Define lr of identity as ll, we have 3 alternatives for lr sweeping: 0.5l0.5l,lland$2l$(8 seeds each). It shows that not all conclusions of the submission were stable. On protein, the submission (each order at its own rate) reported an alternative significantly better than left-to-right; with a shared rate the sign reverses and left-to-right is slightly better, though only by about 0.01%. This led us to re-examine how the learning rates were selected. The selection used a single seed and 20M tokens, on a grid spaced by a factor of two. On protein, some orders were assigned twice the identity rate although, in tuning, that rate was better than the identity rate by only 0.0002%–0.05%. In the 100M-token runs, these two rates change the codelength by 0.12%–0.14%, more than the differences between orders reported in the submission (−0.10% and −0.12%). The protein result of the submission therefore came from the difference in learning rate, not from the order. We have revised our conclusion accordingly. We have added one rule, applied uniformly to every corpus: an order uses the identity learning rate unless its own optimum is better in tuning by more than 0.1%, and the same 0.1% tolerance is used when judging differences between orders. Applied to the existing tuning results, the rule gives all 21 orders on both protein corpora the same rate, 1.5e-4. The value 0.1% is set with reference to the effect of one grid step of the learning rate on protein (it changes the codelength of identity by 0.13%–0.17%): this is the corpus with the smallest differences between orders and the one most exposed to the learning rate, and a difference below this size cannot be separated from the effect of the rate. The rule was adopted after the ablation; it was not specified in advance. Under this rule, no corpus receives opposite verdicts across the three shared rates. The verdicts that change do so only between resolved and unresolved: chain-of-thought and RNA become unresolved at 2×, and arithmetic, unresolved in the main experiment and at 0.5×, has an alternative better at 1× and 2×. The full ablation is in the new appendix "One learning rate for all orders".

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear reviewers,

We sincerely thank you all for the insightful and valuable feedback. We have revised the manuscript in response, with all changes marked in blue, and summarize the main points below.

Contribution Summary

We propose order-learnability, measured by prequential codelength, a cumulative quantity that tracks the whole course of learning, to quantify how efficiently a fixed learner learns the same corpus under different prediction orders. Comparing 21 fixed orders across 12 corpora, we find that left-to-right is rarely privileged, that text is far more sensitive to local shuffling than to reversal, and that the gaps between orders change with training.

Strengths Acknowledged by the Reviewers

We are glad the reviewers found that:

  1. We study a meaningful and well-motivated question: how factorization order affects learning efficiency under a controlled protocol (jFXq, 5W3C, AxUM).
  2. Prequential codelength is a principled and interpretable measure (jFXq, AxUM), supported by synthetic calibration and sanity checks (jFXq, 5W3C, AxUM).
  3. Our experiments are broad (jFXq, 5W3C) and the statistical protocol is careful (AxUM).

Summary of Changes

  1. We added the requested learning-rate ablation with one rate shared by all orders. The main conclusions hold; only a ~0.1% difference on protein is now reported as equivalent (Appendix K).
  2. Following Reviewer 5W3C's suggestion, we removed path-star (one of thirteen corpora) and cleaned up the chain-of-thought and arithmetic setups; their conclusions are unchanged (Section 3.1, Appendix I).
  3. We added a mutual-information analysis, a word-level tokenization experiment and a block-reversal control to explain the locality result (Section 4.2, Appendix F).
  4. We added a model-size experiment showing that the main contrast holds across sizes (Appendix L).
  5. We improved readability: terms are defined before use, and the visualization of the orders is moved into the main text (Sections 2 and 3.2).

Thank you again for your contributions to improving our work. We are happy to address any further concerns or queries.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.