Discovering Learning-Friendly Generation Orders for Sequential Computation

Sequential computation via autoregressive generation can make hard tasks learnable, but the generation order of intermediate steps strongly affects whether training succeeds. We address the problem of discovering a learning-friendly target order automatically, rather than designing it manually. Our key observation is that learning-friendly orders often exhibit a faster loss drop in the early stage of training. We exploit this by loss profiling, which ranks candidate orders by the early-stage loss of a single short run. To handle the factorial candidate space, we wrap loss profiling in a hierarchical global–local search over block- and within-block-level orderings. Experiments on benchmark tasks show that, in the best case, our method recovers the optimal forward order for sequences of length $L=13$ from random initialization and $L=40$ from structured initialization, lifting success rates from about 10% to near 100%. It also rediscovers the reverse-digit order for integer multiplication, reported to be efficient in prior studies. A case study in delay dynamical systems shows that learnability varies sharply even among valid topological sorts of the dependency graph. Namely, computational admissibility is not a sufficient condition for learning-friendliness.