- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies how to automatically discover a generation order that makes sequential computation tasks easier to learn. The proposed method is based on loss profiling: first briefly train a small Transformer on a mixture of targets written in different orders, then select the orders with the fastest validation loss decrease. Because the number of possible orders grows factorially with sequence length, they combine this with a hierarchical search that first chooses an order over blocks and then an order within each block. On six recurrence tasks where the forward order is known to be best, the method recovers it in the best cases for lengths up to L = 13 when starting from random candidates, and up to L = 40 when starting from structured candidates. It also rediscovers the reverse-digit order for multiplication. Finally, a case study on delay dynamical systems shows that orders consistent with the dependency graph can still differ substantially in how easy they are to learn.
Critical strengths
The observation is useful: the early validation loss of a model trained on a mixture of orders predicts which order will be learnable when trained alone. On RELU and MLP, the profiling ranking correlates well with final accuracy. The signal also appears very early, so ranking orders is cheap.
Critical weaknesses
-
The main search results seem inconsistent with the paper's seed study. Table 3 reports single runs, while the three-seed study (Table 29) gives much lower numbers for many of the same settings. For example, SQUARE and TRIANGLE with at drop from 100% to 2.7% and 3.5%. The paper does not explain this gap, and the results have no multi-seed check.
-
The paper does not show that the method can discover new orders. In the recurrence tasks the best order is the forward causal order, and in multiplication it is the known reverse-digit order. In the delay systems, where the best order is unknown, the profiling ranking barely correlates with final accuracy on two tasks, and global search is worse than the best hand-designed orders on three of five tasks.
Questions for authors
- Could the authors report Table 3 as means over seeds with validation-based selection, and explain why it differs so much from Table 29?