- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper argues that Transformer training graphs should be treated as explicit experimental objects and proposes a framework for comparing them across quality, cost, execution changes, and composition.
Critical strengths
1.The paper provides a clear methodological distinction between training-graph specification, weight lifecycle, and fixed-weight execution edits, which is useful for interpreting results in non-serial Transformer architectures. 2.The empirical discussion is careful and transparent about its limitations and confounding factors, avoiding overclaiming beyond what the experiments actually establish.
Critical weaknesses
- The comparison contract largely formalizes existing good experimental practices rather than introducing a clearly new methodology.
- The paper explicitly states that no experiment trains a multi-operation graph, so its proposed “composition value” remains untested.
- The controlled experiments use 16-dimensional synthetic teacher functions and feed-forward students without self-attention, limiting their relevance to realistic Transformer training.
- The main looping-vs-serial comparison does not fully control width, parameter sharing, arithmetic cost, or training time, making it difficult to isolate the effect of looping itself.
Questions for authors
- In Table 2, Dense-64 and Looping-32x2 have similar parameter counts and equal block calls, but differ substantially in width (1,280 vs. 1,792) and training time (15.76 vs. 24.66 hours). Could the authors provide a compute- or wall-clock-matched comparison to show whether the NLL advantage of Looping-32x2 remains under an equal training-cost budget?
- Figure 1 specifies a concrete multi-operation graph that combines skipping, parallel execution, and looping, but this graph is never trained. Could the authors train this exact candidate and compare it against Serial, skipping-only, parallel-only, and looping-only baselines under the same resource allowance to directly test the proposed “composition value”?
2 comments
Thanks the authors for clarification. I remain the original score.
Matched-Time Comparisons and Evidence from Trained Compositions
Q1 / W4: the equal-token advantage does not carry over to our matched-time comparison.
Dense-64 has lower validation negative log-likelihood than Looping-32x2 in the same-host, matched-time comparison. Both retain the original token-based learning-rate schedule: Looping processes fewer tokens and stops before completing its annealing, without time-specific retuning. Section 5.3 and Appendix D.2, Table 8, provide the paired losses, times, tokens, and schedule. This compares complete designs under that recipe, not looping-only causality or separately time-optimized recipes. Historical equal-token and new time-limited studies remain separate because their hardware/software environments differ.
Q2 / W2: training the Figure 1 computation supports retaining a simpler alternative.
We trained it against Serial, its constituent operations, and simpler combinations. The full composition offers no consistent gain over validation-selected simpler graphs in our short-budget settings. This supplies a concrete decision, rather than assuming that combining plausible operations adds value. Section 5.6 and Appendix D.3, Tables 9-10, provide all graphs, the selection rule, and paired results. The branches are scheduled sequentially, so this is a composition-quality test, not a hardware-parallel speedup claim.
W1: an evidence-backed evaluation position, not a new ablation method.
We agree that ablations and cost matching are established practices, not new methods introduced here (Section 2.1). Our contribution is a Transformer-specific evaluation position: the trained execution and weight origin determine which comparison can justify a graph claim (Sections 3-4). For example, reduced selected-path skipping damage coexists with failed full-execution quality requirements (Section 5.5; Appendix C.4, Table 6). Reporting only the improved edit tolerance would miss that reason not to adopt the design under the quality rule.
W3: what the controlled evidence can establish.
The 16-dimensional synthetic teachers specify intended dependencies, allowing us to test which conclusion follows from each named comparison; they do not establish realistic pretraining rankings. With small attention students on strict order, Serial still has lower test error than Parallel-concat, whereas Looping has lower error than Serial. The complete rankings do not all persist (Section 5.2; Appendix D.1, Table 7). Thus the example no longer relies only on feed-forward students, while the separate short-budget language-model studies test time and composition choices on natural-language data.