acceptodds
Under review as a conference paper at ICLR 2027

Position: Transformer Depth Graphs Should Be First-Class Training Variables

Abstract

Transformer depth can be changed through skipping, looping, reordering, and parallel execution, but studies differ in which graph they train, which pretrained weights they edit, and what resources they allow. An operation name therefore does not identify a comparable experiment. We argue that the complete training graph, including its block calls, dependencies, parameter-sharing identities, merge rules, execution order, path-selection rule, and settings attached to individual block calls, should be treated as a shared experimental object. Claims about this object require graph-aware comparisons that separate whether training on the graph learns the intended computation, what quality its cost buys, how fixed weights respond to an execution edit, and what a multi-operation graph adds over simpler alternatives. In four shared runs of a controlled strict-order task, Serial has lower test error than Parallel-concat in all four but lower error than Looping in only one; the Looping comparison is not matched on estimated arithmetic. In language modeling, a wider shared-looping model has lower validation loss than a narrower serial model at similar parameter counts and matched training tokens and block calls, but width and sharing remain confounded and the looping design takes about 1.56 times as long to train. In the tested checkpoints, fixed-weight loop-count edits increase loss, while training with skipping lowers later skipping damage but misses the full-execution quality criterion. These experiments address task fit, quality and cost, and execution dependence, but none trains a multi-operation graph, so its value beyond simpler graphs remains untested. Together, these cases support a comparison contract that can select a proposed graph or retain a serial or simpler alternative.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Transformer depth graphs, graph priors, nonlinear depth, recurrent computation, conditional computation, parameter sharing, matched-budget evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_AgByOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that Transformer training graphs should be treated as explicit experimental objects and proposes a framework for comparing them across quality, cost, execution changes, and composition.

Critical strengths

1.The paper provides a clear methodological distinction between training-graph specification, weight lifecycle, and fixed-weight execution edits, which is useful for interpreting results in non-serial Transformer architectures. 2.The empirical discussion is careful and transparent about its limitations and confounding factors, avoiding overclaiming beyond what the experiments actually establish.

Critical weaknesses
  1. The comparison contract largely formalizes existing good experimental practices rather than introducing a clearly new methodology.
  2. The paper explicitly states that no experiment trains a multi-operation graph, so its proposed “composition value” remains untested.
  3. The controlled experiments use 16-dimensional synthetic teacher functions and feed-forward students without self-attention, limiting their relevance to realistic Transformer training.
  4. The main looping-vs-serial comparison does not fully control width, parameter sharing, arithmetic cost, or training time, making it difficult to isolate the effect of looping itself.
Questions for authors
  1. In Table 2, Dense-64 and Looping-32x2 have similar parameter counts and equal block calls, but differ substantially in width (1,280 vs. 1,792) and training time (15.76 vs. 24.66 hours). Could the authors provide a compute- or wall-clock-matched comparison to show whether the NLL advantage of Looping-32x2 remains under an equal training-cost budget?
  2. Figure 1 specifies a concrete multi-operation graph that combines skipping, parallel execution, and looping, but this graph is never trained. Could the authors train this exact candidate and compare it against Serial, skipping-only, parallel-only, and looping-only baselines under the same resource allowance to directly test the proposed “composition value”?
2 comments
AuthorsOpenReview ↗

Matched-Time Comparisons and Evidence from Trained Compositions

Q1 / W4: the equal-token advantage does not carry over to our matched-time comparison.

Dense-64 has lower validation negative log-likelihood than Looping-32x2 in the same-host, matched-time comparison. Both retain the original token-based learning-rate schedule: Looping processes fewer tokens and stops before completing its annealing, without time-specific retuning. Section 5.3 and Appendix D.2, Table 8, provide the paired losses, times, tokens, and schedule. This compares complete designs under that recipe, not looping-only causality or separately time-optimized recipes. Historical equal-token and new time-limited studies remain separate because their hardware/software environments differ.

Q2 / W2: training the Figure 1 computation supports retaining a simpler alternative.

We trained it against Serial, its constituent operations, and simpler combinations. The full composition offers no consistent gain over validation-selected simpler graphs in our short-budget settings. This supplies a concrete decision, rather than assuming that combining plausible operations adds value. Section 5.6 and Appendix D.3, Tables 9-10, provide all graphs, the selection rule, and paired results. The branches are scheduled sequentially, so this is a composition-quality test, not a hardware-parallel speedup claim.

W1: an evidence-backed evaluation position, not a new ablation method.

We agree that ablations and cost matching are established practices, not new methods introduced here (Section 2.1). Our contribution is a Transformer-specific evaluation position: the trained execution and weight origin determine which comparison can justify a graph claim (Sections 3-4). For example, reduced selected-path skipping damage coexists with failed full-execution quality requirements (Section 5.5; Appendix C.4, Table 6). Reporting only the improved edit tolerance would miss that reason not to adopt the design under the quality rule.

W3: what the controlled evidence can establish.

The 16-dimensional synthetic teachers specify intended dependencies, allowing us to test which conclusion follows from each named comparison; they do not establish realistic pretraining rankings. With small attention students on strict order, Serial still has lower test error than Parallel-concat, whereas Looping has lower error than Serial. The complete rankings do not all persist (Section 5.2; Appendix D.1, Table 7). Thus the example no longer relies only on feed-forward students, while the separate short-budget language-model studies test time and composition choices on natural-language data.

Thanks the authors for clarification. I remain the original score.

Reviewer_8jR1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that Transformer depth modifications (skipping, looping, reordering and parallel execution) should be specified as a complete training graph rather than by operation name. It introduces a two-level specification, separating the depth graph from the training graph, and a “comparison contract” that asks four questions: task fit, quality and cost, execution dependence, and composition value. The position is illustrated with feed-forward experiments and language-model runs.

Critical strengths

S1: I agree with the authors that matching parameter counts or block calls is not equivalent to matching compute or wall-clock time. This is an important distinction that the paper makes clear.

S2: I appreciate that the authors report both positive and negative experimental results.

S3: The distinction between training a model on a new graph and editing the execution of a pretrained model is useful, especially since results from fixed-weight edits are sometimes used to motivate new architectures.

Critical weaknesses

W1: Composed multi-operation graphs are the main motivation, yet none is trained, and the remaining experiments mostly show confounds introduced by the authors' own designs rather than the contract changing a real decision.

W2: The controlled experiments use small feed-forward networks without self-attention, while the language models are trained for relatively few tokens. I am not convinced that the observed rankings would hold for realistically trained Transformers.

W3: The proposed comparison contract seems expensive to follow in practice. It requires independently training the serial graph, each constituent graph, and simpler alternatives. The paper does not explain how this could be done at the scale of modern language models.

W4: The paper introduces many new terms in dense prose and repeats similar caveats across sections, which makes the core proposal harder to follow than its content requires.

Questions for authors

Q1: Can the authors apply the contract to two or three published looping or skipping papers and show a case where it changes the reported conclusion?

Q2: Can you train at least one multi-operation graph, such as the Figure 1 candidate, against its constituent graphs under a declared allowance?

Q3: Do the controlled-task rankings hold when the students are small Transformers rather than feed-forward networks?

Q4: What does the comparison contract reveal in the controlled tasks that a standard ablation against all alternatives would not?

2 comments
AuthorsOpenReview ↗

New Composition and Attention Results, with Clarified Scope

We trained the Figure 1 block-call computation against Serial, its constituent operations, and all two-operation combinations, using a common token/update allowance within each scale. The full composition fails the declared composition-value rule on quality, supporting a validation-selected simpler graph in these short-budget settings. Section 5.6 and Appendix D.3, Tables 9-10, provide the budgets, full comparisons, selection rule, and paired results. Branches are scheduled sequentially, so this tests composition quality, not hardware-parallel acceleration.

Q3 / W2: Serial still beats Parallel-concat with attention students, but not Looping.

On strict order, attention-based Serial still has lower test error than Parallel-concat, but Looping has lower error than Serial throughout the paired runs. Parallel-mix also changes from losing to Serial to beating it in most pairs. Thus the full rankings do not transfer unchanged, while the need to name the alternative remains. Section 5.2 and Appendix D.1, Table 7, provide the complete results and protocol. This remains synthetic coordinate regression, not evidence of realistic pretraining rankings.

Q1: published evidence supports conditional choices, not a demonstrated reversal.

Our examination of Mixture-of-Recursions, LayerDrop, and Transformer Layers as Painters does not establish a reversal of their own reported conclusions. Their existing controls deserve credit: respectively, matched-compute trained designs, inherited shallower networks versus independently trained small models, and fixed-weight execution edits (Section 2).

A narrower, actionable example comes from Mixture-of-Recursions Table 3: among its fixed-compute rows, minimizing validation loss selects the two-recursion expert/cache configuration; imposing an additional 120M non-embedding-parameter cap selects three recursions. This illustrates how a declared objective changes the model choice, not how the original conclusion was wrong.

Q4 / W3: what the contract adds, and how to keep it practical.

The contract does not reveal measurements unavailable to correctly designed standard ablations. Its contribution is to make the required comparison and permitted inference explicit. In the strict-order example, Serial's lower error than Parallel-concat does not establish lower error than Looping (Section 5.2; Appendix D.1, Table 7). A complete ablation already exposes this distinction; our recommendation prevents the first comparison from being reported as a general serial advantage. Sections 3-4 apply the same discipline to trained quality, fixed-weight edits, and composition.

The revised Introduction and Sections 6-7 recommend a finite, claim-relevant set of alternatives and explicit selection/training cost accounting, not exhaustive training over an unbounded graph space. If necessary comparisons are unaffordable, we would limit the claim to the evaluated setting rather than assert a general graph advantage.

W4: simplifying the presentation.

We reduced the opening's terminology inventory by moving hypothesis/objective/allowance definitions to Section 3.2. The revised Introduction states the recommendation first, and its contribution paragraph now gives the supported lessons instead of repeating the specification and contract. Appendix D carries full protocols behind the main-text summaries. These are concrete reductions in repeated exposition, not a claim that every repetition has been removed.

I would like to thank the authors for their detailed rebuttal. After reading the responses across all reviews, my concerns are only partially solved.

Regarding W1 and Q1/Q2, while I appreciate that the authors trained the Figure 1 multi-operation candidate on small models, the broader concern in W1 remains unresolved: the contract does not change the reported conclusion of any published paper, and the empirical examples still largely reflect confounds in the experimental setup (for example, the new matched-time run cuts off the slower looping model mid-way through an un-retuned learning-rate schedule).

Regarding W2/Q3 and W3/Q4, replacing feed-forward students with small Transformers changes the synthetic rankings, the language models remain heavily undertrained, and as noted in the other reviews, the contract mainly restates standard ablation practice while handling large-scale cost only by narrowing claims.

Thus, I will keep my original scores unchanged.

Reviewer_KGF8OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors of this position paper mainly propose including the training graph as a “shared experimental object.” They define a training graph as a depth graph, the full block dependency structure of a Transformer model + extra information about the blocks: “training semantics” like execution order, path-selection rules, etc., which are defined on top of this depth/dependency graph (although the paper doesn’t provide a crisp line delineating “depth graphs” and “training graphs”). They also propose a “comparison contract” with four questions: task fit, quality vs. cost, sensitivity to execution changes, and the value of composing operations vs. serial baselines. They provide experimental results to back up the first three questions in their proposed “comparison contract.”

Critical strengths
  • At a higher level, the paper does make a good point regarding the need for standardizing comparison across different transformer models that use different operations like looping, reordering, parallel execution etc…
  • The paper makes a reasonable attempt to support its proposed framework with experiments showing how the choice of baseline, resource budget, and evaluation criterion can affect conclusions about a training graph.
Critical weaknesses

Here are the main weaknesses I found in the paper (in no particular order):

  • Proposed framework lacks precision

    • As briefly mentioned in the summary, I find the distinction between “depth graphs” and “training graphs” insufficiently precise. The “training graph” appears to augment the depth graph with additional execution information rather than introduce a separate graph structure (e.g., see the bottom panel in Figure 1). However, one could argue that block dependencies already partly constrain execution order; for instance, the fact that F5 is executed twice is already encoded in the depth graph. A more precise definition would help establish what this distinction adds beyond a fully specified architecture and training procedure.
  • Another major issue with the paper is the lack of clarity in the writing and the disjointed flow of ideas — getting through this paper took up a lot of my reviewing time, even though the ideas explored weren’t particularly complicated.

    • Even from the beginning, I found it difficult to identify the paper’s goal from the abstract alone. At a minimum, the abstract should clearly communicate the central objective, yet several sentences lack the context needed to interpret them. For example:
      • “In four shared runs of a controlled strict-order task, Serial has lower test error than Parallel-concat in all four but lower error than Looping in only one;” — it is not clear what the task is, what is “shared” across the runs, etc.
    • The paper lacks basic writing practices like proper cross-referencing. These issues accumulate and distract the reader from engaging with the ideas the paper is arguing for.
      • For instance, in Section 3, “the looping model below” should have explicitly referenced Section 5.3 (which is three pages away!) and Appendix C.1 so readers could quickly identify the model and its implementation details, but neither reference was provided.
  • Insufficient positioning:

    • The authors don’t clearly situate the paper’s main claim in relation to previous work proposing similar comparison frameworks or explain how their framework relates to or improves on them. This would have helped me better understand the motivation for the paper and the significance of the work.
  • I am not sure whether some of the experiments provide additional insight to support the proposed questions in the “comparison contract.”

    • For instance, Table 2 in Section 5.3 compares a baseline model with a looping model with a similar number of parameters and reports that the looping model achieves lower validation NLL while taking 1.56× as long to train (I assume to illustrate the quality–cost trade-off). However, existing work on looping models already includes comparisons at fixed total training FLOPs (see Mixture-of-Recursions). The authors acknowledge that their own comparison does not match compute, but I am not sure what additional insight this experiment provides beyond checking a box.
Questions for authors

No questions.

2 comments
AuthorsOpenReview ↗

Clarifying Our Position and Its Practical Value

W4: what the comparisons add to a design decision.

Table 2 establishes an observed quality-versus-training-time trade-off at matched tokens, not a looping-only causal benefit. The same-host follow-up tests whether that quality advantage justifies choosing the model when its measured training-time cost is constrained.

Dense-64 has lower validation negative log-likelihood than Looping-32x2 under the matched-time allowance. Both retain the original token-based learning-rate schedule, so Looping processes fewer tokens and stops before completing its annealing; neither recipe was retuned for time. Section 5.3 and Appendix D.2, Table 8, provide the paired results and protocol. This tests elapsed time, complementing rather than replacing fixed-arithmetic comparisons such as Mixture-of-Recursions.

The trained Figure 1 computation likewise offers no consistent gain over validation-selected simpler graphs in our short-budget settings (Section 5.6; Appendix D.3, Tables 9-10). These comparisons yield concrete decisions to retain serial or a simpler trained graph. The original edit and skipping studies explain why edit tolerance also cannot substitute for trained quality (Sections 5.4-5.5; Appendices C.2-C.4). The contribution is therefore a claim-specific route from a favorable measurement to a defensible design choice, not a universal ranking of operations.

W1: a precise specification with a limited requirement.

We agree that repeated calls and parameter sharing, including the two calls to shared F5, are already represented in the depth graph. The training graph completes that structure with path selection and call-specific training settings; it is not another topology. Section 3 now states this directly and points to the looping implementation in Section 5.3 and Appendix C.1. The revised Introduction and Section 6 also make the scope clear: a complete conventional architecture and training description requires no additional reporting format.

W3: how this relates to existing evaluation frameworks.

Section 2.1 now explicitly connects the position to the neural architecture search best-practices framework and The Efficiency Misnomer, and credits Mixture-of-Recursions as a positive matched-compute example. We apply these principles to a Transformer-specific interpretation problem: independently trained graphs, inherited subnetworks, and fixed-weight execution edits are different experimental objects. The contract states which object a claim concerns and which alternatives and resource conditions could change the choice; it does not propose cost matching or ablations as new techniques.

W2: making the argument easier to read.

The revised Introduction gives the recommendation and scope before Section 3's operational definitions. We moved the opening's hypothesis/objective/allowance definitions to Section 3.2 and replaced the contribution paragraph's repeated specification list with the lessons supported by the comparisons. The abstract now states the purpose and findings without the isolated "four shared runs" example. Section 3 also points explicitly to Section 5.3 and Appendix C.1 instead of saying "the looping model below."

Reviewer_KGF8OpenReview ↗

Reply to Authors

Thank you for your response and the effort towards addressing my earlier points.

After reading the rebuttal, I would be keeping my original score. My main concern is how much the proposed framework adds beyond established evaluation practices. Additionally, some of my previous points we only partly addressed (for instance I still find the central argument hard to follow in places). I do however believe that making the suggested changes has made the manuscript better.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.