acceptodds
Desk rejected at ICLR 2027

Position: Pretrained LLMs Should Move from Linear to Graph-Structured Execution

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) typically execute their layers in a fixed linear sequence. Post-training methods modify this structure through skipping, looping, reordering, or parallel execution, offering different capability and resource benefits without training a new LLM from scratch. However, these operations are often studied separately. This position paper argues that research on post-training LLM restructuring should systematically explore how multiple structural operations can be combined to pursue their different benefits within the same LLM. We introduce model execution graphs to realize this position, jointly specifying which pretrained layers execute, how often, and how they are connected. We also develop an evaluation framework that assesses multiple capabilities and resource costs and examines whether composition provides benefits beyond applying the operations separately. Experiments across pretrained LLMs show that combined operations can achieve simultaneous resource savings, but these savings can accompany task-performance losses and do not necessarily confer an advantage over simpler alternatives. The same graph can also affect task performance differently across LLMs. Finally, we outline future research directions for designing and evaluating model execution graphs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language model, model restructuring

Related papers

Discussion (0)

Sign in to comment.