Familiar Operations, Unseen Structures: How Task Compositions Affect Task Generalization?

Large language models are trained on data from various tasks and can generalize to unseen test tasks by reusing knowledge from training. Different training tasks demonstrate different ways to compose the knowledge pieces, and thus are central in shaping both the knowledge pieces a model acquires and the ways it learns to reuse them. Yet how different compositions in training affect generalization to unseen tasks remains elusive. We thus introduce schema-based data composition, a methodology that constructs structurally diverse tasks from a small library of reusable operations (i.e., primitives) according to predefined composition structures (i.e., schemas). A schema specifies dependencies and output assembly independently of the particular primitives and inputs; executing an instantiated schema provides its reference output. This representation allows us to study how the structural content of training data shapes generalization. Our empirical study yields three key insights. (1) Topology shapes learnability. Schemas of the same size can differ substantially in difficulty, and the tested primitive-sampling policies do not extend the observed direct-training frontier. (2) Structural complementarity enables transfer. Training on complementary forms of depth and branching enables transfer to unseen structures. It can even outperform direct training on those structures with the same prompt budget per run. (3) Source learnability matters. We observe stronger transfer when complementary training schemas continue to provide learning signals. Including intermediate schemas can also help the model begin learning from harder schemas. These findings identify computational structure as a distinct dimension of training-data design for generalization to new tasks.