The Order Effect and the Order Imprint Are Not the Same Thing
Abstract
Post-training a language model with two capability operators in different orders produces measurably different models, and a growing line of work uses Lie-bracket surrogates to predict which order scores better. We show that this validation target is largely the wrong object. We compose two on-policy distillation operators (math and code teachers) from a shared start over six seeds, and find that one intervention—resetting optimizer state at the phase boundary—pulls apart two quantities the literature treats as one. It collapses the benchmark order effect: ΔJ_M falls from −0.0107 to −0.0038±0.0027, inside our preregistered indistinguishable-from-zero band, and uniform weight averaging of the two single-operator models matches the better order in any case. The same intervention leaves the parameter order imprint u = θ_MC − θ_CM within 4% of unchanged on every seed. An audit of that imprint shows that its "recency" share depends on a free parameter no prior work controls, namely where operator displacements are measured: 2–3% of ‖u‖² at initialization, 12–15% at the phase boundary. The rest survives every null we can build: it replicates across seeds (mean φ = 0.44), an SFT pair on the same data leaves a near-orthogonal imprint, random directions of equal norm are inert, and its geometry replicates on a 4B student. Benchmarks thus register mostly optimizer bookkeeping and miss the part of the order effect that replicates. We release the audit protocol as a measurement standard for composition claims. Code is available at https://anonymous.4open.science/r/pdc-2027.
Then back it, or bet against it.
Related papers
Open the market on this paper to see 7 more related papers.