Same Fit, Different Functions: Evidence Structure and Training Order Shape Held-Out Behavior
Abstract
Training fit alone does not determine behavior away from observed examples. We study this in controlled contradictory-supervision tasks where compared training runs reach matched or saturated training fit but can express different held-out predictive mappings. In Study I, using Qwen2.5-0.5B and a fixed budget of 200 exceptions, increasing the fraction of exceptions aligned with a shared secondary relation raises mean held-out transfer from 0.0808 to 1.0000 across five coherence levels, strictly in all three seeds. A second modular-composition family reproduces the positive fully coherent versus fully idiosyncratic endpoint contrast in three fresh seeds, but not the graded response. In Study II, after first establishing a primary predictor, all 21 Phase-2 arms fit their training examples, yet 200 fully coherent corrections yield complete primary retention and secondary transfer in every seed, whereas 100 coherent corrections produce highly variable transfer. Finally, in an exploratory fixed-source intervention, six schedules with identical correction counts at every optimizer step and matched total exposure yield terminal transfer from 0.1775 to 0.9900; selected extreme schedules replay exactly. Thus, in these controlled tasks, evidence organization and training order can alter which held-out predictive mapping emerges despite matched terminal fit. The results are behavioral and establish neither an internal mechanism nor a universal or cross-model law.
Then back it, or bet against it.
Related papers
Open the market on this paper to see 7 more related papers.