A Recipe Is Not Equivalent to Itself: Item-Level Intervals Overstate Parity Between Fine-Tuning Recipes

Claims that one fine-tuning recipe "matches" another are equivalence claims, and they are usually certified with an interval obtained by resampling test items while the trained models are held fixed. We show that this interval answers a different question from the one the claim asks. It measures how a particular set of trained adapters would fare on new questions, not how the recipe would fare on a new training run, and when training-seed variance is large it is badly anti-conservative. We establish this on a pre-registered study of LLM abstention: six instruction-tuned models (1B–8B), a label-supervised and a label-free abstention recipe, five seeds each, 60 trained adapters. The registered item-level test finds the two recipes equivalent within ±0.05 on five of six models, and keeps finding five of six when the denominator is corrected; an interval over training runs finds equivalence on at most one. Across the twelve recipe–model combinations, hallucination rates vary across seeds 1.4–3.9× more than item sampling explains, so the nominal 90% item-level interval covers the recipe-level difference only 28–78% of the time. Most directly, applied to two seeds of the same recipe, the item-level test calls the recipe significantly different from itself on 84 of 120 seed pairs (about 12 expected) and fails to find it equivalent to itself on 79 of 120. Two seeds of one recipe differ about as much as the two recipes do. We give a simple variance decomposition that predicts when the problem arises, document two scoring defects the audit exposed, and propose a short reporting protocol.