acceptodds
Withdrawn from ICLR 2027

The Free-Recipe Limit: Every Recipe Effect Measures Which Premise Broke

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Post-training pipelines must choose which skills to train together, whether to run one blocked stage per skill or interleave them, and in what order, and a large literature reports that these choices matter. Whether any of them can be decided depends on two quantities that are never reported together: how far apart the best and worst recipes can be, and how finely a re-execution of the whole protocol can tell two recipes apart. Where the second exceeds the first, no quantity of search converts into a decision, and the signature is not an absence of winners but winners that do not survive re-running. We measure both sides for supervised fine-tuning on competition mathematics, pricing the resolution from nested variance components over repeated executions of the protocol rather than from seed spread alone. Across six skill pairs the largest order contrast on any single corpus is 0.062 while one execution resolves only 0.081 — a ratio of 0.76 that is an upper bound, since its numerator is the maximum of a search — and re-running the winner reverses its sign. Most of that noise is cheap to remove: at four samples per problem, evaluation sampling rather than training stochasticity carries 0.955 of the variance of the order contrast itself, so quadrupling the samples costs minutes and beats a fourth training seed costing hours. One departure clears the wall: split a corpus so its halves answer under incompatible but equally correct conventions, and arrangement stops moving capability and starts moving which convention the model commits to, by two orders of magnitude over its same-convention control, across four scales in two pretraining families. Recipe ablations should therefore publish their re-execution resolution beside their effect; a public scorecard grades all 30 preregistered claims, of which 7 failed.

Primary area:
foundation or frontier models, including LLMs
Keywords:
recipe search, curriculum learning, data ordering, supervised fine-tuning, large language models, statistical resolution, evaluation variance, pre-registration, catastrophic forgetting, knowledge conflict

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_nZ2qOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper contends that post-training recipe ablations (skill composition, blocked vs. interleaved, ordering) are only decidable when the reachable spread across recipes exceeds the resolution of re-executing the full protocol, and that the failure mode is non-replicating winners rather than null effects. Measuring both sides for SFT on competition mathematics, it finds the largest single-corpus order contrast (0.062) falls below single-execution resolution (0.081), with the winner's sign reversing on re-run, and that 0.955 of the order-contrast variance at k=4 comes from evaluation sampling rather than training stochasticity. The one premise whose violation clears the wall is coherence: splitting a corpus into halves with incompatible but equally correct answer conventions shifts which convention the model commits to by about two orders of magnitude over a same-convention control, across four scales in two pretraining families; the paper also ships a scorecard grading 30 preregistered claims, 7 of which failed.

Critical strengths
  1. The research question is important. The paper asks how a post-training pipeline should keep and improve performance across several skills or domains at once: which skills to train together, whether to train them in blocked stages or interleave them, and in what order. Practitioners face this decision in nearly every multi-domain fine-tuning setup, and the existing literature contradicts itself, reporting that order matters, does not matter, or matters in reverse. The paper treats this as a question about when such decisions can be made at all, rather than adding one more ablation. That framing helps explain why earlier findings disagree and is useful to the community.

  2. The paper looks at the problem from an interesting angle. It argues that a recipe effect means little on its own and should be compared against how well the evaluation protocol can tell two recipes apart. The authors re-run the whole training-and-evaluation protocol several times and use nested variance components (seed-level and run-level) to set an explicit threshold for a meaningful change (the resolution δ). They then report the effect relative to that threshold, Δ/δ. This shows that a "winning" recipe can be an artifact of noise: in their grid, the best single-corpus order contrast (0.062) falls below the re-run resolution (0.081), and re-running the winner reverses its sign. The resulting recommendation is practical: ablation studies should report re-run resolution alongside effect size. The variance breakdown also adds a useful finding, that most of the noise comes from evaluation sampling, so drawing more samples per problem is a much cheaper fix than training more seeds.

Critical weaknesses
  1. Training data is too small. Every wall measurement uses only 622 rows per stage, far below the 10^4–10^7 examples typical of production post-training. The authors themselves note that at this scale, small recipe effects are consistent with both arms being far from saturation, and their experiments cannot rule this out. The central claim may therefore not transfer to realistic post-training.
  2. The skills are not diverse enough. All skills are sub-areas of competition mathematics (algebra, calculus, combinatorics, etc.), so they likely overlap heavily. Small order and arrangement effects are expected between such closely related skills and say little about genuinely different domains (code, dialogue, instruction following, safety). The "disagreement" conditions are also synthetic formatting conflicts within math (numerals vs. spelled-out numbers, Roman vs. Arabic numerals), not natural conflicts between domains.
  3. The evaluation set is too small. Each skill is tested on only 120 problems with k=4 and exact match on the final answer. By the authors' own estimate, evaluation sampling accounts for ~95.5% of the order contrast's variance. The reported "wall" may therefore reflect this small, noisy test set rather than a limit of recipe search itself. Larger, standard benchmarks, including non-math tasks to measure retention, are needed.
  4. The paper is hard to read. It relies heavily on metaphors and coined terms ("free-recipe limit", "patience/coherence/room", "wall", "floors"). The prose is extremely dense, packing many numbers into each sentence, and it leans on the appendix for key definitions and results. Even the main conclusions are hard to pull out, which seriously limits accessibility.
  5. The main quantitative conclusion is statistically fragile. The headline ratio of 0.76 depends on a run-level variance estimated with only seven degrees of freedom. Its bootstrap 95% CI is [0.25, 1.21], with Pr(ρ>1) ≈ 0.09. As the authors admit, this is "weaker than settled," so the central claim is suggestive rather than established.
Questions for authors
  1. This paper's motivation is keeping and improving performance across domains, but the evaluation covers only the trained math skills. Did the authors measure how different recipes affect general capabilities outside the training skills?
Reviewer_fuddOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors study how training order, arrangement, and composition affect supervised fine-tuning on mathematics tasks. In this paper, they compare observed recipe effects with estimated re-execution variability, investigate convention conflicts using controlled relabellings, and report replication, replay, and capacity controls across several model scales and families. The authors also present an idealised linear account and a ledger of supported, failed, and unresolved claims.

Critical strengths

The authors aim to use useful controls in their experiments, including matched corpus halves, same-convention controls, and alternative answer formats.

Critical weaknesses

The mathematics does not establish the claimed search limit and the abstract further presents a variance decomposition as a finding even though the paper later acknowledges that it is not identified.

Questions for authors

See above.

Reviewer_fpsnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Post-training papers keep reporting that recipe choices matter (which skills to train together, blocked vs interleaved, what order), but the results contradict each other. This paper argues the field never checks two things side by side: how far apart the best and worst recipes actually are, and how finely one re-run of the whole protocol can tell recipes apart. If the second is bigger, searching harder doesn’t help, and the tell is that the “winner” flips when you re-run it. The authors measure both sides for full SFT on competition math (mostly Qwen2.5-7B, 6 skill pairs, 622 rows per stage) using nested variance components over repeated executions. The largest order effect is 0.062 against a re-run resolution of 0.081 (ratio 0.76), and the best cell reverses sign when re-run. They also show that the coherent-domain “interleaving” gain is really revisiting the first skill, i.e. forgetting recovered by replay. The one thing that clears the wall is a constructed convention conflict inside one corpus (numeral vs spelled, Arabic vs Roman): there, arrangement changes which convention the model commits to by 84–139× over a same-convention control, across four scales and two model families.

Critical strengths
  • It probes an important question for this literature. Reporting effect size next to re-execution resolution, with actual re-runs instead of just seed spread, is the comparison that’s been missing. The best cell flipping sign on re-run makes the point concretely.
  • The rigor and honesty are above average. There’s preregistration, a public scorecard of 30 claims (7 failed), and an errata table. They built a replay arm that ended up renaming their own headline result, and they added a separability precondition after the instrument failed on a real-data corpus. You can trust the reported numbers much more than usual.
Critical weaknesses

The scope is much narrower than the framing. Everything is 622 rows per stage, 120 problems per skill, four math skills, and mostly one model, against production post-training. Their own volume result (+0.046 per doubling) means the small spread between recipes could just be under-training rather than recipes not mattering, and they say the design can’t separate the two. The noise budget, which they call the main takeaway (raise k before adding a seed), is also undercut by their own failed claim: evaluation draws shared one seed, and the estimated eval share exceeds 1 in 232/433 cells.

Questions for authors
Reviewer_wm8xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether post-training recipe choices (ordering, blocked vs interleaved, composition) can be resolved at all, by comparing the spread of outcomes a recipe family can reach against the resolution of a re-executed protocol. On SFT of Qwen2.5-7B over four competition-math skills, the largest order contrast is 0.0619 against a stated re-run resolution of 0.081 (ratio 0.76) and flips sign on re-execution. The positive result is a conflict switch: when halves of one corpus answer under incompatible conventions, arrangement moves the committed convention (SHARE) by -0.27 to -0.44 while capability stays flat, and a write-time key removes the effect. A lazy-regime theory scores 5/5 on signs and 0/6 on magnitudes; a pre-registered scorecard of 29 claims plus one graded assumption records 7 failures.

Overall: The conflict switch is real and the reporting is transparent, but the headline negative result rests on a resolution whose estimand is unclear and on an upper-bound claim the proof does not support, and the abstract leads with advice the scorecard grades as unidentified. A revision that fixes the estimand, replaces the bound with an interval, conditions the k advice, and centres on the switch with uncertainty on dSHARE would be acceptable.

Critical strengths

The conflict switch (Sec. 5.2) is a clean within-distribution experiment: random halves of one pool, identical rows and budget, difficulty and domain-distance nulls (Table 9), a write-time key that takes dSHARE from -0.2677 to -0.0008 (Table 2), convention-agnostic re-scoring showing the effect is allocation rather than capability (Table 13), and replication on a disjoint draw and on Qwen3-8B (App. G). This is the result worth keeping.

The paper tests its own alternatives and reports the failures: a replay control that recovers +0.043 against an interleaving gain of +0.038 (Sec. 5.1), volume matching that removes the capacity crossing (Sec. 5.4), a pre-registered scorecard with named failure branches (Table 16) and an errata table (Table 18). That transparency is worth crediting.

Critical weaknesses
  1. The resolution delta_run and the "wall" built on it are not established. Definition 1 sets delta = 2.8 * sigma * sqrt(2/R) with sigma the s.d. of a single arm; the sqrt(2) prices a contrast between two arms. But Table 5's sigma_seed = 0.0332 is defined as the dispersion of ASYM, which is already an order contrast, and the "run mean" 0.0205 built from it is used in Sec. 3 as the s.e. of a three-seed ASYM mean (z = 1.10). delta_run then applies 2.8 * sqrt(2) to that again and gets 0.081. If the estimand is the already-formed contrast tested against zero, the MDE is 2.8 * 0.0205 = 0.057 and the ratio is 1.08, not 0.76; the paper's own delta_seed = 0.054 for the same three-seed design (built from the single-arm 0.0234) is consistent with that reading, and Corollary 1's sigma_run = 0 limit (0.076) is not. The authors need to say which estimand and which replication decision the extra factor prices, and make the numerator match it. Separately, Proposition 3 (App. C) proves that the plug-in ratio is biased upward in expectation and then treats the realized 0.76 as an upper bound; a positive bias does not make one observed value a bound on the truth, and the paper's own interval for the ratio is [0.25, 1.21] (Sec. 5.3). An MDE at 80% power is a design choice, not an information boundary, so "unresolvable at every value of the unidentified component" (Secs. 5.3, 7) overstates what Proposition 1 shows. The sign flip on re-run is still good evidence that the selected effect is mostly noise; the quantitative wall is not.

  2. The two broadest claims are stated unconditionally while the paper withdraws their basis. The abstract says evaluation sampling carries 0.955 of the variance and that quadrupling samples beats a fourth seed; Sec. 5.3, Table 16 and Table 18 record that no driver passes an engine seed, the share exceeds one in 232 of 433 cells, and k = 16 re-scoring left dispersion unchanged. Sec. 7 conditions the advice; the abstract does not. The title promises that every recipe effect measures which premise broke, but the theory is 0/6 on magnitudes and c*rho < gamma fails in 144 of 144 cells (Table 11); the right conclusion is limited empirical applicability of an exact linear-model result, and order is in fact resolvable at the one-epoch budget (Tables 7, 14), so what is shown is budget dependence.

Other issues: dSHARE has no interval (Table 2); the 84x/139x ratios divide by the coherent anchor, not the control the abstract cites (52x/87x against the control); "nineteen of nineteen seeds across four scales" includes 3B at 1.38 floors (Table 9); Table 10 still labels NuminaMath/OpenR1 independent after Sec. 5.2 corrects the lineage; sigma_run rests on 7 df with CI [0, 0.025].

Presentation: The experimental argument can be followed, but it takes real work: the main text is a stream of numbers with definitions spread across sections, five different quantities called "six" (App. J), a 24-row errata table, and "Proof in App. D" for Proposition 1 where there is none.

Questions for authors
  1. [clarification] Is sigma_seed = 0.0332 the per-seed s.d. of ASYM (a contrast) or of a single arm? If a contrast, what does the extra sqrt(2) in delta_run price, and how do you reconcile Corollary 1's sigma_run = 0 limit (0.076) with delta_seed = 0.054?
  2. [clarification] Proposition 3 establishes bias in expectation. What justifies treating the observed 0.76 as an upper bound rather than reporting the [0.25, 1.21] interval?
  3. [clarification] Why do the abstract and Contribution 2 state the 0.955 share and the "raise k" advice without the condition Sec. 7 attaches?
  4. [clarification] Please give an interval for dSHARE in each row of Table 2, and ratios against the actual control.
  5. [suggestion] An independently seeded re-evaluation of fixed checkpoints at k = 4 and k = 16, with a contrast-level covariance estimate, would settle the noise budget (App. H's cheapest cells).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.