- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper contends that post-training recipe ablations (skill composition, blocked vs. interleaved, ordering) are only decidable when the reachable spread across recipes exceeds the resolution of re-executing the full protocol, and that the failure mode is non-replicating winners rather than null effects. Measuring both sides for SFT on competition mathematics, it finds the largest single-corpus order contrast (0.062) falls below single-execution resolution (0.081), with the winner's sign reversing on re-run, and that 0.955 of the order-contrast variance at k=4 comes from evaluation sampling rather than training stochasticity. The one premise whose violation clears the wall is coherence: splitting a corpus into halves with incompatible but equally correct answer conventions shifts which convention the model commits to by about two orders of magnitude over a same-convention control, across four scales in two pretraining families; the paper also ships a scorecard grading 30 preregistered claims, 7 of which failed.
Critical strengths
-
The research question is important. The paper asks how a post-training pipeline should keep and improve performance across several skills or domains at once: which skills to train together, whether to train them in blocked stages or interleave them, and in what order. Practitioners face this decision in nearly every multi-domain fine-tuning setup, and the existing literature contradicts itself, reporting that order matters, does not matter, or matters in reverse. The paper treats this as a question about when such decisions can be made at all, rather than adding one more ablation. That framing helps explain why earlier findings disagree and is useful to the community.
-
The paper looks at the problem from an interesting angle. It argues that a recipe effect means little on its own and should be compared against how well the evaluation protocol can tell two recipes apart. The authors re-run the whole training-and-evaluation protocol several times and use nested variance components (seed-level and run-level) to set an explicit threshold for a meaningful change (the resolution δ). They then report the effect relative to that threshold, Δ/δ. This shows that a "winning" recipe can be an artifact of noise: in their grid, the best single-corpus order contrast (0.062) falls below the re-run resolution (0.081), and re-running the winner reverses its sign. The resulting recommendation is practical: ablation studies should report re-run resolution alongside effect size. The variance breakdown also adds a useful finding, that most of the noise comes from evaluation sampling, so drawing more samples per problem is a much cheaper fix than training more seeds.
Critical weaknesses
- Training data is too small. Every wall measurement uses only 622 rows per stage, far below the 10^4–10^7 examples typical of production post-training. The authors themselves note that at this scale, small recipe effects are consistent with both arms being far from saturation, and their experiments cannot rule this out. The central claim may therefore not transfer to realistic post-training.
- The skills are not diverse enough. All skills are sub-areas of competition mathematics (algebra, calculus, combinatorics, etc.), so they likely overlap heavily. Small order and arrangement effects are expected between such closely related skills and say little about genuinely different domains (code, dialogue, instruction following, safety). The "disagreement" conditions are also synthetic formatting conflicts within math (numerals vs. spelled-out numbers, Roman vs. Arabic numerals), not natural conflicts between domains.
- The evaluation set is too small. Each skill is tested on only 120 problems with k=4 and exact match on the final answer. By the authors' own estimate, evaluation sampling accounts for ~95.5% of the order contrast's variance. The reported "wall" may therefore reflect this small, noisy test set rather than a limit of recipe search itself. Larger, standard benchmarks, including non-math tasks to measure retention, are needed.
- The paper is hard to read. It relies heavily on metaphors and coined terms ("free-recipe limit", "patience/coherence/room", "wall", "floors"). The prose is extremely dense, packing many numbers into each sentence, and it leans on the appendix for key definitions and results. Even the main conclusions are hard to pull out, which seriously limits accessibility.
- The main quantitative conclusion is statistically fragile. The headline ratio of 0.76 depends on a run-level variance estimated with only seven degrees of freedom. Its bootstrap 95% CI is [0.25, 1.21], with Pr(ρ>1) ≈ 0.09. As the authors admit, this is "weaker than settled," so the central claim is suggestive rather than established.
Questions for authors
- This paper's motivation is keeping and improving performance across domains, but the evaluation covers only the trained math skills. Did the authors measure how different recipes affect general capabilities outside the training skills?