- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents Puro-2B, a 2B dense model trained from scratch on 1.4T tokens using blockwise FP8 on a consumer-grade RTX 5090 cluster, based on the Qwen3-1.7B architecture with untied embeddings. The authors state their main contribution as showing that "affordable, open pretraining is practical today", with the deliverable being a low-cost, reproducible pretraining "recipe" for resource-constrained labs, the authors' own term, rather than a new model architecture or algorithm. The cost efficiency comes from four co-designed components. These are RTX 5090 infrastructure, where the authors modify the open-source driver to raise intra-node bandwidth from 32 to 111 GB/s and enable cross-node GPUDirect RDMA, together with blockwise FP8, the MuonH optimizer, and Curriculum Model Averaging, which places higher-quality data later and averages several late checkpoints. The authors also argue that MuonH helps mainly through a better effective learning-rate schedule rather than its hyperball normalization. Across 15 benchmarks uniformly re-run by the authors, Puro-2B averages 57.81, exceeding Qwen2-1.5B by 2.7 points and approaching Qwen2.5-1.5B, at roughly $6.9K under the authors' cost accounting, with a scaled-down $ 4.4K checkpoint already exceeding Qwen2-1.5B. The paper also fits a cost-versus-performance scaling law and reports a post-training case study showing the CMA-over-uniform advantage persists after matched SFT. Data, code, and weights are released.
Critical strengths
1. A genuine, reproducible systems contribution that lowers the hardware barrier to pretraining. RTX 5090 gaming cards lack the high-speed interconnect of data-center cards. There is no NVLink and no direct GPU-to-GPU or node-to-node transfer. The authors modify the open-source driver to raise intra-node bidirectional bandwidth from 32 to 111 GB/s (near the 128 GB/s theoretical limit) and fix the driver to enable cross-node GPUDirect RDMA, and they run blockwise FP8 stably from random init at 73% MFU. This systems work is solid, and combined with a full open release (data, code, weights) it is a useful reference for the community interested in pretraining on cheap hardware.
2. The optimizer analysis offers transferable insight. Rather than stopping at "MuonH works better", the authors run a clean bridging experiment showing MuonH's benefit comes mainly from giving the learning rate a better effective-decay schedule, not from the hyperball normalization itself, and use this to guide LR-schedule design. This explains the mechanism behind an empirical observation, and the insight carries over to other training setups.
Critical weaknesses
1. The core CMA result carries a data-leakage risk that the authors acknowledge but do not fully address. D.4 notes that part of the advantage of CMA [2] "could be attributed to benchmark contamination in the later-stage data" and offers two responses, that it is "discussed in Section B" and that the advantage persists after post-training. However, Section B is a Limitations section without a decontamination check, and the paper does not provide an overlap analysis between the training data and the evaluation sets. In addition, the fact that contamination cannot explain the entire gain does not mean it contributes nothing, and its contribution is not quantified. What the authors perform is within-source deduplication, that is, removing duplicate documents, which differs from removing content that overlaps the evaluation sets. Since CMA places high-quality data later in training, and high-quality corpora have a relatively higher chance of overlapping evaluation sets, this risk is left unquantified. It is therefore unclear how much of the CMA gain reflects genuine capability.
2. The paper does not adequately show that findings obtained by iterating new ideas on this small-scale platform transfer to larger scale. The paper positions itself as a platform that lowers the barrier for the community to do controlled pretraining research, and it uses the UD-versus-CMA case study to argue the platform can support end-to-end studies. To support the claim that a 2B finding informs larger models, a common approach is a scaling ladder that checks whether a conclusion holds in the same direction across several increasing scales. The paper already uses this approach for the FP8 and MuonH [1] compute-saving ablations, each with a five-point ladder from 0.17B to 1.7B, and the data comparison uses a ladder as well, but these ladders all stop at 1.7B. For the core CMA claim and for the "platform informs larger-scale research" claim, no scale-ladder verification is provided, and CMA is evaluated only at a single 2B point. If someone iterates a new method on this recipe, whether the conclusion transfers to a larger model is a question researchers care about, and the paper provides little evidence on this point.
Questions for authors
- Was contamination actually checked? D.4 points to Section B for the contamination analysis, but it is absent there. Could you provide an n-gram overlap decontamination analysis between the late-stage training data and the 15 evaluation sets, and report how much of the CMA-over-uniform gain remains after removing overlaps.
- Scale-transfer evidence for CMA. Your FP8 and MuonH ablations already use five-point ladders from 0.17B to 1.7B, but CMA is evaluated only at a single 2B scale. Could you evaluate CMA across increasing scales, or otherwise provide evidence that the 2B findings transfer to larger models, to support the claim that this platform informs larger-scale research.
- The key missing baseline. Could you shrink a fully-open recipe such as SmolLM3 [3] or OLMo [4] to the same scale as Puro-2B, that is 2B and about 1.4T tokens, and train it on your own RTX 5090 plus FP8 platform under the same cost accounting and evaluation, to show the net increment of your recipe over simply shrinking an existing open recipe. Among the four cost-saving components, MuonH [1] and CMA [2] are not original here, and the main differentiation comes from hardware and systems choices such as the gaming card plus FP8. Without this comparison, how is the novelty of the core "low-cost recipe" contribution supported.
- Can the cost be compared on one fair basis? Could you redo the core Figure-1 comparison with both sides on the same accounting basis, either measured GPU-hours or theoretical estimation, covering Puro-2B and 2 to 3 open-recipe baselines, and report whether the conclusion holds.
- Why can the four savings be presented together? Your own footnote says the four multipliers are not from a single joint experiment, and that the ablation training length differs from production by about 35 times. Could you give a single-basis, end-to-end speedup estimate at the production training length.
- How robust is the cost and performance curve? With 5 points, one fixed parameter, and a fit RMSE of 0.209, how sensitive is the curve to that fixed parameter and to point selection. Is "$4.4K beats Qwen2-1.5B" an interpolation or an extrapolation. Please provide confidence intervals or a leave-one-out check.
References:
[1] Wen et al. Fantastic pretraining optimizers and where to find them II: Hyperball optimization. arXiv preprint arXiv:2606.16899, 2026.
[2] Luo et al. How learning rate decay wastes your best data in curriculum-based LLM pre-training. In The Fourteenth International Conference on Learning Representations (ICLR), 2026.
[3] Bakouch et al. SmolLM3: smol, multilingual, long-context reasoner. 2025.
[4] Walsh et al. 2 OLMo 2 Furious. In Second Conference on Language Modeling (COLM), 2025.
5 comments
Response to Reviewer aF9A [2/2]
Q4: Can the cost be compared on a fair basis?
A4: As discussed in General Response, Q1, we would like to answer from two aspects: (1) price basis; (2) baseline cost accounting.
As detailed in General Response, Q1, we recompute reproduction costs under one unified price basis (Verda and AutoDL), using common commercial GPU rental rates. The common rental rates actually align with our amortizing accounting since we actually compared to market price after our amortizing accounting. After the accounting correction, the cost efficiency remains for RTX 5090, and Puro-2B retains its substantial cost advantage.
Moreover, we would like to emphasize that the cost of baseline models is underestimated, since we use a theoretical compute account with an optimistic 70% MFU. This means that our estimation of the baseline costs is the lower bound of their actual costs. In practice, the cost efficiency of Puro should be even higher compared to the baseline.
These results support the robustness of the cost frontier in Figure 1, and the conclusion still holds.
Q5: Why can the four cost-saving factors be presented together without a joint production-scale ablation?
A5: We agree with the reviewer’s concern and identified this issue as a limitation in the original manuscript.
(1) As explicitly clarified in the footnote of Section 3.2, the four cost-saving factors should not be interpreted as a directly measured end-to-end speedup, since these factors are derived from separate analyses and are intended to illustrate the contributions of different components. However, it is still informative that these cost-saving factors are presented together. For example, the hardware cost-efficiency and the algorithm speedup can be primarily decoupled. Therefore, these estimates allow readers to compare the component-level cost effects. Hence, with our careful footnote, we present this speedup decomposition in this way to provide an illustrative comparison.
(2) The four factors are therefore component-level accounting estimates instead of a measured multiplicative end-to-end speedup. Because the ablations use 20 tokens per parameter while production training uses approximately 700 tokens per parameter, a reliable joint estimate at production scale would require additional matched full-length runs. However, we leave this as an explicit limitation, due to limited compute and time budget.
Q6: Robustness of the Puro Cost Scaling Law and the $4.4K checkpoint.
A6: As discussed in General Response, Q3, our additional sensitivity analyses support the robustness of CMA's cost-efficiency advantage, while the \$4.4K checkpoint is a directly measured result rather than an interpolation or extrapolation.
**First, the \4.4K. Second, the reported 2.40× CMA cost-efficiency is a fit-based extrapolation, whereas the same-budget CMA improvement (57.81 vs. 56.00 at \$6.89K) is directly measured.
For the fitted scaling law, we fix the Phase 1 offset at its measured cost of approximately \$1.84K. Our additional leave-one-out analysis yields cost-efficiency estimates ranging from 1.83× to 2.86×, so the advantage is persistent. Allowing this offset to be fitted instead produces 2.86× cost-efficiency (with only a marginal reduction in fitting RMSE, from 0.209 to 0.192). These results consistently support CMA's substantial cost-efficiency advantage, although the precise multiplier remains sensitive to the fitting configuration.
Dear Reviewer aF9A,
Thank you again for your detailed feedback. As the discussion period nears its end, we would like to briefly summarize how our rebuttal addresses your main concerns:
- Decontamination check & actual retraining (Weakness 1 / Q1): Beyond the 13-gram overlap analysis and GSM8K-excluded evaluation in
General Response [3/n], we have now completed actual retraining after 13-gram benchmark-overlap filtering inGeneral Response [5/n]. Full-budget decontaminated CMA reaches 56.97 Avg15, outperforming Released UD (55.67) by +1.30 points and closely matching Released CMA (57.44, within 0.47 points). A matched 1/16-budget pair on the exact same decontaminated corpus also confirms a +1.19-point CMA improvement (56.18 vs. 54.99, winning on 9 of 12 tasks). - Scale-transfer ladder for CMA (Weakness 2 / Q2): In
Response to Reviewer aF9A [1/2](A2), we extended our CMA ablation to a four-point model ladder (0.60B, 1.09B, 1.70B, and 2.60B, including a scale larger than 2B). CMA improves average downstream performance at three of the four scales, with the largest gain (+1.21 points) at 2.60B. - Unified cost accounting & scaling-law sensitivity (Q3–Q6): In
General Response [1/n]andGeneral Response [3/n], we recomputed Figure 1 under unified commercial rental rates and a 6ND (@ 70% MFU) compute basis, verified scaling-law robustness via leave-one-out analysis (1.83×–2.86×) and offset fitting (2.86×), and clarified the scope of our component-level speedup estimates.
Please let us know if any questions remain. If these updates resolve your concerns, we would be grateful if you would consider raising your soundness and overall ratings.
Response to Reviewer aF9A [1/2]
We thank the reviewer for recognizing the value of the Puro recipe, our open-source effort, optimizer analysis, systems work, and the paper’s presentation. We sincerely thank you for your effort and thoughtful feedback on our work.
Q1: Check benchmark contamination and quantify CMA improvement after decontamination.
A1: As detailed in
General Response, Q4, we have added the missing overlap analysis and expanded the limitations discussion in the revised manuscript. Specifically, we have conducted a 13-gram overlap analysis on the late-stage training data against the 15 evaluation benchmarks. We further propose a conservative analysis to exclude the most overlapped GSM8K task and reassess the conclusions. The results suggest that the overlaps are unlikely to undermine or take over the CMA advantage. Following your suggestion, we have now completed actual retraining after 13-gram benchmark-overlap filtering and reported these results (SeeGeneral Response [5/n]for full details). After decontamination and retraining, the efficiency gain of CMA is persistent, and the final performance is close to our primary results.Q2: Scale-transfer evidence for CMA.
A2: Thank you for this suggestion. We have extended the CMA study to a four-point ladder spanning 0.60B, 1.09B, 1.70B, and 2.60B parameters, including a model larger than the original 2B setting.
Each UD–CMA pair follows an approximately 1× Chinchilla setup (20 tokens per parameter). CMA uses the scaled-down production recipe: curriculum ordering, a constant-learning-rate tail beginning at 97% of training, and six-checkpoint parameter averaging over the final 0.3%.
Our FP8 and MuonH ladders quantify compute efficiency using validation loss. But the terminal learning does not match between CMA and UD, which can weaken the effect of validation loss as an indicator. We therefore use a downstream evaluation of the CMA ablation ladder. And for these smaller models and shorter training budgets, some benchmarks (MMLU-Pro, BBH, etc.) are too hard. We evaluate knowledge, commonsense, and mathematical reasoning using MMLU, ARC-C, ARC-E, CSQA, PIQA, SIQA, WinoGrande, and GSM8K, following OLMES [1]. We use the same eight tasks and their unweighted arithmetic mean throughout the ladder.
CMA improves the average at three of the four scales, with the largest observed gain at 2.60B, while the 1.70B result is slightly lower.
These experiments provide evidence of benefits at multiple scales, including beyond 2B, and a positive fitted compute-efficiency shift over the tested ladder. This ladder analysis complements the efficiency comparison of CMA by extending the Phase 2 study to multiple model scales.
Q3: A full-scale comparison with a scaled-down existing open recipe.
A3: Thank you for this suggestion. We first clarify what Puro contributes.
The novelty of Puro-2B lies in providing and validating an affordable, fully reproducible end-to-end pretraining recipe. This includes our RTX 5090 systems optimizations (
Appendices D.1.2–D.1.3), the effective learning-rate analysis and schedule design for MuonH (Section 2.2andAppendix I), and the multi-source, block-wise adaptation of CMA (Section 2.3andAppendix D.4). We support these choices with ablations and a completed 1.4T-token training run.Then we would like to clarify that adapting an existing open recipe to a new scale and hardware regime is not a direct baseline. It requires choices about depth, width, data mixture, optimizer, learning-rate schedule, parallelism, and numerical stability. These choices require experimentation and validation. Making these choices work together under a limited budget is an important part of Puro’s contribution. Hence, a fair comparison requires more than one additional training run: it also requires tuning and validating the adapted recipe. Without this work, a poor result could reflect insufficient adaptation rather than a weakness of the original recipe.
Moreover, our system measurements and component ablations already provide evidence for specific component benefits of Puro. The hardware adaptation and low-precision efficiency, as well as the MuonH and CMA recipe adaptation, are supported by substantial ablation experiments. The value of each recipe component is validated in our results. In addition, it is challenging to complete both adaptation experiments and another 1.4T-token run within the rebuttal’s time and compute budget.
[1] Gu et al. OLMES: A Standard for Language Model Evaluations. (ACL), 2025.