acceptodds
Under review as a conference paper at ICLR 2027

Puro-2B: Poor Lab’s Qwen2-1.5B Trained on RTX 5090 within $5090

15 views

Abstract

Language model pretraining has become almost synonymous with prohibitive cost, placing it out of reach for much of the academic and open-source communities. Although strong open-source efforts already exist, including open-weight models and open-source training recipes, a cost-efficient, hardware-accessible, and open-source pretraining recipe has long been missing. Even at a small scale, training Llama-3.2-3B costs over 700K. In this report, we present an open pretraining recipe designed to lower this barrier. Using this recipe, we train a collection of Puro-2B models from scratch on up to 1.4 trillion tokens with FP8 precision on consumer-grade RTX 5090 GPUs. The models differ in token budgets and selected recipe variants. Our best model is trained for less than 4.4K, less than $5,090, is sufficient to reach the performance of Qwen2-1.5B. Second, our end-to-end case study shows that the curriculum-based pretraining advantage persists under matched supervised fine-tuning, with higher performance on mathematical and general-purpose tasks. Access to the full pretraining pipeline rather than model weights alone enables such controlled studies. We release the full Puro-2B training recipe, including data, code, and model weights.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Pretraining, Fully Open-Source, Cost-Efficient

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_aF9AOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Puro-2B, a 2B dense model trained from scratch on 1.4T tokens using blockwise FP8 on a consumer-grade RTX 5090 cluster, based on the Qwen3-1.7B architecture with untied embeddings. The authors state their main contribution as showing that "affordable, open pretraining is practical today", with the deliverable being a low-cost, reproducible pretraining "recipe" for resource-constrained labs, the authors' own term, rather than a new model architecture or algorithm. The cost efficiency comes from four co-designed components. These are RTX 5090 infrastructure, where the authors modify the open-source driver to raise intra-node bandwidth from 32 to 111 GB/s and enable cross-node GPUDirect RDMA, together with blockwise FP8, the MuonH optimizer, and Curriculum Model Averaging, which places higher-quality data later and averages several late checkpoints. The authors also argue that MuonH helps mainly through a better effective learning-rate schedule rather than its hyperball normalization. Across 15 benchmarks uniformly re-run by the authors, Puro-2B averages 57.81, exceeding Qwen2-1.5B by 2.7 points and approaching Qwen2.5-1.5B, at roughly $6.9K under the authors' cost accounting, with a scaled-down $ 4.4K checkpoint already exceeding Qwen2-1.5B. The paper also fits a cost-versus-performance scaling law and reports a post-training case study showing the CMA-over-uniform advantage persists after matched SFT. Data, code, and weights are released.

Critical strengths

1. A genuine, reproducible systems contribution that lowers the hardware barrier to pretraining. RTX 5090 gaming cards lack the high-speed interconnect of data-center cards. There is no NVLink and no direct GPU-to-GPU or node-to-node transfer. The authors modify the open-source driver to raise intra-node bidirectional bandwidth from 32 to 111 GB/s (near the 128 GB/s theoretical limit) and fix the driver to enable cross-node GPUDirect RDMA, and they run blockwise FP8 stably from random init at 73% MFU. This systems work is solid, and combined with a full open release (data, code, weights) it is a useful reference for the community interested in pretraining on cheap hardware.

2. The optimizer analysis offers transferable insight. Rather than stopping at "MuonH works better", the authors run a clean bridging experiment showing MuonH's benefit comes mainly from giving the learning rate a better effective-decay schedule, not from the hyperball normalization itself, and use this to guide LR-schedule design. This explains the mechanism behind an empirical observation, and the insight carries over to other training setups.

Critical weaknesses

1. The core CMA result carries a data-leakage risk that the authors acknowledge but do not fully address. D.4 notes that part of the advantage of CMA [2] "could be attributed to benchmark contamination in the later-stage data" and offers two responses, that it is "discussed in Section B" and that the advantage persists after post-training. However, Section B is a Limitations section without a decontamination check, and the paper does not provide an overlap analysis between the training data and the evaluation sets. In addition, the fact that contamination cannot explain the entire gain does not mean it contributes nothing, and its contribution is not quantified. What the authors perform is within-source deduplication, that is, removing duplicate documents, which differs from removing content that overlaps the evaluation sets. Since CMA places high-quality data later in training, and high-quality corpora have a relatively higher chance of overlapping evaluation sets, this risk is left unquantified. It is therefore unclear how much of the CMA gain reflects genuine capability.

2. The paper does not adequately show that findings obtained by iterating new ideas on this small-scale platform transfer to larger scale. The paper positions itself as a platform that lowers the barrier for the community to do controlled pretraining research, and it uses the UD-versus-CMA case study to argue the platform can support end-to-end studies. To support the claim that a 2B finding informs larger models, a common approach is a scaling ladder that checks whether a conclusion holds in the same direction across several increasing scales. The paper already uses this approach for the FP8 and MuonH [1] compute-saving ablations, each with a five-point ladder from 0.17B to 1.7B, and the data comparison uses a ladder as well, but these ladders all stop at 1.7B. For the core CMA claim and for the "platform informs larger-scale research" claim, no scale-ladder verification is provided, and CMA is evaluated only at a single 2B point. If someone iterates a new method on this recipe, whether the conclusion transfers to a larger model is a question researchers care about, and the paper provides little evidence on this point.

Questions for authors
  1. Was contamination actually checked? D.4 points to Section B for the contamination analysis, but it is absent there. Could you provide an n-gram overlap decontamination analysis between the late-stage training data and the 15 evaluation sets, and report how much of the CMA-over-uniform gain remains after removing overlaps.
  2. Scale-transfer evidence for CMA. Your FP8 and MuonH ablations already use five-point ladders from 0.17B to 1.7B, but CMA is evaluated only at a single 2B scale. Could you evaluate CMA across increasing scales, or otherwise provide evidence that the 2B findings transfer to larger models, to support the claim that this platform informs larger-scale research.
  3. The key missing baseline. Could you shrink a fully-open recipe such as SmolLM3 [3] or OLMo [4] to the same scale as Puro-2B, that is 2B and about 1.4T tokens, and train it on your own RTX 5090 plus FP8 platform under the same cost accounting and evaluation, to show the net increment of your recipe over simply shrinking an existing open recipe. Among the four cost-saving components, MuonH [1] and CMA [2] are not original here, and the main differentiation comes from hardware and systems choices such as the gaming card plus FP8. Without this comparison, how is the novelty of the core "low-cost recipe" contribution supported.
  4. Can the cost be compared on one fair basis? Could you redo the core Figure-1 comparison with both sides on the same accounting basis, either measured GPU-hours or theoretical estimation, covering Puro-2B and 2 to 3 open-recipe baselines, and report whether the conclusion holds.
  5. Why can the four savings be presented together? Your own footnote says the four multipliers are not from a single joint experiment, and that the ablation training length differs from production by about 35 times. Could you give a single-basis, end-to-end speedup estimate at the production training length.
  6. How robust is the cost and performance curve? With 5 points, one fixed parameter, and a fit RMSE of 0.209, how sensitive is the curve to that fixed parameter and to point selection. Is "$4.4K beats Qwen2-1.5B" an interpolation or an extrapolation. Please provide confidence intervals or a leave-one-out check.

References:

[1] Wen et al. Fantastic pretraining optimizers and where to find them II: Hyperball optimization. arXiv preprint arXiv:2606.16899, 2026.

[2] Luo et al. How learning rate decay wastes your best data in curriculum-based LLM pre-training. In The Fourteenth International Conference on Learning Representations (ICLR), 2026.

[3] Bakouch et al. SmolLM3: smol, multilingual, long-context reasoner. 2025.

[4] Walsh et al. 2 OLMo 2 Furious. In Second Conference on Language Modeling (COLM), 2025.

5 comments
AuthorsOpenReview ↗

Response to Reviewer aF9A [1/2]

We thank the reviewer for recognizing the value of the Puro recipe, our open-source effort, optimizer analysis, systems work, and the paper’s presentation. We sincerely thank you for your effort and thoughtful feedback on our work.

Q1: Check benchmark contamination and quantify CMA improvement after decontamination.

A1: As detailed in General Response, Q4, we have added the missing overlap analysis and expanded the limitations discussion in the revised manuscript. Specifically, we have conducted a 13-gram overlap analysis on the late-stage training data against the 15 evaluation benchmarks. We further propose a conservative analysis to exclude the most overlapped GSM8K task and reassess the conclusions. The results suggest that the overlaps are unlikely to undermine or take over the CMA advantage. Following your suggestion, we have now completed actual retraining after 13-gram benchmark-overlap filtering and reported these results (See General Response [5/n] for full details). After decontamination and retraining, the efficiency gain of CMA is persistent, and the final performance is close to our primary results.

Q2: Scale-transfer evidence for CMA.

A2: Thank you for this suggestion. We have extended the CMA study to a four-point ladder spanning 0.60B, 1.09B, 1.70B, and 2.60B parameters, including a model larger than the original 2B setting.

Each UD–CMA pair follows an approximately 1× Chinchilla setup (20 tokens per parameter). CMA uses the scaled-down production recipe: curriculum ordering, a constant-learning-rate tail beginning at 97% of training, and six-checkpoint parameter averaging over the final 0.3%.

Our FP8 and MuonH ladders quantify compute efficiency using validation loss. But the terminal learning does not match between CMA and UD, which can weaken the effect of validation loss as an indicator. We therefore use a downstream evaluation of the CMA ablation ladder. And for these smaller models and shorter training budgets, some benchmarks (MMLU-Pro, BBH, etc.) are too hard. We evaluate knowledge, commonsense, and mathematical reasoning using MMLU, ARC-C, ARC-E, CSQA, PIQA, SIQA, WinoGrande, and GSM8K, following OLMES [1]. We use the same eight tasks and their unweighted arithmetic mean throughout the ladder.

Model sizeTraining tokensUD AvgCMA AvgCMA−UD
0.60B11.325B43.64443.845+0.201
1.09B21.391B47.96148.861+0.901
1.70B34.393B51.46351.373−0.090
2.60B51.800B55.56256.775+1.213

CMA improves the average at three of the four scales, with the largest observed gain at 2.60B, while the 1.70B result is slightly lower.

These experiments provide evidence of benefits at multiple scales, including beyond 2B, and a positive fitted compute-efficiency shift over the tested ladder. This ladder analysis complements the efficiency comparison of CMA by extending the Phase 2 study to multiple model scales.

Q3: A full-scale comparison with a scaled-down existing open recipe.

A3: Thank you for this suggestion. We first clarify what Puro contributes.

The novelty of Puro-2B lies in providing and validating an affordable, fully reproducible end-to-end pretraining recipe. This includes our RTX 5090 systems optimizations (Appendices D.1.2–D.1.3), the effective learning-rate analysis and schedule design for MuonH (Section 2.2 and Appendix I), and the multi-source, block-wise adaptation of CMA (Section 2.3 and Appendix D.4). We support these choices with ablations and a completed 1.4T-token training run.

Then we would like to clarify that adapting an existing open recipe to a new scale and hardware regime is not a direct baseline. It requires choices about depth, width, data mixture, optimizer, learning-rate schedule, parallelism, and numerical stability. These choices require experimentation and validation. Making these choices work together under a limited budget is an important part of Puro’s contribution. Hence, a fair comparison requires more than one additional training run: it also requires tuning and validating the adapted recipe. Without this work, a poor result could reflect insufficient adaptation rather than a weakness of the original recipe.

Moreover, our system measurements and component ablations already provide evidence for specific component benefits of Puro. The hardware adaptation and low-precision efficiency, as well as the MuonH and CMA recipe adaptation, are supported by substantial ablation experiments. The value of each recipe component is validated in our results. In addition, it is challenging to complete both adaptation experiments and another 1.4T-token run within the rebuttal’s time and compute budget.

[1] Gu et al. OLMES: A Standard for Language Model Evaluations. (ACL), 2025.

AuthorsOpenReview ↗

Response to Reviewer aF9A [2/2]

Q4: Can the cost be compared on a fair basis?

A4: As discussed in General Response, Q1, we would like to answer from two aspects: (1) price basis; (2) baseline cost accounting.

As detailed in General Response, Q1, we recompute reproduction costs under one unified price basis (Verda and AutoDL), using common commercial GPU rental rates. The common rental rates actually align with our amortizing accounting since we actually compared to market price after our amortizing accounting. After the accounting correction, the cost efficiency remains for RTX 5090, and Puro-2B retains its substantial cost advantage.

Moreover, we would like to emphasize that the cost of baseline models is underestimated, since we use a theoretical 6ND6ND compute account with an optimistic 70% MFU. This means that our estimation of the baseline costs is the lower bound of their actual costs. In practice, the cost efficiency of Puro should be even higher compared to the baseline.

These results support the robustness of the cost frontier in Figure 1, and the conclusion still holds.

Q5: Why can the four cost-saving factors be presented together without a joint production-scale ablation?

A5: We agree with the reviewer’s concern and identified this issue as a limitation in the original manuscript.

(1) As explicitly clarified in the footnote of Section 3.2, the four cost-saving factors should not be interpreted as a directly measured end-to-end speedup, since these factors are derived from separate analyses and are intended to illustrate the contributions of different components. However, it is still informative that these cost-saving factors are presented together. For example, the hardware cost-efficiency and the algorithm speedup can be primarily decoupled. Therefore, these estimates allow readers to compare the component-level cost effects. Hence, with our careful footnote, we present this speedup decomposition in this way to provide an illustrative comparison.

(2) The four factors are therefore component-level accounting estimates instead of a measured multiplicative end-to-end speedup. Because the ablations use 20 tokens per parameter while production training uses approximately 700 tokens per parameter, a reliable joint estimate at production scale would require additional matched full-length runs. However, we leave this as an explicit limitation, due to limited compute and time budget.

Q6: Robustness of the Puro Cost Scaling Law and the $4.4K checkpoint.

A6: As discussed in General Response, Q3, our additional sensitivity analyses support the robustness of CMA's cost-efficiency advantage, while the \$4.4K checkpoint is a directly measured result rather than an interpolation or extrapolation.

**First, the \4.4Kcheckpointwasobtainedfromanactualtrainingrun,achievinganAvg15scoreof55.54comparedwith55.14forQwen2−1.5B.∗∗Andasexplainedin‘GeneralResponse,Q3‘,thescalinglawinterpolationcanserveasasupportingresultfortheclaimthatthePurorecipematchesQwen2−1.5Bfor4.4K checkpoint was obtained from an actual training run, achieving an Avg15 score of 55.54 compared with 55.14 for Qwen2-1.5B.** And as explained in `General Response, Q3`, the scaling law interpolation can serve as a supporting result for the claim that the Puro recipe matches Qwen2-1.5B for \\4.4K. Second, the reported 2.40× CMA cost-efficiency is a fit-based extrapolation, whereas the same-budget CMA improvement (57.81 vs. 56.00 at \$6.89K) is directly measured.

For the fitted scaling law, we fix the Phase 1 offset at its measured cost of approximately \$1.84K. Our additional leave-one-out analysis yields cost-efficiency estimates ranging from 1.83× to 2.86×, so the advantage is persistent. Allowing this offset to be fitted instead produces 2.86× cost-efficiency (with only a marginal reduction in fitting RMSE, from 0.209 to 0.192). These results consistently support CMA's substantial cost-efficiency advantage, although the precise multiplier remains sensitive to the fitting configuration.

Dear Reviewer aF9A,

Thank you again for your detailed feedback. As the discussion period nears its end, we would like to briefly summarize how our rebuttal addresses your main concerns:

  1. Decontamination check & actual retraining (Weakness 1 / Q1): Beyond the 13-gram overlap analysis and GSM8K-excluded evaluation in General Response [3/n], we have now completed actual retraining after 13-gram benchmark-overlap filtering in General Response [5/n]. Full-budget decontaminated CMA reaches 56.97 Avg15, outperforming Released UD (55.67) by +1.30 points and closely matching Released CMA (57.44, within 0.47 points). A matched 1/16-budget pair on the exact same decontaminated corpus also confirms a +1.19-point CMA improvement (56.18 vs. 54.99, winning on 9 of 12 tasks).
  2. Scale-transfer ladder for CMA (Weakness 2 / Q2): In Response to Reviewer aF9A [1/2] (A2), we extended our CMA ablation to a four-point model ladder (0.60B, 1.09B, 1.70B, and 2.60B, including a scale larger than 2B). CMA improves average downstream performance at three of the four scales, with the largest gain (+1.21 points) at 2.60B.
  3. Unified cost accounting & scaling-law sensitivity (Q3–Q6): In General Response [1/n] and General Response [3/n], we recomputed Figure 1 under unified commercial rental rates and a 6ND (@ 70% MFU) compute basis, verified scaling-law robustness via leave-one-out analysis (1.83×–2.86×) and offset fitting (2.86×), and clarified the scope of our component-level speedup estimates.

Please let us know if any questions remain. If these updates resolve your concerns, we would be grateful if you would consider raising your soundness and overall ratings.

Reviewer_wLviOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The recipe trains a 2B-total / 1.4B-non-embedding dense model from scratch on up to 1.4T tokens, entirely on consumer RTX 5090 GPUs

  • infrastructure work to make 5090s trainable at scale. patching the open-source driver to enable PCIe P2P and the CUDA user-space driver to enable GPUDirect RDMA, topology-aware PP/DP layout avoiding tensor parallelism, and a reported 73% effective MFU on 24 GPUs across 3 nodes
  • blockwise FP8 training from random init following DeepSeek-V3 (1.34× net speedup after the fitted quality penalty, κ=0.98)
  • the MuonH hyperball optimizer, plus the paper's most original observation. it's a Muon variant that merely matches MuonH's effective learning-rate (ELR) schedule, without any normalization or projection, nearly reproduces MuonH's loss curve, suggesting the hyperball benefit is mostly an ELR-schedule effect
  • a data recipe selected by proxy experiments (2,000-step runs from a Qwen3-0.6B checkpoint over candidate slices), and a multi-source extension of Curriculum Model Averaging (CMA) in Phase 2: within-source quality-ascending ordering with a stable source mixture, LR decay, then a constant-LR continuation with six-checkpoint averaging.
Critical strengths
  • data composition, training and evaluation code, intermediate checkpoints, the checkpoint ledger for the scaling runs, proxy-experiment results, and an anonymous repository are all provided. Most "open" releases stop at weights
  • Muon-ELR diagnostic is a genuine scientific finding. MuonH beats Muon; instead of stopping there, the authors construct Muon-ELR, plain Muon rescaled step-by-step to match MuonH's effective LR trajectory, with no hyperball projection and no update normalization. It nearly reproduces MuonH's convergence across ELR schedules and weight-decay settings. That reframes a concurrent optimizer's mechanism
  • 5090 systems work is concrete and measured, not aspirational. PCIe P2P enabled via driver work (32→111 GB/s bidirectional, latency 14.3µs→0.4µs), GDR enabled inter-node (AllReduce busbw 8.87→19.93 GB/s on 24 GPUs), topology-aware rank mapping, LM-head-aware pipeline balancing, memory-aware optimizer placement to fit training state in 32 GB — and a disclosed MFU definition (238 TFLOP/s/GPU median in Phase 1 against a precision-weighted effective peak of ~327 TFLOP/s
  • candor in the accounting footnotes and limitations. The paper states plainly that the component factors in Fig. 2a should be treated as illustrative transfers rather than measurements of a single sequential end-to-end speedup, that the ablations are not a joint factorial design and were run at 20 tokens/parameter while production is 700, that the regime is overtrained and not compute-optimal, and that the cost law is a fixed-2B scale-down curve, not a scale-up law
Critical weaknesses
  • The price comparison isn't apples-to-apples. Their own cost uses $0.31 per GPU-hour for the 5090 — but that's not a price anyone can pay. There's no public rental market for 5090s, so they took the cost of owning their machines, spread over five years. the $6.9K counts only the one successful final run — not the experiments, failed runs, and data work that got them there

  • The wins are smaller than the error bars nobody reported. The $4.4K model beats Qwen2-1.5B by 0.4 points on a 15-benchmark average. A single benchmark like GSM8K wobbles by more than a point on its own just from sampling. With no error bars and one training run per number, the honest claim is matches Qwen2-1.5B at $4.4K, which is still a good result

  • The 2.40× comes from extending a curve, not from a run. at the same $6.9K budget, the curriculum recipe does beat plain training in a real run, 57.8 vs. 56.0. But turning that gap into 2.40× more cost-efficient means trusting a two-parameter curve fitted to five single runs and reading it beyond where the data ends.

  • The ingredient tests were run in a different kitchen. The speedups credited to FP8 and to the optimizer were measured on small runs at 20 tokens per parameter. The real model trains at 700. Effects measured in the small regime don't automatically carry over, and the paper's own footnote admits the factors were each measured against a different baseline and can't simply be multiplied together

  • most of the recipe is other people's ingredients. The optimizer is Wen et al.'s, the curriculum-and-averaging is Luo et al.'s, the FP8 scheme is DeepSeek-V3's, the architecture is Qwen3's, the data is public. The genuinely new parts are the effective-LR analysis, the multi-source extension, the 5090 systems work, and the integration itself. That's a legitimate contribution for a recipe paper — but a "co-designed system" claim should come with at least one result showing the combination does something the parts don't

Questions for authors
  • RTX 5090 price ($0.31/GPU-hour) is an amortized cost of owning your own hardware, while every model you compare against is priced at market rental rates. What would PURO-2B's total cost be under a market-style 5090 rate comparable to the rental rates in Table 5? And which costs are left out of the $6.9K figure — for example, the proxy data experiments, the ablation runs, any failed or restarted training, and data processing?

  • $4.4K checkpoint's lead over Qwen2-1.5B is 0.4 points on the 15-benchmark average (55.5 vs. 55.1), and no error bars are reported anywhere in Table 2. How large is the noise on that average — from evaluation sampling, or from retraining with a different seed? Is 0.4 points bigger than that noise?

  • table 19/20 do include a Uniform–Decay run at the same full budget as CMA (56.00 vs. 57.81 at $6.9K), so the same-budget comparison is measured. But the headline "2.40× cost efficiency" comes from projecting your fitted cost curve beyond the measured points. How sensitive is that 2.40× to the fit — for example, if the offset C_P1 is fitted rather than fixed? Separately, Appendix K.2 notes that part of the curriculum ordering's advantage could be benchmark contamination in the late-stage data. Can you report a decontamination check of that data against the 15 benchmarks

4 comments
AuthorsOpenReview ↗

Response to Reviewer wLvi

We thank the reviewer for recognizing the value of our open-source effort, optimizer analysis, systems work, and empirical documentation. Your suggestions and feedback are important for us to clarify our contributions and make them more convincing.

Q1: The RTX 5090 price is not comparable with baselines, and the cost accounting needs to go beyond the reproduction run.

A1: As detailed in General Response, Q1 and Q2, we provide additional cost comparisons under unified accounting assumptions and clarify the scope of our reported reproduction costs.

(1) We use unified pricing (Verda and AutoDL) and compute estimation methods, and find that the cost advantage of Puro-2B is maintained. We recomputed reproduction costs using updated commercial GPU rental rates. In our work, we underestimate the baseline cost, since we use theoretical 6ND6ND estimates with an optimistic 70% MFU. Hence, the cost efficiency of Puro can be even higher in practice.

(2) We clarify the accounting boundary and also provide development cost accounting. This accounting boundary applies consistently to both Puro-2B and baseline models in Figure 1. As detailed in General Response, Q2, our primary goal is to provide an affordable, fully open pretraining recipe that others can directly reproduce. Reporting reproduction cost is useful for the open-source community. To further improve transparency, we have also separately estimated the costs of our development experiments in our manuscripts, as reported in General Response, Q2.

Q2: Is the 0.4-point advantage over Qwen2-1.5B larger than evaluation noise?*

A2: **We would like to emphasize matching instead of surpassing Qwen2-1.5B with \4.4K.** As detailed in `General Response, Q5`, our \4.4K run robustly matches Qwen2-1.5B under the evaluated settings. (1) Through the scaling law fitting, we can mitigate the training randomness across different scale runs; (2) We conducted additional evaluations with repeated sampling at different temperatures to assess evaluation variability.

Hence, we agree with the reviewer's suggestion and have revised our claim to "matching Qwen2-1.5B performance at approximately $4.4K" (Summary of Revisions).

Q3: How robust is the 2.40× CMA cost-efficiency estimate, and report the benchmark contamination?

A3: As discussed in General Response, Q3 and Q4, we have further examined both the scaling-law sensitivity and potential benchmark contamination.

(1) Scaling-law robustness. The CMA improvement is directly supported by a same-budget comparison (57.81 vs. 56.00 at $6.89K), independent of curve fitting. The reported 2.40× cost-efficiency is an extrapolation-based estimate. We then test scaling law robustness with leave-one-out experiments. The leave-one-out analysis yields estimated gains of CMA ranging from 1.83× to 2.86×, so the advantage of CMA is persistent. In addition, as the reviewer suggested, we refit the scaling law allowing the Phase 1 cost offset to be fitted, and it yields 2.85×. These results consistently support a substantial cost-efficiency advantage for CMA, although the precise multiplier can vary with fitting choices.

(2) Benchmark contamination diagnosis. As detailed in General Response, Q4, we have conducted a 13-gram overlap analysis on the late-stage training data against the 15 evaluation benchmarks, and excluded the most overlapped GSM8K to reassess the conclusions. The results suggest that the identified overlaps are unlikely to dominate the CMA advantage. We have included the decontamination analysis in the revised manuscript. We also have our new post-decontamination retraining results in General Response [5/n] to support our diagnosis. After decontamination and retraining, the efficiency gain of CMA is persistent, and the final performance is close to our primary results.

Q4: Concerns about the "co-designed system" claim.

A4: We agree that the term "co-designed" may imply stronger synergistic effects than we intended to claim, and we have revised the wording accordingly. Our intended point in the Introduction is that these components are not entirely independent and require coordination, rather than that their combination produces superadditive gains. For example, MuonH and CMA require careful coordination of their learning-rate schedules. We have replaced "co-designed" with the more precise term "coordinated" (Summary of Revisions) to avoid implying unverified synergistic improvement claims. We appreciate the reviewer's recognition of the value of integrating these components into a practical, reproducible pretraining recipe.

Dear Reviewer wLvi,

Thank you again for your thoughtful review. As the discussion period nears its end, we would like to briefly summarize how our rebuttal addresses all four points you raised:

  1. Unified cost accounting & development costs (Weakness 1 / Q1): In General Response [1/n], we recomputed all model costs under unified commercial rental rates (Verda and AutoDL) and a common 6ND (@ 70% MFU) compute basis, confirming Puro-2B's substantial cost advantage. In General Response [2/n], we also provided an itemized breakdown of development and search runs in the paper.
  2. Evaluation noise & $4.4K checkpoint claim (Weakness 2 / Q2): We revised our claim to state that the $4.4K checkpoint matches (rather than surpasses) Qwen2-1.5B, supported by our scaling-law fit and repeated sampling across temperatures in General Response [4/n].
  3. CMA scaling-law robustness & decontamination retraining (Weakness 3 / Q3): In General Response [3/n], leave-one-out checks (1.83×–2.86×) and fitting the Phase 1 offset (2.86×) confirm that CMA's efficiency advantage is robust. Furthermore, in General Response [5/n], we have now completed actual model retraining after 13-gram benchmark-overlap filtering: full-budget decontaminated CMA achieves 56.97 Avg15 (+1.30 over Released UD at 55.67, and within 0.47 of Released CMA at 57.44). In addition, we run a matched 1/16-budget pair on identical decontaminated data, and confirm a +1.19-point CMA gain (56.18 vs. 54.99, winning on 9 of 12 benchmarks).
  4. Clarifying component coordination (Weakness 5 / Q4): We replaced "co-designed" with "coordinated" in the revised manuscript to avoid implying unverified superadditive gains.

Please let us know if any questions remain. If these updates resolve your concerns, we would be grateful if you would consider raising your overall ratings.

I am raising my rating to weak accept.

I will be honest about where I started: my biggest worry was the cost comparison. Pricing your own run at an amortized $0.31 per GPU-hour while everything you compare against sits at market rental rates is not a fair fight, and I said so. So the unified-rental recompute is exactly what I wanted to see, and the gap survives it

The other two fixes are just as welcome. "Matches" instead of "surpasses" for the $4.4K checkpoint is the honest word, and the repeated sampling backs it up. And I did not expect a full decontaminated retraining inside a rebuttal — the CMA gain holding up on filtered data (56.97 vs. the released UD's 55.67, and +1.19 in the matched pair on identical data) settles the contamination question for me. "Coordinated" instead of "co-designed" is a small change that says a lot; thank you for making it.

None of this changes what I liked from the start: a complete, released recipe at a budget normal labs can actually afford, and a genuinely neat finding in the Muon-ELR analysis. I am glad the rebuttal let me move up.

Dear Reviewer wLvi,

Thank you very much for taking the time to review our rebuttal and new experiments, and for raising your rating. We truly appreciate your constructive feedback throughout the discussion. Your suggestions helped make the paper much more rigorous and clear.

Best regards,
The Authors

Reviewer_q1eeOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Please give the exact branch and merged computations, identify how cross terms are avoided, and report numerical output agreement before and after merging. Can you also reconcile the inference shapes and training specifications, and provide a compute-matched training comparison?

Critical strengths
  1. The engineering documentation is unusually useful. Data mixtures, checkpoint lineage, hardware configurations, and software changes provide information that is often absent from model releases. This gives the work value beyond the final benchmark score.
  2. The consumer-hardware demonstration is substantive. Completing a run of this scale with FP8 on RTX 5090 GPUs establishes a practical capability. Itemized cost accounting also lets readers reconsider the result under their own pricing assumptions.
Critical weaknesses
  1. The cost frontier combines quantities that are not directly comparable. PURO uses measured runtime with a particular owned-hardware pricing basis, whereas competitors use estimated compute and assumed accelerator rental rates. These choices can affect the frontier in different directions, so the figure does not yet isolate a cost advantage. A common pricing basis, supplemented by sensitivity ranges for estimated quantities, would make the comparison more informative.
  2. The reported budget and headline performance refer to different checkpoints. The sub-$5,090 claim applies to the $4.4K model, whose mean score is 55.5 versus 55.1 for Qwen2-1.5B; the stronger 57.8 result costs $6.9K. Both are legitimate results, but they should remain clearly distinguished throughout the paper. The small cheaper-checkpoint margin also needs an indication of evaluation variability, while the comparison with Qwen2.5-1.5B should acknowledge the 2.9-point average gap.
  3. The resource and evaluation boundaries need clearer disclosure. The stated GPU-hours and duration imply approximately 53 continuously active GPUs, but the actual GPU/node count and utilization are unclear. The final-run cost also excludes recipe search. Separately reporting search costs would distinguish reproducing an established recipe from developing it. Baseline prompting, shot settings, and answer parsing should be explicit because protocol differences can affect the reported margins.
Questions for authors

Please specify the hardware scale and utilization, distinguish final-run and search costs, and clarify whether all benchmark scores use the same harness. Can the frontier be recomputed under common pricing assumptions with uncertainty for estimated competitor costs?

2 comments
AuthorsOpenReview ↗

Response to Reviewer q1ee

We would like to thank the reviewer for the appreciation of the open-source value of our work and recognition of our extensive empirical validation. Moreover, your valuable and constructive suggestions are important for us to further improve our work.

Q1: Does the cost advantage in Figure 1 persist under unified GPU pricing and compute estimation methods?

A1: The explanation is detailed in General Response, Q1. We address this in two respects: (1) price basis; (2) baseline cost accounting.

(1) Under the unified commercial price basis, Puro-2B retains a substantial cost advantage. The updated RTX 5090 quote is reasonably close to our amortized estimate, while the relative price ratios remain comparable to those used in the paper. The conclusion therefore persists under the stated sensitivity analyses.

(2) We emphasize that the baseline estimates derived from 6ND6ND compute and an optimistic 70% MFU are favorable to the competing models. Their actual costs may be higher, but we do not quantify that gap. Even under this favorable assumption, Puro-2B retains a substantial cost advantage.

These results support the robustness of the cost frontier in Figure 1 and hopefully address the concern that the advantage arises from inconsistent accounting assumptions.

Q2: Distinguish two Puro-2B checkpoints and explain the evaluation variability.

A2: (1) The distinction between the two checkpoints was present in the original manuscript, but we agree that it should have been made more prominent. The \4.4KUDcheckpointscores55.54,comparedwith55.14forQwen2−1.5B;thecanonical4.4K UD checkpoint scores 55.54, compared with 55.14 for Qwen2-1.5B; the canonical \\6.9K CMA checkpoint scores 57.81. We have made this distinction more explicit in the Abstract, Introduction, Figures 1–2, Tables 19–20, and the Conclusion.

(2) The $4.4K checkpoint matches Qwen2-1.5B under the tested evaluation settings, and the differences remain small across the tested sampling temperatures. As detailed in General Response, Q5, we use two sources of evidence. First, apart from the actual training run, the cost scaling law prediction matches Qwen2-1.5B at \$4.4K. Second, across the three sampling temperatures, the mean Puro–Qwen2 differences are +0.08, +0.05, and −0.16 points, respectively, over five decoding runs. These additional runs can quantify decoding-time variability.

(3) We appreciate the reviewer's concern regarding the performance gap with Qwen2.5-1.5B. We have further qualified this comparison by explicitly acknowledging the remaining 2.92-point gap in the revised manuscript (Summary of Revisions).

Q3: The hardware utilization, reproduction-cost boundary, and evaluation protocol require clearer disclosure.

A3: In the original manuscript, we have provided detailed documentation of both the production training configuration and the unified evaluation protocol in Appendix F. We clarify these points as follows.

(1) Hardware configuration and utilization. As detailed in Appendix F.2 (Table 7), Phase 1 uses 24 RTX 5090 GPUs across 3 nodes (TP/PP/DP = 1/2/12), achieving 238 TFLOP/s/GPU and approximately 73% mixed-precision MFU. Phase 2 scales to 96 GPUs (TP/PP/DP = 1/4/24), achieving 192 TFLOP/s/GPU. The increase in GPU count is because additional compute resources became available.

(2) Unified evaluation protocol. As explicitly described in Appendix F.1, all pretrained models are evaluated using the same OpenCompass framework, with consistent benchmark-specific prompts, few-shot examples, generation settings, and answer postprocessors. These shared settings ensure that the reported model comparisons follow a consistent evaluation protocol.

(3) Reproduction-cost boundary. As clarified in General Response, Q2, the reported costs cover the pretraining runs required to reproduce the checkpoints, excluding other development expenses. We have made this accounting boundary more explicit in the revised manuscript. Nevertheless, we also provide a lower-bound estimate of quantified development compute based on the experiments reported in the paper.

Dear Reviewer q1ee,

Thank you again for your constructive review and for noting that your soundness concerns are minor and fixable. As the discussion period nears its end, we would like to briefly summarize how our rebuttal and revised manuscript address all three points you raised:

  1. Unified cost comparison (Weakness 1): In General Response [1/n], we recomputed all model costs under common commercial rental quotes (Verda and AutoDL) as well as a unified 6ND6ND (@ 70% MFU) compute basis. Puro-2B still maintains a substantial cost advantage (\4.37K–4.37K–\\6.89K vs. baselines).
  2. Clear checkpoint distinction & evaluation stability (Weakness 2): Throughout the revised PDF, we clearly distinguish the $4.4K UD checkpoint (55.54, matching Qwen2-1.5B) from the $6.9K CMA checkpoint (57.81), and explicitly acknowledge the 2.92-point gap to Qwen2.5-1.5B. In General Response [4/n], repeated sampling runs across temperatures confirm that the $4.4K checkpoint reliably matches Qwen2-1.5B within evaluation noise.
  3. Hardware scale, search costs & evaluation protocol (Weakness 3): We clarified the exact cluster configuration (24 GPUs in Phase 1 at 73% MFU; 96 GPUs in Phase 2), added an itemized breakdown in General Response [2/n], and confirmed that all models use the exact same evaluation settings (Appendix F.1).

Please let us know if any questions remain. If these updates resolve your concerns, we would be grateful if you would consider raising your soundness and overall ratings.

Comments on the paper

6 comments
AuthorsOpenReview ↗

Summary of Revisions

We sincerely thank all reviewers for their valuable feedback and constructive suggestions. Based on your comments and to clarify the changes discussed below, we have revised the manuscript as follows (section references correspond to the revised version):

  • Appendix B: We expanded the discussion of benchmark overlap, clarified our original assumptions about upstream decontamination, and added an analysis excluding GSM8K. We report the updated model comparisons and the refitted CMA cost-efficiency multiplier. (wLvi; aF9A)
  • Appendix E: We clarified the scope of reproduction cost, explicitly distinguishing it from recipe search, debugging, failed experiments, and other development expenses. We also state that the same accounting boundary applies to Puro-2B and the comparison models. (q1ee; wLvi)
  • Introduction & Section 2.3.1: We revised the co-design descriptions to emphasize coordination among components, avoiding implications of an unverified superadditive gain claim. (wLvi)
  • Throughout the manuscript: We clarified the distinction between the $4.4K UD checkpoint and the $6.9K CMA checkpoint, updating the abstract, introduction, performance discussion, conclusion, and relevant captions. We revised the comparison wording to state that the UD checkpoint matches Qwen2-1.5B’s overall performance, while the canonical CMA checkpoint exceeds Qwen2-1.5B by 2.7 points and trails Qwen2.5-1.5B by 2.9 points. (q1ee; wLvi)

We hope these revisions address your concerns and clarify the points addressed below.

AuthorsOpenReview ↗

General Response [1/n]

We sincerely thank all reviewers for recognizing the open-source value of our work. Before answering each specific question, we want to highlight the main contributions of this paper:

  • Full-stack, cost-efficient pretraining: We provide a complete end-to-end recipe that trains a 2B model from scratch on 1.4T tokens for under $6.9K (matching Qwen2-1.5B at $4.4K; here, the accounting follows the original version), cutting reproduction costs by more than 10× compared to existing open baselines.
  • Fully open-source release: Unlike releases that only share final weights, we open-source the entire pipeline, including training framework, model weights, training dataset, and intermediate checkpoints. So other labs can reproduce or build on our work.
  • Clear technical novelty: Beyond system integration, we contribute: (i) RTX 5090 hardware selection and driver-level adaptation (enabling PCIe P2P and GPUDirect RDMA); (ii) the Muon-ELR analysis and experiments showing that hyperball optimization gains mainly come from its effective learning-rate schedule; and (iii) multi-source CMA adaptation and experiments, including our new multi-scale ladder (0.60B–2.60B) and post-decontamination retraining.

Across General Responses [1/n]–[5/n] and our individual replies, we have resolved your technical questions with new experiments and unified analyses. We hope the reviewers will consider these complete contributions when updating their final ratings.


Q1: Are the GPU prices and cost estimates in Figure 1 fair across models? (q1ee; wLvi; aF9A)

A1:

(1) Our GPU pricing assumptions are reasonable, and updated market rental rates further support the RTX 5090's cost advantage. As noted in Appendix D.1.1 (Table 3), NVIDIA's EULA limits RTX 5090 availability on conventional data-center rental platforms. We therefore amortize our hardware and electricity costs over five years for the RTX 5090 while using publicly listed rental rates for other GPUs. To validate this assumption, we collected November 6, 2026 quotes from Verda (data-center-class GPUs) and AutoDL (RTX 5090):

GPUPaper ($/GPU-hour)Nov. 6 quote ($/GPU-hour)
H200 SXM46.44
H100 SXM3.254.938
A100 80GB1.792.316
A100 40GB1.291.657
RTX PRO 60001.892.725
RTX 50900.310.41

The updated RTX 5090 quote is reasonably close to our amortized estimate, while data-center GPUs exhibit higher quoted prices. Therefore, it yields price ratios relative to the RTX 5090 comparable to or higher than assumed in the paper.

(2) Our accounting provides a lower-bound estimate of baseline costs, so the actual efficiency advantage can be even larger. As detailed in Appendix E, when reported costs or GPU-hours are unavailable, we estimate theoretical compute via C=6NDC = 6ND and convert to H100-equivalent GPU-hours assuming an optimistic 70% model FLOPs utilization (MFU)—challenging to achieve in practice, thus providing a favorable estimate for baseline models. To ensure full comparability, we recomputed reproduction costs under two consistent approaches: (i) applying updated commercial rental rates (Unified Rental); and (ii) uniformly estimating GPU-hours from 6ND6ND compute at 70% MFU with those same rental rates (6ND6ND (@70% MFU)):

ModelTokensPricing BasisPaper CostUnified Rental6ND6ND (@70% MFU)Avg15 Score
Puro-2B (UD)0.92TRTX 5090$4.37K$5.85K (@$0.41)$3.84K (@$0.41)55.54%
Puro-2B (CMA)1.40TRTX 5090$6.89K$9.23K (@$0.41)$5.85K (@$0.41)57.81%
Qwen2-1.5B7.0TH100 equivalent$84.3K$128.1K (@$4.938)$128.1K55.14%
Instella-3B4.123TH100 equivalent$100.3K$152.3K (@$4.938)$152.3K55.20%
Qwen2.5-1.5B18.0TH100 equivalent$216.8K$329.4K (@$4.938)$329.4K60.73%
SmolLM3-3B11.2TH100$718.8K$1,092.2K (@$4.938)$399.2K65.85%

Puro-2B retains a substantial cost advantage under both unified accounting approaches, whether applying commercial rental rates alone or uniformly estimating compute via 6ND6ND at 70% MFU (where the advantage becomes even more pronounced). Together, the updated market quotes and unified comparisons confirm the central conclusion of Figure 1: Puro-2B greatly reduces the reproduction cost required to achieve competitive model performance.

AuthorsOpenReview ↗

General Response [2/n]

Q2: The reported cost excludes recipe search, ablation studies, and failed runs. (q1ee;wLvi)

A2:

We would like to clarify that for both Puro-2B and the competing models, the reproduction costs reported in Figure 1 cover only the production pretraining runs required to reproduce the reported checkpoints.

Our focus on reproduction costs aligns with the primary goal of this work: making end-to-end pretraining affordable and reproducible for other researchers. As emphasized in Appendix C.2, we aim to provide a fully open pretraining recipe that others can directly reproduce. So we did not measure the total research and development expenses behind its creation in the original manuscript. Moreover, accurately accounting for these costs is inherently challenging, as they involve numerous exploratory experiments, ablations, and failed runs, which are difficult to track and quantify consistently across different projects. We therefore believe that reproduction cost provides a clearer and more comparable measure of the computational resources for reproducing an training recipe.

Nevertheless, to provide greater transparency, we have carefully calculated the experimental costs, accounting for both major experiments and smaller-scale runs reported in the paper, as detailed below:

Experiment categoryRunsEstimated cost (USD)
Optimizer / FP8 scaling and hyperparameter search46$1,008.64
WSD schedule search (peak LR, token budget, decay ratio)131$9,375.49
MuonH / Muon-ELR control experiments15$95.67
Data-selection proxy training39$691.57
Reduced-budget uniform-data Phase 2 runs4$4,062.47
Full-budget curriculum Phase 2 training1$4,329.19
Constant-LR continuations (steps 215k and 218k)2$344.38
SFT ablations (3 settings × 2 initializations × 3 repetitions)18$54.12
Additional training subtotal256$19,961.52
Reference training (Phase 1 + full-budget uniform-data Phase 2)2 stages$6,979.47
Total$26,940.99

We acknowledge that the accounting boundary could have been stated more clearly in the original manuscript. We have therefore made it more explicit in the revised manuscript Appendix E.1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.