- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies how post-training and test-time refinement can improve language models on challenging competitive programming problems. The authors train Nano-CC and Ultra-CC using synthetic reasoning data, with SFT applied to both models and additional RL for Nano-CC. They also introduce GenCorrect, which selects diverse candidate solutions and uses execution feedback to guide further attempts. Experiments on IOI, ICPC, and LiveCodeBench Pro show improvements from the overall pipeline, with additional ablations examining the contributions of training and inference strategies.
Critical strengths
-
Practical end-to-end system. The paper brings together data curation, post-training, and inference in a working system. Its live evaluation during IOI 2026, though unofficial, provides a useful test under real contest constraints.
-
Substantial engineering effort. The large training corpus and extensive compute investment represent a considerable amount of work. Experiments at this scale offer useful reference points for practitioners developing large models.
-
Broad experimental coverage. Evaluation across IOI, ICPC, and LiveCodeBench Pro, together with training and inference ablations, provides reasonably strong support for the main findings. Repeated runs and uncertainty estimates further strengthen the evaluation.
Critical weaknesses
-
The data pipeline is not described in enough detail for others to follow, particularly the deduplication procedure, difficulty assignment, and filtering of teacher-generated traces. Since the training data cannot be released, these missing details make the work difficult to reproduce and reduce its value as a reference for future data construction.
-
The contest score is impressive, but it does not establish that the underlying models outperform existing leading models. General Ultra-CC still trails its smaller teacher, DeepSeek-V4-Flash, on IOI and ICPC, though it does better on LCB Pro. Given the substantial training investment, this raises concerns about how efficiently the recipe uses its training data.
-
I appreciate the engineering effort and the practical value of the results. As a research contribution, however, the paper feels more like a large-scale application of existing techniques than a methodological advance. The ablations are useful, but I find the new insights limited relative to the scale of the work.
Questions for authors
-
Could you clarify how the 22,000 problems are turned into the final SFT datasets? In particular, how is difficulty determined, how many traces are sampled per problem at each difficulty level, and which traces are retained after execution-based checks, if any? I would also like to understand how the self-improvement examples are constructed—what feedback the teacher receives and whether a revision must improve the score to be retained. Since the data cannot be released, these details would make the training recipe much easier to build on.
-
Could you offer a deeper analysis of the modest RL gains? The explanations in §4.4 seem plausible, but it is unclear which factors actually limit performance beyond the available compute budget. I would also be interested in whether a GenCorrect-style, feedback-conditioned rollout process could improve on independent sampling during RL. This seems worth exploring both as a methodological direction and as a way to better align training with how the models are used at inference time.
-
Could you evaluate other strong models, particularly the teachers, with the same GenCorrect pipeline under comparable inference budgets and contest constraints? It would also be helpful to include the Ultra initialization checkpoint without the proposed domain-specific SFT. The live result is impressive; these comparisons would help establish how much the specialized training adds beyond applying the inference pipeline to an already strong model.
3 comments
Thank you for the detailed response. The new results and clarifications address some of my concerns, especially the comparison between Ultra-CC and its initialization checkpoint under the same GenCorrect setup. The additional data details are helpful, but they also raise questions about data quality, since the SFT traces are not execution-verified and self-improvement examples are not required to show actual improvement. I strongly encourage the authors to describe the full data pipeline in the paper and include more analysis of the resulting data. Such analysis may be more valuable to future research than performance gains alone. I would also encourage the authors to report teacher-model performance under the same inference setup and budget. This would make the comparisons more informative. The paper should also carefully scope its claims to avoid the system-level result being interpreted as broader superiority in model capability. Overall, I consider this a solid and practically valuable work. I will raise my score, while keeping my reservations about the experimental rigor, analysis, and framing expected from a research paper.
Thank you for the constructive follow-up and updated assessment! We will expand the pipeline description and data analysis, clarify the limits of trace filtering, and carefully scope our system-level claims. We are hoping to complete teacher evaluations under the same GenCorrect setup for the camera-ready version.
Thank you for recognizing the system’s practical value, prospective evaluation, and experimental coverage.
W1 / Q1: Data construction and reproducibility
Difficulty labels were initialized from source metadata and reviewed by language-model agents using statements, constraints, and required algorithms. Each generation pass allocated 5, 20, and 50 traces per easy, medium, and hard entry. We retained complete outputs with reasoning and syntactically valid C++, removing exact prompt–response duplicates and duplicate normalized code within each prompt. Execution checks validate problem environments, not SFT traces. Self-improvement examples prompt revisions without requiring execution-verified correctness or measured improvement. We will document these procedures, prompts, teacher-specific generation and retention counts, and training–evaluation deduplication in Sections 3.1–3.2 and Appendices A–B.
W2 / Q3 Contribution of specialized training
We additionally evaluated Ultra’s actual initialization checkpoint with GenCorrect on IOI 2025.
Both results average five runs using the same GenCorrect procedure and maximum budget of 1,000 candidates and 50 submissions per problem. Domain-specific SFT improves the mean by 87.94 points, demonstrating benefits beyond applying GenCorrect to the initialization checkpoint. Single-sample performance also improves from 45.1% to 50.7% on IOI 2025, 50.1% to 57.4% on ICPC 2025, and 70.2% to 74.5% on LiveCodeBench Pro.
Our IOI 2026 claim concerns a historical, prospective achievement where this is the first AI system to outscore the highest-scoring human contestant on an IOI problem set. We do not claim superiority over all existing AI systems and will make this limitation explicit. Our study addresses a practical question of how we can turn an existing model into a strong competitive-programming system and evaluate post-training and test-time scaling approaches on IOI 2025. We then apply this recipe in a live evaluation at IOI 2026, using a system developed before the problems were revealed and following the same time and submission constraints as contestants.
W3: Research contribution
Our contribution is an empirical study that answers: how can post-training and test-time scaling turn an existing model into a strong competitive-programming system? Beyond demonstrating a combined system, we analyze the contributions of data generation, SFT, RL, and test-time scaling across model settings and training initializations. Matched-budget ablations further show when diversity selection and feedback-conditioned generation help. Together with the new initialization comparison, these findings inform how to adapt existing models and allocate training and inference effort. We will foreground these empirical insights and distinguish them from established methods in the introduction and Related Work.
Q2: Modest RL gains and feedback-conditioned training
Figure 6b shows that initialization strongly affects RL outcomes. After 30 steps, base-initialized Nano reaches 24.9% on IOI 2025, compared with 43.0%, 47.1%, and 48.7% after initialization from successive SFT epochs. Under this budget, RL neither closes these gaps nor consistently improves every SFT checkpoint. The modest gains therefore should not be interpreted as evidence that RL generally offers little value. Our reward structure suggests two further diagnostics. Groups with uniformly unsuccessful or successful rollouts provide no relative task-reward signal, while terminal rewards on trajectories of up to 255K tokens offer limited credit assignment.
Feedback-conditioned RL could better align training with GenCorrect. Toshniwal et al. (2026), “Learning Generative Selection for Best-of-N,” shows that targeted RL improves inference-time candidate selection. Extending this principle to generating revisions conditioned on previous attempts and execution feedback is a promising, untested direction that we will discuss.
We hope these results and clarifications address your concerns. If so, we would be grateful if you would consider revisiting your score.