- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces CHIMERA, a synthetic reasoning dataset with 9,225 problems across eight subjects and 1,179 topics. The authors use LLMs to expand topics, generate problems, and check the problems and answers, then use an open reasoning model to produce long reasoning traces. They report gains on several reasoning benchmarks after training Qwen3-4B with SFT and RL.
Critical strengths
-
The paper addresses the need for reasoning data beyond mathematics and coding. CHIMERA combines broad subject coverage with long reasoning traces, and its automated pipeline reduces reliance on expert annotation.
-
The results across several models and benchmarks show that a relatively small synthetic dataset can improve existing reasoning models. This gives the work practical value as a data resource for further training.
Critical weaknesses
-
I find the methodological novelty limited. CHIMERA combines topic expansion, problem generation, verification with LLMs, and reasoning distillation. Earlier work such as s1 and LIMO has already explored the value of small, carefully curated reasoning datasets. The paper should clarify what new ideas or design choices it adds.
-
The added benefit of subject diversity is not yet clear. In Table 3, full CHIMERA exceeds training on mathematics alone by 0.7 points on GPQA Diamond and 0.9 on HLE. Mathematics alone already improves these scores over the base model by 2.3 and 0.8 points. This is insufficient to demonstrate CHIMERA’s advantage in terms of broader subject coverage. Evaluations on OOD tasks (such as code and general reasoning) should be added.
Questions for authors
-
Can you evaluate the trained models on additional code and other general reasoning benchmarks?
-
Can you report the number of samples retained after each filtering stage, including problem validation, correctness verification, and decontamination, as well as the final dataset sizes used for SFT and RL?
Since the authors have not given any response to my comments, I will maintain my original score here.