- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Open-Sci, a 186k-example dataset for scientific reasoning, constructed using the proposed PrecSci pipeline. The pipeline extracts questions from scientific sources, formalizes them, filters out low-quality samples, and generates reasoning traces through a generator-critic refinement process. Experiments across several model families and scientific benchmarks show that Open-Sci improves downstream performance and outperforms substantially larger datasets such as MegaScience. The authors also provide matched-size comparisons, pipeline ablations, and evaluations on general knowledge and mathematical reasoning tasks.
Critical strengths
-
The paper presents a useful scientific reasoning dataset. Open-Sci covers four scientific disciplines and a diverse range of question types. Building on existing scientific problems rather than generating everything from scratch is a sensible approach to reducing hallucinations in training data.
-
The experimental results are encouraging. Open-Sci achieves substantial improvements across multiple models and benchmarks, despite being much smaller than several competing datasets. I also appreciate the matched-size comparisons and pipeline ablations, which provide useful evidence beyond simple full-dataset comparisons.
Critical weaknesses
-
The claimed data quality is not independently validated. The reasoning chains are generated and evaluated by the same Qwen3 model, and matching the final answer does not necessarily mean that the reasoning is correct. More importantly, the two critics agree on only 71% of the sampled accept/reject decisions (Appendix C.4). Given that scientific precision is the central motivation of the paper, I would expect at least some human expert evaluation of the final dataset, ideally with a comparison against existing datasets.
-
The data-efficiency comparisons are not entirely fair. Table 5 compares datasets using approximately the same number of examples, but Open-Sci has considerably longer answers than MegaScience (1,122 vs. 693 tokens on average). This means the models may receive substantially more supervision tokens from Open-Sci, even with similar sample counts. A token-matched comparison seems necessary to support the claim that the improvements come from data quality rather than supervision volume.
-
The pipeline ablations leave some important questions unanswered. It is unclear whether the ablated datasets are matched in size, answer length, and difficulty distribution. Without these controls, the performance differences cannot be cleanly attributed to individual components. I also found it interesting that dual refinement, which receives considerable emphasis in the method, contributes the smallest improvement among the three ablated stages.
-
The methodological novelty is somewhat incremental. Most components of PrecSci, including question reformulation, deduplication, difficulty filtering, and generator-critic selection, are already widely used in LLM data curation. I see the main contribution as the resulting dataset and its empirical performance rather than a substantially new curation method. Stronger evidence about what makes Open-Sci's supervision particularly effective would help establish its contribution.
Questions for authors
-
Could the authors provide a human expert evaluation of Open-Sci's scientific correctness and reasoning quality, preferably compared with MegaScience or other existing datasets?
-
How does Open-Sci compare with MegaScience under matched training-token or compute budgets, rather than matched sample counts alone?
-
Were dataset size, answer length, and difficulty controlled in the pipeline ablations? If not, how should we interpret the contribution of each stage?
Reminder
Dear authors,
Your resonses haven't received by any reviewers. Please submit your rebuttal as soon as possible .