acceptodds
Withdrawn from ICLR 2027

Open-Sci: Beyond Scaling with High-Density Datasets for Precise Scientific Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large Language Models (LLMs) have made remarkable progress on reasoning tasks such as coding and mathematics, yet their scientific reasoning ability remains significantly limited, hampered by the scarcity of high-quality scientific datasets. While LLM training has long been guided by scaling laws, attention is shifting toward data quality: high-density supervision complements scale and at the margin can outperform noisier corpora. Constructing such datasets, however, remains challenging. Existing efforts either rely on LLM-generated synthetic data prone to hallucination and logical inconsistency, or on manual curation that struggles with scalability and standardization. To overcome these hurdles, we propose PrecSci, a systematic data curation pipeline centred on precision and internal consistency. It extracts knowledge from reliable scientific documents, refines questions for completeness and precision, applies multi-stage filtering to eliminate redundancy and noise, and refines answers through a generator and a critic that scores candidates along multi-dimensional quality criteria, retaining only high-quality samples. Leveraging PrecSci, we build Open-Sci, a compact corpus of 186k scientifically precise question–answer pairs. Despite being less than one-sixth the size of the state-of-the-art MegaScience dataset ( 14.88%), Open-Sci consistently improves LLM performance, lifting Qwen3-8B by 7.08 points on average across diverse scientific benchmarks (e.g., PHYSICS, ChemBench, MMLU-Pro).

Primary area:
datasets and benchmarks
Keywords:
ai4science, scientific dataset, scientific foundation model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bVLDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Open-Sci, a 186k-example dataset for scientific reasoning, constructed using the proposed PrecSci pipeline. The pipeline extracts questions from scientific sources, formalizes them, filters out low-quality samples, and generates reasoning traces through a generator-critic refinement process. Experiments across several model families and scientific benchmarks show that Open-Sci improves downstream performance and outperforms substantially larger datasets such as MegaScience. The authors also provide matched-size comparisons, pipeline ablations, and evaluations on general knowledge and mathematical reasoning tasks.

Critical strengths
  1. The paper presents a useful scientific reasoning dataset. Open-Sci covers four scientific disciplines and a diverse range of question types. Building on existing scientific problems rather than generating everything from scratch is a sensible approach to reducing hallucinations in training data.

  2. The experimental results are encouraging. Open-Sci achieves substantial improvements across multiple models and benchmarks, despite being much smaller than several competing datasets. I also appreciate the matched-size comparisons and pipeline ablations, which provide useful evidence beyond simple full-dataset comparisons.

Critical weaknesses
  • The claimed data quality is not independently validated. The reasoning chains are generated and evaluated by the same Qwen3 model, and matching the final answer does not necessarily mean that the reasoning is correct. More importantly, the two critics agree on only 71% of the sampled accept/reject decisions (Appendix C.4). Given that scientific precision is the central motivation of the paper, I would expect at least some human expert evaluation of the final dataset, ideally with a comparison against existing datasets.

  • The data-efficiency comparisons are not entirely fair. Table 5 compares datasets using approximately the same number of examples, but Open-Sci has considerably longer answers than MegaScience (1,122 vs. 693 tokens on average). This means the models may receive substantially more supervision tokens from Open-Sci, even with similar sample counts. A token-matched comparison seems necessary to support the claim that the improvements come from data quality rather than supervision volume.

  • The pipeline ablations leave some important questions unanswered. It is unclear whether the ablated datasets are matched in size, answer length, and difficulty distribution. Without these controls, the performance differences cannot be cleanly attributed to individual components. I also found it interesting that dual refinement, which receives considerable emphasis in the method, contributes the smallest improvement among the three ablated stages.

  • The methodological novelty is somewhat incremental. Most components of PrecSci, including question reformulation, deduplication, difficulty filtering, and generator-critic selection, are already widely used in LLM data curation. I see the main contribution as the resulting dataset and its empirical performance rather than a substantially new curation method. Stronger evidence about what makes Open-Sci's supervision particularly effective would help establish its contribution.

Questions for authors
  1. Could the authors provide a human expert evaluation of Open-Sci's scientific correctness and reasoning quality, preferably compared with MegaScience or other existing datasets?

  2. How does Open-Sci compare with MegaScience under matched training-token or compute budgets, rather than matched sample counts alone?

  3. Were dataset size, answer length, and difficulty controlled in the pipeline ablations? If not, how should we interpret the contribution of each stage?

Reviewer_sFi2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PrecSci, a four-stage pipeline that builds a compact scientific training set from textbook, exam, and competition problems: extracting existing QA pairs, formalizing questions, removing duplicates and benchmark contamination, and refining reasoning chains with a generator–critic filter. The resulting Open-Sci contains 186k QA pairs across physics, chemistry, biology, and medicine. The authors then fine-tune on it, improving Qwen3-8B by +5.09 points on average across 12 scientific benchmarks, despite using only about 15% of the data, with additional gains on math. The main contribution is a precision-oriented curation pipeline and a smaller, stronger scientific SFT corpus.

Critical strengths

The overall line of the work is clear. It is a series of efforts based on thinking about domain scientific and physical reasoning ability, and it finally produces a small but precise training set to improve general models’ scientific reasoning. The training-set curation pipeline is scientific: it relies on the authority of textbooks and papers to hold the lower bound of precision, so the resulting training set is relatively reliable. The experiments include a large-dataset comparison (MegaScience), a matched-scale comparison of about 200k examples, results across five models, stage ablations, and a disciplinary-coverage comparison (medicine-only versus mixed). Decontamination was performed, and small benchmarks were evaluated with repeated averages. The main conclusion that “small and precise outperforms large and noisy” still holds at matched scale, rather than relying only on data volume.

Critical weaknesses

The main problem is that the dataset was curated without human checking. In many cases textbook answers are not necessarily suitable for model training. First, textbook problems themselves can be wrong. Even highly authoritative textbooks in physics, such as solid-state physics, contain many incorrect exercises, so someone still needs to look at the actual cases. Second, conventions are lost once the item is taken out of its original context. Textbook chapters have their own conventions: how symbols are used, what units are assumed, how a quantity is defined in that chapter, and whether the result of a previous part can be used directly. A problem that is correct inside the chapter loses these conventions when extracted alone. Common issues when adapting textbook problems include dependence on symbols or formulas defined earlier, default unit systems, unspecified normalization or commutation relations, and multi-part problems that need the conclusion of a previous part to have a unique answer. In the curation pipeline, the later formalization and dual refinement only ensure that the final answer of the reasoning chain matches the extracted reference answer. They do not ensure that this reference answer remains uniquely correct once removed from the original context. As a result, a model may write a correct reasoning chain under another common convention, fail to match the textbook answer, and be filtered out. Or the missing conventions that the original answer depends on may not be supplemented, so the retained supervision is actually misleading. Automated stages have difficulty detecting this kind of error, because both sides can be self-consistent under their own conventions.

Questions for authors

When extracting textbook problems, how do you handle cases where the item depends on earlier definitions, default units, or results from a previous part? Did you supplement missing conventions so that the reference answer remains unique outside the original context? In the dual-refinement stage, a generated chain is kept only if it matches the extracted reference answer. How do you distinguish a wrong chain from a correct chain written under a different but common convention?

Reviewer_ZGTSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PrecSci, a data curation pipeline that extracts knowledge from scientific documents to improve scientific reasoning in LLMs. It focuses on increasing the information density and supervision precision of training data rather than simply scaling up data volume. Using PrecSci, the authors construct Open-Sci, a compact scientific reasoning dataset containing 186K instances. Fine-tuning on Open-Sci achieves improved performance on several scientific benchmark.

Critical strengths

S1. The motivation is reasonable. Constructing a well-structured corpus from reliable sources with high-quality reasoning chains is valuable for LLM fine-tuning. S2. The experimental evaluation covers a variety of scientific benchmarks and includes comparisons with multiple baseline datasets.

Critical weaknesses

W1. The novelty is limited. a) The contribution is primarily a resource construction and engineering effort that combines standard techniques (heterogeneous data extraction, type-specific question generation, and QA pair refinement), rather than introducing substantial methodological innovations. b) The core idea closely resembles the recent work LIMO (COLM 2025, Less is More for Reasoning), which is not discussed by the authors. Although LIMO focuses on mathematical reasoning, its pipeline, including multi-stage data filtering and reasoning chain construction (Figure 2), can be adapted to other scientific reasoning tasks. W2. The performance improvements are inconsistent across models and benchmarks. Although the authors evaluate two model families, Llama and Qwen, most of the reported results focus on Qwen. The improvements are primarily observed with Qwen3 and on the MATH and PHYSICS benchmarks. In contrast, the gains on ChemBench and ProteinLM are marginal or even negative. The paper lacks a deeper analysis of these performance differences and does not sufficiently demonstrate the generalizability of the proposed approach.

Questions for authors
  1. The following claims lack sufficient quantitative evidence and supporting references. Please clarify the criteria used to distinguish simple from challenging problems and overly concise from lengthy answers. a) Section 3.3: "Some problems are too simple and offer little challenge for training, while others are too difficult for the model to learn." b) Section 3.4: "Many answers are too concise or lengthy, and some answers may contain errors, which undermines the accuracy of science."
  2. In Section 4.3 (Ablation Study), the baseline datasets are sampled to match the training data scale (e.g., 200K out of 1.2M instances from MegaScience). What sampling strategy is used? Different sampling strategies may introduce distributional bias and affect performance across benchmarks.

Comments on the paper

1 comment
Area_Chair_yQEzOpenReview ↗

Reminder

Dear authors,

Your resonses haven't received by any reviewers. Please submit your rebuttal as soon as possible .

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.