acceptodds
Under review as a conference paper at ICLR 2027

GlassBench: Benchmarking Large Language Models for Domain-Specific Glass Materials

Abstract

Large language models (LLMs) have shown substantial potential for scientific question answering, reasoning, and knowledge integration, yet materials science still lacks standardized benchmarks for assessing domain-specific capabilities. This limitation is especially pronounced in glass science, where terminology is specialized, composition-structure-property relations are highly nonlinear, and most available resources are structured numerical databases rather than natural-language evaluation sets. Here, we introduce GlassBench, a multitask benchmark for evaluating LLMs in glass materials science. Based on this, GMatLLM is constructed as a domain-specific LLM for the glass research. First, GlassBench was constructed by extracting question-answer pairs from authoritative textbooks and research literature, followed by multistage quality control combining LLM-based scoring, confident-learning-based screening, domain-consistency checks, and expert review. The resulting dataset contains 19,974 quality-screened samples spanning judgement, single-choice, multiple-choice, short-answer, and open-ended tasks. To test the benchmark's sensitivity to domain adaptation, we used Qwen2.5-7B-Instruct as the base model and applied supervised fine-tuning with LoRA through the LLaMA-Factory framework, yielding GMatLLM. We then evaluated general-purpose, chemistry-specific, materials-specific, and encoder-only models on GlassBench and complementary general benchmarks. GMatLLM achieved approximately 0.72 accuracy on GMat-MCQ and 0.31 on GMat-QA, while its reduced scores on several general benchmarks exposed a domain-specialization/general-capability trade-off. These results indicate that GlassBench can distinguish model families and quantify both the gains and costs associated with domain adaptation. GlassBench therefore provides a reproducible framework for evaluating and developing LLMs for glass materials research.

Primary area:
datasets and benchmarks
Keywords:
large language models, benchmark, question-answering dataset, supervised fine-tuning, materials science

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KP8TOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GlassBench is a glass-materials benchmark containing 19,974 question-answer pairs across five formats. Its construction combines textbook/literature extraction, LLM-based filtering, confident-learning screening, consistency checks, and a 5% expert audit. The paper evaluates general-purpose and scientific models and uses LoRA adaptation of Qwen2.5-7B-Instruct into GMatLLM as a validation probe. It reports improved glass-domain scores alongside reduced general-benchmark performance, and adds an external Matbench-glass comparison.

Critical strengths
  1. A useful evaluation target. Glass science involves specialized processing and composition–structure–property knowledge that broad language benchmarks and numerical materials databases do not directly assess. The five question formats and separate MCQ/QA reporting could make this a useful domain resource (Sections 2.1 and 2.4).

  2. Evaluation goes beyond a single adapted-model score. The comparison spans general and adjacent-domain models, and reports general-capability retention alongside glass-domain gains (Figures 3–5 and 10). This is a useful evaluation perspective, even though the evidence does not yet establish a general specialization trade-off.

Critical weaknesses
  1. Benchmark independence and reliability are insufficiently established. Although the paper reports stratified splits and near-duplicate detection, it does not document source-disjoint splitting or a cross-split overlap audit, leaving potential leakage unresolved. Section 2.2.4 reports a 5% expert audit but omits reviewer qualifications and quantitative error/correction rates, limiting confidence in factual reliability.

  2. Broader conclusions about domain adaptation exceed the evidence. Adaptation is studied only on Qwen2.5-7B-Instruct under one LoRA configuration. This provides a useful validation case but does not establish that domain fine-tuning generally outperforms increasing model size, as suggested in Section 3.2. The observed general-capability declines also remain configuration-specific without controlled ablations of the training settings.

Questions for authors
  1. What source-level separation and cross-split deduplication were enforced? Please provide an overlap audit and, if available, source-disjoint results for the base-versus-adapted comparison.

  2. For the 5% expert audit in Section 2.2.4, how many materials/glass-science experts participated, and what were their qualifications and review protocol? Please report error/correction rates and, where items were independently reviewed by multiple experts, inter-rater agreement and how disagreements were resolved.

Reviewer_4VKLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This article mainly discusses the contributions of their proposed GlassBench, which is a benchmark for question-answering in the vertical field of glass materials science. It is automatically generated through multi-stage filtering from textbooks and research papers in their field, including 19,974 entries with true or false questions, multiple-choice questions, short answer questions, and open-ended tasks. They also use the LoRA fine-tuning of Qwen2.5-7B-Instruct as a probe to test whether the benchmark can distinguish models before and after domain adaptation. It also involves different control group models to make comparisons on different benchmarks. The conclusion is that GlassBench can distinguish model families and quantify the benefits and costs of domain adaptation and general capability.

Critical strengths

There is indeed a lack of natural language evaluation sets in the field of glass materials. The overall approach is relatively coherent and self-consistent, including constructing a benchmark from papers and textbooks within the field, and the subsequent process of domain-adaptation probe is indeed a closed-loop logic. The evaluation design in Section 2.4 is quite impressive. The designed control levels are rich and align with their evaluation objectives. The comparison and control are not limited to their own benchmark but also involve general and adjacent fields, with a relatively comprehensive design approach.

Critical weaknesses

In the initial phase of building the benchmark, the Qwen-Plus scores were not evaluated by glass experts to assess the actual effectiveness, and relying solely on a general large model is somewhat unconvincing. Although this is a preliminary screening, the scoring effect does have an impact on the later stages. The results of the subsequent manual inspections, which did not report how many were inspected, how many were modified, the error rate, and the consistency with experts, are not reported. Moreover, only 5% were manually inspected, which I believe is not reasonable; the confidence in the ground truth obtained through this process is not very high. In Section 2.3, if the goal is to test whether GlassBench can distinguish models before and after adaptation, the training should be as clean as possible, and a control with conservative settings should be provided. However, in practice, using rank 16 LoRA, training on approximately 14,000 questions with the same distribution at a learning rate of 2e-4 for 10 epochs, the domain improvement is limited, while the general capability has significantly decreased. The paper does not provide a rationale for the necessity of 10 such epochs and cannot rule out overfitting to the training distribution of GlassBench. Finally, the experimental results show little improvement, and the judges for the open questions are difficult to replicate. There is no fixed judge model and version, fixed prompts and scoring criteria, and no consistency with expert judgments. The conclusions supported by this part are very weak.

Questions for authors

Please show the results of the 5% expert review: how many items were reviewed, by how many experts, how many items were corrected or deleted, the error rate found, and the consensus among experts. Did the Qwen-Plus filter itself check the sample against expert labels, including the items it discarded? Please provide the LLM-judge configuration for GMat-QA (model and version, prompts, scoring criteria, and how it is integrated with BERTScore) as well as any validations related to expert judgments. Please show how training loss, validation loss, or retained GlassBench performance changes with training steps, and explain why 10 epochs are needed. If available, also report a more conservative fine-tuning setup as a control.

Reviewer_JGhbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

GlassBench is a QA benchmark for glass materials science. Two Chinese glass textbooks and about 3,000 papers were split into 150-300 character chunks, and Easy Dataset turned these into 30,801 candidate QA pairs. The candidates were filtered in three steps, Qwen-Plus scoring, iterative Cleanlab auditing that uses the judge's factuality score as the label, and term-density, contradiction, and 5% expert checks. The final set has 19,974 items in five formats, split 14,289/1,587/4,098. The test set is reported as GMat-MCQ and GMat-QA. To validate the benchmark, the authors LoRA-tune Qwen2.5-7B-Instruct on the training split for 10 epochs and call the result GMatLLM. GMat-MCQ improves from 0.671 to 0.719 and GMat-QA from 0.238 to 0.307, while MMLU drops from 0.796 to 0.368. The paper compares 18 models and also reports GMatLLM on Matbench-glass.

Critical strengths

1. Targets a real gap. Existing glass resources such as SciGlass and INTERGLAD are composition-property tables, and most materials LLM benchmarks focus on crystalline systems or general chemistry. A text benchmark that covers processing, thermal history, defects, and structure-property reasoning in glasses would be useful, and the mix of textbook and literature sources across five task formats is a sensible design.

2. Reports domain gains together with general-capability loss. Scoring MCQ and open QA separately, and placing both next to general benchmarks, is the right way to evaluate domain adaptation. The authors are open about the drop on general tasks. If the numbers held up, this would be the most useful takeaway of the paper.

Critical weaknesses

1. Not convinced the benchmark measures correct glass knowledge.

(a) Item correctness rests almost entirely on LLMs. One LLM writes the items from 1-2 sentence fragments and another LLM filters them. The only human check is a 5% expert review with no reported number of experts, agreement, or correction rate, and there is no estimate of the test-set error rate or a human baseline. The Cleanlab step does not fill this gap. It runs only on the Golden pool, where the factuality label can only be 4 or 5, so a classifier on MiniLM and TF-IDF features can at best guess 4 vs. 5 from surface text. A flagged "label issue" means "hard to predict from wording," not "factually wrong."

(b) GMat-QA is not defined, and its results look off. Section 2.4 says the judge setup "should be reported explicitly" but never reports the judge model, rubric, BERTScore model, or how the two scores are combined. All models fall between 0.20 and 0.31, and GPT-4.1 (0.20) scores below Qwen2.5-7B (0.24). This suggests the metric rewards matching the style of the generated references rather than being correct, which would also favor a model trained on those references.

(c) Some cross-model conclusions are likely parsing failures. LlaSMol scores 0.10, ether0 0.16, and llamat-3-chat 0.25 on GMat-MCQ. Part of this set is true/false, so these scores are below chance and point to the regex failing to extract an answer. The claim in Sec. 3.2 that chemistry or materials pretraining does not transfer cannot be drawn from these numbers.

(d) The data and code are not available. Sec. 4 leaves provenance, deduplication and contamination checks, expert agreement, and evaluation scripts to future releases. For a benchmark paper, these are the main evidence.

2. The fine-tuning validation is close to circular, and the main numbers disagree across figures.

(a) The tuned model is trained on the same distribution it is tested on. The paper does not say whether the split is by document, chunk, or item. If it is by item, paraphrases from neighboring chunks will appear in both train and test. Even so, the gain is only 4.8 points on MCQ, and GMatLLM (0.72) ties the untuned Qwen2.5-14B and 32B and stays below GPT-4.1 (0.74). This is weak evidence that the benchmark is sensitive to domain adaptation.

(b) The same model gets different scores in different figures. In Figs. 3 and 10, GMatLLM gets 0.37 on MMLU and 0.69 on CMMLU. In Fig. 5 it gets 0.66 and 0.79, which is higher than the base model's 0.75. Sec. 2.4 says answers are extracted with a regex, while Sec. 3.3 says decoder models are scored by log-probability. If MMLU is 0.66 under log-prob scoring, the drop from 0.80 to 0.37 may mostly reflect the model losing the answer format after 10 epochs of free-text training, not losing knowledge. That weakens the trade-off claim, which is one of the two main messages.

(c) Some general benchmark numbers are hard to believe. GPT-4.1 gets 1.00 on TruthfulQA and 0.80 on GPQA-Diamond, while OpenAI reports about 0.66 on GPQA-Diamond. For GPT-4.1 and Qwen2.5-32B, every score on AIME-25, GPQA, and TruthfulQA is a multiple of 0.2, which looks like evaluation on about 5 items. Sample sizes are not reported.

(d) The Matbench-glass section does not support its claim. matbench_glass predicts glass formation in metallic alloys, which is far from the oxide glasses in GlassBench. The paper does not explain what v0.1 and v0.2 are, whether the model was tuned per fold, or how predictions were extracted. The gap to gptchem (0.0044 ROC-AUC) is well within GMatLLM's own std (0.028), and the text presents this as showing that "GlassBench is capable of distinguishing" models (l.231, 237, 240), even though Matbench is not GlassBench.

Questions for authors
  1. Which protocol produced each figure? Please state for Figs. 3, 4, 5, and 10 whether answers were extracted by regex or scored by log-probability, and on which subset (1,118 or 223 items). Please also report parse-failure rates per model on GMat-MCQ and MMLU, and explain why GMatLLM's MMLU is 0.37 in Fig. 3 but 0.66 in Fig. 5. This addresses my main concern in Weakness 2.
  2. How is GMat-QA computed? Please give the judge model, prompt, rubric, and temperature, the BERTScore model, and the formula that combines them. Please also report agreement with expert ratings on a sample, and explain why GPT-4.1 scores below Qwen2.5-7B.
  3. How was the data split? Was it by source document, chunk, or item? Please report near-duplicate overlap between train and test, and GMatLLM's scores on a test subset whose source documents do not appear in training.
  4. How reliable are the test items? Please report the number and background of the experts, the share of the 5% sample that was corrected or deleted, inter-annotator agreement, and an estimated error rate for the test set.
  5. How many items were used for each general benchmark and model? In particular, how did GPT-4.1 get 1.00 on TruthfulQA and 0.80 on GPQA-Diamond?
  6. What happens at each filtering stage? Please give the actual counts from 30,801 to 19,974 after each stage and each Cleanlab round, and state the exact Cleanlab label and input text (Q, A, or Q+A).
  7. Will the dataset, prompts, and evaluation code be released, and under what license given the copyrighted sources?
  8. Could you clarify the Matbench and training setup? What distinguishes GMatLLM v0.1 from v0.2, and which version is used elsewhere in the paper? Was it tuned on each fold, and how were predictions read out? Which checkpoint is reported, the last epoch or the best on validation?

Suggestions

  • Report a 1-2 epoch checkpoint to separate forgetting from format drift.
  • Add a human expert baseline and a contamination check of the source texts against the evaluated models.
  • Evaluate Darwin on GlassBench, since it already appears in Table 1.
  • Compare with existing materials and chemistry QA benchmarks such as MaScQA and ChemBench. The claim that materials science "lacks standardized benchmarks" is too broad.

Minor Issues

  • Fig. 2b says the training set has n = 15,876, which is train plus validation. The text calls 13,435 "94.0%" of training, but in the figure it is 84.6%.
  • The text lists 731 and 481 as training counts, but in Fig. 2b these are test counts. The 875 listed for the test set is a training count.
  • The split is called "stratified" (l.322) but also "intentionally" shifted (l.330). About 36% of the test set is outside core glass.
  • In Fig. 6a, the line labeled "threshold = 4.0" is drawn at about 3.2. Fig. 6c is a hypothetical 10,000-sample example, and its counts do not match Fig. 6b.
  • In the 542-sample example, 439 = 542 - 83 - 20, so near-duplicates were kept. This contradicts the "reduced semantic redundancy" claim (l.170).
  • In Eq. 2-1, B and A should be d x r and r x k, not both d x k. The alpha/r factor also appears twice (l.197).
  • Fig. 10c shows error bars, but no repeated runs are described, and its 7 domains do not match the 10 in Fig. 2b.
  • Fig. 2a and the Fig. 3 caption say "General MCQ/QA" instead of GMat-MCQ/QA.
  • There are several typos: "DOMIAN ADAPTATION" (Fig. 1), "strat egies" (l.862), the broken sentence at l.220-221, reversed quotation marks (l.125, 133, 326), and spaces before punctuation in Sec. 2.2-2.3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.