- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
GlassBench is a glass-materials benchmark containing 19,974 question-answer pairs across five formats. Its construction combines textbook/literature extraction, LLM-based filtering, confident-learning screening, consistency checks, and a 5% expert audit. The paper evaluates general-purpose and scientific models and uses LoRA adaptation of Qwen2.5-7B-Instruct into GMatLLM as a validation probe. It reports improved glass-domain scores alongside reduced general-benchmark performance, and adds an external Matbench-glass comparison.
Critical strengths
-
A useful evaluation target. Glass science involves specialized processing and composition–structure–property knowledge that broad language benchmarks and numerical materials databases do not directly assess. The five question formats and separate MCQ/QA reporting could make this a useful domain resource (Sections 2.1 and 2.4).
-
Evaluation goes beyond a single adapted-model score. The comparison spans general and adjacent-domain models, and reports general-capability retention alongside glass-domain gains (Figures 3–5 and 10). This is a useful evaluation perspective, even though the evidence does not yet establish a general specialization trade-off.
Critical weaknesses
-
Benchmark independence and reliability are insufficiently established. Although the paper reports stratified splits and near-duplicate detection, it does not document source-disjoint splitting or a cross-split overlap audit, leaving potential leakage unresolved. Section 2.2.4 reports a 5% expert audit but omits reviewer qualifications and quantitative error/correction rates, limiting confidence in factual reliability.
-
Broader conclusions about domain adaptation exceed the evidence. Adaptation is studied only on Qwen2.5-7B-Instruct under one LoRA configuration. This provides a useful validation case but does not establish that domain fine-tuning generally outperforms increasing model size, as suggested in Section 3.2. The observed general-capability declines also remain configuration-specific without controlled ablations of the training settings.
Questions for authors
-
What source-level separation and cross-split deduplication were enforced? Please provide an overlap audit and, if available, source-disjoint results for the base-versus-adapted comparison.
-
For the 5% expert audit in Section 2.2.4, how many materials/glass-science experts participated, and what were their qualifications and review protocol? Please report error/correction rates and, where items were independently reviewed by multiple experts, inter-rater agreement and how disagreements were resolved.