- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces TMEX, a benchmark for extracting structured information from topological-magnetism papers. It combines an expert-defined seven-section schema, expert-annotated records for 112 papers, and a field-level semantic judge. The proposed extraction framework uses material-aware routing, figure-to-field indexing, seven parallel specialist agents, and programmatic record assembly and validation. The workflow is iteratively refined using expert analysis of extraction errors.
Validation includes multi-backbone comparisons and component ablations on Eval 53, a subset of the 112 gold papers. The paper reports an improvement from 0.773 to 0.927 F1 for DeepSeek-V4-Pro when moving from single-shot extraction to the proposed workflow. It also describes a database extracted from 8,082 papers using DeepSeek-V3.2-Exp.
Critical strengths
Potentially useful domain-specific evaluation resource. The expert-defined schema explicitly separates material identity, magnetic properties, sample preparation, experimental evidence, and theory, while requiring evidence to be associated with the appropriate material. Together with the 112 expert-annotated papers, this could provide a useful resource for evaluating scientific extraction beyond isolated entity or property recognition. This resource contribution is the strongest aspect of the submission in my assessment
Critical weaknesses
-
Eval 53 may not be independent of development. Figure 1 says the workflow is frozen once extraction on the 112 gold papers becomes stable, yet Eval 53 comes from those same papers. It is unclear whether these 53 papers informed prompt or topology revisions. As presented, the paper provides no clearly independent held-out evaluation to support its main performance claims.
-
I cannot reconcile Table 4 with the metric definitions. If the three metrics use the same verdicts and denominators and differ only in PAR credit, Partial F1 should equal the midpoint of Main and Strict, even after macro-averaging. For fixed-topology DeepSeek-V4-Pro, this gives 0.798, not the reported 0.729. Either the scoring procedure needs further explanation or the table needs correction.
Questions for authors
-
Were any Eval 53 papers used to tune prompts, revise the topology, or diagnose errors during development? When was this subset selected, and was it kept separate from those steps?
-
Do the three metrics in Table 4 differ only in the credit assigned to PAR? If so, why is Partial F1 not the midpoint of Main and Strict? If the matching rules or denominators also change, please clarify those differences.