acceptodds
Under review as a conference paper at ICLR 2027

The TMEX Benchmark and a Fixed-Topology Multi-Agent Extraction Framework for Building an AI-Ready Topological Magnetism Database

Abstract

Traditional scientific databases have largely relied on experimental data collected and curated by researchers. As data-driven and AI-assisted research has expanded, large language models and multi-agent systems have made it feasible to construct AI-ready databases by extracting structured information from scientific literature at scale. However, topological-magnetism papers span diverse material systems, experimental methods, and physical descriptions, so deciding which scientific dimensions to preserve and how to organize the evidence still requires domain expertise. Together with physics experts, we develop TMEX (Topological Magnetism EXtraction), an expert benchmark that integrates a domain schema, expert-annotated gold records, and a semantic judge. Guided by TMEX, we iteratively refine a fixed-topology multi-agent workflow that routes single-material and multi-material papers, indexes figure evidence to schema fields, and assigns seven scientific dimensions to parallel specialists. The judge's field-level scores localize extraction errors and guide targeted revisions to the agent topology and prompts. Instantiated on GPT-5.5, the fixed topology reaches 0.924 F1 on TMEX (0.942 precision, 0.910 recall). On Eval 53, with the backbone held fixed, DeepSeek-V4-Pro improves from 0.773 to 0.927 F1 (+15.4) at $0.19 per paper. We release TMEX, the pipeline code, and the TMEX database of 8,082 papers.

Primary area:
datasets and benchmarks
Keywords:
large language models, scientific information extraction, datasets and benchmarks, materials database, topological magnetism

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CaS4OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces TMEX, a benchmark for extracting structured information from topological-magnetism papers. It combines an expert-defined seven-section schema, expert-annotated records for 112 papers, and a field-level semantic judge. The proposed extraction framework uses material-aware routing, figure-to-field indexing, seven parallel specialist agents, and programmatic record assembly and validation. The workflow is iteratively refined using expert analysis of extraction errors.

Validation includes multi-backbone comparisons and component ablations on Eval 53, a subset of the 112 gold papers. The paper reports an improvement from 0.773 to 0.927 F1 for DeepSeek-V4-Pro when moving from single-shot extraction to the proposed workflow. It also describes a database extracted from 8,082 papers using DeepSeek-V3.2-Exp.

Critical strengths

Potentially useful domain-specific evaluation resource. The expert-defined schema explicitly separates material identity, magnetic properties, sample preparation, experimental evidence, and theory, while requiring evidence to be associated with the appropriate material. Together with the 112 expert-annotated papers, this could provide a useful resource for evaluating scientific extraction beyond isolated entity or property recognition. This resource contribution is the strongest aspect of the submission in my assessment

Critical weaknesses
  1. Eval 53 may not be independent of development. Figure 1 says the workflow is frozen once extraction on the 112 gold papers becomes stable, yet Eval 53 comes from those same papers. It is unclear whether these 53 papers informed prompt or topology revisions. As presented, the paper provides no clearly independent held-out evaluation to support its main performance claims.

  2. I cannot reconcile Table 4 with the metric definitions. If the three metrics use the same verdicts and denominators and differ only in PAR credit, Partial F1 should equal the midpoint of Main and Strict, even after macro-averaging. For fixed-topology DeepSeek-V4-Pro, this gives 0.798, not the reported 0.729. Either the scoring procedure needs further explanation or the table needs correction.

Questions for authors
  1. Were any Eval 53 papers used to tune prompts, revise the topology, or diagnose errors during development? When was this subset selected, and was it kept separate from those steps?

  2. Do the three metrics in Table 4 differ only in the credit assigned to PAR? If so, why is Partial F1 not the midpoint of Main and Strict? If the matching rules or denominators also change, please clarify those differences.

Reviewer_qikJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work addresses information extraction from scientific literature in the field of topological magnetism by constructing a comprehensive expert-curated benchmark, TMEX. The authors designed a fixed-topology multi-agent pipeline, conducted rigorous controlled experiments, and released a large-scale dataset. The study is thorough, the experiments are extensive, and it holds significant value for the field.

Critical strengths
  1. The field of topological magnetism lacks a structured database tailored for machine learning. TMEX addresses this gap by providing a domain-specific schema, expert-curated gold standards, and field-level semantic evaluators.
  2. The study employs a fixed backbone model to compare single-pass extraction against a multi-agent pipeline, with ablation studies isolating the contributions of individual modules. Furthermore, by benchmarking against a "free-form" multi-agent baseline, the authors compellingly demonstrate that performance gains stem from the domain-specific organizational topology rather than merely the underlying model's inherent capabilities. The work ensures high reproducibility by providing comprehensive reporting on computational costs, instances of null extractions, and multiple scoring criteria.
  3. Beyond establishing a benchmark, the project processes 8,082 research papers and provides an open-access database, source code, and an interactive browser, resulting in a genuinely useful resource for scientific research.
Critical weaknesses
  1. The work lacks sufficient innovation. Concepts such as domain-specific schemas, modular multi-agent extraction, and the use of an LLM as a judge have already been extensively explored in existing literature[1-4]. The primary innovation claimed that a "fixed-topology multi-agent" system, and is essentially a manually predefined workflow rather than a new paradigm.
  2. The evaluation relies on an LLM to act as a judge, which raises concerns regarding credibility. Although some human alignment was performed using a small sample, the scale is too limited.
  3. The study lacks a comprehensive system-level comparison against existing information extraction systems for materials science[1-3]; it merely cites related work without providing direct experimental benchmarking.
  4. The large-scale database is generated entirely through model-based extraction. As there is no statistical analysis of the actual error rate based on human-verified samples, the reliability of the database cannot be substantiated.
Questions for authors
  1. Could you explain why a fixed topology leads to a significant improvement in F1 scores, whereas autonomous multi-agent scheduling results in poorer performance? What is the underlying mechanism driving this?
  2. The paper validates the evaluator’s alignment with human experts using only 466 fields. What potential biases might the semantic evaluator itself introduce, and how can the evaluation noise stemming from this component be further mitigated?
  3. Ablation experiments indicate that the seven-dimensional role-based decomposition is the most critical component for performance, while the gains from FigIndex and FieldChecker are relatively minor. What specific pain points of LLMs does this seven-parallel-agent decomposition actually address?
  4. While the model achieves high metrics on the small-scale gold-standard dataset Eval-53, the large-scale database is processed using a lower-performing version, DeepSeek-V3.2-Exp. What types of distribution shifts might arise when transitioning from small-scale gold-standard evaluation to a large corpus of real-world literature?
Reviewer_27PHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces TMEX, an expert-defined schema and benchmark for extracting structured topological-magnetism information from scientific papers. The proposed fixed-topology workflow combines material-aware routing, figure indexing, and seven specialist agents. The authors also report a database covering 8,082 papers. Experiments show extraction improvements across several backbones, although evaluation independence and the attribution of these gains require further clarification.

Critical strengths
  1. Meaningful application and resource contribution. Organizing scattered scientific evidence into structured material records is practically useful. The expert-defined schema and annotated benchmark provide a concrete basis for evaluating this task.
  2. Targeted technical design. Material-aware routing, figure-to-field indexing, and specialist extraction address practical challenges in associating multimodal evidence with the correct material and field.
  3. Encouraging experimental results. The paper reports improvements across multiple backbones, includes same-backbone comparisons and component ablations, and provides costs and alternative, stricter scoring results.
Critical weaknesses
  1. Unclear evaluation independence. The framework is refined using the 112 gold papers, while Eval 53 is selected from this collection. The authors should clarify the development/test separation and evaluate the frozen system on papers not used to refine the prompts, topology, or judge.
  2. Incomplete attribution to the expert-defined topology. Holding the backbone fixed does not control for additional calls, output capacity, or prompt specialization. Equal-budget schema-decomposition or iterative-extraction baselines would help distinguish the specific scientific organization from the general benefits of dividing a large extraction task.
  3. Limited validation of the released database. The primary metric gives full credit to underspecified answers, while stricter scores are substantially lower. Moreover, the 8,082-paper database uses a different, cheaper backbone. An independent expert audit of this database would better establish its suitability for downstream scientific use.
Questions for authors
  1. Which Eval 53 papers, if any, were excluded from every stage of prompt, topology, and judge refinement? If none were held out, can you evaluate the frozen system on an independently annotated set of previously unused papers?
  2. Does the proposed expert-defined topology outperform simple schema decomposition or iterative extraction when the backbone, total inference budget, and output capacity are comparable? Such a comparison would help isolate scientific organization from the general benefits of additional calls and smaller extraction subtasks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.