acceptodds
Under review as a conference paper at ICLR 2027

StructVista: A Benchmark for Molecular Reconstruction and Assembly Reasoning in Scientific Figures

Abstract

Accurate interpretation of molecular structures in scientific figures is fundamental to molecule-centric literature mining and scientific discovery. Existing chemical structure recognition and Markush parsing methods can recover molecules when each molecule is drawn completely or when its components are linked by explicit correspondences. Beyond these settings, many real figures provide the structural components needed to recover the target molecule, but do not explicitly specify how those components map onto the final molecular graph. Correctly reconstructing such molecules requires figure-level integration and structural reasoning. To characterize this transition, we introduce , a benchmark of 419 expert-verified figures specifying 3,412 compound identifier–structure pairs, organized into three levels ranging from local self-contained recognition, to global instantiation with explicit relations, and finally to context-dependent assembly reasoning. We benchmark seven frontier multimodal large language models and two agent frameworks equipped with chemistry tools. When queried directly, the best-performing model, GPT-6 Sol, achieves only 50.8 F1 overall and exactly recovers 26.5% of figures. Deployed within the Codex framework, the same model improves to 89.2 F1 and 74.9% exact figure recovery. Even so, it exactly recovers only 62.9% of figures requiring context-dependent assembly reasoning, compared with 81.7% of figures with self-contained depictions. These results show that current models still struggle with chemical structure reasoning, particularly when implicit assembly relations must be inferred from figure context. thus provides a testbed for moving molecular structure interpretation beyond local recognition toward figure-level reconstruction and reasoning.

Primary area:
datasets and benchmarks
Keywords:
Chemical Structure Reasoning, Scientific Foundation Model

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_dETpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces StructVista, a benchmark for reconstructing molecular structures from scientific figures. It contains 419 expert-verified figures and 3,412 compound–structure pairs, divided into three levels: local structure recognition, global reconstruction with explicit assembly relations, and context-dependent reconstruction requiring implicit assembly reasoning. The authors evaluate multimodal language models and tool-augmented agents, showing that current systems struggle with complete figure-level reconstruction, particularly when structural information must be integrated across different regions.

Critical strengths

Useful benchmark construction. The dataset is carefully curated and expert-verified, with a meaningful distinction between local recognition, explicit structural instantiation, and context-dependent assembly.

Systematic evaluation. The evaluation of direct multimodal models and tool-augmented agents provides useful evidence about current limitations in extracting complete molecular structures from scientific literature. The authors also investigate the contributions of recognition errors and target multiplicity.

Critical weaknesses

Limited demonstrated scientific significance. While extracting molecular structures from scientific figures is practically useful, the paper does not convincingly establish why this particular benchmark represents a substantial advance for machine learning research. The contribution is primarily dataset construction and evaluation of existing models, with limited methodological or conceptual insights beyond identifying another challenging extraction setting. The paper would benefit from a stronger argument for the broader significance of this task beyond literature digitization.

Insufficient formalization of molecular assembly reasoning. The central distinction between L2 and L3 relies on whether assembly relations are explicitly specified or inferred from context, but these relations are described largely in natural language. A more rigorous formulation in graph or grammar notation could characterize the structural operations involved, such as fragment substitution, attachment-site matching, and graph fusion. Relevant precedents include Representing Molecules as Random Walks Over Interpretable Grammars (ICML 2024) and Foundation Molecular Grammar (ICML 2025), which demonstrate how molecular composition can be represented through interpretable graph grammars and how foundation models can interact with such representations. The paper should clarify why it evaluates only final SMILES reconstruction rather than explicitly representing intermediate assembly operations. Without this formalization, it is difficult to isolate failures of molecular assembly reasoning from bottlenecks in representation/generation.

Questions for authors
  1. What broader ML research questions does StructVista enable beyond evaluating molecular structure extraction from scientific figures? How does the benchmark's difficulty translate into scientific significance or methodological insights?

  2. Can the authors formalize L3 assembly reasoning using graph operations or a molecular grammar, explicitly defining the relations and transformations that must be inferred? Why is the task formulated solely around final SMILES outputs rather than an intermediate graph-based assembly representation? Would evaluating foundation models with explicit molecular grammar representations, as explored in the ICML 2024 and 2025 works mentioned above, provide a more direct assessment of their assembly reasoning capabilities?

5 comments
AuthorsOpenReview ↗

Response to Reviewer dETp (1/2)

We thank the reviewer for the constructive suggestions on significance and formalization. We have clarified how StructVista supports research on native molecular understanding, using controlled experiments to connect its difficulty to model capabilities. We also formalize assembly as graph operations and explain how intermediate representations can complement final-graph evaluation. We welcome any further questions or suggestions.

W1 and Q1: Broader significance for machine learning.

The broader objective of StructVista is to study native molecular understanding: whether a multimodal model can interpret molecular entities and their relationships from visual evidence and recover the structure they jointly specify. This connects local perception to compositional reasoning. A model must determine what each component represents, what is inherited or replaced, and which interfaces correspond. Molecular graphs make the outcome of these decisions precisely testable.

To establish why this capability merits separate study, we added an experiment that fixes molecular identity, molecular size, and target count while varying how the same structures are presented. For 1,379 targets from 105 figures, we constructed complete L1 depictions, explicit L2 decompositions, and L3 depictions retaining context-dependent relationships. Direct GPT-6 Sol, evaluated with the same prompt and configuration and one output per condition, obtains 55.53, 54.32, and 34.35 F1, respectively. L3 is 20.67 points below L2 (95% paired figure-bootstrap interval [9.20, 32.05]). Thus, the underlying molecules alone cannot explain the difficulty: performance also depends on how structural information and relationships are expressed.

A second experiment examines how this capability interacts with external tools. On all 419 original figures, removing the chemistry toolkit while retaining the Codex backbone, code execution, and iteration reduces overall F1 from 83.77 to 78.98. Tools provide clear gains on L1 and L2, while the L3 comparison does not resolve a benefit. This suggests that access to recognition and validation tools does not automatically yield reliable reconstruction from implicit relationships.

Together, these results motivate models that develop native molecular interpretation alongside tool use, so that they can determine the structural decisions a tool must execute and assess whether the resulting graph matches the figure. StructVista provides an evaluation setting for studying this interaction between perception, compositional interpretation, and tool-assisted construction. We have strengthened this motivation in Sections 1 and 5.4 and report the experiments in Appendix L.1–L.2, Tables 18–19.

AuthorsOpenReview ↗

Response to Reviewer dETp (2/2)

W2 and Q2: Graph formalization and intermediate assembly representations

We have added a graph-based formulation in Appendix H, Equations (2)–(5). For figure II and target tt, reconstruction is expressed as

Gt=A(F(I),zt),G_t = \mathcal{A}(\mathcal{F}(I), z_t),

where F(I)\mathcal{F}(I) contains component graphs and templates grounded in the figure, and ztz_t specifies their roles, replacement domains, attachment correspondences, and shared interfaces. Role/reference selection determines which components are used and what they represent. The graph transformations are:

1. Subgraph replacement: remove a region UU from reference graph GG, insert replacement graph HH, and reconnect the retained structure through boundary bonds BβB_\beta:

= \bigl(G[V(G)\setminus U]\sqcup H\bigr)+B_\beta.$$ **2. Attachment:** add specified labeled bonds $B$ between selected atoms of component union $K$, including ring closure: $$\operatorname{Attach}(K;B)=(V(K),E(K)\cup B).$$ **3. Shared-interface fusion:** identify atoms or edges that depict the same molecular entities through a compatible correspondence $\phi$: $$\operatorname{Fuse}(G,H;\phi)=(G\sqcup H)/{\sim_\phi}.$$ Here $\sqcup$ denotes disjoint union; the operations preserve the required chemical labels and stereochemical constraints. This makes the L2/L3 distinction precise: both can use the same operations, but L2 supplies their arguments explicitly, whereas L3 requires at least one argument to be inferred from figure context. **Final structures and intermediate evaluation.** SMILES provides a common output interface. Scoring compares parsed, standardized molecular structures and their identifiers, allowing different valid SMILES, decompositions, and construction orders. This supports comparisons among direct prediction, programmatic assembly, and grammar-based approaches. Intermediate representations would provide complementary evidence about which structural decisions a model gets right. We have incorporated the suggested [ICML 2024 grammar work](https://proceedings.mlr.press/v235/sun24c.html) and [Foundation Molecular Grammar](https://proceedings.mlr.press/v267/sun25aa.html). Their compositional representations motivate an extension in which a model receives reference component graphs, predicts $z_t$, and a deterministic assembler produces the final graph. Evaluating both the relations and the assembled graph would reduce recognition and SMILES-generation burdens and more directly assess assembly decisions. Scoring would accommodate atom renumbering, symmetry, and alternative valid programs. The additional challenge in StructVista is grounding the components and operation arguments in visual evidence. Appendix H connects this task to molecular grammars and specifies the proposed intermediate diagnostic; the completed matched-presentation experiment provides complementary end-to-end evidence about the effect of implicit relationships. Overall, we thank the reviewer for the constructive feedback and helpful suggestions. We would be happy to address any remaining questions or concerns.
Reviewer_dETpOpenReview ↗

Reply to Authors

Thanks for the additional experiments and formulation. Your proposed extension to have the model predict the assembly program is the missing L3 result. All baselines clearly struggle with L3 using SMILES, but without the graph formalization we cannot conclude if their failures are due to the implicit information, or the ill representation. Now that you have the formalization, please repeat the L3 experiments to see whether your conclusions still hold.

Reviewer_ddniOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents StructVista, a benchmark for reconstructing molecules from figures in scientific publications. Given a figure, a system must output every labeled compound shown on it as an (identifier, SMILES) pair, including all members of compound series defined through a shared scaffold and variable substituents. The benchmark contains 419 figures from 244 articles and 3,412 expert-verified identifier–structure pairs. The figures were selected from 29,124 candidate images found in 8,175 articles from 265 journals: domain experts selected the candidates, the reference annotations were first drafted by a Codex-based assistant using supplementary information and ChEMBL records, and were then independently verified by two experts. The authors evaluate seven multimodal large language models under direct prompting (DeepSeek V4.1 Flash, GPT-6 Luna, GPT-6 Sol, Gemini 3.8 Flash, Qwen 3.8 27B, Seed 2.1 Pro, MiMo 2.6 Pro) and agentic frameworks with chemistry tools (Codex with GPT-6 Luna or GPT-6 Sol, and Cursor with Composer 2.5), which can call a molecule detector (MolDetv2), a structure recognizer (MolParser-Mobile) and RDKit. Two metrics are used: pair-level F1 (a predicted identifier–SMILES pair is counted as correct only if both of its parts match a reference pair) and exact figure recovery (EFR), the share of figures for which the predicted set of pairs coincides exactly with the reference set. The best result is obtained by the agentic framework Codex with GPT-6 Sol (overall F1 89.2, EFR 74.9%). The best model under direct prompting is GPT-6 Sol (F1 50.8, EFR 26.5%), which improves to 73.0 F1 / 56.6% EFR when region-level detection and OCSR results are added to the input. This additional recognition information helps most on L1 and least on L3, so the L1–L3 gap widens, and even the best system exactly recovers only 62.9% of L3 figures, compared with 81.7% of L1 figures. The authors conclude that current models and agents still cannot reliably reconstruct molecules from whole figures, especially when assembly relations have to be inferred from context, and propose StructVista as a testbed for the transition from local structure recognition to figure-level reconstruction and reasoning.

Critical strengths
  1. An expert-verified benchmark for a realistic and practically important task. Existing OCSR and Markush-structure benchmarks evaluate isolated structure crops or templates, whereas StructVista evaluates the reconstruction of every labeled compound of an entire figure, bound to its identifier, and separates cases by the locality of the information and the explicitness of assembly relations. The large amount of manual work (29,124 candidates reviewed, 419 figures, 3,412 pairs, two independent expert annotators) has produced a resource that significantly simplifies the evaluation of models that convert structures from figures in scientific publications into SMILES; this is important for knowledge extraction from the literature in chemistry and drug discovery.
  2. A broad and informative evaluation: seven frontier MLLMs and three agentic configurations under direct prompting, with OCSR augmentation, and with tools, supported by detailed analysis in the appendices (strict scoring, bootstrap intervals, error categories, diagnostics on rendered molecules). It gives a clear picture of the current state of the art, shows that the L1–L3 gap emerges when recognition results are supplied, and shows that agents with tools deliver a large improvement.
Critical weaknesses
  1. The paper attributes L3 errors to a lack of assembly reasoning, but does not adequately distinguish this explanation from recognition errors or differences in the number of targets per figure. After standardization, the performance gap for the best-performing system shrinks to approximately 4 percentage points, weakening the evidence that L3 reveals a distinct reasoning limitation. This undermines the paper’s main contribution.
  2. The annotation protocol may favor the model family used for preliminary annotation, raising concerns about the independence of the evaluation. Furthermore, all results are based on a single run, with no assessment of variability across repeated runs. Together, these issues make it difficult to determine whether the reported findings are robust or partly artifacts of the annotation and evaluation setup.
Questions for authors
  1. Only 419 of the 29,124 candidate images (about 1.4%) were included in the benchmark. Please report the full selection funnel: how many candidates were removed at each step and why, and how many figures at each level passed expert screening before L1 and L2 were subsampled. This is needed to judge how representative StructVista is of the real literature.
  2. Section 3.3 states that L3 figures were collected first and that L1/L2 figures were then selected randomly to obtain 180/134/105 figures. How were these sizes chosen, and what are the natural frequencies of the three levels among the screened figures? Could the exhaustive collection of L3 combined with random subsampling of L1/L2 affect representativeness (journals, years, publishers, areas of chemistry) or the comparison between levels?
  3. A lot of candidate figures are excluded because the information needed to determine the structures is contained in the main text or supplementary material rather than in the figure. Real literature-mining pipelines (e.g., OpenChemIE, BioMiner) use this text. Why was the figure-only setting chosen, what share of otherwise relevant figures does it exclude, and would the authors consider adding a text-augmented track or at least discussing the scope that this restriction leaves out?
  4. The preliminary annotation was obtained from a Codex-based assistant, and model-output inspection informed annotation verification (Appendix B). What fraction of the AI-proposed pairs did the experts correct, and are Codex-based systems more likely to agree with the AI-proposed labels? Could the main results be reported on a subset annotated without access to AI proposals and model outputs?
  5. Since levels are assigned per figure and the standardized L1–L3 gap for Codex with GPT-6 Sol is only about 4 points (Table 12), what evidence supports that the difficulty of L3 stems from assembly reasoning rather than from the number of targets or the size of the molecules? Could the authors report results for the targets that genuinely require an inferred relation and/or an oracle experiment in which the model is given the correct structures of the components?
  6. Are the results of the agentic and direct systems stable across repeated runs? And what drives the improvement of the agents – iterative execution, code writing, or access to the tools? An ablation (e.g., Codex without the chemistry tools, or a direct model given the same tools in a single pass) would clarify this.
3 comments
AuthorsOpenReview ↗

Response to Reviewer ddni (1/3)

We thank the reviewer for the detailed suggestions on validity and robustness. Our revision addresses the main validity concerns with a matched-target experiment, reference-sensitivity checks, repeated Codex Sol runs, and a chemistry-tool ablation. We also clarify how the benchmark was selected and what the figure-only setting evaluates. We welcome any further questions or suggestions.

W1 and Q5: The paper attributes L3 errors to a lack of assembly reasoning, but does not adequately distinguish this explanation from recognition errors or differences in the number of targets per figure. After standardization, the performance gap for the best-performing system shrinks to approximately 4 percentage points, weakening the evidence that L3 reveals a distinct reasoning limitation. This undermines the paper’s main contribution.

We added a matched experiment to test whether the difficulty persists when molecular identity, molecular size, and target count are held fixed. The approximately four-point difference in Table 12 concerns figure-level EFR after target-count standardization. The new experiment directly compares different presentations of the same molecules, using pair-level F1.

For each included L3 figure, we constructed three versions: L1 shows the complete target molecules; L2 gives their decomposition with explicit interfaces; and L3 retains the source-specific implicit relationships. All three use rendered molecular depictions and the same direct GPT-6 Sol API configuration, low reasoning effort, task prompt, and scorer, without tools. The cohort contains 105 figures, with one output per figure and condition.

PresentationRecall (%)F1
L1: complete molecules53.3055.53
L2: explicit decomposition51.5654.32
L3: implicit relations32.7834.35

The L2–L3 F1 difference is 19.97 points (95% paired figure-bootstrap interval [9.20, 32.05]), while L1–L2 differs by 1.24 points [−8.08, 10.16]. Thus, differences in molecular content or target count cannot explain this matched comparison. The result supports a substantial challenge associated with context-dependent presentation. It measures the combined effect of relation explicitness, layout, and visibility; isolating relation prediction with correct component graphs would require the complementary oracle diagnostic described in Appendix H. We report the completed experiment in Section 5.4 and Appendix L.1, Table 18.

We also examined saved execution records for all 39 L3 figures with unmatched targets in the original Codex Sol run. The records reveal failures involving attachment correspondence, omitted intermediate fragments, and template bonds, showing how errors can arise at different reconstruction stages. This process evidence helps interpret the aggregate results without assigning every L3 mismatch to a single cause. Appendix M.2 reports the analysis, and Table 3 now explicitly describes mismatch categories rather than verified error causes.

AuthorsOpenReview ↗

Response to Reviewer ddni (2/3)

W2, Q4, and Q6: The annotation protocol may favor the model family used for preliminary annotation, raising concerns about the independence of the evaluation. Furthermore, all results are based on a single run, with no assessment of variability across repeated runs. Together, these issues make it difficult to determine whether the reported findings are robust or partly artifacts of the annotation and evaluation setup.

We examined three aspects of robustness: the reference annotations, variation across executions, and the contribution of the chemistry toolkit.

1. Expert verification and reference sensitivity.

All proposed reference pairs underwent independent review by two experts, who checked identifiers, molecular graphs, and target coverage, corrected proposals, and resolved disagreements. The reported 96.3% is their pre-discussion agreement, rather than agreement between the benchmark and the initial AI proposals.

We added two checks of the annotation evidence. First, archived review records covering 50 distinct targets in five L3 figures document ten structural corrections linked to human comments. These records demonstrate substantive expert correction.

Second, we tested whether the main comparisons depend on the later reference revisions. Keeping predictions from all 17 configurations frozen, we rescored them against the current references, the pre-revision references, references excluding the 130 revised targets, and figures unaffected by those revisions:

Reference populationFigures / targetsDirect Sol F1Augmented Sol F1Codex Sol F1
Current419 / 3,41250.7973.0189.21
Pre-revision419 / 3,40849.4070.8186.28
Excluding revised targets411 / 3,28250.8973.3289.53
Unaffected figures344 / 2,60350.3574.4189.01

The ordering by overall F1 remains unchanged within each evaluation setting—Direct, Augmented, and Agentic—across all four reference populations. This supports the robustness of the model comparisons to the recorded revisions.

On the model-family concern: the GPT-6 Sol backbone reaches only 50.8 F1 when queried directly and 73.0 with OCSR augmentation, and Codex with GPT-6 Luna reaches 47.9, so sharing a model family does not by itself yield agreement with the references. Excluding all revised targets leaves Codex Sol at 89.53, and the review records show experts correcting the AI drafts. We provide the records and results in Appendix K, including Table 17.

2. Stability across repeated runs. We added two full-benchmark Codex Sol executions on the original 419 figures and 3,412 targets and report them alongside the original run:

RunL1 F1L2 F1L3 F1Overall F1Overall EFR (%)
Original retained run90.8190.1187.5089.2174.94
Additional run 190.1183.4079.4383.2169.93
Additional run 289.0587.3577.6283.7769.69
Mean ± sample SD89.99 ± 0.8986.95 ± 3.3781.51 ± 5.2685.40 ± 3.3171.52 ± 2.96

The two additional runs differ by 0.56 overall F1 points, and L3 has the lowest F1 in all three runs. The mean and SD summarize cross-run variation. These results appear in Appendix L.3, Table 20.

3. Contribution of chemistry tools. To measure how much specialized tools contribute, we reran all 419 figures with MolDetv2, MolParser-Mobile, and RDKit removed. Native image inspection, general code execution, and iterative interaction remain available. The comparison holds the backbone, client, reasoning effort, and 600-second budget fixed:

LevelWith chemistry tools F1Without chemistry tools F1
L189.0579.72
L287.3579.06
L377.6278.53
Overall83.7778.98

Tools improve overall F1 by 4.78 points (95% paired source-paper bootstrap interval [0.80, 9.12]) and EFR from 60.86% to 69.69%. The F1 gains are 9.33 points on L1 and 8.29 on L2; the L3 difference is −0.91 [−7.31, 6.53], with no clear benefit in either direction. All 419 figures and 3,412 targets remain in both conditions, including failed executions, and the execution audit found no use of the removed tools.

This comparison isolates the availability of the chemistry toolkit while retaining the agent's ability to write code and iterate. It shows that tools contribute substantially overall, but their gains differ across reconstruction requirements. Appendix L.2, Table 19 reports the ablation; Appendix M documents how agents use iterative checking and programmatic construction in practice.

AuthorsOpenReview ↗

Response to Reviewer ddni (3/3)

Q1 and Q2: Selection records, level allocation, and representativeness.

We first established the eligible L3 cohort, requiring readable, figure-contained information sufficient to recover complete structures and their identifiers. We then selected L1 and L2 figures for complementary coverage, source diversity, and reference completeness. The resulting 180/134/105 allocation supports comparison across reconstruction requirements; it does not estimate their natural frequencies in the literature.

The recorded collection and selection counts are:

StageCandidates and selection
Initial collection29,124 cumulative candidate entries
L3 inventory108 candidates; two exclusions and one figure without an eligible complete target; 105 retained
L1 candidate set306 figures: 95 prioritized complete-reference figures and 199 eligible supplements; 95 + 85 selected, giving 180; 114 eligible supplements unselected and 12 outside the selection criteria
L2 candidate set192 figures; 134 meet the source-completeness rule and 58 do not; 134 retained

We have clarified the source-coverage and reference-completeness rules in Section 3.3 and provided the recorded counts in Appendix K. Accordingly, the benchmark characterizes the selected reconstruction requirements, while the matched-presentation experiment controls molecular content directly when comparing levels.

Q3: A lot of candidate figures are excluded because the information needed to determine the structures is contained in the main text or supplementary material rather than in the figure. Real literature-mining pipelines (e.g., OpenChemIE, BioMiner) use this text. Why was the figure-only setting chosen, what share of otherwise relevant figures does it exclude, and would the authors consider adding a text-augmented track or at least discussing the scope that this restriction leaves out?

The figure-only setting tests whether models can interpret molecular structures and relationships from a common, self-contained source of visual evidence. Inclusion requires the figure to contain enough information to recover the complete target structures and identifiers. Article text, supplementary information, and database records support reference verification but are not supplied to evaluated systems.

Figures whose structural definitions are available only outside the figure fall beyond this setting. A text-augmented track would address the complementary problem of integrating visual structures with article-level definitions and retrieved evidence. It could retain the same identifier–graph outputs while expanding the available inputs, allowing the two capabilities to be studied together. We have clarified this scope and extension in Section 3.1 and Appendix K.

Overall, we thank the reviewer for the constructive feedback and helpful suggestions. We would be happy to address any remaining questions or concerns.

Reviewer_NKqLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces StructVista, a benchmark for evaluating multimodal LLMs and agent systems on molecular structure reconstruction from scientific figures. Its main contribution beyond existing OCSR benchmarks is the inclusion of context-dependent assembly reasoning: structural components may be distributed across a figure, and the relationships needed to assemble them must be inferred from context. The benchmark contains 419 expert-verified figures, grouped into three levels: local, self-contained reconstruction (L1); global reconstruction with specified relations (L2); and global reconstruction requiring relations to be inferred from context (L3). The authors evaluate a range of frontier MLLMs, as well as two agent harnesses—Codex and Cursor—with access to chemistry tools. Their results show that tool-using agent workflows improve reconstruction performance over direct prompting, while L3 remains challenging.

Critical strengths
  • The L3 setting—context-dependent assembly reasoning—is well motivated and appears underexplored in prior OCSR work.

  • The benchmark is carefully curated and expert-verified. Its figures were collected from 8,175 articles across 265 journals, a substantial source corpus.

  • The evaluation covers a broad range of frontier models and agent harnesses. The results are clearly presented and offer useful insights into where tool-using workflows help and where challenges remain.

  • The paper is clearly written and easy to follow.

Critical weaknesses
  • The L3 subset contains 105 figures, but the paper does not quantify how these examples are distributed across the different types of context-dependent assembly relations it identifies. A breakdown by relation type would help readers assess the diversity and representativeness of the subset.

  • The main novelty appears to lie in the L3 formulation and benchmark, while L1 and L2 overlap with established OCSR and Markush-instantiation settings. The paper could more clearly explain what new capabilities or evaluation insights the benchmark provides beyond these existing settings.

Questions for authors

See the Weaknesses section.

1 comment
AuthorsOpenReview ↗

Response to Reviewer NKqL

We thank the reviewer for the constructive feedback. We now quantify the four relation types across all 105 L3 figures and use the matched-target experiment to explain the capability evaluated beyond local recognition and explicit Markush instantiation. These additions clarify the diversity of L3 and its value for studying native molecular understanding. We welcome any further questions or suggestions.

W1: The L3 subset contains 105 figures, but the paper does not quantify how these examples are distributed across the different types of context-dependent assembly relations it identifies. A breakdown by relation type would help readers assess the diversity and representativeness of the subset.

We have added a breakdown of all 105 L3 figures according to the structural decision that must be inferred from context. The four categories describe what the model needs to determine to reconstruct the target:

Relation typeRequired structural decisionFigures
Structural role and referenceWhat a depicted region represents and which structural reference applies12 (11.4%)
Replacement and inheritance scopeWhich subgraph is replaced and which surrounding structure is retained40 (38.1%)
Attachment correspondenceWhich atoms or fragment ends connect, including their orientation46 (43.8%)
Shared-atom or shared-edge fusionWhich depicted atoms or edges represent the same graph elements7 (6.7%)
Total105 (100.0%)

Each figure receives one primary category, so these percentages sum to 100%. Figures can also involve additional relations; the corresponding multi-label counts are 13, 44, 61, and 7.

The distinction is between the structural decision and the visual evidence supporting it: color, spatial orientation, legends, or structural similarity can indicate a relationship whose precise molecular interpretation must still be inferred. The distribution therefore characterizes several forms of context-dependent reconstruction, including replacement, attachment, and fusion, rather than grouping figures only by their visual cues.

We provide the definitions in Appendix H, the distribution in Appendix I, Table 16, and a source-linked visual example of each category in Appendix J.

W2: The main novelty appears to lie in the L3 formulation and benchmark, while L1 and L2 overlap with established OCSR and Markush-instantiation settings. The paper could more clearly explain what new capabilities or evaluation insights the benchmark provides beyond these existing settings.

The key capability introduced by L3 is native molecular understanding: interpreting the components shown in a figure and inferring the roles, scopes, and interfaces needed to recover the target molecular graph. L1 and L2 provide complementary reference settings for local recognition and explicitly specified assembly. L2 and L3 can involve the same graph operations; their distinction is whether the required arguments are supplied explicitly or must be inferred from visual context. Sections 3.2 and 5.4 now make this capability and its evaluation role more explicit.

To test whether this distinction produces a measurable challenge beyond differences in molecular content, we constructed three presentations of the same 1,379 targets from 105 figures: complete L1 structures, L2 decompositions with explicit interfaces, and L3 depictions retaining implicit relationships. All three use rendered molecular depictions. Direct GPT-6 Sol was evaluated without tools, at the same low reasoning effort, with one output per figure and condition and the same prompt and scorer.

The resulting F1 scores are 55.53 for L1, 54.32 for L2, and 34.35 for L3. L3 is 19.97 points below L2 (95% paired figure-bootstrap interval [9.20, 32.05]), even though molecular identity, molecular size, and target count are fixed (Appendix L.1, Table 18).

These results help explain the added value of L3: recovering the same molecules becomes substantially harder when their construction depends on interpreting contextual relationships. The benchmark therefore enables research on how models connect visual evidence to molecular structure, including determining what to inherit, replace, connect, or fuse. This motivates developing native molecular understanding alongside recognition and tool use, and provides a concrete way to evaluate progress toward that capability.

Overall, we thank the reviewer for the constructive feedback and helpful suggestions. We would be happy to address any remaining questions or concerns.

Reviewer_uPJ6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces StructVista, a benchmark for evaluating how well multimodal LLMs can reconstruct molecular structures from scientific figures. Unlike existing benchmarks that focus on recognizing individual molecules, StructVista also tests whether models can combine information scattered across a figure and infer how different molecular components fit together. The authors evaluate seven multimodal LLMs and two agent frameworks, Codex and Cursor, equipped with specialized chemistry tools. Direct LLM predictions perform relatively poorly, but using agentic frameworks substantially improves performance. The benchmark uses two main metrics, F1 (individual molecular reconstruction) and EFR (exact figure recovery), and demonstrates that complete figure recovery remains much more challenging, especially for Level 3 (L3) figures. Overall, the results highlight the limitations of current models in molecular structure recognition and figure-level reasoning, while demonstrating the benefits of integrating specialized tools into agentic workflows.

Critical strengths
  1. The paper benchmarks several state-of-the-art frontier models and identifies key limitations in extracting molecular structures from figures in scientific articles.

  2. The authors categorize common error patterns (Table 3) and provide some representative failure cases in the appendix.

  3. The paper is generally clear and easy to follow, and the benchmark offers new insights compared to previous studies.

  4. The categorization of figures into the proposed three levels is reasonable.

  5. AI-assisted annotation workflow follows a thorough human verification with two independent experts.

Critical weaknesses
  1. The scope of the current benchmark does not include new methodological contributions aimed at improving reconstruction performance.

  2. The study did not explore different harness configurations and self-improvement strategies to better understand and potentially improve performance on this task.

  3. There is no detailed comparison of inference cost, token usage, or runtime between direct MLLMs and agentic frameworks.

  4. It is not fully clear whether humans can achieve near-perfect scores, especially for the L3 category. A human baseline evaluated on a smaller subset of figures under the same conditions could strengthen the analysis.

  5. The paper does not analyze whether variations in image resolution and quality across samples affect the results.

  6. (Minor) The plots in Figure 3 are missing y-axis labels. Although the quantities are explained in the captions, the plots are harder to interpret without them.

Questions for authors
  1. Why were only 419 figures included in the final benchmark out of the 29,124 candidate images? Could the authors clarify why so many figures were left out (e.g. what kind of failures were observed)?
  2. For the agentic frameworks, could examples of the full trajectories or their summaries be included in the appendix, covering both successful and failed cases?
5 comments
AuthorsOpenReview ↗

Response to Reviewer uPJ6 (1/3)

We thank the reviewer for recognizing the value of the reconstruction task, the three-level formulation, and the expert verification. We have strengthened the paper with controlled experiments, an analysis of agent execution, and a clearer account of the benchmark's contribution. Changes are marked in blue in the revised manuscript.

W1: The scope of the current benchmark does not include new methodological contributions aimed at improving reconstruction performance.

StructVista contributes a task formulation and evaluation framework for studying native molecular understanding: interpreting molecular components and their relationships from visual evidence. To establish the implications for model development, we added two controlled comparisons and an analysis of the agents’ construction programs.

1. Matched molecular targets across presentation levels. To examine whether reconstruction difficulty depends on how structural information and assembly relations are presented, we constructed three presentations of the same molecular targets from 105 figures. L1 shows each complete molecule; L2 decomposes the targets into components with explicit interfaces; L3 redraws the source-specific presentation while retaining the implicit relationships that must be inferred. All three conditions use rendered molecular depictions and hold molecular identity, molecular size, and target count fixed.

We evaluated direct GPT-6 Sol by API, without tools, using the same low reasoning effort, task prompt, and scorer, with one output per figure and condition:

Presentation of the same targetsF1
L1: complete molecular structures55.53
L2: components with explicit assembly relations54.32
L3: components with context-dependent relations34.35

L1 and L2 have similar observed scores, while L3 is 19.97 points below L2 (95% paired figure-bootstrap interval [9.20, 32.05]). The gap therefore persists with identical target molecules and counts. This identifies a substantial challenge in reconstructing molecules from their context-dependent visual presentation (Section 5.4; Appendix L.1).

2. Chemistry-tool ablation. To measure how specialized tools contribute across reconstruction requirements, we evaluated Codex Sol on all 419 original figures and 3,412 targets with and without MolDetv2, MolParser-Mobile, and RDKit, retaining image inspection and general code execution. The comparison holds the backbone, client, reasoning effort, and 600-second budget fixed. Tools increase overall F1 from 78.98 to 83.77, with gains of 9.33 points on L1 and 8.29 on L2. The L3 difference is −0.91 points [−7.31, 6.53], so this comparison does not resolve an L3 benefit. This establishes the toolkit’s overall value while showing that its measured gains differ across reconstruction requirements (Appendix L.2).

3. Implications for model design. The matched-target experiment helps explain what makes this task challenging. With molecular identity and target count held fixed, L1 and L2 achieve similar scores, whereas L3 is 20.67 F1 points below L2. Thus, the underlying molecules alone cannot explain the difference: performance also depends on how structural information and assembly relationships are expressed in the figure. This highlights the importance of interpreting visual context to determine which components belong together and how they form the target structure.

The toolkit ablation provides complementary evidence: chemistry tools yield clear gains on L1 and L2, while no clear L3 gain is observed in this comparison. This suggests that access to recognition and validation tools does not automatically translate into reliable context-dependent reconstruction. These findings motivate developing native molecular understanding alongside tool use, so that models can infer component roles, replacement scopes, and attachment correspondences, then use tools to construct and check the resulting graphs. StructVista provides a concrete setting for evaluating progress toward this capability.

Together, these findings motivate models with native molecular understanding that can ground component roles and attachment correspondences in the figure, and verify the assembled graph against that visual evidence. StructVista makes these capabilities concrete evaluation targets and provides controlled comparisons for studying their interaction with recognition tools. We have strengthened this contribution in the introduction and Section 5.4.

AuthorsOpenReview ↗

Response to Reviewer uPJ6 (2/3)

W2: The study did not explore different harness configurations and self-improvement strategies to better understand and potentially improve performance on this task.

Evaluation across different harnesses. Our original evaluation covers two agent harnesses, Codex and Cursor, with three model–harness configurations on the full 419-figure benchmark (Table 2):

HarnessBackboneOverall F1Overall EFR (%)
CodexGPT-6 Luna47.930.8
CodexGPT-6 Sol89.274.9
CursorComposer 2.551.036.8

These evaluations use the harnesses’ native multi-step workflows, with image inspection, code execution, and access to MolDetv2, MolParser-Mobile, and RDKit within a 600-second per-figure budget. Agents can inspect tool outputs, construct molecular graphs, validate and render them, and revise their predictions. The table compares complete model–harness configurations; implementation details appear in Appendix C.

Self-checking and correction in the evaluated workflow. We further examined the recorded trajectories to determine whether these capabilities were actually exercised. Across all 419 original Codex Sol sessions, executed commands document explicit validation checks in 188 sessions (44.9%), molecular rendering in 155 (37.0%), and checking after a draft answer in 48 (11.5%). These categories overlap.

The trajectories additionally show concrete revisions following inspection or validation feedback. For example, Figure 10 documents a reconstruction in which the agent identifies a conflict between an OCSR candidate and the source figure, enlarges the relevant region, constructs an alternative parent structure, validates and renders it, and then expands the series to recover all 14 targets.

Thus, the evaluated workflow already includes within-task self-improvement through autonomous checking and correction, with direct execution evidence that these capabilities are used. We have made this behavior explicit in Appendix M, which provides chronological summaries of two successful and two incomplete reconstructions, including cases where errors persist after checking. It also reports an exploratory default-versus-restricted-review prompt comparison. Together, the harness results and trajectory analysis characterize both the systems evaluated and how they inspect and revise their outputs.

W3: There is no detailed comparison of inference cost, token usage, or runtime between direct MLLMs and agentic frameworks.

We added recorded resource use for all 17 configurations in Appendix L.5, Table 22. The GPT-6 Sol comparison is:

SettingMedian runtime (s)Mean input tokensMean output tokens
Direct19.041,7991,696
OCSR-augmented10.822,280972
Codex with tools277.44272,2494,481

Codex input includes an average of 244,532 cached tokens. Agent totals accumulate across turns and include tool execution; augmented runtime excludes offline OCSR preprocessing. These measurements make the resources accompanying agent performance explicit.

A detailed table can be found in Appendix L.5, Table 22.

W4: It is not fully clear whether humans can achieve near-perfect scores, especially for the L3 category. A human baseline evaluated on a smaller subset of figures under the same conditions could strengthen the analysis.

We conducted an expert evaluation on the full benchmark ([419 figures, 3,412 targets]). The evaluator holds a PhD in chemistry and was not involved in constructing or annotating StructVista. The evaluator was instructed to maximize accuracy and was given no time limit. As a result, the expert achieved 98.6 F1 overall and 98.9 F1 on L3. Since no time limit was imposed, these scores serve as a reference for human-achievable accuracy rather than a time-matched comparison with the 600-second agent budget.

The recoverability of these targets is also supported by our dataset construction criteria. Figure-level self-containment is required for inclusion: the information needed to determine each target’s molecular graph and identifier must be available within the figure under the stated chemical conventions. Two experts checked this requirement during annotation verification. L3 retains this information completeness while requiring the relationships between components to be inferred from contextual cues, such as color correspondence, spatial arrangement, or structural similarity. Thus, the human evaluation measures experts’ ability to interpret and integrate information already present in the figure. We have clarified the construction criteria in Section 3.3 and provided source-linked L3 examples in Appendix J.

AuthorsOpenReview ↗

Response to Reviewer uPJ6 (3/3)

W5: The paper does not analyze whether variations in image resolution and quality across samples affect the results.

We completed a resolution experiment on all 419 evaluation figures and 3,412 targets. We rendered the corresponding source-PDF regions at 150, 300, and 600 DPI and ran Codex GPT-6 Sol with the same prompts, molecular tools, medium reasoning setting, and 600-second budget, once per condition. A fresh original-image control was subsequently run with the same current core configuration:

InputOverall F1 (%)L3 F1 (%)Overall EFR (%)
Original images83.2179.4369.93
150 DPI82.0578.4268.50
300 DPI83.8679.3269.45
600 DPI82.9880.9667.30

All assigned outcomes, including execution failures, are retained. The overall 600-minus-150-DPI F1 difference is +0.94 percentage points (95% source-paper bootstrap interval: [−2.90,4.88][-2.90,4.88]). For L3, it is +2.54 points ([−4.63,10.07][-4.63,10.07]); exact L3 recovery at 600 DPI remains 51/105 figures (48.57%). Higher DPI therefore yields no consistent overall gain, and substantial L3 failures remain at 600 DPI.

We also completed analyses of native image dimensions, contrast, and sharpness proxies across all 17 configurations, which will accompany the controlled experiment.

W6: (Minor) The plots in Figure 3 are missing y-axis labels. Although the quantities are explained in the captions, the plots are harder to interpret without them.

We have added the missing y-axis labels to Figure 3 and verified that the plotted values are unchanged.

Q1: Why were only 419 figures included in the final benchmark out of the 29,124 candidate images? Could the authors clarify why so many figures were left out (e.g. what kind of failures were observed)?

The 29,124 candidates formed the initial discovery pool. We curated the final benchmark to ensure figure-level self-containment, readable structural information, clear compound identifiers, and diversity of reconstruction patterns. Exclusions included partially specified structures, unresolved variable definitions, and figures requiring structural information available only in the article text or supplementary material.

We first established the eligible L3 cohort of 105 figures. We then selected 180 L1 and 134 L2 figures for complementary coverage and a reasonably balanced comparison across levels. The resulting 419-figure benchmark therefore reflects both eligibility screening and deliberate selection for evaluation coverage. We have clarified these selection principles in Section 3.3.

Q2: For the agentic frameworks, could examples of the full trajectories or their summaries be included in the appendix, covering both successful and failed cases?

We have added chronological summaries of two successful and two incomplete reconstructions in Appendix M.1. Figure 10 (p. 35) additionally presents an illustrated trajectory showing the recorded agent–tool interactions, intermediate structures, validation, and revisions leading to the final predictions.

Overall, we thank the reviewer for the constructive feedback and helpful suggestions. We would be happy to address any remaining questions or concerns.

Comments on the paper

1 comment
Area_Chair_JHXCOpenReview ↗

Please Engage in the Discussion

Dear Reviewers,

As the authors have now submitted their rebuttal, I would kindly ask you to read their responses and engage in the discussion.

Thank you!

Best regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.