acceptodds
Under review as a conference paper at ICLR 2027

ChemDialog: Benchmarking Chemical Reaction Understanding in LLMs through Dialogue

Abstract

Despite the promise of large language models (LLMs) for chemical reaction understanding, current benchmarks rely heavily on question-answering (QA) for isolated tasks, such as molecule property prediction and stoichiometry. However, real-world chemistry applications require LLMs to engage in interactive dialogue and integrate information across related tasks. To address this gap, we introduce ChemDialog, a reaction-grounded dialogue benchmark for evaluating LLMs’ ability to integrate chemical information across dialogue turns and related tasks. Specifically, we construct ChemDialog by organizing chemical reaction subtasks into coherent dialogue scenarios with LLM assistance, followed by extensive human review for quality control. We further develop a multidimensional evaluation framework that separately assesses task-specific chemical competence and dialogue-level interaction logic. Extensive experimental results indicate that mainstream LLMs still struggle with logical consistency and dynamic information integration in multi-turn chemical dialogues, and providing multi-turn, related task dialogue history significantly enhances model performance in synthesis tasks. We hope that ChemDialog and our findings encourage the research community to further advance the untapped potential of LLMs toward complex, real-world chemical interactions. The dataset and code will be released.

Primary area:
datasets and benchmarks
Keywords:
Multi-turn chemical dialogue, Reaction-grounded benchmark, Cross-task information integration, LLM-as-a-judge validation, Interaction logic evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wP6VOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ChemDialog, a benchmark of 396 multi-turn dialogues (2,725 turns) built from 205 Open Reaction Database reactions. It covers forward synthesis, reaction procedure, and retrosynthesis. An LLM generates the dialogues from predefined task chains and inserts six types of interaction logic, such as anaphora resolution, error correction, and reliability checks. Humans (chemistry students) then reviewed them (about 200 hours; 291 dialogues modified, 20 removed).

Scoring combines exact SMILES and numeric matching with an LLM judge for open-ended answers; capability turns are scored as min(chemistry, logic). Across 13 LLMs evaluated with gold history and no reasoning, the best overall score is about 55%. The authors find that chemical accuracy does not imply interaction competence, and that gold history improves final predictions, which they read as cross-task integration. The judge is validated by rescoring with a second LLM judge and by a 20-question study with five student raters.

Critical strengths
  1. A well-motivated shift from isolated QA to interaction-based evaluation in scientific workflows. The benchmark links related reaction tasks into dialogues where later turns depend on earlier ones, and adds interaction behaviours such as error correction and reliability checks. This matters to my assessment because LLMs are increasingly used as collaborators in research, and single-turn benchmarks cannot capture whether a model carries information across steps or responds properly to user corrections. The paper’s appendix analyses, especially the model-history and thinking × history experiments, already show behaviour that static benchmarks would miss, such as large drops when models condition on their own earlier answers.

  2. Careful, traceable grounding and review. Every dialogue is anchored to an ORD reaction record, and human review is fully logged (per-turn edits, reasons, deletions with provenance) through a dedicated review platform. This matters because the dialogues are LLM-generated, so the benchmark’s credibility depends on auditability. The change log makes the dataset’s quality inspectable and correctable by others, and the review infrastructure could be reused to build similar benchmarks in other scientific domains.

Critical weaknesses
  1. The evaluation does not yet test the interactive ability the benchmark claims to measure. All main results use gold history, so a model’s own answers never shape the dialogue and errors cannot propagate. That removes the defining property of interaction. The appendix shows this matters a lot: with model-generated history, Logic Avg falls from about 58 to 39–44, and error correction from about 50 to under 22, yet this is reported for only two models. The cross-task integration claim also lacks an information-matched control. Intermediate gold answers often nearly state the final answer (e.g., the disconnection names the precursors), so the history gains may reflect extra information rather than integration across turns. Finally, the primary LLM judge (DeepSeek-v4-flash) is also an evaluated model and has the top Logic score. Logic is not separately rescored by an independent judge. This is consequential because the paper’s central contribution is interaction-based evaluation, and the current protocol cannot distinguish it from scripted, oracle-conditioned QA.

  2. The robustness and reproducibility of dialogue generation are not established. Reviewers modified 73.5% of retained dialogues (88% in retrosynthesis), and most review notes were chemical-fact errors. However, no residual error rate after review and no inter-reviewer agreement are reported. The generator model is only mentioned in the AI-use statement, and it is a GPT-5.6 model, a family that is also evaluated. The generation prompts and code are deliberately omitted. This is consequential because the construction pipeline is the part the field would reuse or extend. Without evidence that it produces reliable dialogues, and without the means to rerun it, the benchmark’s value as a foundation for further LLM-for-science research is limited.

Questions for authors

Questions:

  1. Interaction under realistic conditions. Can you report model-history results for all 13 models, and compare gold history against a single-turn prompt containing the same intermediate facts?

  2. Judge independence. Can you rescore the interaction-logic score for all models with a judge that is not itself evaluated?

  3. Generation robustness and reproducibility. What is the residual error rate on a blinded random sample after review? Will the generator model, prompts and validator rules be released?

Suggestion for future work: report thinking-enabled results as a main setting, and validate dialogue realism against real chemist–LLM interactions.

5 comments

Thank you for your specific comments on interaction evaluation protocols, judge independence, and dataset construction quality and reproducibility.

1.1. Weakness 1 & Question 1: All main experiments use Gold-history, so models' own answers cannot affect later context. The appendix shows large Model-history drops for only two models. Can you extend this to all 13 models to assess interaction performance under self-generated history?

Answer 1.1. Gold-history compares current-turn capability under shared correct context; Model-history tests effects of earlier model answers. Appendix A.12 of the original manuscript already reports Model-history results for DeepSeek-v4-pro (Pro) and GPT-5.6-sol (GPT-sol). We added GLM-5.2 (GLM), Kimi-K2.6 (Kimi) and Qwen3.8-max-0902 (Qwen), expanding to five models—the five families' top performers in our evaluation. Scores are higher under Gold-history than Model-history on FS/RS; Model-history gains over No-history vary by model/task. We will present the five-model comparisons, six-capability results and turn-level cases in §4.1 and the appendix, clarifying what each protocol evaluates and the five-model FS/RS scope.

  1. Shared correct history controls comparison; self-generated history tests sequential performance. Gold-history holds prior facts correct and identical across models, consistent with MT-Bench-101, §3.4. Model-history generates answers sequentially from the seed along the original question chain, adding each to later context. No-history retains the seed and final question. Each Gold/Model condition covers 139 FS/137 RS dialogues and 1,620 scored turns (829/791) per model, including 551 capability turns.

  2. History gains do not simply track baseline prediction ability. In Table 1, relative to No-history, Model-history raises scores for four models on FS (one lower) and three on RS (two lower). Kimi has the highest No-history accuracy but gains 2.88 pp on FS and loses 6.57 pp on RS. Table 2 shows decreases in all six joint capability scores, including 30.16–61.41 pp for EC and 39.86–54.73 pp for RC. The six Gold/Model final-structure comparisons for the three added models have Holm-adjusted McNemar p < 0.012.

  3. Response trajectories locate errors entering later verification and arguments. In FS D4, both conditions yield an incorrect 100°C; the model then accepts the reference 90°C under Gold-history but maintains 100°C under Model-history. In FS D19, the model cites its earlier incorrect Pd/Suzuki judgment to reject copper-catalysis evidence (Table 3). These cases support short-range persistence; aggregate gaps include factual errors, omissions and fixed-follow-up compatibility, without isolating propagation's causal effect.

Table 1. Final-structure accuracy (%)

ModelFS NoFS GoldFS ModelRS NoRS GoldRS Model
Pro14.3931.6512.9511.6839.425.84
GPT-sol15.8332.3720.149.4940.1512.41
GLM6.4728.7812.955.1130.6613.87
Kimi28.0643.1730.9421.1745.2614.60
Qwen8.6329.5015.838.0332.1210.95

No/Gold/Model denote No-history/Gold-history/Model-history. FS/RS contain 139/137 final targets, scored by strict equality of canonicalized structural component sets with stereochemistry retained.

Table 2. Joint scores for six capabilities (%, Gold → Model)

ModelARCMELECRCSK
Pro53.17→43.3153.58→45.4057.50→38.5955.00→21.5672.43→31.7666.58→43.70
GPT-sol60.43→55.2554.60→50.6657.50→47.1946.88→16.7272.70→32.8456.03→40.82
GLM54.24→44.1751.24→45.0459.22→31.0968.75→22.9778.78→27.0366.30→41.78
Kimi55.11→45.6152.63→45.4069.06→51.8880.94→19.5371.08→19.7364.38→51.37
Qwen57.70→49.9355.40→47.1567.97→40.3181.09→30.3191.89→37.1661.51→35.75

AR/CM/EL/EC/RC/SK denote anaphora resolution, context memory, elaboration, error correction, reliability checks, and skeptical reasoning, with 139/137/64/64/74/73 turns per model. Joint scores incorporate chemical correctness and interaction requirements, describing overall performance on these tasks.

Table 3. DeepSeek-v4-pro cases under self-generated history

DialoguePreceding answerLater requirementObserved behavior
FS D4Gives 100°C instead of 90°CVerify whether the reaction occurs at 90°CMaintains 100°C and rejects 90°C
FS D19Incorrectly identifies a Pd-catalyzed Suzuki couplingExplain how reaction conditions support copper catalysisCites its earlier Pd/Suzuki judgment to reject copper catalysis

1.2. Weakness 1 & Question 1: Intermediate gold answers may name the precursors, so history gains may reflect extra information or answer cues. Without comparison to a single-turn input containing the same facts, what supports cross-task integration and the benchmark's distinctive value?

Answer 1.2. ChemDialog's value lies in assessing linked, history-dependent chemical subtasks and interaction responses within one reaction; history gains reflect intermediate chemical information use. The Introduction, §2.2 and Appendix A.4 describe task organization and benchmark differences. We added precursor-name removal and same-target No-history controls: removal lowers scores, which remain above No-history, supporting contributions from explicit cues and remaining information. We will clarify task organization and evaluation targets in §2.2, add the ablation to §4.2, and use §4.1's interaction cases to explain the benchmark's value, without inferring an independent cross-turn integration advantage from prediction gains alone.

  1. Task organization, information flow and evaluation requirements distinguish ChemDialog. The paper describes eight static ChemLLMBench tasks and multimodal, step-by-step reasoning in SUPERChem. ChemDialog links functional groups, reaction types, conditions, and structure predictions into successive exchanges about one ORD reaction and related compounds. Each turn has a chemical objective; later turns depend on earlier facts, references, and interaction intents. CoT steps within one answer and later user requests to explain, correct, or verify impose different requirements.

  2. Interaction behaviors add information beyond a final prediction score. Appendix A.4, Table 8 gives “Which new linkage specifically reflects that conversion?”, followed by “What reaction type does that transformation belong to?” The model must explain earlier bond formation and resolve a historical reference. In RS D7 T6, DeepSeek-v4-pro correctly rejects water as the solvent but gives an incorrect alternative. FS D19 in Comment 1 shows an earlier error becoming evidence in a later argument. Task definitions, turn-level scores and trajectories jointly support assessing these behaviors. History may be one contextual input; its value depends on historical dependencies and later response requirements.

  3. Precursor-name removal quantifies the contribution of explicit naming cues. Across 106 editable RS targets, we remove explicit precursor enumerations and repetitions, retaining products, bond locations, conditions, and final questions; 67 cases name both precursors and 39 name one. Both models score lower after removal but above No-history on the same targets (Table 4). Retained bond locations can imply precursor identities, so we interpret the remaining gain as intermediate information use. Comment 1's three conditions, capability results, and cases separately examine self-generated history.

Table 4. Explicit precursor-name removal and No-history comparison

ModelNo-historyFull GoldGold after removalFull−removed, pp
DeepSeek-v4-pro6.6051.8934.9116.98
GLM-5.28.4933.9621.7012.26

Scores are reference-precursor coverage hit rates (%) on the same 106 targets; Table 1 uses strict set matching. No-history retains the product seed and final question. Source-reaction paired-bootstrap 95% CIs for removal are [5.66,27.36]/[1.89,22.64] pp.

1.3. Weakness 1 & Question 2: DeepSeek-v4-flash is both a judge and an evaluated model, with the highest Logic Avg among evaluated models. Can a judge outside the evaluated set rescore all models' raw Logic to test dependence on the primary judge?

Answer 1.3. With Kimi-K2.6 (Kimi) as judge, DeepSeek-v4-flash (Flash) retains its advantage over GPT-5.6-sol (GPT-sol) on the paper's Logic Avg. The original paper used DeepSeek/GPT judges on frozen answers. We added Kimi from a different family to rescore 4,658 turns from two models (1,582 capability turns), examining raw Logic, complete final scores and model differences. For each judge, Flash's joint Logic Avg advantage has a 95% CI above zero; raw Logic also shows positive cross-judge rank correlations. We will add these results, task-level agreement and metric definitions to §4.5/Appendix A.13, noting the two-model scope and all three judges' inclusion in the evaluated set.

  1. Frozen answers make the comparison directly about scoring differences. Answers are not regenerated; Kimi recomputes content scores and reassesses interaction logic. In Table 5, primary/Kimi all-turn Spearman correlations are 0.952/0.962, with exact agreement of 82.70%/85.23%. Table 6 separates capability turns so rule-based structure/numerical scores do not mask interaction differences. Each judge's final scores combine its own content and logic assessments.

  2. Raw Logic supplies a separate interaction check. Table 7 compares 1,575 capability turns valid across judges: Spearman 0.668–0.780; exact agreement 59.37%–69.52%. No content/joint scores replace raw Logic; all-turn and capability-turn agreement provides complementary evidence.

  3. The joint-score difference persists beyond the primary judge. All three judges give Flash a higher Logic Avg (Table 8), with source-reaction clustered paired-bootstrap intervals above zero. Logic Avg includes chemical-correctness constraints; raw-Logic point-estimate ordering changes with the judge. We report these metrics separately and discuss capability-level sensitivity. The study tests scoring stability for two frozen models, without independent validation of all 13 by a judge outside the evaluated set.

Table 5. Primary/Kimi per-turn final-score agreement

GeneratorTaskNSpearman ρExact (%)
FlashFS8290.97191.44
FlashRP7090.96275.60
FlashRS7910.91979.90
FlashAll2,3290.95282.70
GPT-solFS8290.97692.40
GPT-solRP7090.97478.28
GPT-solRS7910.93183.94
GPT-solAll2,3290.96285.23

Table 6. Final-score agreement across three judges

ScopeNKendall WICC(A,1)
All scored turns4,6580.9690.953
Capability turns only1,5820.8920.820

Kendall W measures rank concordance; ICC(A,1) measures absolute agreement. Capability-turn final scores retain chemical-correctness constraints.

Table 7. Cross-judge agreement on raw Logic

Judge AJudge BSpearman ρExact (%)
FlashGPT-sol0.78069.52
FlashKimi0.72261.21
GPT-solKimi0.66859.37

Table 8. Logic Avg and model differences after complete rescoring

JudgeFlash (%)GPT-sol (%)Flash−GPT-sol, pp95% CI
Flash57.1652.53+4.63[2.67, 6.53]
GPT-sol56.6654.11+2.55[0.52, 4.34]
Kimi52.9550.09+2.86[0.67, 4.85]

Logic Avg is the weighted mean final score over capability turns. Table 8 follows complete-score aggregation; Table 7 uses the commonly valid raw-Logic sample.

Reviewer_4XFcOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces ChemDialog, a new benchmark focused on evaluating the performance of LLMs in answering chemical reaction-related questions through dialogue. It also presents extensive experiments analyzing the performance of frontier LLMs on these tasks.

Critical strengths
  1. The paper introduces a new benchmark, ChemDialog, that evaluates LLMs’ chemical competence from a novel perspective.
  2. Extensive experiments are conducted, with analyses from multiple perspectives.
Critical weaknesses

First, the paper’s description of the data format remains vague throughout, leaving the specific form of the tasks and questions unclear to me. The paper also does not explicitly provide data examples. Furthermore, I was unable to access the data link provided in the paper, which poses a major obstacle to assessing the value of the benchmark.

Second, the analyses presented do not appear to focus on the distinctive features and value of ChemDialog.

  1. In Section 4.1, the analysis of interactive capabilities is limited to broad descriptions, without examining the details. These appear to be the tasks unique to ChemDialog, and their analysis would therefore seem to warrant particular attention. In addition, accuracy when answering questions sequentially could be an interesting aspect to investigate, but the paper does not examine it in depth.
  2. The third finding in Section 4.1 is somewhat puzzling. It appears to suggest that LLMs are better at explanation than at information extraction. However, because the paper does not present data examples for the different tasks, I cannot assess how this conclusion was reached or whether it supports the claims made in the paper.
  3. Dialogue history is itself a form of context. Adding other reaction-related questions to the dialogue history does not seem different from providing additional information in the context. From this perspective, the analysis in Section 4.2 can be interpreted as showing that providing more reaction-related information helps LLMs perform chemical tasks more effectively. This conclusion does not appear to require a new benchmark for verification. Moreover, the conclusions in all subsequent subsections are unrelated to ChemDialog’s distinctive features and could equally be drawn by analyzing results from existing benchmarks.
Questions for authors

N/A

5 comments

Weakness 1. The manuscript's task and data formats and concrete question examples are unclear, and its anonymous data link is inaccessible. Can the authors explain the actual inputs and outputs and provide accessible data and code for inspection?

Answer 1. The original manuscript already provides task definitions, complete dialogues and scoring examples. Section 3.1 introduces task construction; Appendix A.3 describes data fields. Tables 7–8 (Appendix A.4) present question chains for the three tasks and a complete dialogue. Tables 10–12 and Figure 5 (Appendix A.6) give scoring rules and scored examples. At each turn, the model receives the preceding dialogue and current question and generates an answer. For example, T4 in Table 8 requires resolving "that transformation" from the history and identifying the reaction type. We will strengthen the pointers to these materials, move a concise version of the Table 8 example earlier in the paper, and update the link.

We again provide a link to the anonymous code repository, which contains selected evaluation code, two human-evaluation questionnaire sets and the validation-interface code.

Weakness 2.1. Section 4.1 gives only broad interaction analysis. Which behaviors need explanation, and how do accuracy and error persistence change when models answer sequentially using prior responses?

Answer 2.1. Capability results and turn-level cases show chemical errors reused in later verification and arguments. Dialogue/scoring cases are in Table 8 (Appendix A.4) and Table 12 and Figure 5 (Appendix A.6); Appendix A.12 reports two-model Model-history experiments. We added GLM-5.2, Kimi-K2.6 and Qwen3.8-max-0902, reaching five models. Gold-history outperforms Model-history in FS/RS final-structure accuracy and all six capability joint scores for all five models. Section 4.1 and its appendix will include five-model comparisons, six-capability cases and turn-level trajectories.

  1. Specific behaviors. In Table 2.1a, EL explains C(sp²)–CF₃ formation as C–F formation; EC correctly rejects water as the solvent but supplies a wrong replacement. SK cites a prior Pd/Suzuki error against copper catalysis. CM's product-structure error occurs at the first scored turn.
  2. Sequential setup and final prediction. Model-history follows the question chain from a given seed, passing generated answers forward. Gold-history uses that chain with correct prior answers; No-history keeps the seed and final question. Per model, each Gold/Model run covers 139 FS and 137 RS dialogues, with 1,620 scored turns (551 capability turns). The three added models use matched Gold/Model questions, prompts, decoding and evaluation settings. Against No-history, Model-history improves for four FS/three RS models and declines for one FS/two RS (Table 2.1b). The six Gold/Model final-structure comparisons for the three added models have Holm-adjusted McNemar p<0.012.
  3. Interaction scores. EC and RC joint scores fall by 30.16–61.41 and 39.86–54.73 percentage points, respectively (Table 2.1c). Joint scores are constrained by chemical correctness and interaction requirements (Table 2.1a).
  4. Turn-level trajectory. Such error propagation is difficult to quantify, so we use turn-level cases to illustrate and analyze it. In FS D4 (Table 2.1d), both settings wrongly answer 100 °C at T5. At T6, Gold-history supplies T5's correct 90 °C and the model accepts it; Model-history retains the actual wrong answer and the model insists on 100 °C.

Table 2.1a. DeepSeek-v4-pro Model-history interaction cases

Capability/turnReference/requestResponse summaryFailure
Anaphora resolution (AR), FS D87/T3Classify "that transformation"; gold: Buchwald–Hartwig aminationEsterificationPrior transformation error persists into the current reaction classification.
Context memory (CM), RS D32/T2Product functional groups, incl. secondary alcoholMisclassified as primary alcoholProduct-structure error at first scored turn
Elaboration (EL), FS D27/T3Explain the new C(sp²)–CF₃ bondAryl C–F formationWrong bond fact in explanation
Error correction (EC), RS D7/T6Water as solvent? Correct: tolueneRejects water; dichloromethane/DMFCorrect rejection; wrong replacement
Reliability check (RC), FS D4/T6Verify the 90 °C claimMaintains prior 100 °CPrior error enters verification
Skeptical reasoning (SK), FS D19/T5Explain how the conditions support copper catalysisCites a prior Pd/Suzuki error against copperOwn error reused as evidence

Table 2.1b. Final-structure accuracy in three histories (%)

ModelFS NoFS GoldFS ModelRS NoRS GoldRS Model
DeepSeek-v4-pro14.3931.6512.9511.6839.425.84
GPT-5.6-sol15.8332.3720.149.4940.1512.41
GLM-5.26.4728.7812.955.1130.6613.87
Kimi-K2.628.0643.1730.9421.1745.2614.60
Qwen3.8-max-09028.6329.5015.838.0332.1210.95

FS has 139 product and RS 137 precursor targets; exact canonical component-set matching retains stereochemistry. The original two and added three models ran separately; comparisons are within each model.

Table 2.1c. Six-capability joint scores (%, Gold/Model)

ModelARCMELECRCSK
DeepSeek-v4-pro53.17/43.3153.58/45.4057.50/38.5955.00/21.5672.43/31.7666.58/43.70
GPT-5.6-sol60.43/55.2554.60/50.6657.50/47.1946.88/16.7272.70/32.8456.03/40.82
GLM-5.254.24/44.1751.24/45.0459.22/31.0968.75/22.9778.78/27.0366.30/41.78
Kimi-K2.655.11/45.6152.63/45.4069.06/51.8880.94/19.5371.08/19.7364.38/51.37
Qwen3.8-max-090257.70/49.9355.40/47.1567.97/40.3181.09/30.3191.89/37.1661.51/35.75

AR/CM/EL/EC/RC/SK: 139/137/64/64/74/73 turns per model.

Table 2.1d. DeepSeek-v4-pro temperature verification, FS D4

TurnQuestionReferenceGoldModel
T5What is the reaction temperature?90 °C100 °C100 °C
T6Is the reaction carried out at 90 degrees Celsius?Yes, 90 degrees Celsius.Yes, 90 degrees Celsius.No. The reaction is carried out at 100 °C, not 90 °C.

Weakness 2.2. The third finding in Section 4.1 seems to imply that LLMs are better at explanation than information extraction. With task examples unclear, what is being compared, and what evidence supports that interpretation?

Answer 2.2. The comparison is between implicit history dependencies (AR/CM) and explicitly stated interaction intents (EL/RC). Section 4.1 states, "LLMs struggle more with implicit dependencies on dialogue history than with explicitly stated interaction intents." Table 8 (Appendix A.4) provides corresponding examples; our supplementary 13-model breakdown shows lower raw Logic means for AR/CM than for EL/RC (Table 2.2a). We will clarify the comparison and metric definitions in Section 4.1, use Table 8 to explain the two groups' interaction requirements, and add the results in Table 2.2a.

  1. Interaction requirements define the groups. AR resolves anaphoric references and CM recovers earlier facts. EL explicitly requests elaboration, while RC supplies a claim to verify. In Table 8, T3 asks "Which new linkage specifically reflects that conversion?", requesting bond elaboration. T4 asks "What reaction type does that transformation belong to?", requiring anaphora resolution before reaction classification. Both concern one reaction, with different current chemical targets and answer forms.
  2. Score breakdown and metric interpretation. Table 2.2a summarizes valid answers from 13 models. Raw Logic is the interaction judge's assessment of whether the relevant interaction requirements are fulfilled; final scores combine chemical content and interaction requirements. Both metrics show performance differences between the interaction groups defined above.

Table 2.2a. Score breakdown for implicit history dependencies and explicit interaction intents

GroupCapabilityRequirementValid answersRaw LogicFinal
Implicit history dependenciesAnaphora resolution (AR)Resolve an anaphoric reference in the current question2,60063.7846.03
Implicit history dependenciesContext memory (CM)Recover and use chemical facts supplied earlier2,54761.0543.00
Explicit interaction intentsElaboration (EL)Supply explanations and details explicitly requested1,20989.2359.18
Explicit interaction intentsReliability check (RC)Assess the stated claim and provide supporting evidence1,37889.4775.46
Reviewer_tfPaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ChemDialog, a multi-turn dialogue benchmark designed to evaluate LLMs' chemical reaction understanding and interactive reasoning capabilities. The benchmark contains human-verified dialogues covering 3 chemical tasks and 6 interaction capabilities. The authors propose a multidimensional evaluation framework that assesses both chemical correctness and dialogue-level logical consistency. Extensive experiments on 13 mainstream LLMs reveal significant limitations in context-dependent reasoning and cross-task information integration. Furthermore, the results demonstrate that incorporating accurate dialogue history substantially improves model performance in forward synthesis and retrosynthesis tasks.

Critical strengths
  1. The paper addresses an important limitation of existing chemistry benchmarks by introducing multi-turn dialogues that evaluate both chemical understanding and interactive reasoning. This provides a more realistic and comprehensive evaluation setting than traditional single-turn QA benchmarks.

  2. The authors evaluate 13 mainstream LLMs and conduct extensive analyses of cross-task information integration, sampling strategies, and evaluation reliability. The finding that accurate dialogue history substantially improves chemical prediction performance provides valuable insights into the capabilities and limitations of current LLMs.

Critical weaknesses
  1. The benchmark contains only 396 dialogues derived from 205 chemical reactions, with dialogues following predefined task structures. This limits the diversity and complexity of interactions. So I have concerns about whether the benchmark adequately represents realistic chemical reasoning scenarios.

  2. The main experiments use ground-truth dialogue histories rather than models’ own generated responses, preventing errors from propagating across turns. Though authors provided limited experiments using model-generated histories, the evaluation doesn't sufficiently investigate long-term consistency and error accumulation in realistic interactions.

Questions for authors

Please refer to the weakness part. I'd love to have your response on those. Thank you!

3 comments

We thank the reviewer for the comments on dataset coverage and history-based evaluation.

Weakness 1. Can 396 predefined dialogues from 205 reactions cover interaction diversity and complexity in realistic chemical reasoning scenarios? What evidence supports their chemical and interaction coverage?

Answer 1. ChemDialog evaluates dialogue capabilities in organic synthesis through chemical coverage, cross-task interaction requirements, and inspectable turn-level history dependencies. Sections 3.1–3.2 and Appendices A.4–A.5 contain task definitions, a complete dialogue, and capability distributions. We add coverage statistics from existing data and use actual answers to explain interaction requirements. The 205 reactions correspond to 188 scaffolds; 204 classifiable records span 41 families; all six capabilities are assessed in each of the three chemical tasks. We will add coverage statistics and highlight the dialogue in Sections 3.1–3.2 and present Tables A–B in Appendices A.4–A.5.

1. Chemical variation spans transformations and product structures. Source selection required complete, verifiable records and diverse common transformations. The families include 57 Buchwald–Hartwig aminations, 39 amide formations, 13 esterifications, and 10 reductive aminations. Reaction families describe transformations; Table A reports structural variation through scaffolds and Morgan similarities. The 205 distinct canonical major products correspond to 188 distinct scaffolds, with a median nearest-neighbor Morgan similarity of 0.306. Products contain 14–71 heavy atoms and 1–10 rings. These results show transformation and structural variation, primarily in aromatic and heteroaromatic organic synthesis.

2. All six capabilities are assessed in each chemical task. The 396 dialogues cover forward synthesis (FS), reaction procedure (RP), and retrosynthesis (RS), comprising 2,329 scored non-seed turns, including 791 capability turns. Anaphora resolution (AR), context memory (CM), elaboration (EL), error correction (EC), reliability checking (RC), and skeptical reasoning (SK) have 200, 196, 93, 93, 106, and 103 turns, respectively. The same interaction requirements are evaluated under different chemical objectives. Predefined dialogue structures give different models the same questions and reference facts.

3. Question chains connect chemical subtasks and history dependencies within one reaction. The complete example in Appendix A.4, Table 8 asks about a functional-group conversion, then requires explaining the bond formed by “that conversion” and resolving “that transformation” to identify the reaction type. Later turns require models to use preceding facts and resolve references, not merely answer isolated questions. The comparison with SUPERChem and other benchmarks in Section 2.2 reflects this focus: ChemDialog links subtasks of the same reaction into consecutive exchanges with explicit interaction requirements.

Table B links requirements to observed behavior. The first two rows show how elaboration and reference resolution depend on history; the last two use archived DeepSeek-v4-pro answers to show an incorrect temperature persisting and an earlier incorrect claim being used as evidence. The full dialogue history, reference facts, and capability labels support turn-level analysis of how chemical answers relate to history use.

Table A. Structural coverage of major products

MetricResult
Distinct canonical major products, retaining stereochemistry205
Bemis–Murcko scaffolds188
Largest group of products sharing a scaffold4
Median pairwise Morgan similarity0.123
Median nearest-neighbor Morgan similarity0.306
Products with nearest-neighbor similarity ≥0.86/205 (2.93%)
Products with identifiable tetrahedral stereocenters98/205 (47.80%)
Products containing aromatic rings200/205 (97.56%)

Families are grouped from reference descriptions and reaction_basis, allowing multiple labels; Morgan similarities do not encode chirality.

Table B. History dependencies and observed behavior

LocationEarlier informationCurrent requirementDependency or behavior
FS D2 T3, ELPrior reference answer: aryl iodide → aryl nitrileExplain the bond formed by “that conversion”Connect the conversion to aryl–C≡N formation
FS D2 T4, AREarlier turns describe the conversion and new bondIdentify the reaction type of “that transformation”Resolve the reference; reference answer: palladium-catalyzed aryl cyanation
FS D4 T6, RCDeepSeek-v4-pro answers 100°C against the 90°C referenceVerify whether the reaction is carried out at 90°C.Repeats 100°C and rejects 90°C
FS D19 T5, SKDeepSeek-v4-pro incorrectly identifies Pd catalysis and Suzuki couplingExplain how reaction conditions support copper catalysisCites its earlier error to reject copper catalysis

Weakness 2. Why do the main experiments use ground-truth histories that prevent models from propagating their own errors? Do the limited Model-history experiments sufficiently assess long-term consistency and error accumulation in realistic interactions?

Answer 2. Gold-history compares current-turn capabilities under a shared correct context; Model-history tests how a model's preceding answers affect later performance. The original paper reports Model-history results for DeepSeek-v4-pro and GPT-5.6-sol. We added GLM-5.2, Kimi-K2.6, and Qwen3.8-max-0902, reaching five models, with No-history comparisons and turn-level analyses. Gold-history yields higher final-structure scores than Model-history for all five models; error-correction and reliability-checking joint scores drop substantially under Model-history. We will present five-model comparisons, six-capability results, and error-propagation cases in Section 4.1 and the history-analysis appendix.

1. Gold-history provides common conditions for comparing current-turn capabilities. Chemical QA, contextual understanding, and interaction responses depend on preceding facts. With different preceding answers, current-turn scores also reflect differences in historical content. Shared reference history controls this factor. This aligns with the golden-context design in MT-Bench-101, Section 3.4.

2. Three-condition comparisons show that benefits of self-generated history vary by model and task. No-history retains the seed and final question; Gold-history supplies intermediate reference exchanges; Model-history uses the model's preceding answers. In Table 1, relative to No-history, Model-history raises FS scores for four models and lowers them for one; on RS, it raises scores for three and lowers them for two. Kimi has the highest No-history scores among the five models, but its history gain is positive on FS and negative on RS. Baseline prediction scores therefore do not map directly to history benefits; useful information, errors, and how models handle history may all affect final performance.

All six joint capability scores decrease under Model-history (Table 2). Error correction and reliability checking drop by 30.16–61.41 and 39.86–54.73 percentage points, respectively. These scores combine chemical correctness and interaction requirements, linking history effects to correction and verification turns.

3. Such error-propagation behavior is difficult to quantify, so we illustrate and analyze it through turn-level cases (Table 3). In FS D4, both runs answer 100°C instead of 90°C in the preceding turn. At the next turn, the model accepts 90°C under Gold-history but maintains 100°C and rejects 90°C under Model-history. In FS D19, the model cites its earlier Pd/Suzuki error to reject copper-catalysis evidence. These trajectories show earlier errors persisting in later verification and justification.

Table 1. Final-structure hit rates under three history conditions

ModelFS NoFS GoldFS ModelFS ΔM−NRS NoRS GoldRS ModelRS ΔM−N
DeepSeek-v4-pro14.3931.6512.95−1.4411.6839.425.84−5.84
GPT-5.6-sol15.8332.3720.14+4.329.4940.1512.41+2.92
GLM-5.26.4728.7812.95+6.475.1130.6613.87+8.76
Kimi-K2.628.0643.1730.94+2.8821.1745.2614.60−6.57
Qwen3.8-max-09028.6329.5015.83+7.198.0332.1210.95+2.92

Scores (%); ΔM−N = Model-history − No-history.

Table 2. Joint scores for six interaction capabilities

ModelARCMELECRCSK
DeepSeek-v4-pro53.17→43.3153.58→45.4057.50→38.5955.00→21.5672.43→31.7666.58→43.70
GPT-5.6-sol60.43→55.2554.60→50.6657.50→47.1946.88→16.7272.70→32.8456.03→40.82
GLM-5.254.24→44.1751.24→45.0459.22→31.0968.75→22.9778.78→27.0366.30→41.78
Kimi-K2.655.11→45.6152.63→45.4069.06→51.8880.94→19.5371.08→19.7364.38→51.37
Qwen3.8-max-090257.70→49.9355.40→47.1567.97→40.3181.09→30.3191.89→37.1661.51→35.75

Scores (%); arrows indicate Gold-history→Model-history. Scores incorporate chemical correctness and interaction requirements.

Table 3. Error-propagation cases under self-generated history

DialogueEarlier errorSubsequent interaction requirementBehavior under Model-history
DeepSeek-v4-pro, FS D4Answers 100°C instead of the reference temperature of 90°CVerify whether the reaction occurs at 90°CRepeats 100°C and rejects 90°C
DeepSeek-v4-pro, FS D19Identifies the reaction as a Pd-catalyzed Suzuki couplingExplain how reaction conditions support copper catalysisCites its earlier Pd/Suzuki claim to reject copper catalysis

Dear Reviewer tfPa,

We provided a breakdown of the reaction families and molecular scaffolds in ChemDialog and expanded the model-history evaluation from two models to five. Could you let us know whether these results address your concerns about chemical diversity and how errors carry over across turns? We would be happy to discuss either point further.

Thank you for your time and feedback.

Comments on the paper

2 comments
Area_Chair_KExeOpenReview ↗

Reminder to Reviewers: Please Engage in the Discussion

Dear Reviewers,

As the authors have now submitted their rebuttal, I would kindly ask you to read their responses and engage in the discussion.

Thank you!

Best regards, AC

Dear Area Chair and Reviewers,

Thank you for your careful reading of ChemDialog and for your constructive feedback. Reviewer tfPa described the joint evaluation of chemical understanding and interactive reasoning as “a more realistic and comprehensive evaluation setting” and highlighted the gains from accurate dialogue history. Reviewer 4XFc recognised that ChemDialog evaluates chemical competence “from a novel perspective” and noted the breadth of the experiments and analyses. Reviewer wP6V explained why linking reaction tasks matters for scientific workflows. It lets us examine “whether a model carries information across steps or responds properly to user corrections.” The reviewer cited the drops when models relied on their own earlier answers as “behaviour that static benchmarks would miss.”

In response, we clarified the dialogue tasks and scoring, expanded the sequential history study, audited the reviewed dialogues, and examined the effect of judge choice on fixed answers. Revisions are marked in blue.

Main Text

  • Sections 1–3. We clarify how subtasks are linked within a reaction and how different answer types are scored. Section 3.1 uses a question from an actual forward-synthesis dialogue to illustrate what “that transformation” refers to and directs readers to the complete dialogue in Appendix A.4. We also describe the generator’s role, disclose its family overlap with the evaluated models, and summarise the post-review audit.
  • Sections 4.1–4.2. Section 4.1 uses two consecutive turns from the same dialogue to illustrate elaboration and anaphora resolution. Section 4.2 corrects the retrosynthesis comparison in Table 2 and adds Table 3, which summarises final prediction gains, error correction, and reliability checking for five models using their own earlier answers. A turn-level example shows how an incorrect temperature answer carries into a later verification turn.
  • Section 4.5. We report how closely the original two judges agree when scoring the same answers. Exact final-score agreement was 93.8% and 96.3% on forward synthesis for the two sets of model answers, compared with 75.5% and 80.3% on reaction procedure (Table 6). With Kimi-K2.6 added, all three judges gave DeepSeek-v4-flash a higher joint Logic Avg than GPT-5.6-sol, with gaps of 4.63, 2.55, and 2.86 percentage points, respectively. The ordering by raw interaction score varied by judge. Table 7 presents the separate human evaluation.

Appendix A

  • A.1 and A.3–A.6. We add a comparison with independent chemical QA, explain what each dialogue turn receives and returns, and point readers to the full dialogue and scored examples. The major products span 188 scaffolds, 204 classifiable reactions cover 41 families, and each of the three tasks includes all six interaction capabilities.
  • A.7–A.8. Three chemistry-trained raters audited 20 dialogues covering 140 turns. They confirmed one unit error, which we corrected in the release data, and unanimously judged 18 of the 20 dialogues suitable for retention (Tables R4–R5). Across 13 models, context memory and anaphora resolution also scored lower than elaboration and reliability checking when interaction logic was scored on its own (Table R6).
  • A.12. We expanded the sequential history study from two to five models. All five performed better with reference history on both final prediction tasks and across all six interaction capabilities (Tables R7–R10). On the same 137 retrosynthesis targets, the corrected gains in precursor coverage over No history ranged from 23.36 to 31.39 percentage points across four models (Table R11). On a separate set of 106 targets, removing explicit precursor names reduced coverage for both tested models, and both remained above their No-history baselines on those same targets (Table R12).
  • A.13. We report the three-judge comparison on 4,658 fixed answer turns from two models. Tables R13–R16 give agreement by task, agreement on final scores and raw interaction scores, and paired confidence intervals for the difference in joint Logic Avg.

The anonymous repository provides six sample dialogues, selected evaluation code, two human-evaluation questionnaires, and the validation-interface code. Upon acceptance, we will release the complete generation pipeline, including code, prompts, validator rules, and running instructions. Anonymous repository

Thank you again for helping us clarify what ChemDialog measures and strengthen the evidence for our conclusions. We address each comment in the individual responses.

Yours sincerely,
The Authors of Submission 14451

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.