- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces ChemDialog, a benchmark of 396 multi-turn dialogues (2,725 turns) built from 205 Open Reaction Database reactions. It covers forward synthesis, reaction procedure, and retrosynthesis. An LLM generates the dialogues from predefined task chains and inserts six types of interaction logic, such as anaphora resolution, error correction, and reliability checks. Humans (chemistry students) then reviewed them (about 200 hours; 291 dialogues modified, 20 removed).
Scoring combines exact SMILES and numeric matching with an LLM judge for open-ended answers; capability turns are scored as min(chemistry, logic). Across 13 LLMs evaluated with gold history and no reasoning, the best overall score is about 55%. The authors find that chemical accuracy does not imply interaction competence, and that gold history improves final predictions, which they read as cross-task integration. The judge is validated by rescoring with a second LLM judge and by a 20-question study with five student raters.
Critical strengths
-
A well-motivated shift from isolated QA to interaction-based evaluation in scientific workflows. The benchmark links related reaction tasks into dialogues where later turns depend on earlier ones, and adds interaction behaviours such as error correction and reliability checks. This matters to my assessment because LLMs are increasingly used as collaborators in research, and single-turn benchmarks cannot capture whether a model carries information across steps or responds properly to user corrections. The paper’s appendix analyses, especially the model-history and thinking × history experiments, already show behaviour that static benchmarks would miss, such as large drops when models condition on their own earlier answers.
-
Careful, traceable grounding and review. Every dialogue is anchored to an ORD reaction record, and human review is fully logged (per-turn edits, reasons, deletions with provenance) through a dedicated review platform. This matters because the dialogues are LLM-generated, so the benchmark’s credibility depends on auditability. The change log makes the dataset’s quality inspectable and correctable by others, and the review infrastructure could be reused to build similar benchmarks in other scientific domains.
Critical weaknesses
-
The evaluation does not yet test the interactive ability the benchmark claims to measure. All main results use gold history, so a model’s own answers never shape the dialogue and errors cannot propagate. That removes the defining property of interaction. The appendix shows this matters a lot: with model-generated history, Logic Avg falls from about 58 to 39–44, and error correction from about 50 to under 22, yet this is reported for only two models. The cross-task integration claim also lacks an information-matched control. Intermediate gold answers often nearly state the final answer (e.g., the disconnection names the precursors), so the history gains may reflect extra information rather than integration across turns. Finally, the primary LLM judge (DeepSeek-v4-flash) is also an evaluated model and has the top Logic score. Logic is not separately rescored by an independent judge. This is consequential because the paper’s central contribution is interaction-based evaluation, and the current protocol cannot distinguish it from scripted, oracle-conditioned QA.
-
The robustness and reproducibility of dialogue generation are not established. Reviewers modified 73.5% of retained dialogues (88% in retrosynthesis), and most review notes were chemical-fact errors. However, no residual error rate after review and no inter-reviewer agreement are reported. The generator model is only mentioned in the AI-use statement, and it is a GPT-5.6 model, a family that is also evaluated. The generation prompts and code are deliberately omitted. This is consequential because the construction pipeline is the part the field would reuse or extend. Without evidence that it produces reliable dialogues, and without the means to rerun it, the benchmark’s value as a foundation for further LLM-for-science research is limited.
Questions for authors
Questions:
-
Interaction under realistic conditions. Can you report model-history results for all 13 models, and compare gold history against a single-turn prompt containing the same intermediate facts?
-
Judge independence. Can you rescore the interaction-logic score for all models with a judge that is not itself evaluated?
-
Generation robustness and reproducibility. What is the residual error rate on a blinded random sample after review? Will the generator model, prompts and validator rules be released?
Suggestion for future work: report thinking-enabled results as a main setting, and validate dialogue realism against real chemist–LLM interactions.
5 comments
1.2. Weakness 1 & Question 1: Intermediate gold answers may name the precursors, so history gains may reflect extra information or answer cues. Without comparison to a single-turn input containing the same facts, what supports cross-task integration and the benchmark's distinctive value?
Answer 1.2. ChemDialog's value lies in assessing linked, history-dependent chemical subtasks and interaction responses within one reaction; history gains reflect intermediate chemical information use. The Introduction, §2.2 and Appendix A.4 describe task organization and benchmark differences. We added precursor-name removal and same-target No-history controls: removal lowers scores, which remain above No-history, supporting contributions from explicit cues and remaining information. We will clarify task organization and evaluation targets in §2.2, add the ablation to §4.2, and use §4.1's interaction cases to explain the benchmark's value, without inferring an independent cross-turn integration advantage from prediction gains alone.
-
Task organization, information flow and evaluation requirements distinguish ChemDialog. The paper describes eight static ChemLLMBench tasks and multimodal, step-by-step reasoning in SUPERChem. ChemDialog links functional groups, reaction types, conditions, and structure predictions into successive exchanges about one ORD reaction and related compounds. Each turn has a chemical objective; later turns depend on earlier facts, references, and interaction intents. CoT steps within one answer and later user requests to explain, correct, or verify impose different requirements.
-
Interaction behaviors add information beyond a final prediction score. Appendix A.4, Table 8 gives “Which new linkage specifically reflects that conversion?”, followed by “What reaction type does that transformation belong to?” The model must explain earlier bond formation and resolve a historical reference. In RS D7 T6, DeepSeek-v4-pro correctly rejects water as the solvent but gives an incorrect alternative. FS D19 in Comment 1 shows an earlier error becoming evidence in a later argument. Task definitions, turn-level scores and trajectories jointly support assessing these behaviors. History may be one contextual input; its value depends on historical dependencies and later response requirements.
-
Precursor-name removal quantifies the contribution of explicit naming cues. Across 106 editable RS targets, we remove explicit precursor enumerations and repetitions, retaining products, bond locations, conditions, and final questions; 67 cases name both precursors and 39 name one. Both models score lower after removal but above No-history on the same targets (Table 4). Retained bond locations can imply precursor identities, so we interpret the remaining gain as intermediate information use. Comment 1's three conditions, capability results, and cases separately examine self-generated history.
Table 4. Explicit precursor-name removal and No-history comparison
| Model | No-history | Full Gold | Gold after removal | Full−removed, pp |
|---|---|---|---|---|
| DeepSeek-v4-pro | 6.60 | 51.89 | 34.91 | 16.98 |
| GLM-5.2 | 8.49 | 33.96 | 21.70 | 12.26 |
Scores are reference-precursor coverage hit rates (%) on the same 106 targets; Table 1 uses strict set matching. No-history retains the product seed and final question. Source-reaction paired-bootstrap 95% CIs for removal are [5.66,27.36]/[1.89,22.64] pp.
1.3. Weakness 1 & Question 2: DeepSeek-v4-flash is both a judge and an evaluated model, with the highest Logic Avg among evaluated models. Can a judge outside the evaluated set rescore all models' raw Logic to test dependence on the primary judge?
Answer 1.3. With Kimi-K2.6 (Kimi) as judge, DeepSeek-v4-flash (Flash) retains its advantage over GPT-5.6-sol (GPT-sol) on the paper's Logic Avg. The original paper used DeepSeek/GPT judges on frozen answers. We added Kimi from a different family to rescore 4,658 turns from two models (1,582 capability turns), examining raw Logic, complete final scores and model differences. For each judge, Flash's joint Logic Avg advantage has a 95% CI above zero; raw Logic also shows positive cross-judge rank correlations. We will add these results, task-level agreement and metric definitions to §4.5/Appendix A.13, noting the two-model scope and all three judges' inclusion in the evaluated set.
-
Frozen answers make the comparison directly about scoring differences. Answers are not regenerated; Kimi recomputes content scores and reassesses interaction logic. In Table 5, primary/Kimi all-turn Spearman correlations are 0.952/0.962, with exact agreement of 82.70%/85.23%. Table 6 separates capability turns so rule-based structure/numerical scores do not mask interaction differences. Each judge's final scores combine its own content and logic assessments.
-
Raw Logic supplies a separate interaction check. Table 7 compares 1,575 capability turns valid across judges: Spearman 0.668–0.780; exact agreement 59.37%–69.52%. No content/joint scores replace raw Logic; all-turn and capability-turn agreement provides complementary evidence.
-
The joint-score difference persists beyond the primary judge. All three judges give Flash a higher Logic Avg (Table 8), with source-reaction clustered paired-bootstrap intervals above zero. Logic Avg includes chemical-correctness constraints; raw-Logic point-estimate ordering changes with the judge. We report these metrics separately and discuss capability-level sensitivity. The study tests scoring stability for two frozen models, without independent validation of all 13 by a judge outside the evaluated set.
Table 5. Primary/Kimi per-turn final-score agreement
| Generator | Task | N | Spearman ρ | Exact (%) |
|---|---|---|---|---|
| Flash | FS | 829 | 0.971 | 91.44 |
| Flash | RP | 709 | 0.962 | 75.60 |
| Flash | RS | 791 | 0.919 | 79.90 |
| Flash | All | 2,329 | 0.952 | 82.70 |
| GPT-sol | FS | 829 | 0.976 | 92.40 |
| GPT-sol | RP | 709 | 0.974 | 78.28 |
| GPT-sol | RS | 791 | 0.931 | 83.94 |
| GPT-sol | All | 2,329 | 0.962 | 85.23 |
Table 6. Final-score agreement across three judges
| Scope | N | Kendall W | ICC(A,1) |
|---|---|---|---|
| All scored turns | 4,658 | 0.969 | 0.953 |
| Capability turns only | 1,582 | 0.892 | 0.820 |
Kendall W measures rank concordance; ICC(A,1) measures absolute agreement. Capability-turn final scores retain chemical-correctness constraints.
Table 7. Cross-judge agreement on raw Logic
| Judge A | Judge B | Spearman ρ | Exact (%) |
|---|---|---|---|
| Flash | GPT-sol | 0.780 | 69.52 |
| Flash | Kimi | 0.722 | 61.21 |
| GPT-sol | Kimi | 0.668 | 59.37 |
Table 8. Logic Avg and model differences after complete rescoring
| Judge | Flash (%) | GPT-sol (%) | Flash−GPT-sol, pp | 95% CI |
|---|---|---|---|---|
| Flash | 57.16 | 52.53 | +4.63 | [2.67, 6.53] |
| GPT-sol | 56.66 | 54.11 | +2.55 | [0.52, 4.34] |
| Kimi | 52.95 | 50.09 | +2.86 | [0.67, 4.85] |
Logic Avg is the weighted mean final score over capability turns. Table 8 follows complete-score aggregation; Table 7 uses the commonly valid raw-Logic sample.
Thank you for your specific comments on interaction evaluation protocols, judge independence, and dataset construction quality and reproducibility.
1.1. Weakness 1 & Question 1: All main experiments use Gold-history, so models' own answers cannot affect later context. The appendix shows large Model-history drops for only two models. Can you extend this to all 13 models to assess interaction performance under self-generated history?
Answer 1.1. Gold-history compares current-turn capability under shared correct context; Model-history tests effects of earlier model answers. Appendix A.12 of the original manuscript already reports Model-history results for DeepSeek-v4-pro (Pro) and GPT-5.6-sol (GPT-sol). We added GLM-5.2 (GLM), Kimi-K2.6 (Kimi) and Qwen3.8-max-0902 (Qwen), expanding to five models—the five families' top performers in our evaluation. Scores are higher under Gold-history than Model-history on FS/RS; Model-history gains over No-history vary by model/task. We will present the five-model comparisons, six-capability results and turn-level cases in §4.1 and the appendix, clarifying what each protocol evaluates and the five-model FS/RS scope.
Shared correct history controls comparison; self-generated history tests sequential performance. Gold-history holds prior facts correct and identical across models, consistent with MT-Bench-101, §3.4. Model-history generates answers sequentially from the seed along the original question chain, adding each to later context. No-history retains the seed and final question. Each Gold/Model condition covers 139 FS/137 RS dialogues and 1,620 scored turns (829/791) per model, including 551 capability turns.
History gains do not simply track baseline prediction ability. In Table 1, relative to No-history, Model-history raises scores for four models on FS (one lower) and three on RS (two lower). Kimi has the highest No-history accuracy but gains 2.88 pp on FS and loses 6.57 pp on RS. Table 2 shows decreases in all six joint capability scores, including 30.16–61.41 pp for EC and 39.86–54.73 pp for RC. The six Gold/Model final-structure comparisons for the three added models have Holm-adjusted McNemar p < 0.012.
Response trajectories locate errors entering later verification and arguments. In FS D4, both conditions yield an incorrect 100°C; the model then accepts the reference 90°C under Gold-history but maintains 100°C under Model-history. In FS D19, the model cites its earlier incorrect Pd/Suzuki judgment to reject copper-catalysis evidence (Table 3). These cases support short-range persistence; aggregate gaps include factual errors, omissions and fixed-follow-up compatibility, without isolating propagation's causal effect.
Table 1. Final-structure accuracy (%)
No/Gold/Model denote No-history/Gold-history/Model-history. FS/RS contain 139/137 final targets, scored by strict equality of canonicalized structural component sets with stereochemistry retained.
Table 2. Joint scores for six capabilities (%, Gold → Model)
AR/CM/EL/EC/RC/SK denote anaphora resolution, context memory, elaboration, error correction, reliability checks, and skeptical reasoning, with 139/137/64/64/74/73 turns per model. Joint scores incorporate chemical correctness and interaction requirements, describing overall performance on these tasks.
Table 3. DeepSeek-v4-pro cases under self-generated history