- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SciConvBench is a benchmark for multi-turn clarification in scientific task formulation. It covers four computational science domains: fluid mechanics, solid mechanics, materials science and PDEs. It has 1,142 cases built by manually perturbing clean tasks, either by removing information (disambiguation) or by planting conflicts (inconsistency resolution). An LLM plays the user, answering only from a hidden reference specification. The assistant asks questions and produces a final specification, which a rubric-based LLM judge scores. The key metrics separate final resolution (FRR) from conversation-grounded resolution (CGRR), so that silent assumptions show up as the gap between them. Eight models are evaluated. The authors report that no model leads on both tasks, that disambiguation is much harder than inconsistency resolution, and that every model has a persistent FRR–CGRR gap. They back this with a human-agreement study on 80 cases, and with ablations over judge, prompt and user simulator.
Critical strengths
[1] The benchmark targets a real issue, since scientific queries often carry hidden assumptions and a wrong assumption can lead to a wrong solution. It supports multi-turn problem solving across fluid mechanics, solid mechanics, materials science and PDEs.
[2] The evaluation section is detailed and clearly written. It allows an in-depth assessment of the scenarios that can arise in a multi-turn conversation.
[3] The experiments are thorough, with ablations and evaluations that anticipate the questions a reader is likely to raise, such as the user simulator ablation and the guided vs. unguided comparison.
Critical weaknesses
[1] The “missing Reynolds number” example in the introduction (line 52) is specific to fluid dynamics. This makes the motivation feel narrower than the benchmark’s claimed scope across four domains.
[2] The phrase “plausible but incorrect flow regime” only holds for the user with that particular goal. Another user might happily accept a result based on an assumed Reynolds number. The authors should clarify that intent and ambiguity resolution depend on the user and may not be wrong in every setting.
[3] Scientific tasks mix mathematical, logical and informational content, and most prior work already has a benchmark for each of these. The paper does not convince the reader that clarification and dialogue in science differ enough from these to need its own benchmark and evaluation criteria.
[4] The paper does not explain why fluid mechanics, solid mechanics, materials science and PDEs were chosen.
[5] Nit: The labels in Figure 3 are too small to read.
Questions for authors
How many experts reviewed the benchmark for quality control? How many samples were rejected or retained during the review?
5 comments
Response to Reviewer
W3: Why does clarification in science need its own benchmark?
We agree that a single CLAMBER subset, dominated by lexical ambiguity, cannot on its own show that scientific clarification is harder: it differs from SciConvBench in domain knowledge, ambiguity type and task complexity at the same time. To separate these factors, we added three more general-domain conversational clarification benchmarks:
- GSM-Q holds the ambiguity type fixed. Like our disambiguation cases, each problem is missing one required parameter, but recovering it needs no scientific knowledge.
- IN3 holds task complexity fixed. Each task has 2–5 missing details that must all be elicited, like the multiple planted issues per SciConvBench case.
- AmbigQA tests a case where a plausible default reading exists. Each question has valid readings, so a model can answer without asking. It shows how much silent resolution occurs outside science.
All of them were run through the exact SciConvBench disambiguation pipeline, with the same assistant (Gemini 2.5 Pro), simulated user, turn budget, rubric judges and scoring rules.
Sampling. These benchmarks are large: CLAMBER has 3,202 queries and AmbigQA has 2,002 validation questions. Each evaluation is a full multi-turn conversation followed by three judge calls, so we evaluate a random subset of each benchmark. To make sure the conclusions do not depend on which cases were sampled, we drew three independent subsets of the same size from each benchmark, using the same selection rules. Each draw has different cases and newly drawn hidden answers, and each was run end to end. We report the pooled rates with case-bootstrap 95% CIs.
- GSM-Q (QuestBench): math word problems with one required quantity removed. This is missing-parameter ambiguity, the same type as our disambiguation task, but without scientific domain knowledge. Each draw covers all 34 problems and withholds a different quantity in each.
- AmbigQA: factoid questions with annotated readings. Of the 2,002 validation questions, annotators labelled 1,002 as ambiguous. Each draw samples 50 of these uniformly.
- IN3: vague everyday agent tasks. 89 test tasks have 2–5 important missing details, which is closest to SciConvBench's multiple issues per case. Each draw samples 50 of these uniformly.
- CLAMBER: of its 3,202 queries, 1,601 require clarification. Each draw samples 115 of these.
| Set | Ambiguity type | ||||
|---|---|---|---|---|---|
| GSM-Q | missing quantity (math) | 82.4 [74.5, 89.2] | 82.4 [74.5, 89.2] | 0.0 [0.0, 0.0] | |
| CLAMBER | lexical / semantic | 86.4 [82.6, 89.9] | 85.2 [81.4, 88.7] | 1.2 [0.3, 2.3] | |
| AmbigQA | multiple interpretations | 68.7 [61.3, 76.0] | 64.7 [56.7, 72.0] | 4.0 [1.3, 7.3] | |
| IN3 | missing task details | 34.0 [26.7, 41.3] | 27.3 [20.7, 34.7] | 6.7 [2.7, 10.7] | |
| SciConvBench (all) | missing scientific requirements | 570 | 38.9 [34.9, 43.0] | 33.5 [29.6, 37.4] | 5.4 [3.7, 7.4] |
| — Fluid mechanics | 151 | 23.8 [17.2, 30.5] | 13.9 [8.6, 19.9] | 9.9 [5.3, 15.2] | |
| — Solid mechanics | 228 | 29.8 [24.1, 36.0] | 27.6 [21.9, 33.3] | 2.2 [0.4, 4.4] | |
| — Materials science | 130 | 57.7 [49.2, 66.2] | 52.3 [43.8, 60.8] | 5.4 [1.5, 9.2] | |
| — PDEs | 61 | 70.5 [59.0, 82.0] | 63.9 [50.8, 75.4] | 6.6 [1.6, 13.1] |
What differs in science.
To our knowledge, SciConvBench is the first benchmark for conversational task formulation in computational science. The comparison shows two properties that the mathematical, logical and informational benchmarks do not exhibit.
- Science is harder where domain knowledge is required. SciConvBench's CI lies entirely below those of others by – points. Fluid mechanics is the hardest set of all ( [8.6, 19.9]), with a CI below even IN3's (27.3 [20.7, 34.7]). Difficulty also varies across scientific domains, with materials science and PDEs scoring higher.
- Science exposes silent resolution with physical consequences. Where no sensible default exists (GSM-Q, CLAMBER): the model has to ask. Silent resolution appears where defaults exist (AmbigQA, IN3), and it is highest in fluid mechanics () despite that domain having the lowest . There, textbook defaults let the model fill gaps without asking.
SciConvBench also provides what none of these benchmarks have:
- planted issues tied to an ontology of scientific specification components;
- per-issue references that separate grounded from silent resolution;
- inconsistency resolution with physically contradictory requirements.
We will add these results to the revised PDF.
Response to Reviewer
W4: Choice of domains
We chose fluid mechanics, solid mechanics, materials science and PDEs for the following reasons:
- They are core areas of computational science and engineering. Together they cover a large share of routine simulation work: CFD, structural and finite-element analysis, materials modelling and numerical PDE solution.
- Each domain stresses different specification requirements.
- Fluid mechanics: the governing regime, for example compressibility, turbulence and steady vs. transient flow.
- Solid mechanics: constitutive and kinematic assumptions, for example linear elastic vs. plastic behaviour, plane stress vs. plane strain, and small vs. finite strain.
- Materials science: composition, structure and thermodynamic state.
- PDEs: the mathematical requirements common to the other domains, namely boundary conditions on, initial conditions the domain and the numerics.
- Together they cover the full ontology of specification components, rather than a single type of missing parameter.
- Specifications are well defined and checkable. Established simulation tools in these domains require explicit inputs. This lets us construct complete reference specifications, plant missing or contradictory requirements, and score resolution for each issue. It also supports tool-oriented subsets alongside general problem statements.
- Domain expertise supports validation. Annotators with expertise in these areas constructed and reviewed the cases, to make sure every planted issue is scientifically meaningful.
The benchmark format can also be extended to other domains, such as chemistry and climate modelling.
W5: Figure 3 label size
Thank you for pointing this out. We will enlarge the axis labels, tick labels and legend in Figure 3 so they are readable at print size, and upload the same in the revised.
Q1: Expert quality control
Two domain experts, reviewed every case. For each case they checked that:
- the hidden or conflicting information is scientifically meaningful
- the case admits a clear intended resolution
- the prompt does not leak the answer through trivial cues
- the conversational variant remains realistic
Cases that failed these checks were revised or discarded.
Of roughly drafted cases, were retained and about were removed:
- disambiguation cases (fluids and PDEs) that every evaluated model already solved were removed
- inconsistency cases with for all evaluated models were removed
- solid-mechanics tool-use prompts longer than k characters;
- about further cases dropped during review and revision.
We will report these counts per domain in the appendix.
Response to the Reviewer
We thank the reviewer for their thorough review and constructive questions, particularly on the comparison with general domain benchmarks, which helped us strengthen the paper's empirical claims.
W1: Fluid-specific motivating example
We agree that the Reynolds number example gives a narrow first impression. It was meant as a simple illustration of why clarification matters. Leaving out one parameter in Re=ρUL/μ can move the problem from laminar to turbulent flow, which changes the physics being studied. The example does not define the benchmark's scope.
SciConvBench covers fluid mechanics, solid mechanics, materials science and PDEs, and the same failure appears in each of these domains. Example: In solid mechanics, a request may not say whether the problem is plane stress (σzz=0) or plane strain (εzz=0), or which constitutive model applies. In materials science, the alloy composition or the reference temperature Tref may be missing. In PDEs, the boundary conditions , the initial condition or the domain itself may be left unspecified. The planted issues follow an ontology of specification components that all four domains share:
They are not restricted to fluid specific parameters. We will make this breadth clear in the introduction by adding a short example from each of the other three domains.
W2: Acceptability of silent assumptions depends on user intent
We agree that whether a default is acceptable depends on the user's intent. A default Reynolds number Re may suit one user, but for another it answers a different scientific question. Without asking, the assistant cannot tell these situations apart. A silent assumption may match the intended task, but nothing in the conversation confirms that it does.
This matters because the task has to be formulated before it is executed. If a mismatch is found only after a downstream simulation finishes, hours or days of computation, and their cost, may have gone into the wrong problem. A clarifying question could settle the intended choice before that computation begins.
SciConvBench therefore does not treat every silent resolution as wrong. The two metrics differ as follows:
We will revise the introduction to explain that an assumed value is not inherently wrong, but stays unverified against the user's intent until the dialogue addresses it.