acceptodds
Under review as a conference paper at ICLR 2027

SciConvBench: Benchmarking LLMs on Multi-Turn Clarification for Task Formulation in Computational Science

Abstract

Large Language Models (LLMs) are increasingly studied as scientific assistants, with benchmarks assessing knowledge, reasoning, code generation, and tool use. These evaluations, however, typically assume the scientific problem is already well-posed, whereas practical scientific assistance often begins with an ill-posed user request that must be refined through dialogue before any computation, analysis, or experiment can be carried out reliably. We introduce SciConvBench, a benchmark for multi-turn clarification in scientific task formulation across four computational science problem domains: fluid mechanics, solid mechanics, materials science, and partial differential equations (PDEs). It is designed to evaluate two complementary capabilities: eliciting missing information (disambiguation) and detecting and correcting erroneous requests containing internally contradictory information (inconsistency resolution). Our benchmark combines a detailed task ontology with a rigorous rubric-based evaluation framework to deliver reproducible results and quantify LLM performance across clarification behavior, conversational grounding, and final-specification fidelity. Current frontier models perform relatively well on inconsistency resolution, but even the best model resolves only 52.7% of the disambiguation cases through dialogue, averaged across the domains. We further find that frontier LLMs often make silent assumptions and perform implicit repairs that are not grounded in the conversation. SciConvBench establishes a foundation for evaluating the upstream conversational reasoning that a reliable computational science assistant requires.

Primary area:
datasets and benchmarks
Keywords:
computational science, large language models, benchmark, clarification, conversational agent

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_iRccOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

SciConvBench is a benchmark for multi-turn clarification in scientific task formulation. It covers four computational science domains: fluid mechanics, solid mechanics, materials science and PDEs. It has 1,142 cases built by manually perturbing clean tasks, either by removing information (disambiguation) or by planting conflicts (inconsistency resolution). An LLM plays the user, answering only from a hidden reference specification. The assistant asks questions and produces a final specification, which a rubric-based LLM judge scores. The key metrics separate final resolution (FRR) from conversation-grounded resolution (CGRR), so that silent assumptions show up as the gap between them. Eight models are evaluated. The authors report that no model leads on both tasks, that disambiguation is much harder than inconsistency resolution, and that every model has a persistent FRR–CGRR gap. They back this with a human-agreement study on 80 cases, and with ablations over judge, prompt and user simulator.

Critical strengths

[1] The benchmark targets a real issue, since scientific queries often carry hidden assumptions and a wrong assumption can lead to a wrong solution. It supports multi-turn problem solving across fluid mechanics, solid mechanics, materials science and PDEs.

[2] The evaluation section is detailed and clearly written. It allows an in-depth assessment of the scenarios that can arise in a multi-turn conversation.

[3] The experiments are thorough, with ablations and evaluations that anticipate the questions a reader is likely to raise, such as the user simulator ablation and the guided vs. unguided comparison.

Critical weaknesses

[1] The “missing Reynolds number” example in the introduction (line 52) is specific to fluid dynamics. This makes the motivation feel narrower than the benchmark’s claimed scope across four domains.

[2] The phrase “plausible but incorrect flow regime” only holds for the user with that particular goal. Another user might happily accept a result based on an assumed Reynolds number. The authors should clarify that intent and ambiguity resolution depend on the user and may not be wrong in every setting.

[3] Scientific tasks mix mathematical, logical and informational content, and most prior work already has a benchmark for each of these. The paper does not convince the reader that clarification and dialogue in science differ enough from these to need its own benchmark and evaluation criteria.

[4] The paper does not explain why fluid mechanics, solid mechanics, materials science and PDEs were chosen.

[5] Nit: The labels in Figure 3 are too small to read.

Questions for authors

How many experts reviewed the benchmark for quality control? How many samples were rejected or retained during the review?

5 comments
AuthorsOpenReview ↗

Response to the Reviewer

We thank the reviewer for their thorough review and constructive questions, particularly on the comparison with general domain benchmarks, which helped us strengthen the paper's empirical claims.

W1: Fluid-specific motivating example

We agree that the Reynolds number example gives a narrow first impression. It was meant as a simple illustration of why clarification matters. Leaving out one parameter in Re=ρUL/μ\mathrm{Re} = \rho U L / \mu can move the problem from laminar to turbulent flow, which changes the physics being studied. The example does not define the benchmark's scope.

SciConvBench covers fluid mechanics, solid mechanics, materials science and PDEs, and the same failure appears in each of these domains. Example: In solid mechanics, a request may not say whether the problem is plane stress (σzz=0\sigma_{zz} = 0) or plane strain (εzz=0\varepsilon_{zz} = 0), or which constitutive model applies. In materials science, the alloy composition or the reference temperature TrefT_{\mathrm{ref}} may be missing. In PDEs, the boundary conditions , the initial condition or the domain itself may be left unspecified. The planted issues follow an ontology of specification components that all four domains share:

  • governing physics and regime
  • material properties
  • boundary and initial conditions
  • geometry and domain
  • numerics

They are not restricted to fluid specific parameters. We will make this breadth clear in the introduction by adding a short example from each of the other three domains.

W2: Acceptability of silent assumptions depends on user intent

We agree that whether a default is acceptable depends on the user's intent. A default Reynolds number Re\mathrm{Re} may suit one user, but for another it answers a different scientific question. Without asking, the assistant cannot tell these situations apart. A silent assumption may match the intended task, but nothing in the conversation confirms that it does.

This matters because the task has to be formulated before it is executed. If a mismatch is found only after a downstream simulation finishes, hours or days of computation, and their cost, may have gone into the wrong problem. A clarifying question could settle the intended choice before that computation begins.

SciConvBench therefore does not treat every silent resolution as wrong. The two metrics differ as follows:

  • FRR\mathrm{FRR} credits a correct final specification however it was reached.
  • CGRR\mathrm{CGRR} additionally requires that each choice was grounded in the conversation.

We will revise the introduction to explain that an assumed value is not inherently wrong, but stays unverified against the user's intent until the dialogue addresses it.

AuthorsOpenReview ↗

Response to Reviewer

W3: Why does clarification in science need its own benchmark?

We agree that a single CLAMBER subset, dominated by lexical ambiguity, cannot on its own show that scientific clarification is harder: it differs from SciConvBench in domain knowledge, ambiguity type and task complexity at the same time. To separate these factors, we added three more general-domain conversational clarification benchmarks:

  • GSM-Q holds the ambiguity type fixed. Like our disambiguation cases, each problem is missing one required parameter, but recovering it needs no scientific knowledge.
  • IN3 holds task complexity fixed. Each task has 2–5 missing details that must all be elicited, like the multiple planted issues per SciConvBench case.
  • AmbigQA tests a case where a plausible default reading exists. Each question has ≥2\geq 2 valid readings, so a model can answer without asking. It shows how much silent resolution occurs outside science.

All of them were run through the exact SciConvBench disambiguation pipeline, with the same assistant (Gemini 2.5 Pro), simulated user, turn budget, rubric judges and scoring rules.

Sampling. These benchmarks are large: CLAMBER has 3,202 queries and AmbigQA has 2,002 validation questions. Each evaluation is a full multi-turn conversation followed by three judge calls, so we evaluate a random subset of each benchmark. To make sure the conclusions do not depend on which cases were sampled, we drew three independent subsets of the same size from each benchmark, using the same selection rules. Each draw has different cases and newly drawn hidden answers, and each was run end to end. We report the pooled rates with case-bootstrap 95% CIs.

  • GSM-Q (QuestBench): math word problems with one required quantity removed. This is missing-parameter ambiguity, the same type as our disambiguation task, but without scientific domain knowledge. Each draw covers all 34 problems and withholds a different quantity in each.
  • AmbigQA: factoid questions with ≥2\geq 2 annotated readings. Of the 2,002 validation questions, annotators labelled 1,002 as ambiguous. Each draw samples 50 of these uniformly.
  • IN3: vague everyday agent tasks. 89 test tasks have 2–5 important missing details, which is closest to SciConvBench's multiple issues per case. Each draw samples 50 of these uniformly.
  • CLAMBER: of its 3,202 queries, 1,601 require clarification. Each draw samples 115 of these.
SetAmbiguity typennFRR\mathrm{FRR}CGRR\mathrm{CGRR}SRR\mathrm{SRR}
GSM-Qmissing quantity (math)34×334 \times 382.4 [74.5, 89.2]82.4 [74.5, 89.2]0.0 [0.0, 0.0]
CLAMBERlexical / semantic115×3115 \times 386.4 [82.6, 89.9]85.2 [81.4, 88.7]1.2 [0.3, 2.3]
AmbigQAmultiple interpretations50×350 \times 368.7 [61.3, 76.0]64.7 [56.7, 72.0]4.0 [1.3, 7.3]
IN3missing task details50×350 \times 334.0 [26.7, 41.3]27.3 [20.7, 34.7]6.7 [2.7, 10.7]
SciConvBench (all)missing scientific requirements57038.9 [34.9, 43.0]33.5 [29.6, 37.4]5.4 [3.7, 7.4]
— Fluid mechanics15123.8 [17.2, 30.5]13.9 [8.6, 19.9]9.9 [5.3, 15.2]
— Solid mechanics22829.8 [24.1, 36.0]27.6 [21.9, 33.3]2.2 [0.4, 4.4]
— Materials science13057.7 [49.2, 66.2]52.3 [43.8, 60.8]5.4 [1.5, 9.2]
— PDEs6170.5 [59.0, 82.0]63.9 [50.8, 75.4]6.6 [1.6, 13.1]

What differs in science.

To our knowledge, SciConvBench is the first benchmark for conversational task formulation in computational science. The comparison shows two properties that the mathematical, logical and informational benchmarks do not exhibit.

  1. Science is harder where domain knowledge is required. SciConvBench's CGRR\mathrm{CGRR} CI lies entirely below those of others by 3030–5050 points. Fluid mechanics is the hardest set of all (CGRR=13.9\mathrm{CGRR} = 13.9 [8.6, 19.9]), with a CI below even IN3's (27.3 [20.7, 34.7]). Difficulty also varies across scientific domains, with materials science and PDEs scoring higher.
  2. Science exposes silent resolution with physical consequences. Where no sensible default exists (GSM-Q, CLAMBER): the model has to ask. Silent resolution appears where defaults exist (AmbigQA, IN3), and it is highest in fluid mechanics (SRR=9.9\mathrm{SRR} = 9.9) despite that domain having the lowest FRR\mathrm{FRR}. There, textbook defaults let the model fill gaps without asking.

SciConvBench also provides what none of these benchmarks have:

  • planted issues tied to an ontology of scientific specification components;
  • per-issue references that separate grounded from silent resolution;
  • inconsistency resolution with physically contradictory requirements.

We will add these results to the revised PDF.

AuthorsOpenReview ↗

Response to Reviewer

W4: Choice of domains

We chose fluid mechanics, solid mechanics, materials science and PDEs for the following reasons:

  1. They are core areas of computational science and engineering. Together they cover a large share of routine simulation work: CFD, structural and finite-element analysis, materials modelling and numerical PDE solution.
  2. Each domain stresses different specification requirements.
    • Fluid mechanics: the governing regime, for example compressibility, turbulence and steady vs. transient flow.
    • Solid mechanics: constitutive and kinematic assumptions, for example linear elastic vs. plastic behaviour, plane stress vs. plane strain, and small vs. finite strain.
    • Materials science: composition, structure and thermodynamic state.
    • PDEs: the mathematical requirements common to the other domains, namely boundary conditions on, initial conditions the domain and the numerics.
  3. Together they cover the full ontology of specification components, rather than a single type of missing parameter.
  4. Specifications are well defined and checkable. Established simulation tools in these domains require explicit inputs. This lets us construct complete reference specifications, plant missing or contradictory requirements, and score resolution for each issue. It also supports tool-oriented subsets alongside general problem statements.
  5. Domain expertise supports validation. Annotators with expertise in these areas constructed and reviewed the cases, to make sure every planted issue is scientifically meaningful.

The benchmark format can also be extended to other domains, such as chemistry and climate modelling.

W5: Figure 3 label size

Thank you for pointing this out. We will enlarge the axis labels, tick labels and legend in Figure 3 so they are readable at print size, and upload the same in the revised.

Q1: Expert quality control

Two domain experts, reviewed every case. For each case they checked that:

  • the hidden or conflicting information is scientifically meaningful
  • the case admits a clear intended resolution
  • the prompt does not leak the answer through trivial cues
  • the conversational variant remains realistic

Cases that failed these checks were revised or discarded.

Of roughly 1,6121{,}612 drafted cases, 1,1421{,}142 were retained and about 470470 were removed:

  • 292292 disambiguation cases (fluids and PDEs) that every evaluated model already solved were removed
  • 8484 inconsistency cases with FR=1\mathrm{FR} = 1 for all evaluated models were removed
  • 6464 solid-mechanics tool-use prompts longer than 2020k characters;
  • about 3030 further cases dropped during review and revision.

We will report these counts per domain in the appendix.

Reviewer_HTsTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SciConvBench evaluates whether LLMs can turn an ill-posed scientific request into a well-specified task through multi-turn dialogue, rather than assuming (as most scientific benchmarks do) that the task is already well-formed. The authors construct 1,142 cases across fluid mechanics, solid mechanics, materials science, and PDEs by perturbing clean task specifications with either deleted entries (disambiguation) or planted contradictions (inconsistency resolution), up to 8 issues per case. An assistant LLM interacts with a simulated user for up to 10 turns, one question per turn, then emits a final specification scored by a rubric-based LLM judge against the case’s planted ground truth. The central design move is separating Final Resolution Rate (correct end state) from Conversation-Grounded Resolution Rate (correct and every planted issue surfaced in dialogue), with their difference defined as Silent Resolution Rate. Across eight models, FRR exceeds CGRR in every domain and task.

Critical strengths
  • The FRR/CGRR decomposition is a transferable evaluation idea, not just a domain benchmark. Separating “reached the right end state” from “reached it through an auditable process” addresses a failure mode that end-state metrics structurally cannot see, and applies to any agentic setting where a correct outcome can be reached by unexamined inference.

  • Statistical reporting is handled properly, and the judge-and-simulator pipeline is validated along all the axes, making the experiment results and conclusions reliable.

Critical weaknesses
  • The measurement error on CGRR is comparable to the effect being reported. The three expert annotators score CGRR at 55.0 / 40.0 / 45.0 on the same 80 cases — a 15-point spread on the paper’s primary metric — with Fleiss’ κ = 0.65. The headline FRR–CGRR gap is 12.2pp and 18.4pp.

  • The Pareto analysis is circular: its axes are re-aggregations of the judgments that define CGRR.

Questions for authors

Can you report the pairwise correlations among the Capability, Robustness, and Usability axes, and clarify how the Pareto analysis should be read given that Usability is a factor inside CGRR and that (1−AR)=CR(1-\text{AR}) = \text{CR} whenever FR = 1? Would redefining Robustness over only MCR and CP preserve the §5.1 conclusions?

2 comments
AuthorsOpenReview ↗

Response to Reviewer

We thank the reviewer for the valuable comments on the manuscript. The following addresses the reviewer concerns:

W1: Annotator spread on CGRR

The 15-point spread is in the absolute CGRR\mathrm{CGRR} across raters. Our claim is about the gap FRR−CGRR\mathrm{FRR} - \mathrm{CGRR} within each rater. Since CGR=1\mathrm{CGR} = 1 requires FR=1\mathrm{FR} = 1, this gap is simply the share of cases that each rater labels as silently resolved:

AnnotatorFRR\mathrm{FRR}CGRR\mathrm{CGRR}FRR−CGRR\mathrm{FRR} - \mathrm{CGRR} [95% Wilson CI]
A175.055.020.0 [12.7, 30.0]
A268.840.028.8 [20.0, 39.5]
A373.845.028.8 [20.0, 39.5]

Every rater sees a gap of at least 20 points, and every lower bound is well above zero. Stricter raters lower CGRR\mathrm{CGRR}, but none removes the gap. The disagreement lies almost entirely in grounding. It comes from different views on what counts as "surfacing" an issue, the same rubric ambiguity that splits the LLM judges in Case 5 (Appendix F.3). We will make this criterion more precise in the rubric.

  • The subset gaps should not be compared directly with the headline gaps (12.2 and 18.4). The 80-case subset was stratified by outcome, so its rates differ from the full benchmark by design.
  • Judge–human agreement on CGR\mathrm{CGR} (κ=0.67\kappa = 0.67–0.720.72) matches agreement between humans (Fleiss' κ=0.65\kappa = 0.65). The judge is therefore as reliable as an individual expert.

W2: Circularity of the Pareto analysis

We agree that the axes re-aggregate the same judge outputs, so the Pareto analysis is not independent evidence. Our main claim, the FRR\mathrm{FRR}–CGRR\mathrm{CGRR} gap, does not rely on it. The analysis is diagnostic. CGRR\mathrm{CGRR} says whether a model succeeds, and the axes say why it falls short:

  • whether it asks the right questions (Capability);
  • whether it tracks earlier answers and conflicts (Robustness);
  • whether it preserves what the user wants (Usability).

The overlap with CGRR\mathrm{CGRR} is also only partial. Clarification precision, plan completeness, memory consistency and error detection are separate judgments that do not enter CGRR\mathrm{CGRR}.

Q1: Correlations, reading the plot, and Robustness over MCR and CP

Pairwise correlations (Pearson rr over all 64 model ×\times domain ×\times task cells in Tables 3–4):

CapabilityRobustnessUsability
Capability10.730.66
Robustness10.27
Usability1

The axes are related but not redundant.

Reading the plot. Usability is one of the conditions for CGR\mathrm{CGR}, so the plot is not a second measure of success. It breaks a model's CGRR\mathrm{CGRR} down into contributing factors, and CGRR\mathrm{CGRR} remains the primary metric.

Robustness =mean(MCR,CP)= \mathrm{mean}(\mathrm{MCR}, \mathrm{CP}). These are the top three models per task, averaged across domains:

TaskRankOriginal Robustnessmean(MCR,CP)\mathrm{mean}(\mathrm{MCR}, \mathrm{CP})
Disambiguation1GPT-5.2 (0.92)GPT-OSS-120B (0.82)
2Gemini 2.5 Pro (0.88)GPT-5.2 (0.81)
3GPT-OSS-120B (0.88)Gemini 2.5 Pro (0.78)
Inconsistency1Gemini 2.5 Pro (0.79)Gemini 2.5 Pro (0.80)
2GPT-5.2 (0.67)Gemini 2.5 Flash (0.76)
3Gemini 2.5 Flash (0.65)GPT-5.2 (0.67)

These are the changes from disambiguation to inconsistency resolution, averaged over all models and domains:

AxisDisambiguationInconsistencyChange
Capability0.690.54−0.15-0.15
Robustness (original)0.810.58−0.23-0.23
Robustness =mean(MCR,CP)= \mathrm{mean}(\mathrm{MCR}, \mathrm{CP})0.760.60−0.16-0.16
Usability0.610.63+0.02+0.02

The model-level conclusions hold:

  • Gemini 2.5 Pro remains the most robust model on inconsistency resolution.
  • GPT-5.2 keeps the strongest disambiguation profile. It is first on Capability (0.80) and Usability (0.77) and nearly tied for first on Robustness (0.81 vs. 0.82).
AuthorsOpenReview ↗

Follow up on our responses

Thank you again for your valuable comments. As the discussion period is coming to a close, we wanted to check whether our responses have addressed your concerns. If they have, we would be grateful if you would consider updating your score. If anything remains unclear, we would be happy to clarify further. The pdf of the manuscript has also been updated based on the response.

Reviewer_8UdsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SciConvBench, a benchmark for evaluating the ability of LLMs to clarify incomplete or inconsistent scientific requests through multi-turn conversations. The benchmark contains 1,142 cases across four computational science domains, covering two tasks: disambiguation and inconsistency resolution. The authors introduce a rubric-based evaluation framework that distinguishes final task correctness from conversation-grounded resolution, allowing them to identify cases where models silently make assumptions or correct inconsistencies without consulting the user. Experiments on eight LLMs show that models generally perform better on inconsistency resolution than disambiguation, and that correct final specifications do not always result from explicit clarification.

Critical strengths
  1. The paper is easy to follow. The motivation, benchmark construction, and evaluation framework are clearly presented.

  2. The paper focuses on a meaningful problem that is often overlooked in existing scientific benchmarks. Evaluating whether LLMs can clarify incomplete or inconsistent scientific requests before task execution is relevant to developing reliable scientific assistants.

  3. The distinction between final resolution and conversation-grounded resolution is interesting. It allows the benchmark to identify cases where models produce correct final specifications without explicitly clarifying missing or conflicting information with users.

  4. The evaluation is relatively comprehensive. The benchmark covers four scientific domains and eight LLMs, and the authors provide additional analyses using human annotations, different LLM judges, prompt variations, and user simulators to examine the reliability of their findings.

Critical weaknesses
  1. The evaluation criteria may not fully reflect successful scientific task formulation. The paper uses Conversation-Grounded Resolution Rate (CGRR) as its primary metric, requiring all annotated issues to be explicitly clarified through dialogue. However, in practical scientific workflows, some missing parameters can be reasonably determined from established conventions or delegated to the assistant by the user. In such cases, making an assumption does not necessarily indicate a failure. The current evaluation does not clearly distinguish unjustified assumptions from acceptable scientific defaults, which may affect the interpretation of the reported results.

  2. The practical significance of the reported findings is not sufficiently demonstrated. The paper shows that models frequently produce correct final specifications without explicit clarification, but it does not establish whether these silent resolutions lead to incorrect scientific outcomes. Since the benchmark does not evaluate downstream simulation or task execution, it remains unclear how strongly CGRR relates to the reliability of scientific workflows. This weakens the practical implications of the main findings.

  3. The comparison with existing clarification benchmarks is not entirely convincing. The paper compares SciConvBench with an adapted subset of CLAMBER and reports a substantial performance gap. However, the two datasets differ in domain knowledge requirements, ambiguity types, and task complexity. In addition, the adapted CLAMBER subset is dominated by lexical ambiguity cases. Therefore, the observed difference may not necessarily demonstrate that scientific clarification is inherently more challenging, and the comparison provides limited evidence of the benchmark's distinctive difficulty.

Questions for authors
  1. Some scientific parameters can be determined from standard conventions without asking users. How does the benchmark distinguish such reasonable assumptions from cases where clarification is necessary? Would these cases still be considered failures under CGRR?

  2. The paper reports a consistent gap between FRR and CGRR. Is there any evidence that silent resolution leads to incorrect or unreliable downstream scientific results? This would help clarify the practical importance of the observed gap.

  3. Appendix F.5 states that CLAMBER does not originally evaluate multi-turn resolution, so the authors construct hidden complete requests for 115 selected cases. How were these cases selected, and how was the correctness of the constructed reference answers verified? Could this adaptation affect the reported 86.1% resolution rate and its comparison with SciConvBench?

4 comments
AuthorsOpenReview ↗

Response to Reviewer

We thank the reviewer for the careful and detailed review, and for recognizing the relevance of clarification for reliable scientific assistants. We address each point below and will incorporate the changes in the revised manuscript.

W1 / Q1: Acceptable scientific defaults

We agree that some parameters in scientific tasks can reasonably be assumed. Our evaluation does not count such assumptions as failures. A correct final specification that relies on a silent default is counted as resolved (FRR\text{FRR}) but not conversationally grounded (CGRR\text{CGRR}). The benchmark separates these assumptions from choices that need clarification in four ways.

  • Planted issues are ones conventions cannot settle. During expert filtering, each planted issue was checked to depend on the user's intent, such as the flow regime, constitutive model or boundary conditions. Unit conventions and values recoverable from the prompt are excluded from the silent resolution check, so they never count against CGRR\text{CGRR}.
  • Correct silent defaults receive credit. FRR\text{FRR} credits a correct final specification even when it contains a silent default. They do not receive CGRR\text{CGRR} credit because they lack conversational grounding.
  • CGRR\text{CGRR} measures whether a choice is traceable. Before execution, a correct silent default and an incorrect one can look equally plausible. Asking the user makes the choice explicit and confirms that it matches the intended task. This is why CGRR\text{CGRR} is our primary metric, with FRR\text{FRR} and SRR\text{SRR} reported alongside it to account for cases correctly resolved but not conversationally grounded. We will make these points explicit in the revised manuscript.

W2 / Q2: Does silent resolution lead to incorrect scientific results?

We agree that silent resolution can produce both a correct specification and a correct scientific result. Our benchmark allows for this: FRR\text{FRR} gives these cases full credit.

To test this, we took 45 fluid mechanics cases resolved by Gemini 2.5 Pro whose target quantity follows from a closed form relation. For each, we hand coded that relation and evaluated it with parameters from the ground-truth specification (q⋆q^{\star}), the assistant's final specification (qq), and the underspecified prompt with generic defaults. Everything is computed in code, without an LLM, and a result counts as correct when ∣q−q⋆∣/∣q⋆∣≤5%|q - q^{\star}|/|q^{\star}| \leq 5\%. where q⋆q^{\star} is the value obtained from the ground-truth specification. The results are as follows:

SpecificationCorrect within 5%
Final specification, grounded (FR=1, CGR=1\text{FR}=1,\ \text{CGR}=1)20/22 (91%)
Final specification, silent (FR=1, SR=1\text{FR}=1,\ \text{SR}=1)5/6 (83%)
Final specification, unresolved (FR=0\text{FR}=0)9/17 (53%)
Original underspecified prompt11/40 (28%)

Grounded and silent resolutions reproduce the reference result at similar rates (91% and 83%). Silent resolution therefore does not, by itself, make the scientific result wrong.

What CGRR\text{CGRR} additionally measures is traceability. When a model silently fills in a missing value or repairs a contradiction, the final specification may be correct, but the user is never told that a choice was made. They cannot check the assumption, correct it if it conflicts with their intent, or reproduce how the specification was reached. Grounded resolution makes these decisions visible in the dialogue. So a positive SRR=FRR−CGRR\text{SRR} = \text{FRR} - \text{CGRR} does not mean the results are wrong. It identifies correct results that depend on choices the user never saw.

We will state this distinction explicitly, add the execution study to the appendix.

AuthorsOpenReview ↗

Response to Reviewer

W3: Comparison with existing clarification benchmarks

We agree that one CLAMBER subset, dominated by lexical ambiguity, is not enough on its own. We therefore added three more general domain conversational clarification benchmarks covering different ambiguity types: GSM-Q, AmbigQA and IN3. We ran all of them, through the exact SciConvBench disambiguation pipeline: the same assistant (Gemini 2.5 Pro), simulated user, turn budget, rubric judges and scoring rules.

Sampling. These benchmarks contain many cases (example: 3,202 queries in CLAMBER and 2,002 validation questions in AmbigQA). Each SciConvBench evaluation is a full multi-turn conversation followed by three judge calls, so we evaluate a random subset of each benchmark. To make sure that the conclusions do not depend on which cases were sampled, we drew three independent subsets of the same size for each benchmark with the same selection rules. Each draw has different cases and newly drawn hidden answers and was run end to end.

  • GSM-Q (QuestBench): math word problems with one required quantity removed. This is missing parameter ambiguity, the closest general domain analogue to our disambiguation task. Each draw covers 34 problems, withholding a different quantity per problem.
  • AmbigQA: factoid questions with two or more annotated readings. Of the 2,002 validation questions, 1,002 are labelled ambiguous. 50 cases per draw are sampled uniformly from these.
  • IN3: vague everyday agent tasks with 2–5 important missing details. Of the test tasks, 89 are vaugue with 2–5 important missing details. 50 cases per draw are sampled uniformly from these.
  • CLAMBER: of its 3,202 queries, 1,601 require clarification; 115 cases per draw are sampled from these.
SetAmbiguity typenFRRCGRRSRR
GSM-Qmissing quantity (math)34 × 382.4 [74.5, 89.2]82.4 [74.5, 89.2]0.0 [0.0, 0.0]
CLAMBERlexical / semantic115 × 386.4 [82.6, 89.9]85.2 [81.4, 88.7]1.2 [0.3, 2.3]
AmbigQAmultiple interpretations50 × 368.7 [61.3, 76.0]64.7 [56.7, 72.0]4.0 [1.3, 7.3]
IN3missing task details50 × 334.0 [26.7, 41.3]27.3 [20.7, 34.7]6.7 [2.7, 10.7]
SciConvBench (all)missing scientific requirements57038.9 [34.9, 43.0]33.5 [29.6, 37.4]5.4 [3.7, 7.4]
— Fluid mechanics15123.8 [17.2, 30.5]13.9 [8.6, 19.9]9.9 [5.3, 15.2]
— Solid mechanics22829.8 [24.1, 36.0]27.6 [21.9, 33.3]2.2 [0.4, 4.4]
— Materials science13057.7 [49.2, 66.2]52.3 [43.8, 60.8]5.4 [1.5, 9.2]
— PDEs6170.5 [59.0, 82.0]63.9 [50.8, 75.4]6.6 [1.6, 13.1]

These results provide further evidence of the need for science specific benchmark.

  1. Science is harder where domain knowledge is required. SciConvBench's CGRR CI lies entirely below those of GSM-Q, CLAMBER and AmbigQA, by 30–50 points. Fluid mechanics is the hardest set of all (CGRR 13.9), with a CI below even IN3's. Difficulty also varies across scientific domains (materials science and PDEs score higher), which is why we report per-domain results rather than a single score.

  2. Science exposes silent resolution with physical consequences. Where no sensible default exists (GSM-Q, CLAMBER), SRR is near zero: the model has to ask, so FRR and CGRR coincide. Silent resolution appears where defaults exist (AmbigQA, IN3), and is highest in fluid mechanics despite its lowest FRR. There, textbook defaults let the model fill gaps without asking. A benchmark that scores only the final answer cannot see this, and the FRR–CGRR split is designed to expose it.

SciConvBench also provides what none of these benchmarks have:

  • planted issues tied to an ontology of scientific specification components;
  • per-issue references that separate grounded from silent resolution;
  • inconsistency resolution with physically contradictory requirements.

We will add these results to the revised PDF.

Q3: CLAMBER subset construction and its effect on the comparison

The 115 cases were sampled at random from CLAMBER's clarification-requiring queries and span four subtypes. Each hidden complete request adds exactly one clarification that answers CLAMBER's own reference question, so a case is resolved only if that single answer is recovered. The clarifications were generated for each query, and generic or templated answers that did not commit to a specific interpretation were removed.

Effect on the reported rate.

  • Independence from our adaptation. Three further benchmarks with their own reference answers (GSM-Q, AmbigQA, IN3), run under the same protocol, give the same picture. The comparison therefore does not rest on the CLAMBER adaptation.

  • Robustness to the adaptation. Two further 115-case draws from the original CLAMBER release, with the same subtype mix and newly constructed hidden answers, give metrics consistent with the reported subset.

AuthorsOpenReview ↗

Follow up on our responses

Thank you again for your valuable comments. As the discussion period is coming to a close, we wanted to check whether our responses have addressed your concerns. If they have, we would be grateful if you would consider updating your score. If anything remains unclear, we would be happy to clarify further. The pdf of the manuscript has also been updated based on the response.

I thank the authors for their thoughtful response. It resolves most of my concerns hence I raise my score. Thanks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.