acceptodds
Under review as a conference paper at ICLR 2027

FlowIntentBench: Can LLM Agents Analyze Flow Fields to Answer Scientific Questions?

Abstract

Answering scientific questions about numerical flow fields requires choosing an appropriate analysis and determining what the resulting evidence supports. Evaluating LLM agents on this task is challenging because different valid analyses can yield different findings from the same data. We introduce FlowIntentBench, a benchmark of 24 scientific question families across 15 flow-field datasets. Each family is instantiated in four conditions that vary which analysis decisions the question specifies and how findings are to be selected and reported, and is paired with a set of accepted operationalizations (definitions, measurements, and decision rules) together with the findings each yields. Responses are scored on the validity of the stated analysis, on recall of the findings it requires, and on the consistency between findings and operationalization. Across nine tool-using LLM agents, every model scores lower when all principal analysis decisions are left to it. Mean validity falls from 0.846 to 0.742 and recall from 0.910 to 0.863. Our results show that current agents perform reliably when an analysis is prescribed, but struggle when they must determine how to analyze the scientific question.

Primary area:
datasets and benchmarks
Keywords:
large language models, scientific data analysis, benchmarks, flow fields

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_9SfGOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

FlowIntentBench is a benchmark for evaluating whether LLM agents can analyze numerical flow fields and answer scientific questions. The benchmark contains 96 cases from 24 question families across 15 flow-field datasets. Each question family has four conditions that vary how much of the analysis is prescribed and which findings the agent must report. A key feature of the benchmark is that it allows multiple valid analyses of the same scientific question, each associated with its own reference findings.

Critical strengths

I would like to regard the following point as the critical strength of this paper: it makes a useful distinction between executing a prescribed analysis and deciding how a scientific question should be analyzed, allowing different operationalizations to produce different valid findings.

Critical weaknesses
  1. The main contribution of this benchmark is its ability to accommodate multiple valid analyses. However, the number of stored reference branches is quite limited.
  2. The operationalization scores depend heavily on an LLM evaluator judging whether the stated analysis is scientifically appropriate. Unlike numerical verification, these judgments are not deterministic, and even human experts may disagree on what constitutes an appropriate analysis for an underspecified scientific question.
Questions for authors

Could the authors report evaluator–expert agreement separately for each of the five metrics, especially the operationalization score and URS?

5 comments
AuthorsOpenReview ↗

Response to Review (1/2)

Q1. The main contribution of this benchmark is its ability to accommodate multiple valid analyses. However, the number of stored reference branches is quite limited.

A1. Thank you for raising this concern. The stored reference branches provide a finite set of verified analyses, rather than exhaustive coverage of all scientifically valid operationalizations. Each O2–F1 case contains 2–3 reference branches, and each O3–F1 case contains 2–4 (Appendix F.6). We agree that this limits the coverage of reference-based finding verification.

Importantly, an operationalization does not need to match a stored branch to receive credit. The operationalization score (SOS_O) assesses whether the analysis is clearly defined, satisfies task constraints, and is scientifically appropriate; URS evaluates the condition-designated open dimensions under the corresponding rubric. For example, GPT-5.6 Luna’s excess-speed analysis for Blunt Fin O2–F1 falls outside the stored peak/mean branches but receives SO=0.979S_O=0.979 and URS=0.958\mathrm{URS}=0.958 (Appendix G.4). This illustrates that operationalization scoring can recognize alternatives beyond the stored branches.

Scientific findings are verified separately. When compatible reference results or frozen supplementary evidence are insufficient, affected metric scores remain N/A. Thus, the benchmark accommodates alternative analyses, but does not guarantee verification of every valid analysis. The coverage counts and paired analysis added to Appendix H.4 make this limitation explicit and assess its effect on the reported O1–F1 versus O3–F1 comparison.

Q2. The operationalization scores depend heavily on an LLM evaluator judging whether the stated analysis is scientifically appropriate. Unlike numerical verification, these judgments are not deterministic, and even human experts may disagree on what constitutes an appropriate analysis for an underspecified scientific question.

A2. Thanks for your professional suggestions. To support reliable operationalization scoring, we use explicit criteria for each attribute. Fixed attributes are assessed against task constraints, while open attributes are assessed for scientific appropriateness given the target and available data. Each attribute receives a grade under the rubric in Appendix A.1.

To assess scoring reliability, we kept 96 answers generated by GPT-6 Astra and their corresponding case-specific ground truth fixed. The GPT-6 Astra evaluator scored each answer three times using the configuration in Appendix G.2. The table reports ICC(A,1) and mean pairwise MAE. Scores range from 0 to 1, and URS applies to 48 of these answers. We have added the design and results of this experiment to Appendix I.1 (Table 31) of the revised manuscript.

MetricSample sizeICC(A,1)MAE
Operationalization score (SOS_O)960.85490.0057
Unresolved-choice score (URS\mathrm{URS})480.82530.0082

These results show small variation across repeated evaluations of this fixed input set.

AuthorsOpenReview ↗

Response to Review (2/2)

Q3. Could the authors report evaluator–expert agreement separately for each of the five metrics, especially the operationalization score and URS?

A3. Thank you for requesting this breakdown. For GPT-6 Astra, the evaluator used in our experiments, Pearson correlations with the final expert scores range from 0.871 to 0.914 across the five metrics, including 0.914 for operationalization quality and 0.871 for URS. The table below reports each correlation and its valid pair count. Only answers with numerical scores from both the evaluator and the expert reference are included; N/A scores are excluded.

MetricValid pairsPearson rr
Operationalization score (SOS_O)930.9135
Unresolved-choice score (URS)450.8706
Finding recall (RFR_F)920.9012
Finding precision (PFP_F)940.8867
O–F consistency (COFC_{OF})910.8929

These results show strong positive correlations with expert scores for both operationalization and finding evaluation.

We have added the metric-specific correlations and valid pair counts for all nine candidate evaluators to Appendix I.1 (Table 30) of the revised manuscript.

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end with less than 24 hours remaining, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you’d like us to consider, please let us know. Your insights are invaluable to us, and we’re eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

Reviewer_4xbQOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents FlowIntentBench, a benchmark for evaluating LLM agents on scientific analysis in computational fluid dynamics, specifically the analysis of flow fields. The core question behind the benchmark is whether an LLM agent can not only execute a specified scientific analysis, but also determine what analysis should be performed, execute it, and report conclusions that are supported by that analysis. To build the benchmark, the authors rely on existing flow-field datasets and design 24 scientific-question families, each instantiated in four conditions that progressively vary how much responsibility the agent has for making analysis choices and selecting findings to report. The authors evaluate several LLM agents on the benchmark and find that performance deteriorates as agents are given greater responsibility for defining the analysis themselves. Finally, they conduct a small-scale human study that provides additional validation of their LLM-based evaluation system.

Critical strengths
  • The experimental decomposition of the benchmark into different levels of agent responsibility is well thought out and allows for a nuanced evaluation of LLM agents' capabilities in flow-field scientific analysis. It is much more informative than benchmarking agents on arbitrary questions with varying difficulty. It directly manipulates how much analytical responsibility belongs to the agent.
  • The benchmark separately evaluates whether the chosen analysis is scientifically appropriate and whether the reported findings are actually supported by that analysis. This handles the important case where multiple valid analyses can produce different defensible answers.
  • The paper (especially in the appendix) provides a clear methodology for constructing the benchmark and for evaluating LLM agents, which enhances the reproducibility and transparency of the benchmark.
Critical weaknesses
  • The main weakness of the paper is the limited scope of scientific analysis it covers, focusing solely on flow fields in computational fluid dynamics. Sometimes, reading the paper gives the impression that the benchmark may not generalize well to other types of scientific analyses.
Questions for authors
  • Is the agent allowed to explore several analyses and settle on an operationalization after observing intermediate results? If so, that means O3 primarily tests whether the agent can iteratively arrive at and report an appropriate operationalization, rather than cleanly testing its ability to independently select an analysis before observing its results.
3 comments

Q1. The main weakness of the paper is the limited scope of scientific analysis it covers, focusing solely on flow fields in computational fluid dynamics. Sometimes, reading the paper gives the impression that the benchmark may not generalize well to other types of scientific analyses.

A1. Thank you for encouraging us to clarify the scope. FlowIntentBench is a domain-specific benchmark, and our empirical conclusions are limited to the flow-field tasks evaluated here. Flow fields provide a concrete setting in which different operationalizations can address the same scientific target on the same data, while their resulting findings can be computationally verified. This makes it possible to examine both the appropriateness of analysis choices and the evidence supporting their conclusions.

The potentially transferable element is the evaluation approach: distinguishing operationalization validity from evidential support and checking their consistency. Applying this approach to another domain would require domain-specific task definitions, valid operationalizations, and computationally verified reference evidence. Thus, cross-domain applicability remains to be validated rather than established by the current experiments.

Q2. Is the agent allowed to explore several analyses?

A2. Thank you for this interesting question. Yes, agents may explore multiple analyses. Within the task constraints, they can call the Python tool repeatedly, try different analyses, and revise their approach before submitting an answer. O1–O3 use the same tool-interaction rules and differ in which analysis decisions the question specifies (Section 5.1 and Appendix B.5).

Q3. Can the agent settle on an operationalization after observing intermediate results?

A3. Yes, your interpretation is consistent with our experimental setting. O3 evaluates whether an agent can determine an appropriate operationalization through iterative exploration and report findings supported by evidence. Evaluation uses the operationalization and scientific findings stated in the final answer (Section 4.1). The experiment does not require agents to fix their analysis before observing results. We have clarified this setting and its evaluation implications in Section 5.1 of the revised manuscript.

Thank you for answering my questions and addressing my main concern. It is totally fine for FlowIntentBench to be a domain-specific benchmark. I was maybe a bit nitpicky when reading the paper the first time. I got a bit misled by sentences like:

  • "The second, benchmarks for scientific data analysis, evaluates agents’ analysis choices and scientific findings."
  • "We develop an evaluation framework that pairs accepted operationalizations with executed reference findings to assess analysis validity, finding recall and precision, and consistency between findings and the stated operationalization."

which seem to hint at being an evaluation framework that can be generalized to other domains. Maybe it is but that's not in the scope of this paper and that's fair. :)

AuthorsOpenReview ↗

Thank You Very Much for Your Feedback

Thank you for your follow-up and for pointing out these two sentences. Your comment has been invaluable in improving our paper. In the revised manuscript, we have clarified that FlowIntentBench is a benchmark for flow-field analysis. Specifically, the second contribution in Section 1 now describes our evaluation as "an evaluation method for flow-field questions". In addition, the opening of Section 2 now presents benchmarks for scientific data analysis as prior work across disciplines and places FlowIntentBench at the intersection of language-guided flow analysis and these benchmarks. If you have any further suggestions or questions, we would be glad to address them.

Reviewer_R4t9OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce FlowIntentBench to assess whether tool-using LLM agents can answer scientific questions about numerical flow fields when they must determine part or all of the analysis themselves. The benchmark contains 24 scientific question families over 15 three-dimensional flow-field datasets. Each family is instantiated in four conditions. O1–F1 specifies all principal analysis decisions, O2–F1 leaves one or two principal decisions open, O3–F1 asks the agent to formulate the analysis under task constraints, and O1–F2 retains the prescribed analysis while requiring the agent to select a sufficient set of findings.

The benchmark represents ground truth as paired operationalization–finding branches. The evaluation checks whether the agent identifies the required findings, whether those findings are accurate and supported by evidence, and whether they are consistent with the agent’s stated analysis approach. This is intended to accommodate cases in which different scientifically valid analyses yield different numerical conclusions.

Nine tool-using LLM agents are evaluated with three runs on every case, for 2592 runs in a common network-isolated environment. Both operationalization quality and recall are lower in O3–F1 than in O1–F1, suggesting that agents struggle more when they must choose how to conduct the analysis themselves. The paper additionally validates its evaluator against two doctoral-student reviewers and conducts a user study with 15 doctoral students, who show qualitatively similar degradation as analysis responsibility increases.

Critical strengths
  • Pairing operationalizations with executed findings ties numerical claims to the analytical choices that produced them. A plausible value can therefore be distinguished from a result supported by the declared measurement, aggregation, or representation. Stored execution evidence and fixed comparison policies make these judgments inspectable.
  • The controlled task-family design examines analysis formulation and finding selection as distinct evaluation dimensions. O1, O2 and O3 hold the scientific target, data, and required finding roles fixed while progressively delegating analysis decisions to the agent. This makes responsibility for analysis formulation a controlled dimension of the task. O1–F2 separately examines whether an agent can select findings that constitute a sufficient answer after the analysis has been prescribed.
Critical weaknesses
  • Finding scores partly depend on whether the frozen reference evidence covers the agent’s chosen analysis. Results from otherwise valid analyses can remain unverified, with affected metric scores excluded from the reported averages. Model and condition comparisons may reflect differences in reference coverage alongside differences in answer quality.
  • The comparison between F1 and F2 mixes responsibility for choosing findings with changes in the required answer content. In the Office example, F1 requires location and strength, whereas F2 additionally requires an informative characterization chosen by the agent. Lower performance under F2 may therefore reflect the additional analytical and reporting demands, rather than difficulty selecting findings alone, which makes the comparison difficult to interpret as a distinct measure of finding-selection ability.
Questions for authors
  • If evaluation is restricted to a shared set of verifiable answers, how will the results be affected? How should findings outside the frozen reference branches contribute to the reported results?
  • How can the F1–F2 comparison separate the difficulty of selecting findings from the difficulty introduced by additional content requirements?
4 comments
AuthorsOpenReview ↗

Response to Review (1/2)

Q1. If evaluation is restricted to a shared set of verifiable answers, how will the results be affected?

A1. Thank you for raising this concern. We have added a paired analysis to examine whether differences in reference coverage affect the O1–F1 versus O3–F1 comparison. This analysis shows that the performance decrease remains essentially unchanged when the comparison is restricted to answers that are verifiable under both conditions.

We matched O1–F1 and O3–F1 answers by model, question family, and run, retaining pairs with numerical scores under both conditions for each metric. This yields 616 jointly verifiable pairs, covering 95.1% of the 648 possible pairs.

MetricReported ΔPaired Δ
Finding recall (RFR_F)−0.0469−0.0467
Finding precision (PFP_F)−0.0470−0.0469
O–F consistency (COFC_{OF})−0.0459−0.0459

Here, Δ denotes the mean score under O3–F1 minus that under O1–F1. Across all three metrics, restricting the comparison to jointly verifiable answers changes the estimated decreases by at most 0.0002, supporting the robustness of the observed decline within this shared subset.

The paired analysis has been added to Appendix H.4 (Table 28) of the revised manuscript.

Q2. How should findings outside the frozen reference branches contribute to the reported results?

A2. Thank you for the helpful comments. Findings outside the frozen reference branches are included in the reported scores when sufficient evidence permits verification. Specifically (Section 4.2 and Appendices A.5–A.6):

  • Verifiable findings: When compatible reference results or frozen supplementary evidence permit verification, findings are scored according to the verification outcome. Confirmed errors and established omissions of required findings receive zero for the corresponding items.
  • Unverified findings: When the available evidence is insufficient, affected metric scores that cannot be determined are recorded as N/A and excluded from the corresponding means. They are not treated as either correct or incorrect.

We agree that the reported means are conditional on verifiability and should be accompanied by coverage information. The table below summarizes valid/N/A counts for the three finding metrics, aggregated across models.

ConditionFinding recall (RFR_F)Finding precision (PFP_F)O–F consistency (COFC_{OF})
O1–F1626/6622/10616/16
O2–F1625/11623/13612/24
O3–F1626/6622/10616/16
O1–F2631/10630/11618/23

We have added the full valid/N/A counts for every metric, model, and condition to Appendix H.4 (Table 27) of the revised manuscript and referenced this table in Section 5.2.

AuthorsOpenReview ↗

Response to Review (2/2)

Q3. The comparison between F1 and F2 mixes responsibility for choosing findings with changes in the required answer content. In the Office example, F1 requires location and strength, whereas F2 additionally requires an informative characterization chosen by the agent. Lower performance under F2 may therefore reflect the additional analytical and reporting demands, rather than difficulty selecting findings alone, which makes the comparison difficult to interpret as a distinct measure of finding-selection ability.

How can the F1–F2 comparison separate the difficulty of selecting findings from the difficulty introduced by additional content requirements?

A3. Thank you for highlighting this distinction. The current F1–F2 comparison does not isolate finding-selection difficulty from additional analysis and reporting demands. In the Office example, F2 requires an informative characterization beyond the location and strength required in F1, so lower performance cannot be attributed to finding selection alone.

We therefore interpret F1–F2 as evaluating the transition from reporting specified findings to selecting and supporting a sufficient set of findings, including the associated analysis and reporting requirements (Appendix F.4). We have clarified this evaluation target and its limitation in Section 5.2 and the caption of the Office example in Appendix D of the revised manuscript.

Thank you for the detailed responses and the additional paired analysis. These substantially address the two concerns raised in my review.

Reference coverage and verifiability. The comparison restricted to jointly verifiable O1–F1 and O3–F1 answers is directly relevant to my concern. The finding-score decreases remain essentially unchanged, with differences of at most 0.0002 from the originally reported estimates. This provides reassuring evidence that differential verifiability does not materially explain the observed decline in this comparison.

The clarification that findings outside the stored branches can still be verified using compatible reference results or frozen supplementary evidence is also helpful. Reporting valid/N/A counts by model, condition, and metric improves transparency. The remaining qualification is that these means are conditional on verifiability: the paired analysis supports the robustness of the O1–F1 versus O3–F1 result within the shared verifiable subset, but does not establish the quality of answers that remain unverified. Please retain this qualification alongside the main results rather than only in the appendix.

Interpretation of the F1–F2 comparison. I appreciate the explicit acknowledgment that this comparison does not isolate finding-selection difficulty from additional analysis and reporting demands. Interpreting it as the transition from reporting specified findings to selecting and supporting a sufficient set of findings is more accurate and remains a useful evaluation target. The clarification addresses my concern about overinterpreting the comparison, although the underlying confounding remains a limitation of the current design. Please ensure that this broader interpretation is used consistently throughout the paper wherever F1–F2 results are discussed.

Overall, the rebuttal strengthens the empirical support for the analysis-formulation result and appropriately narrows the interpretation of the finding-selection comparison. The paired operationalization–finding representation remains a useful contribution to scientific-agent evaluation. I will maintain my weak-acceptance rating. Thank you for incorporating the additional analysis, coverage information, and interpretive clarifications into the revision.

AuthorsOpenReview ↗

Thank You Very Much for Your Feedback

We sincerely thank you for your feedback! We are glad to hear that our responses address your concerns. We deeply appreciate your time and effort in reviewing our paper and providing insightful and constructive comments. If you have any further suggestions or questions, we will make every effort to address them.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Reviews and Author Rebuttal

Dear Reviewers, ACs, and SACs,

We sincerely thank you for the constructive feedback and thoughtful suggestions. We are glad that the reviewers recognized our task design, our evaluation of both analyses and findings, and our clear methodology. The main strengths noted are:

Controlled task design: All reviewers agreed that our task design clearly separates following a given analysis from deciding how to analyze a scientific question.

  • R_R4t9: "The controlled task-family design examines analysis formulation and finding selection as distinct evaluation dimensions."
  • R_4xbQ: "The experimental decomposition of the benchmark into different levels of agent responsibility is well thought out ... It is much more informative than benchmarking agents on arbitrary questions with varying difficulty."
  • R_9SfG: "It makes a useful distinction between executing a prescribed analysis and deciding how a scientific question should be analyzed."

Evaluation that supports multiple valid analyses: All reviewers found it valuable that our evaluation links findings to the analyses that produce them and allows multiple valid analyses.

  • R_R4t9: "Pairing operationalizations with executed findings ties numerical claims to the analytical choices that produced them."
  • R_4xbQ: "This handles the important case where multiple valid analyses can produce different defensible answers."
  • R_9SfG: "A key feature of the benchmark is that it allows multiple valid analyses of the same scientific question, each associated with its own reference findings."

Clear and transparent methodology: Reviewers found our benchmark construction and evaluation clear and inspectable.

  • R_R4t9: "Stored execution evidence and fixed comparison policies make these judgments inspectable."
  • R_4xbQ: "The paper (especially in the appendix) provides a clear methodology ... which enhances the reproducibility and transparency of the benchmark."

We also deeply appreciate the insightful comments of each reviewer and have carefully addressed them. Below, we summarize the main review concerns and our responses/solutions.

Additional quantitative analysis:

  • Conducted a paired analysis on answers verifiable under both O1–F1 and O3–F1 to assess the effect of reference coverage (R_R4t9, R_9SfG).
  • Reported valid/N/A counts for every metric, model, and condition (R_R4t9, R_9SfG).
  • Reported evaluator–expert agreement separately for each of the five metrics (R_9SfG).
  • Evaluated the repeatability of the operationalization score and URS across repeated evaluations (R_9SfG).

Clarifications:

  • Clarified how findings outside the stored reference branches are verified and scored (R_R4t9, R_9SfG).
  • Clarified that an operationalization does not need to match a stored reference branch to receive credit (R_9SfG).
  • Clarified the evaluation target of the F1–F2 comparison and its limitation (R_R4t9).
  • Clarified that agents may explore multiple analyses before answering, and what O3 evaluates under this setting (R_4xbQ).
  • Clarified that FlowIntentBench is a domain-specific benchmark for flow-field analysis (R_4xbQ).

Reviewer feedback after the rebuttal:

  • R_R4t9 noted that our responses and the paired analysis "substantially address the two concerns" and that "the rebuttal strengthens the empirical support for the analysis-formulation result."
  • R_4xbQ confirmed that our responses addressed their main concern and that "it is totally fine for FlowIntentBench to be a domain-specific benchmark."

We are confident that these additional analyses and clarifications will effectively address the reviewers' concerns. We deeply appreciate your careful review, constructive feedback, and valuable time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.