- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
FlowIntentBench is a benchmark for evaluating whether LLM agents can analyze numerical flow fields and answer scientific questions. The benchmark contains 96 cases from 24 question families across 15 flow-field datasets. Each question family has four conditions that vary how much of the analysis is prescribed and which findings the agent must report. A key feature of the benchmark is that it allows multiple valid analyses of the same scientific question, each associated with its own reference findings.
Critical strengths
I would like to regard the following point as the critical strength of this paper: it makes a useful distinction between executing a prescribed analysis and deciding how a scientific question should be analyzed, allowing different operationalizations to produce different valid findings.
Critical weaknesses
- The main contribution of this benchmark is its ability to accommodate multiple valid analyses. However, the number of stored reference branches is quite limited.
- The operationalization scores depend heavily on an LLM evaluator judging whether the stated analysis is scientifically appropriate. Unlike numerical verification, these judgments are not deterministic, and even human experts may disagree on what constitutes an appropriate analysis for an underspecified scientific question.
Questions for authors
Could the authors report evaluator–expert agreement separately for each of the five metrics, especially the operationalization score and URS?
5 comments
Response to Review (2/2)
Q3. Could the authors report evaluator–expert agreement separately for each of the five metrics, especially the operationalization score and URS?
A3. Thank you for requesting this breakdown. For GPT-6 Astra, the evaluator used in our experiments, Pearson correlations with the final expert scores range from 0.871 to 0.914 across the five metrics, including 0.914 for operationalization quality and 0.871 for URS. The table below reports each correlation and its valid pair count. Only answers with numerical scores from both the evaluator and the expert reference are included; N/A scores are excluded.
| Metric | Valid pairs | Pearson |
|---|---|---|
| Operationalization score () | 93 | 0.9135 |
| Unresolved-choice score (URS) | 45 | 0.8706 |
| Finding recall () | 92 | 0.9012 |
| Finding precision () | 94 | 0.8867 |
| O–F consistency () | 91 | 0.8929 |
These results show strong positive correlations with expert scores for both operationalization and finding evaluation.
We have added the metric-specific correlations and valid pair counts for all nine candidate evaluators to Appendix I.1 (Table 30) of the revised manuscript.
Dear Reviewer,
I hope this message finds you well. As the discussion period is nearing its end with less than 24 hours remaining, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you’d like us to consider, please let us know. Your insights are invaluable to us, and we’re eager to address any remaining issues to improve our work.
Thank you for your time and effort in reviewing our paper.
Response to Review (1/2)
A1. Thank you for raising this concern. The stored reference branches provide a finite set of verified analyses, rather than exhaustive coverage of all scientifically valid operationalizations. Each O2–F1 case contains 2–3 reference branches, and each O3–F1 case contains 2–4 (Appendix F.6). We agree that this limits the coverage of reference-based finding verification.
Importantly, an operationalization does not need to match a stored branch to receive credit. The operationalization score (SO) assesses whether the analysis is clearly defined, satisfies task constraints, and is scientifically appropriate; URS evaluates the condition-designated open dimensions under the corresponding rubric. For example, GPT-5.6 Luna’s excess-speed analysis for Blunt Fin O2–F1 falls outside the stored peak/mean branches but receives SO=0.979 and URS=0.958 (Appendix G.4). This illustrates that operationalization scoring can recognize alternatives beyond the stored branches.
Scientific findings are verified separately. When compatible reference results or frozen supplementary evidence are insufficient, affected metric scores remain N/A. Thus, the benchmark accommodates alternative analyses, but does not guarantee verification of every valid analysis. The coverage counts and paired analysis added to Appendix H.4 make this limitation explicit and assess its effect on the reported O1–F1 versus O3–F1 comparison.
A2. Thanks for your professional suggestions. To support reliable operationalization scoring, we use explicit criteria for each attribute. Fixed attributes are assessed against task constraints, while open attributes are assessed for scientific appropriateness given the target and available data. Each attribute receives a grade under the rubric in Appendix A.1.
To assess scoring reliability, we kept 96 answers generated by GPT-6 Astra and their corresponding case-specific ground truth fixed. The GPT-6 Astra evaluator scored each answer three times using the configuration in Appendix G.2. The table reports ICC(A,1) and mean pairwise MAE. Scores range from 0 to 1, and URS applies to 48 of these answers. We have added the design and results of this experiment to Appendix I.1 (Table 31) of the revised manuscript.
These results show small variation across repeated evaluations of this fixed input set.