- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces FireWorldBench, a benchmark for evaluating physical-world reasoning through coupled fire dynamics. It contains 9,074 QA pairs from 494 simulated worlds and 26 real-world-aligned event groups, covering five physical capabilities and three fire-scenario categories. The benchmark pairs choice-based and structured open-report tasks across textual and visual observations, with deterministic scoring of predictions and explanations.
Critical strengths
-
Fire dynamics involve interactions among temperature, smoke, airflow, and ventilation, making them a useful setting for studying continuous physical fields, partial observability, and responses to interventions. Evaluating models in this setting provides a meaningful complement to existing physical reasoning benchmarks.
-
The benchmark covers 47 scene archetypes and 9,074 question–answer instances, with reference answers grounded in simulation states and controlled counterfactual branches for evaluating intervention effects. Paired choice-based and open-report tasks, structured and visual inputs, and distinct physical capability categories enable more detailed analysis of where models struggle across evidence formats and reasoning tasks.
Critical weaknesses
-
The paper lacks in-depth error analysis and case study. The analysis primarily reports performance differences and five qualitative examples show successful responses. Without representative failures and a systematic analysis of recurring failure patterns, it remains unclear why models fail and which limitations future methods should address.
-
The explanation metrics do not adequately establish physical correctness. Mechanism Alignment measures reference-token recall, Evidence-F1 does not verify region–variable relationships, and GLS relies largely on lexical matching rules. So explanations containing correct terminology but incorrect causal relationships may receive high scores.
Questions for authors
Please see the Critical Weaknesses section for my main concerns.
4 comments
Response to Reviewer a2wE (2/2)
W2. Physical correctness of explanations
➡️Metric refinement. We augment the original metrics with physical-relation checks, yielding Evi-F1†, Mech†, and GLS† to assess explanation correctness beyond lexical overlap. Metrics are aggregated using the same model, observation-track, and task weights as in the original paper, with inapplicable samples excluded according to predefined rules.
- Evi-F1† checks region–variable bindings, time, statistical definitions, and numerical values or trends in addition to the original region and variable matching.
- Mech† checks cause–effect relations, causal direction, and necessary conditions in addition to the original reference-mechanism token recall.
- GLS† additionally checks whether each prediction target is supported by evidence and mechanisms with correct physical relations and no direct contradictions, extending the original association between correct predictions and evidence/mechanism anchors.
➡️Alignment with Expert Judgments. Experts assess the physical correctness of model-generated explanations. Using these judgments as references, the newly added metrics achieve AUROC values of 0.827–0.846 as shown in Table 4, supporting their ability to distinguish correct from incorrect explanations.
➡️Conclusion robustness. We compare the original and newly added scores under matched evaluation settings. Table 5 shows slightly lower scores but unchanged capability rankings within each metric, supporting the robustness of our capability-level findings.
Table 4. Agreement between the newly added metrics and expert judgments.
| Supplementary metric | Valid reports | Correct / incorrect explanations | AUROC ↑ | 95% CI |
|---|---|---|---|---|
| Evi-F1† | 190 | 112 / 78 | 0.834 | [0.771, 0.895] |
| Mech† | 172 | 101 / 71 | 0.846 | [0.779, 0.905] |
| GLS† | 184 | 109 / 75 | 0.827 | [0.759, 0.888] |
Table 5. Original and the newly added metrics on the main controlled-simulation set (%).
| Capability | Original Evi-F1 | Evi-F1† | Original Mech | Mech† | Original GLS | GLS† |
|---|---|---|---|---|---|---|
| P1 Temporal forecasting | 58.06 | 56.74 | 36.05 | 34.61 | 23.73 | 22.85 |
| P2 Field perception and grounding | 40.87 | 40.15 | 21.99 | 21.12 | 29.50 | 28.69 |
| P3 Cross-field coupling | 56.33 | 54.48 | 28.16 | 26.93 | 32.12 | 30.74 |
| P4 Mechanism attribution | 30.33 | 29.17 | 15.41 | 14.58 | 29.35 | 28.43 |
| P5 Counterfactual intervention | 9.63 | 9.21 | 18.42 | 17.54 | 27.12 | 26.18 |
We thank the reviewer again and would be glad to clarify anything further.
Thank you for the detailed response. It addresses most of my concerns and I will raise my score.
We sincerely thank you for your constructive review and positive follow-up, which have provided great value for our future research.
Response to Reviewer a2wE (1/2)
We sincerely thank the reviewer for the constructive suggestion.
➡️A systematic analysis of recurring failure patterns. We analyze recurring errors in models’ reasoning about physical fields and identify four failure patterns (Table 1 reports the proportion of each failure pattern among all failed samples.) and their associated causes:
Table 1. Distribution of failure patterns.
➡️Detailed analysis of cross-field integration failure. Table 2 shows that 72 of 158 responses with correct basic field readings still make incorrect cross-field inferences (45.6%). Among these 72 responses, 58 also produce incorrect final answers (80.6%). These results highlight cross-field integration failure as a recurring bottleneck associated with task failure.
Table 2. Visual field-task errors and final-answer outcomes. Parentheses show error counts / corresponding sample counts.
➡️Case study. Table 3 presents representative case analyses illustrating these failure patterns, contrasting models’ incorrect judgments with the corresponding physical-field evidence.
Table 3. Representative cases illustrating each failure pattern.