acceptodds
Under review as a conference paper at ICLR 2027

FloodMark: A Benchmark for Flood Scene Understanding and Reasoning in Multimodal Large Language Models

Abstract

Multimodal Large Language Models (MLLMs) are increasingly used to interpret complex visual scenes and support domain-specific decision making. To com- prehensively evaluate their capabilities in flood-scene imagery understanding, we introduce FloodMark, a benchmark for fine-grained, evidence-aware flood haz- ard assessment. FloodMark contains 1,400 real-world flood and post-flood images and over 16,000 human-annotated image question pairs across 15 expert-authored tasks, covering Evidence Quality, Scene Context, Flood-state Perception, Physical Reasoning, and Operational Reasoning. The benchmark evaluates whether mod- els recognize when visual evidence is insufficient to support a reliable conclusion. We evaluate 13 recent proprietary and open-weight MLLMs. Gemini 3.8 Flash achieves the highest overall accuracy of 73.2%, showing substantial room for im- provement. More importantly, when the image does not provide enough evidence for a reliable answer, on average, models still give a specific answer in 70.0% of cases. This tendency can lead to unsupported conclusions about flood conditions and their operational consequences. These findings show that current MLLMs remain unreliable for fine-grained, evidence-aware flood-scene assessment

Primary area:
datasets and benchmarks
Keywords:
Multimodal Large Language Models, Flood Understanding, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_DALdOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FloodMark evaluates flood-scene understanding with 1,400 real-world images and 16,403 question-answer pairs across 15 tasks. It covers scene context, flood conditions, physical and operational judgments, and image quality, with explicit options for insufficient evidence. The authors evaluate 13 models, and also analyze abstention, capture conditions, and sampled model errors.

Critical strengths
  1. The tasks cover useful aspects of flood assessment beyond detecting whether flooding is present, including depth, passability, and obstruction.
  2. Distinguishing supported absence from insufficient evidence is a useful design choice, especially for images with poor visibility or limited coverage.
  3. The questions are human-authored, annotations include independent labeling and disagreement resolution, and the 13-model evaluation provides detailed task-level results.
Critical weaknesses
  1. The added evaluation value over existing work needs stronger evidence. Table 4 shows broader task coverage, but does not establish what distinct capability gap FloodMark reveals beyond disaster VQA and answerability benchmarks. The findings on physical judgments and insufficient evidence do not clearly show what is specifically learned from the new flood tasks. A focused comparison with overlapping existing tasks, or an analysis showing how the added tasks change our assessment of model capabilities, would strengthen the contribution.
  2. The physical and operational labels are not sufficiently validated. Absolute depth intervals require reliable reference cues, while passability depends on vehicle assumptions and decision criteria. Section 3.2 mentions annotation rules, but these are not provided or validated in the manuscript. Average AC1 establishes consistency, not the physical meaning of the labels. A representative validation subset would help show that the benchmark measures physically grounded hazard reasoning rather than mainly agreement with the authors’ visual judgments.
  3. The data construction leaves coverage and uncertainty selection unclear. Source proportions, event coverage, and duplicate handling are not reported. More importantly, Section 3.4 excludes unresolved questions without characterizing them. For a benchmark centered on insufficient evidence, it matters whether these cases reflect annotation errors, unclear task rules, or genuinely ambiguous imagery. Per-task exclusion counts and an analysis of removed cases are needed to assess whether filtering preserves the uncertainty cases the benchmark aims to evaluate.
  4. The evaluation does not isolate reasoning from simpler visual decisions. Water Depth and Water Flow include no-flood or not-applicable answers, which contribute to physical-task scores without requiring depth estimation or motion inference. Results on applicable cases are not reported separately. Independent question evaluation also leaves unclear whether operational errors arise from perception, physical inference, or inconsistent decisions. Conditional results on relevant flood cases, majority/no-image baselines, and consistency checks across depth, flow, and passability would better support the capability claims.
  5. The metrics do not fully support the operational-reliability conclusions. Matching “Cannot determine” is only a proxy for recognizing insufficient evidence; Water Depth’s separate no-reliable-scale option needs consistent treatment. The paper reports over-abstention, but not errors among answered hazard cases or their operational direction. Incorrect “any vehicle can pass” and “no vehicle can pass” predictions receive the same penalty despite different consequences. Task-specific abstention accounting and a breakdown of unsafe versus conservative judgments would connect the results more directly to the stated motivation.
Questions for authors

Please refer to the Critical Weaknesses.

7 comments
AuthorsOpenReview ↗

Response to Reviewer DALd

We thank the reviewer for the careful analysis and for pointing out where the contribution and protocol needed clearer documentation.

W1 — Evidence for the contribution

Existing flood and disaster VQA benchmarks ask what an image contains. We ask what a responder could justifiably conclude from it, which changes the task set in two ways.

First, other benchmarks ask about damage and passability, but not about the physical observations behind them. We ask for an absolute depth band in metres, for where the water sits relative to a named object in the scene, and for whether the water is still, flowing or fast. Without these questions, a model that misreads the water and one that misapplies the rule produce the same score.

Second, we evaluate the image as evidence. Two tasks rate how far a frame can be read and name what limits it, and insufficient evidence is a correct answer set by a domain expert, not inferred from model uncertainty or induced by breaking the question. We also separate a supported absence from an unreadable frame, which earlier benchmarks score as one answer.

W2 — Validation of the physical and operational labels

The gold labels in FloodMark are based on independent observer agreement, which is the appropriate standard for this setting. In practice, when responders assess flood conditions from camera feeds, the image is all they have. There is no co-located gauge or field report at the time of capture, and physical measurements are not available for these images. The full question text and answer options for all 15 tasks are in Appendices B and D, and the options themselves are the decision criteria. The depth question includes an explicit option for images where nothing in the frame gives a reliable sense of scale, and annotators select it when no reference is available rather than estimating a depth. For passability, the answer options distinguish any vehicle from a high-clearance vehicle such as a pickup or SUV, and the question asks about the worst-affected stretch visible.

W3 — Coverage and the unresolved questions

The table gives each source's share of the released collection, its licence, and accuracy averaged over the 13 models.

SourceImagesLicenceMean acc
FloodIMG34.3%CC0-1.060.3
CVFD31.4%CC-BY-SA-4.066.8
State DOT traffic cameras28.2%Agency public records via the Internet Archive61.4
NOAA NGS aerial6.2%US Government work, public domain64.3

34.4% of the images carry a named hurricane.

EventImages
Hurricane Harvey 201717.0%
Hurricane Ian 202213.3%
Hurricane Michael 20181.8%
Hurricane Ida 20211.5%
Hurricane Florence 20180.7%

The remaining 65.6% come from two public flood datasets that carry no per-image event or location metadata in their original releases. Duplicate removal was performed, and the released collection contains no image that duplicates another.

Our agreement-and-discard protocol ensures that every item in the released benchmark has a gold label that independent observers agreed on. Items without agreement are excluded, so they do not affect the benchmark or its scores. A gold answer of "Cannot determine" means observers independently agreed the image does not support an answer, so it is a finding about evidence quality, not an unresolved question.

W4 — Isolating reasoning from simpler visual decisions

Whether water is present is part of the task rather than a precondition for it. A responder scanning camera feeds is not told which frames contain water, so deciding that there is nothing to measure is one of the judgements we evaluate, and it is reached by the same reading of the scene that a depth estimate would require. Restricting the depth question to frames that already contain water would hand the model the answer to the prior question.

W5 — Metrics and operational reliability

FloodMark evaluates whether VLMs can match human-level observation of flood scenes. It is an evaluation benchmark, not a deployment-readiness assessment, and we did not train any model. Whether a wrong prediction is dangerous or merely conservative depends on the downstream application, which is out of our scope. Our finding that models lag far behind humans on physical reasoning and evidence assessment already identifies where the gap is and what future work should focus on. Connecting that gap to a specific operational cost structure is valuable but requires assumptions about the deployment context that an evaluation benchmark should not prescribe.

Thank you again for proposing the reviews. We hope the additional analyses clarify FloodMark's contribution and protocol. We welcome any follow-up questions.

Dear Reviewer DALd,

We wanted to follow up as the discussion period is coming to an end. We would appreciate it if you could take another look at our response and revised manuscript. Please let us know if you have any remaining concerns. We hope the additional evidence helps address your concerns, and we'd be grateful if you could consider raising your score.

Thank you for your time!

Thank you for the clarifications. I still have three main concerns:

  1. Agreement between annotators shows consistency, but does not establish that the depth and passability labels are physically well grounded. I understand that measurements are unavailable, but clearer rules for estimating depth, judging passability, and identifying flow from a single image, along with some validation examples, would help address this concern.

  2. Reporting results on applicable flood cases would not give models any extra information. The existing predictions can simply be grouped by gold labels after evaluation. Keeping the overall score while adding these results would help separate recognizing no-water/non-applicable cases from judging depth and flow.

  3. The depth task has both “no reliable sense of scale” and “Cannot determine.” From Table 3’s definitions, choosing the former when the gold is the latter may count as false certainty, and the reverse as over-abstention. No-reliable-scale cases may also remain in Section 5.2’s “determinate” subset. Explaining how these options are counted and treating both as insufficient-evidence responses for this analysis would make the findings easier to interpret.

The added details are helpful, but I do not think the current response resolves these concerns about what the benchmark measures and what the results show.

Reviewer_DAYXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FloodMark, a benchmark for evaluating fine-grained and evidence-aware flood-scene understanding in MLLMs, containing 1,400 images and 16,403 human-annotated image–question pairs covering 5 assessment dimensions. It also distinguishes between a supported negative observation and a case in which the image does not provide enough evidence to answer reliably. The authors evaluate 13 MLLMs using a unified zero-shot multiple-choice protocol. They found that current models perform better on scene recognition and directly visible flood conditions than on physical-state estimation and evidence assessment, and that they frequently provide specific answers even when the available visual evidence is insufficient.

Critical strengths

(1) FloodMark goes beyond conventional flood detection by covering water depth, flow, road passability, debris, and damage. More importantly, it distinguishes “nothing is visible and the view is sufficient” from “cannot determine.” Considering that an unsupported negative judgment may be more harmful than an explicit refusal to answer, this distinction is essential for further applications.

(2) The paper gives a broad diagnostic evaluation supported by independent annotation. Each image is initially labeled by two annotators, with disagreements reviewed by a third annotator. The evaluation covers 13 recent mainstream MLLMs and examines task-level performance, naturally occurring capture conditions, abstention failures, and manually categorized errors. These analyses provide more insight into model limitations than a single aggregate accuracy score.

Critical weaknesses

(1) The composition and independence of the image collection are insufficiently documented. Section 3.3 only states that the images come from public websites, traffic cameras, and aerial sources. The paper does not report the contributing sources, flood events, geographic regions, dates, camera sites, licensing conditions, or the number of images from each category. It also does not describe duplicate or near-duplicate removal.

(2) The appendix indicates that water depth, flow, passability, and damage labels were assigned through visual judgment by the authors rather than verified against physical measurements or external records, but we don’t know the annotation protocol, annotator expertise, labeling criteria, adjudication process, and representative ambiguous cases. Maybe reporting task-specific agreement statistics will be more persuasive than only an overall average. If physical measurements or incident reports are available for any subset of images, they should be used for validation.

Questions for authors

(1) In the representative image problem example shown in Figure 1, for the options F, G, and H of Physical Reasoning, if the large model selects F or H, will it also choose G at the same time? For instance, they would be more likely to choose F and G rather than just selecting F. For example, in the description of F, “Nothing in the frame gives a reliable sense of scale”, which is precisely the reason why G cannot determine.

(2) The nighttime image of the "Evidence Quality" example in Figure 1 is extremely blurry and, in the human eye, it can be said to provide no information at all. However, the example option chose B. It seems that something about the image limits what can be judged. So, could more cases about the selection of options A or C for "Evidence Quality" be provided?

(3) Have the authors evaluated a text-only, answer-options-only, or shuffled-image baseline? Since all examples use fixed question templates and answer spaces, models may exploit task-specific answer frequencies without interpreting the image. How much does performance decrease when the image is removed or replaced with an unrelated image?

(4) Since only one primary run is reported for each model and several models use default but potentially different reasoning settings, how stable are the rankings across repeated runs or alternative reasoning configurations?

5 comments
AuthorsOpenReview ↗

Response to Reviewer DAYX (1/2)

We thank the reviewer for the thorough reading and for highlighting that distinguishing supported absence from insufficient evidence is a useful design choice. The detailed questions have helped us strengthen the paper.

W1 — Image collection composition and independence

The table gives each source's share of the released collection, its licence, and accuracy averaged over the 13 models.

SourceImagesLicenceMean acc
FloodIMG34.3%CC0-1.060.3
CVFD31.4%CC-BY-SA-4.066.8
State DOT traffic cameras28.2%Agency public records via the Internet Archive61.4
NOAA NGS aerial6.2%US Government work, public domain64.3

34.4% of the images carry a named hurricane.

EventImages
Hurricane Harvey 201717.0%
Hurricane Ian 202213.3%
Hurricane Michael 20181.8%
Hurricane Ida 20211.5%
Hurricane Florence 20180.7%

The remaining 65.6% come from two public flood datasets that carry no per-image event or location metadata in their original releases.

Duplicate removal was performed, and the released collection contains no image that duplicates another. We have added this to the paper.

W2 — Annotation protocol, expertise, criteria, adjudication

The annotation protocol, question text, answer space for all 15 tasks, and the annotation interface are in Sections 3.2–3.4 and Appendices B and D. Gwet's AC1 for each of the 15 tasks is below.

QuestionAC1QuestionAC1
Legibility0.97Debris0.82
Vantage point0.90Water present0.80
Damage0.90Passability0.79
Main structure0.88Road levels0.77
People in water0.88Water flow0.77
Image limits0.84Road surface0.71
Stranded vehicles0.83Water depth0.58
Water level0.58
Macro0.80

The gold labels are based on independent observer agreement, which is the appropriate standard for this setting. In practice, when responders assess flood conditions from camera feeds, the image is all they have. There is no co-located gauge or field report at the time of capture, and physical measurements or incident reports are not available for these images. Items without agreement are excluded from the benchmark. We have added a representative ambiguous case to Appendix D.

Q1 — Figure 1 options F, G and H

Water depth is a single-choice task, as are 13 of the 15 tasks, so a model selects exactly one option and the case of choosing F and G together does not arise. The three options have distinct triggers. F states that the frame contains no object of known size against which a depth can be read, which is a specific and checkable property of the image. G covers any other reason for being unable to answer, such as darkness or occlusion. H applies when there is no ground or road to assess at all. Requiring a single choice here is deliberate, because it asks the model to commit to why a depth cannot be given, and that carries more information for a responder than a blanket refusal.

Q2 — The Figure 1 night image

This frame is a borderline case, which is why it illustrates the dimension well. On close inspection the roadway is visible and the vantage point is identifiable, while the flood conditions on that road are not. That is what option B describes, partial legibility, rather than option C, which is reserved for frames where not even the scene type can be made out. The full question text and answer options that define each legibility level are in Appendix B.

AuthorsOpenReview ↗

Response to Reviewer DAYX (2/2)

Q3 — No-image and options-only baselines

We evaluated both for all 13 models on the full benchmark. In the no-image condition the model receives the question and the answer options but not the image. In the options-only condition it receives the answer options alone, with neither the image nor the question stem. Everything else is unchanged, including the per-item option shuffle, so the prompts match the reported runs apart from what is withheld.

ModelWith imageNo imageΔOptions onlyΔ
Gemini 3.8 Flash73.227.5−45.729.7−43.5
Kimi K368.915.4−53.511.0−57.9
Claude Opus 568.638.2−30.433.3−35.3
GLM-5.3 Flash67.111.4−55.78.9−58.2
DeepSeek V4.1 Flash65.114.2−50.99.5−55.6
GPT-5.6 Sol60.79.0−51.710.0−50.7
STEP3-VL-10B58.69.0−49.616.9−41.7
InternVL3.5-38B58.225.6−32.626.4−31.8
Phi-4-Reasoning-Vision-15B58.134.2−23.917.0−41.1
Qwen3.5-397B-A17B58.034.9−23.122.2−35.8
Mistral-Small-3.1-24B54.028.3−25.724.5−29.5
Keye-VL-1.5-8B51.332.5−18.832.6−18.7
InternVL3.5-8B47.723.9−23.821.1−26.6

Removing the image costs between 18.8 and 55.7 points, and removing the question stem as well costs a comparable amount. The highest score any model reaches without the image is 38.2, against with-image scores of 47.7 to 73.2. Qwen3.5-397B-A17B is evaluated here in its non-thinking configuration, whose with-image score is 58.0; Table 1 reports the thinking configuration at 60.3.

Q4 — Run-to-run stability and reasoning configuration

Each model is evaluated under its default reasoning configuration, as stated in Section 4.1. Closed-source APIs do allow adjusting reasoning intensity, but running every model under multiple configurations was beyond our compute budget. Models already fall substantially short of human-level observation under default settings, particularly on physical reasoning and evidence assessment. Varying the reasoning budget may shift individual scores but is unlikely to close this gap or change the ranking pattern.

Thanks again for the review, and please let us know if anything needs further clarification.

Reviewer_DAYXOpenReview ↗

Q1 & Q2— More examples are needed

Thank you for the clarifications. Your response and the table in Appendix B provide detailed complete question stems and answer options. The author designed the single-choice options to increase the amount of information provided in the answers. Taking Q1 as an example, I still find it difficult to understand how the model would make a choice when options F and H have overlapping conditions (the frame contains no object of known size against which depth can be measured and no ground or road for assessment). Since the erroneous case demonstrations in Appendix E have been excluded, more objective question-answer pairs and image examples need to be presented.

Reviewer_uBdoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FloodMark evaluates whether multimodal large language models can extract reliable visual evidence from flood images and infer flood conditions and their potential impacts. It contains 1,400 real-world images and over 16,000 human-annotated image–question pairs across 15 tasks. The study evaluates 13 models and systematically analyzes their reliability.

Critical strengths
  1. Practical relevance. Flood-state reasoning and road passability assessment have potential value for disaster assessment and emergency response.

  2. Systematic evaluation. The structured tasks, multi-model experiments, and error analysis reveal weaknesses in physical reasoning and evidence assessment.

Critical weaknesses
  1. Physical-state labels would benefit from further validation. The paper describes an annotation protocol and reports mean agreement of AC1=0.80. However, task-specific or independent validation would help assess labels for depth, flow, and passability, particularly in visually ambiguous cases.

  2. The scope of generalization needs clearer characterization. The paper describes multiple sources and capture conditions, but coverage across geographic regions and flood events is not clearly quantified. More detailed distribution statistics would help establish the scope of its conclusions.

Questions for authors
  1. Could the authors report task-specific agreement for depth, flow, and passability? Is independent validation available for any subset of these labels?

  2. Could the authors provide geographic, flood-event, and image-source statistics, together with performance breakdowns where available?

  3. Table 3 defines abstention using Cannot determine. Appendix B also includes a Water Depth option for images without a reliable scale reference. How is this option handled, and would including it as abstention change the main findings?

4 comments
AuthorsOpenReview ↗

Response to Reviewer uBdo

We thank the reviewer for recognizing the practical relevance of the task design and the value of the systematic multi-model evaluation.

W1 / Q1 — Task-specific agreement and independent validation

We have computed AC1 for all 15 tasks. The full table is now in the appendix.

QuestionAC1QuestionAC1
Legibility0.97Debris0.82
Vantage point0.90Water present0.80
Damage0.90Passability0.79
Main structure0.88Road levels0.77
People in water0.88Water flow0.77
Image limits0.84Road surface0.71
Stranded vehicles0.83Water depth0.58
Water level0.58
Macro0.80

Water depth and water level are the two lowest, reflecting how hard depth is to judge from a single image when scale cues are sparse or the level sits near a category boundary. Our agreement-and-discard protocol is designed for this case: gold labels require annotator agreement, and items that did not reach agreement were excluded rather than given a label we could not defend.

A subset of 100 images was annotated independently by all five annotators rather than two. On this subset AC1 is 0.59 for water depth, 0.81 for water flow, and 0.80 for passability, closely matching the two-annotator values.

W2 / Q2 — Scope of generalization

The table gives each source's share of the released collection, its licence, and accuracy averaged over the 13 models.

SourceImagesLicenceMean acc
FloodIMG34.3%CC0-1.060.3
CVFD31.4%CC-BY-SA-4.066.8
State DOT traffic cameras28.2%Agency public records via the Internet Archive61.4
NOAA NGS aerial6.2%US Government work, public domain64.3

34.4% of the images carry a named hurricane.

EventImages
Hurricane Harvey 201717.0%
Hurricane Ian 202213.3%
Hurricane Michael 20181.8%
Hurricane Ida 20211.5%
Hurricane Florence 20180.7%

The remaining 65.6% come from two public flood datasets that carry no per-image event or location metadata in their original releases.

Q3 — The Water Depth no-reliable-scale option

The two options answer different questions. The no-reliable-scale option records a property of the scene, namely that the frame contains no object of known size against which a depth can be read, and it is set by a separate annotation question about reference objects rather than by a judgement on depth itself. "Cannot determine" records that no judgement could be reached at all. We therefore score the first as a substantive finding about evidence quality, consistent with how the benchmark treats a supported negative elsewhere, and the second as abstention.

To check whether this choice changes the main findings, we recomputed the abstention analysis with the no-reliable-scale option counted as abstention on both the gold and the prediction side. Averaged over the 13 models, correct abstention moves from 30.0 to 31.6, false certainty from 70.0 to 68.4, and over-abstention from 3.8 to 5.2. The findings hold under either definition.

We hope these additions address the concerns. Please let us know if you have any further questions.

Dear Reviewer uBdo,

We wanted to kindly follow up as the discussion period is approaching its end. We would greatly appreciate it if you could take another look at our response and revised manuscript and let us know if you have any remaining concerns. We hope the additional evidence helps address your concerns, and we'd be grateful if you could consider raising your score.

Thank you for your time and consideration.

Dear Reviewer uBdo,

As the rebuttal period ends in the next few hours, we wanted to kindly follow up to see whether you have had a chance to review our responses and revised manuscript.

We sincerely appreciate your thoughtful review. We have provided additional task-specific agreement analyses, independent validation results, and clarifications regarding dataset coverage and abstention definitions to directly address your concerns.

We would greatly appreciate any final feedback you may have. If you feel that our additional evidence and revisions have sufficiently addressed your concerns, we would be very grateful if you could consider revisiting your assessment.

Thank you again for your time and consideration.

Dear Reviewer uBdo,

As this is the last hour of the discussion period, we are following up one final time. Our response and the revised manuscript answer each point in your review with new results: AC1 for all 15 tasks, validation on a five-annotator subset, source and event statistics, and the abstention analysis with the no-reliable-scale option counted as abstention.

Please let us know if any concerns remain. If these additions resolve your concerns, we would be grateful if you would consider updating your score.

Thank you again for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.