- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
FloodMark evaluates flood-scene understanding with 1,400 real-world images and 16,403 question-answer pairs across 15 tasks. It covers scene context, flood conditions, physical and operational judgments, and image quality, with explicit options for insufficient evidence. The authors evaluate 13 models, and also analyze abstention, capture conditions, and sampled model errors.
Critical strengths
- The tasks cover useful aspects of flood assessment beyond detecting whether flooding is present, including depth, passability, and obstruction.
- Distinguishing supported absence from insufficient evidence is a useful design choice, especially for images with poor visibility or limited coverage.
- The questions are human-authored, annotations include independent labeling and disagreement resolution, and the 13-model evaluation provides detailed task-level results.
Critical weaknesses
- The added evaluation value over existing work needs stronger evidence. Table 4 shows broader task coverage, but does not establish what distinct capability gap FloodMark reveals beyond disaster VQA and answerability benchmarks. The findings on physical judgments and insufficient evidence do not clearly show what is specifically learned from the new flood tasks. A focused comparison with overlapping existing tasks, or an analysis showing how the added tasks change our assessment of model capabilities, would strengthen the contribution.
- The physical and operational labels are not sufficiently validated. Absolute depth intervals require reliable reference cues, while passability depends on vehicle assumptions and decision criteria. Section 3.2 mentions annotation rules, but these are not provided or validated in the manuscript. Average AC1 establishes consistency, not the physical meaning of the labels. A representative validation subset would help show that the benchmark measures physically grounded hazard reasoning rather than mainly agreement with the authors’ visual judgments.
- The data construction leaves coverage and uncertainty selection unclear. Source proportions, event coverage, and duplicate handling are not reported. More importantly, Section 3.4 excludes unresolved questions without characterizing them. For a benchmark centered on insufficient evidence, it matters whether these cases reflect annotation errors, unclear task rules, or genuinely ambiguous imagery. Per-task exclusion counts and an analysis of removed cases are needed to assess whether filtering preserves the uncertainty cases the benchmark aims to evaluate.
- The evaluation does not isolate reasoning from simpler visual decisions. Water Depth and Water Flow include no-flood or not-applicable answers, which contribute to physical-task scores without requiring depth estimation or motion inference. Results on applicable cases are not reported separately. Independent question evaluation also leaves unclear whether operational errors arise from perception, physical inference, or inconsistent decisions. Conditional results on relevant flood cases, majority/no-image baselines, and consistency checks across depth, flow, and passability would better support the capability claims.
- The metrics do not fully support the operational-reliability conclusions. Matching “Cannot determine” is only a proxy for recognizing insufficient evidence; Water Depth’s separate no-reliable-scale option needs consistent treatment. The paper reports over-abstention, but not errors among answered hazard cases or their operational direction. Incorrect “any vehicle can pass” and “no vehicle can pass” predictions receive the same penalty despite different consequences. Task-specific abstention accounting and a breakdown of unsafe versus conservative judgments would connect the results more directly to the stated motivation.
Questions for authors
Please refer to the Critical Weaknesses.
7 comments
Dear Reviewer DALd,
We wanted to follow up as the discussion period is coming to an end. We would appreciate it if you could take another look at our response and revised manuscript. Please let us know if you have any remaining concerns. We hope the additional evidence helps address your concerns, and we'd be grateful if you could consider raising your score.
Thank you for your time!
Thank you for the clarifications. I still have three main concerns:
-
Agreement between annotators shows consistency, but does not establish that the depth and passability labels are physically well grounded. I understand that measurements are unavailable, but clearer rules for estimating depth, judging passability, and identifying flow from a single image, along with some validation examples, would help address this concern.
-
Reporting results on applicable flood cases would not give models any extra information. The existing predictions can simply be grouped by gold labels after evaluation. Keeping the overall score while adding these results would help separate recognizing no-water/non-applicable cases from judging depth and flow.
-
The depth task has both “no reliable sense of scale” and “Cannot determine.” From Table 3’s definitions, choosing the former when the gold is the latter may count as false certainty, and the reverse as over-abstention. No-reliable-scale cases may also remain in Section 5.2’s “determinate” subset. Explaining how these options are counted and treating both as insufficient-evidence responses for this analysis would make the findings easier to interpret.
The added details are helpful, but I do not think the current response resolves these concerns about what the benchmark measures and what the results show.
Response to Reviewer DALd
We thank the reviewer for the careful analysis and for pointing out where the contribution and protocol needed clearer documentation.
W1 — Evidence for the contribution
Existing flood and disaster VQA benchmarks ask what an image contains. We ask what a responder could justifiably conclude from it, which changes the task set in two ways.
First, other benchmarks ask about damage and passability, but not about the physical observations behind them. We ask for an absolute depth band in metres, for where the water sits relative to a named object in the scene, and for whether the water is still, flowing or fast. Without these questions, a model that misreads the water and one that misapplies the rule produce the same score.
Second, we evaluate the image as evidence. Two tasks rate how far a frame can be read and name what limits it, and insufficient evidence is a correct answer set by a domain expert, not inferred from model uncertainty or induced by breaking the question. We also separate a supported absence from an unreadable frame, which earlier benchmarks score as one answer.
W2 — Validation of the physical and operational labels
The gold labels in FloodMark are based on independent observer agreement, which is the appropriate standard for this setting. In practice, when responders assess flood conditions from camera feeds, the image is all they have. There is no co-located gauge or field report at the time of capture, and physical measurements are not available for these images. The full question text and answer options for all 15 tasks are in Appendices B and D, and the options themselves are the decision criteria. The depth question includes an explicit option for images where nothing in the frame gives a reliable sense of scale, and annotators select it when no reference is available rather than estimating a depth. For passability, the answer options distinguish any vehicle from a high-clearance vehicle such as a pickup or SUV, and the question asks about the worst-affected stretch visible.
W3 — Coverage and the unresolved questions
The table gives each source's share of the released collection, its licence, and accuracy averaged over the 13 models.
34.4% of the images carry a named hurricane.
The remaining 65.6% come from two public flood datasets that carry no per-image event or location metadata in their original releases. Duplicate removal was performed, and the released collection contains no image that duplicates another.
Our agreement-and-discard protocol ensures that every item in the released benchmark has a gold label that independent observers agreed on. Items without agreement are excluded, so they do not affect the benchmark or its scores. A gold answer of "Cannot determine" means observers independently agreed the image does not support an answer, so it is a finding about evidence quality, not an unresolved question.
W4 — Isolating reasoning from simpler visual decisions
Whether water is present is part of the task rather than a precondition for it. A responder scanning camera feeds is not told which frames contain water, so deciding that there is nothing to measure is one of the judgements we evaluate, and it is reached by the same reading of the scene that a depth estimate would require. Restricting the depth question to frames that already contain water would hand the model the answer to the prior question.
W5 — Metrics and operational reliability
FloodMark evaluates whether VLMs can match human-level observation of flood scenes. It is an evaluation benchmark, not a deployment-readiness assessment, and we did not train any model. Whether a wrong prediction is dangerous or merely conservative depends on the downstream application, which is out of our scope. Our finding that models lag far behind humans on physical reasoning and evidence assessment already identifies where the gap is and what future work should focus on. Connecting that gap to a specific operational cost structure is valuable but requires assumptions about the deployment context that an evaluation benchmark should not prescribe.
Thank you again for proposing the reviews. We hope the additional analyses clarify FloodMark's contribution and protocol. We welcome any follow-up questions.