acceptodds
Withdrawn from ICLR 2027

SARFACT: Benchmarking Task Answers and Visual Evidence for SAR Vision–Language Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Evaluating SAR vision–language models only by the correctness of their task outputs leaves a verification gap: a model may produce the expected answer without identifying the visual support needed to inspect it. We introduce SARFACT, a large-scale object-centric SAR vision–language dataset and benchmark that preserves each task's native Task Answer and defines task-specific Visual Evidence: the localized object support needed to inspect the answer. Built from 200K SAR images and 1.11M foreground objects, SARFACT contains 1.42M examples across nine task interfaces. Task Answers and Visual Evidence are derived from shared localized object facts, while task-specific construction controls reduce shortcut-prone patterns and preserve informative evaluation cases. SARFACT separately evaluates Answer, Evidence, and Joint correctness. Experiments with six public VLMs reveal limited zero-shot performance and frequent cases in which correct Answers fail to satisfy the task-defined Visual Evidence requirements. Explicit Evidence supervision substantially improves joint Answer–Evidence performance. Annotations, task records, and evaluation code will be publicly released.

Primary area:
datasets and benchmarks
Keywords:
Synthetic Aperture Radar (SAR), Vision-Language Models (VLMs), Visual Evidence, Multimodal Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rBZaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SARFACT, an object-centric vision–language benchmark and dataset for SAR imagery, built by merging four public SAR detection datasets into a shared pool of localized object facts. From this pool it derives examples across nine task interfaces: detection, presence, multi-label classification, counting, point-prompted recognition, spatial relation, count comparison, visual grounding, and captioning. The central idea is to pair each task's native answer with a task-specific notion of Visual Evidence, so models can be scored on Answer, Evidence, and Joint correctness. The construction pipeline separates certified absence from missing annotation, checks cross-source consistency, assigns splits by content lineage, and applies task-specific selection controls to suppress dominant answer and composition patterns. Captions are generated under object-fact constraints and evaluated with fact-based metrics alongside CIDEr-D. Experiments cover zero-shot evaluation of six public VLMs, task-specific LoRA fine-tuning with and without evidence supervision, and a transfer study on task-aligned SARLANG subsets.

Critical strengths
  1. The task-dependent definition of Visual Evidence is clean and carefully specified. Instead of a uniform "output a box" requirement, each task gets an evidence contract that matches its semantics, and the matching and joint metrics follow directly from these contracts. This is a sensible way to make answer correctness inspectable.
  2. The dataset construction is thorough and unusually transparent. Absence is separated from missing annotation, duplicated content across sources is checked for consistency rather than merged blindly, splits are assigned by content lineage before task examples are generated, and the shortcut controls come with before/after diagnostics showing that they reshape the targeted distributions. The authors are also candid about limitations, such as batch-approved caption references and the non-probability human audit.
Critical weaknesses
  1. The benchmark largely re-derives existing detection annotations. All nine tasks, their answers, and their Visual Evidence are computed automatically from the boxes of four public detection datasets; no new annotation is collected beyond the model-generated captions. For most tasks, the required evidence is exactly the object set from which the answer is computed (e.g., counting requires the complete counted set, classification requires one witness per positive category), so the Answer–Evidence decomposition and the Joint score may mostly measure detection ability rather than a separate verification skill. The paper never establishes what Joint accuracy adds beyond correct detection plus a correct answer; a pipeline baseline in which a separate detector supplies the evidence is missing.
  2. The empirical study is too narrow to support the claims. Answer–Evidence results are reported for one model on five tasks, although the abstract describes six VLMs; count comparison and visual grounding, the two motivating examples, and captioning, whose evidence is defined in Table 2, are absent. Zero-shot scores sit at or near the floor on most tasks, and the paper offers no error analysis separating genuine task difficulty from output-format and parsing failures, which matters for a benchmark whose interfaces prescribe specific answer and evidence formats. The only external check is a three-task answer-only transfer study on images that partly share sources with SARFACT training data.
Questions for authors
  • What does Joint accuracy capture that a correct answer plus correct detection does not? Can you compare against a pipeline baseline in which answer-only fine-tuning is paired with a separately trained detector producing the evidence boxes?
  • Can you add an error analysis on zero-shot outputs that separates answers below chance and formatting or parsing failures from genuine task errors, and clarify which tasks are primarily tests of localization?
  • Can you report Answer–Evidence results for count comparison, visual grounding, and captioning, given that the first two are the motivating examples in Figure 2?
  • Can you report SARLANG results after removing test images that share upstream sources with SARFACT training images, and with evidence as well as answers?- Was any human verification performed on the structured-task records? If not, can you report error rates on a small random sample?
Reviewer_VBwdOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SARFACT, an object-centric dataset and benchmark for evaluating both task answers and supporting visual evidence in SAR vision–language understanding. It contains 1.42 million examples across nine tasks, constructed from approximately 200,000 SAR images and 1.11 million annotated objects. Task-specific evidence requirements and data construction controls support consistent evaluation while reducing shortcut biases. The benchmark separately measures answer correctness, evidence quality, and joint correctness. Experiments with six public vision–language models reveal limited zero-shot performance and frequent cases where correct answers lack the required visual evidence. Supervised experiments further demonstrate improved joint answer–evidence performance with explicit evidence supervision.

Critical strengths

(1) The dataset integrates approximately 200,000 synthetic aperture radar (SAR) images, 1.11 million objects, and 1.42 million task records across nine task categories. Answers and supporting evidence across tasks are derived from a shared set of object-level facts, helping maintain consistency across task annotations. This design also provides a basis for examining model performance across different tasks within the same scene.

(2) The paper introduces an evaluation framework covering three dimensions: answer correctness, evidence correctness, and the joint correctness of answers and evidence for individual samples. This framework can reveal model limitations that answer accuracy alone may overlook, enabling a more comprehensive assessment of existing vision–language models (VLMs).

Critical weaknesses

(1) The dataset defines nine tasks, but the zero-shot answer–evidence evaluation reports results for only six task categories. Although a separate evaluation is unnecessary for object detection because its answers and evidence coincide, corresponding evaluations for visual grounding and image captioning are missing. These omissions limit the completeness of the experimental evaluation.

(2) The supervised fine-tuning experiments focus primarily on the Qwen3.5 family and lack validation on other model architectures. Consequently, it remains unclear whether the observed performance gains and the benefits of evidence supervision are consistent across model families. I recommend conducting additional fine-tuning experiments with other model families, such as InternVL or LLaVA, and reporting answer correctness, evidence correctness, and joint correctness. Such experiments would provide stronger support for the generalizability of the conclusions.

Questions for authors

(1) Was the entire dataset manually verified? If so, how many annotators were involved, and what were their relevant backgrounds and areas of expertise?

(2) If manual verification was conducted, what was the level of inter-annotator agreement? What was the agreement between human judgments and the automatically generated annotations?

(3) The paper states that ten query templates were created for each task. Could the choice of these templates introduce bias into the evaluation results? What measures were taken to assess and mitigate this potential bias?

(4) The dataset defines nine tasks, but the zero-shot answer–evidence evaluation reports results for only six task categories. Why were results not reported for all nine tasks? Could the authors additionally report supervised answer–evidence evaluation results for visual grounding and count comparison?

(5) For image captioning, multiple descriptions can be valid for the same image. How is the correctness of both the generated caption and its supporting bounding boxes assessed? What criteria determine whether a sample is counted as jointly correct?

Reviewer_XjgbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

SARFACT is a large-scale benchmark for SAR vision-language understanding. It contains 1.42 million examples, built from approximately 200k SAR images and 1.11 m objects. Images come from four existing public SAR datasets. Examples cover nine tasks: object presence, multi-label classification, counting, count comparison, visual grounding, point-prompted recognition, spatial relations, and object detection and captioning.

The goal of the dataset is to measure "Visual Evidence," meaning locating the specific objects supporting the answer. For presence, one matching object is enough; for counting, the model has to locate all relevant instances.

The paper shows that zero-shot performance for six models is weak and that correct answers often lack the required evidence. It also provides a fine-tune model trained on SAR data with bounding boxes as evidence and show an increase in performance.

Critical strengths

The paper introduces a large-scale benchmark containing 1.42 million examples derived from approximately 200k SAR images and 1.1 million annotated objects, covering nine distinct SAR vision-language tasks. It provides a unified framework in which each task is paired with an explicit definition of the visual evidence required to support the answer.

The benchmark scores the answer and the supporting visual evidence separately, and the amount of evidence required depends on the task. Using this setup, the paper shows that current models often give correct answers without finding the objects those answers depend on.

Critical weaknesses

Most of the benchmark comes from existing datasets and from an evidence-based evaluation idea that earlier work, such as SARLANG, already introduced. The only clearly new part is the generated captions, and those are a very small share of the benchmark. The authors should state exactly which parts are new and explain why combining existing resources is a meaningful contribution in itself, or if contribution is primary the new task of asking the questions and bounding boxes together.

Unclear effects of the training and evaluation setup. The original models were never trained to point out every object needed for an answer, so their low scores may reflect an unfamiliar output format rather than an inability to find the right visual evidence. After training on the benchmark the models improve, but the paper does not check whether this training makes them worse at things they could already do.

No large models tested. The study only evaluates small and medium-sized models.

Domain shift and format shift are mixed together. SAR images look very different from the images general models were trained on. The large jump in performance after fine-tuning could come from learning the new image type, from learning the output format, or from both, and the paper cannot tell these apart.

Figures are hard to read. Several figures, especially Figure 1, use text so small that it cannot be read comfortably.

Questions for authors

Does the performance improvement come from the evidence or from training on SAR alone? Fine-tuning a model using SAR data and answers without evidence could help determine whether the main gap is due to evidence or formatting.

The SARLANG comparison uses a different model version from your main experiments, and no model is trained on SARLANG in the same way. Can you report results with the same model version, trained the same way on each dataset?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.