- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces SARFACT, an object-centric vision–language benchmark and dataset for SAR imagery, built by merging four public SAR detection datasets into a shared pool of localized object facts. From this pool it derives examples across nine task interfaces: detection, presence, multi-label classification, counting, point-prompted recognition, spatial relation, count comparison, visual grounding, and captioning. The central idea is to pair each task's native answer with a task-specific notion of Visual Evidence, so models can be scored on Answer, Evidence, and Joint correctness. The construction pipeline separates certified absence from missing annotation, checks cross-source consistency, assigns splits by content lineage, and applies task-specific selection controls to suppress dominant answer and composition patterns. Captions are generated under object-fact constraints and evaluated with fact-based metrics alongside CIDEr-D. Experiments cover zero-shot evaluation of six public VLMs, task-specific LoRA fine-tuning with and without evidence supervision, and a transfer study on task-aligned SARLANG subsets.
Critical strengths
- The task-dependent definition of Visual Evidence is clean and carefully specified. Instead of a uniform "output a box" requirement, each task gets an evidence contract that matches its semantics, and the matching and joint metrics follow directly from these contracts. This is a sensible way to make answer correctness inspectable.
- The dataset construction is thorough and unusually transparent. Absence is separated from missing annotation, duplicated content across sources is checked for consistency rather than merged blindly, splits are assigned by content lineage before task examples are generated, and the shortcut controls come with before/after diagnostics showing that they reshape the targeted distributions. The authors are also candid about limitations, such as batch-approved caption references and the non-probability human audit.
Critical weaknesses
- The benchmark largely re-derives existing detection annotations. All nine tasks, their answers, and their Visual Evidence are computed automatically from the boxes of four public detection datasets; no new annotation is collected beyond the model-generated captions. For most tasks, the required evidence is exactly the object set from which the answer is computed (e.g., counting requires the complete counted set, classification requires one witness per positive category), so the Answer–Evidence decomposition and the Joint score may mostly measure detection ability rather than a separate verification skill. The paper never establishes what Joint accuracy adds beyond correct detection plus a correct answer; a pipeline baseline in which a separate detector supplies the evidence is missing.
- The empirical study is too narrow to support the claims. Answer–Evidence results are reported for one model on five tasks, although the abstract describes six VLMs; count comparison and visual grounding, the two motivating examples, and captioning, whose evidence is defined in Table 2, are absent. Zero-shot scores sit at or near the floor on most tasks, and the paper offers no error analysis separating genuine task difficulty from output-format and parsing failures, which matters for a benchmark whose interfaces prescribe specific answer and evidence formats. The only external check is a three-task answer-only transfer study on images that partly share sources with SARFACT training data.
Questions for authors
- What does Joint accuracy capture that a correct answer plus correct detection does not? Can you compare against a pipeline baseline in which answer-only fine-tuning is paired with a separately trained detector producing the evidence boxes?
- Can you add an error analysis on zero-shot outputs that separates answers below chance and formatting or parsing failures from genuine task errors, and clarify which tasks are primarily tests of localization?
- Can you report Answer–Evidence results for count comparison, visual grounding, and captioning, given that the first two are the motivating examples in Figure 2?
- Can you report SARLANG results after removing test images that share upstream sources with SARFACT training images, and with evidence as well as answers?- Was any human verification performed on the structured-task records? If not, can you report error rates on a small random sample?