- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces RSFaith-Bench, a benchmark for evaluating whether the visual evidence stated by multimodal large language models is consistent with source annotations and sufficient to support their answers. The benchmark contains 13,511 human-validated questions constructed from six remote sensing datasets, covering perception, relational reasoning, and temporal reasoning. Its central methodological component is an evidence contract that specifies the facts required to derive an answer. A frozen language model extracts visual claims from responses, which are then checked against a grounded fact layer within declared closed scopes and matched to the corresponding contract.
The evaluation of 22 models reveals a substantial gap between answer accuracy and faithful accuracy: the reported averages are 50.2% and 24.3%, respectively. Missing support accounts for 63% of the mean gap. The paper supports these findings with human audits, extractor and prompting controls, an open-ended evaluation, and a human preference study.
Critical strengths
The benchmark addresses an important limitation of answer-level evaluation. Evidence contracts explicitly connect the facts stated in a response to the requirements of its answer. This allows the evaluation to identify responses whose individual claims may be correct but do not collectively establish the answer, a limitation that claim-level factuality alone does not capture. Verification against source annotations within declared closed scopes also makes the verification stage traceable. This contribution matters because responses with equally correct final answers can differ substantially in the quality of their supporting evidence.
The empirical study includes substantial and complementary validation. The evaluation covers 22 models and 13,511 human-validated questions from six datasets. The authors distinguish schema-blind human judgments of evidential support from a contract-aware audit of the automated pipeline, and report 89.4% agreement between the schema-blind judgments and automated decisions. The appendix further examines extractor choice, prompt format, a common requested evidence budget, and open-ended answering. In answer-matched comparisons, human raters prefer faithfully supported responses over missing-support responses in 69.0% of pairs. Together, these results provide meaningful support for the benchmark’s usefulness and the practical relevance of evaluating stated evidence.
Critical weaknesses
-
The interpretation of missing support remains sensitive to how evidence is elicited. Missing support accounts for 63% of the mean answer–evidence gap, making its interpretation central to the paper. I appreciate the answer-first/open-format, fixed five-claim, and open-ended controls in Tables A15–A17. However, these controls do not establish how much missing support can be recovered by explicitly requesting complete answer-critical evidence or by an answer-preserving follow-up that does not reveal gold evidence. The baseline requests relevant visual evidence, which may elicit an incomplete selection of facts even when a model could provide additional correct support. The findings therefore establish insufficient stated evidence under the tested protocols, while the distinction between an elicitation-dependent omission and a persistent difficulty in supplying the required facts remains unresolved. This limits the broader interpretation of the model-comparison and capability-level conclusions.
-
Contract completion needs more targeted validation as an operational measure of evidence sufficiency. The scoring rule requires supported claims to fill a predefined contract. Although compatible slot matching and count aggregation are supported, it remains unclear how broadly the framework accepts alternative sufficient justifications that do not reconstruct the question program’s canonical inputs. Verification coverage adds a related concern: the median proportion of decidable claims is 68%, and 74.6% of uncertain claims in Table A14 fall into annotation- or vocabulary-related categories. These statistics do not establish that M-CUR labels are incorrect, but they leave the contribution of relevant yet unverifiable evidence unquantified. The schema-blind audit provides valuable validation; however, aggregate agreement does not reveal how frequently M-CUR reflects genuine omissions versus sufficient evidence that the representation or verifier fails to recognize. A targeted analysis of this category would strengthen the interpretation of the paper’s dominant failure mode.
Questions for authors
- Requested control for the current evaluation: Could you compare pure answer-only generation with the current evidence-first protocol on the same questions, using the same models and decoding settings? Please report answer accuracy and paired answer changes. The existing answer-first and open-ended conditions still request evidence, so they do not isolate how requiring evidence affects answer generation. FA would not be an informative comparison for the answer-only condition, which intentionally provides no evidence.
- Requested diagnostic experiment for the main finding: How much of M-CUR is recoverable through explicit, task-type-specific completeness instructions or an answer-preserving follow-up requesting additional supporting facts? The prompts should not reveal gold evidence values. Reporting transitions from M-CUR to FA, persistent missing support, and newly introduced contradictions would help determine whether the dominant gap is readily reduced by evidence elicitation.
- Clarification and targeted validation of the metric: How does the verifier handle a sufficient justification that differs from the canonical evidence contract? Could you provide representative M-CUR examples and a schema-blind audit focused on this category, separating genuine omissions, alternative valid justifications, and extraction/normalization/coverage failures? A confusion matrix and breakdown by reasoning subtype would help establish how much these issues affect the reported missing-support rate.
4 comments
Official Response to Reviewer 2xmG (2/2)
W2 & Q3: Evidence sufficiency and M-CUR validation
Genuine omissions account for 67% of responses in the targeted M-CUR audit. Two annotators independently classified 100 randomly sampled M-CUR responses (κ = 0.66). The verifier accepts equivalent facts, including inverse spatial relations and counts summed over disjoint regions covering the queried scope (Appendix B.3). The audit also identifies valid alternatives: responses that establish the answer through a different set of facts from those specified by the contract. For example, stating the locations of the building and all candidate fields can establish which field lies below the building. Shares and 95% CIs below are percentages (Table A24).
| Cause | Share | 95% CI |
|---|---|---|
| Genuine omission | 67 | [57.3, 75.4] |
| Valid alternative | 8 | [4.1, 15.0] |
| Extraction / normalization / coverage | 17 | [10.9, 25.5] |
| Unclear | 8 | [4.1, 15.0] |
Across the benchmark, 17.3% of the 70,255 unfilled contract slots in M-CUR responses have a corresponding claim that the source annotations cannot verify. The remaining slots either have no matching extracted claim or have only partial support, such as a subregion count or a broader category label. In the table, n counts slots and Share is a percentage; percentages are rounded independently.
| Evidence | n | Share |
|---|---|---|
| No matched claim | 46,896 | 66.8 |
| Partial support | 11,219 | 16.0 |
| Unverifiable | 12,140 | 17.3 |
Human sufficiency judgments. Raters judge 73.6% of M-CUR responses partially or not supported and 89.1% of FA responses fully supported. In this larger audit of 2,500 responses from five models, raters inspect images and responses without seeing contracts or automated labels. Table A23 compares their judgments with the automatic labels; entries are counts.
| Label | Full | Partial | None | Total |
|---|---|---|---|---|
| FA | 476 | 33 | 25 | 534 |
| C-CUR | 64 | 143 | 86 | 293 |
| M-CUR | 96 | 172 | 96 | 364 |
| Wrong answer | 47 | 378 | 884 | 1,309 |
| Total | 683 | 726 | 1,091 | 2,500 |
Using these human sufficiency ratings while retaining the automatic contradiction check gives a 24.8-point gap, compared with 26.3 under automatic scoring. For the 364 M-CUR responses, the next table gives the ratings by reasoning level and subtype (Table A24). Bold rows summarize each level; n counts responses. Full, Partial, and None are human support ratings (%), and brackets give 95% Wilson CIs for Full.
| Level / subtype | n | Full | Partial | None |
|---|---|---|---|---|
| Perception | 109 | 18.3 [12.2, 26.6] | 45.9 | 35.8 |
| Object presence | 48 | 14.6 [7.2, 27.2] | 43.8 | 41.7 |
| Object counting | 23 | 21.7 [9.7, 41.9] | 47.8 | 30.4 |
| Object localization | 19 | 15.8 [5.5, 37.6] | 52.6 | 31.6 |
| Fine-grained recognition | 19 | 26.3 [11.8, 48.8] | 42.1 | 31.6 |
| Relational Reasoning | 180 | 33.9 [27.4, 41.1] | 45.6 | 20.6 |
| Directional | 39 | 30.8 [18.6, 46.4] | 48.7 | 20.5 |
| Topological | 48 | 33.3 [21.7, 47.5] | 54.2 | 12.5 |
| Proximity | 31 | 25.8 [13.7, 43.2] | 41.9 | 32.3 |
| Projective ordering | 21 | 28.6 [13.8, 50.0] | 38.1 | 33.3 |
| Aggregate distribution | 41 | 46.3 [32.1, 61.3] | 39.0 | 14.6 |
| Temporal Reasoning | 75 | 20.0 [12.5, 30.4] | 53.3 | 26.7 |
| Category turnover | 11 | 9.1 [1.6, 37.7] | 45.5 | 45.5 |
| Semantic transition | 35 | 28.6 [16.3, 45.1] | 51.4 | 20.0 |
| Net change | 29 | 13.8 [5.5, 30.6] | 58.6 | 27.6 |
| All | 364 | 26.4 [22.1, 31.1] | 47.3 | 26.4 |
By reasoning level, 81.7% of M-CUR responses in Perception, 66.1% in Relational Reasoning, and 80.0% in Temporal Reasoning are rated partially or not supported. Appendix G.8 now includes these ratings and the analyses of M-CUR causes and verification coverage.
Representative cases. The three added examples show why the missing fact is needed to distinguish the answer options. In Perception, the stated row places the roundabout in the lower third; its horizontal position distinguishes bottom-center from bottom-right. In the Relational Reasoning case, the evidence identifies two fields on the left side of the image without stating which lies below the building. For Temporal Reasoning, the evidence describes buildings at the later date but omits whether the same area previously contained low vegetation or trees. Appendix E now includes these cases and identifies the unfilled slots.
The authors' responses have adequately addressed my concerns, and the additional experiments further strengthen the paper. I appreciate the authors' efforts in clarifying the methodology and providing further empirical evidence. I have no remaining concerns and will maintain my original rating.
Thank you for your positive assessment and for maintaining your rating. We sincerely appreciate your constructive feedback.
Official Response to Reviewer 2xmG (1/2)
Thank you for the careful assessment and concrete diagnostic suggestions. The new controls quantify how much missing support can be recovered, and targeted human audits examine the causes of M-CUR labels.
Q1: Answer-only control
Mean AA is 52.4% with evidence-first generation and 53.3% with answer-only generation, with answers differing in 9.2% of cases. We compare eight models on the same 500 questions, keeping images, options, and decoding settings fixed. The table reports AA (%) for both conditions. ΔAA is evidence-first minus answer-only, in percentage points (pp); Changed is the percentage of answers that differ between conditions.
The paired mean change, computed before rounding, is −0.975 points (95% CI [−2.00, 0.03]). Across 4,000 responses, switching from answer-only to evidence-first generation changes 2.6% of answers from wrong to right and 3.6% from right to wrong. Another 3.0% change from one wrong answer to another. 95.0% of answer-only responses follow the requested format. Appendix G.7 now reports this control and the paired answer changes (Table A22).
W1 & Q2: Evidence elicitation and model comparisons
With task-specific completeness instructions, 23.4% of baseline M-CUR responses become FA and 59.5% remain M-CUR. The instructions specify fact types, while the model supplies the values. For directional questions, the prompt requests object identities, locations, and relative direction. The table tracks label transitions for 667 baseline M-CUR responses (%), using the first scoring pass for Evidence-first and Completeness.
The 13.9% that move to C-CUR now contain a refuted claim. In total, 490 of 667 responses (73.5%) remain correct but unfaithful under the completeness prompt. For model comparisons, the next table gives Evidence-first / Completeness results on the same 500 questions for each model. Evidence scores are averaged over two scoring passes. Scores are percentages; FA ranks are within these eight models.
FA improves for all eight models, with a mean gain of 4.3 points (95% CI [3.34, 5.20], computed before rounding). M-CUR decreases from 16.6% to 11.5%, and the fidelity gap (AA − FA) narrows from 28.2 to 25.2 points. The FA ranking remains similar across the two prompts (Spearman ρ = 0.93). Section 4.2 now specifies that the 63% M-CUR share refers to the standard evidence request. Appendix G.7 adds the completeness control and per-model results (Table A22).
Across the six within-family upgrades in Table 2, mean C-CUR rises by 3.6 points while M-CUR changes by −0.4 points. Refuted claims account for the widening gap. We have added these paired comparisons to Appendix G.2.