acceptodds
Under review as a conference paper at ICLR 2027

Right, but Can You Show It? Correct Answers from Multimodal LLMs Rarely Come with Faithful Evidence

Abstract

When a multimodal large language model (MLLM) answers a question about an image, it usually also states what it sees, such as which objects are present and where, as evidence for its answer. Users rely on this evidence to decide whether to trust the answer, yet benchmarks grade only the answer. They tacitly assume that *a correct answer comes with trustworthy evidence*. Testing this assumption means checking every stated fact against the image, and being able to prove a false fact false. That requires knowing everything the image contains: if a dataset labels only some of its objects, a claim about an unlabeled one might still be true and cannot be marked wrong, and most image datasets are labeled this way. Remote sensing is a rare exception. By protocol, its datasets label every object of declared categories, together with their spatial relations and their changes over time. Building on this, we introduce *evidence contracts*, which specify for each question the visual facts its answer depends on, and **RSFaith-Bench**, 13,511 human-validated questions over 16,288 images on which evidence is *checked against ground-truth annotation rather than judged by another model*. This check agrees with human raters on 89.4% of responses. Evaluating 22 MLLMs shows that the assumption fails in every one of them. Taken together, **more than half of the answers that accuracy counts as correct would fail a check of their own evidence.** Surprisingly, the dominant failure is not contradiction but missing support: most of these answers state nothing that the annotation refutes, yet their evidence does not establish the answer, a failure that no check of the stated claims alone can detect. More strikingly, **stronger models do not close the gap**. Across six larger or newer model versions, answer accuracy improves more than five times as much as faithful accuracy, and in half of them faithful accuracy *declines*. Human raters blind to our labels prefer the faithfully supported answer in up to 82% of matched pairs. Answer accuracy tells us when a model is right; it cannot tell us when to believe it. The [benchmark](https://huggingface.co/datasets/anonymous-paper-submit/RSFaith-Bench) and [code](https://anonymous.4open.science/r/RSFaith-Bench-submit-7CEE) are publicly available.

Primary area:
datasets and benchmarks
Keywords:
Multimodal large language models, Remote sensing, Faithfulness evaluation, Visual reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_2xmGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RSFaith-Bench, a benchmark for evaluating whether the visual evidence stated by multimodal large language models is consistent with source annotations and sufficient to support their answers. The benchmark contains 13,511 human-validated questions constructed from six remote sensing datasets, covering perception, relational reasoning, and temporal reasoning. Its central methodological component is an evidence contract that specifies the facts required to derive an answer. A frozen language model extracts visual claims from responses, which are then checked against a grounded fact layer within declared closed scopes and matched to the corresponding contract.

The evaluation of 22 models reveals a substantial gap between answer accuracy and faithful accuracy: the reported averages are 50.2% and 24.3%, respectively. Missing support accounts for 63% of the mean gap. The paper supports these findings with human audits, extractor and prompting controls, an open-ended evaluation, and a human preference study.

Critical strengths

The benchmark addresses an important limitation of answer-level evaluation. Evidence contracts explicitly connect the facts stated in a response to the requirements of its answer. This allows the evaluation to identify responses whose individual claims may be correct but do not collectively establish the answer, a limitation that claim-level factuality alone does not capture. Verification against source annotations within declared closed scopes also makes the verification stage traceable. This contribution matters because responses with equally correct final answers can differ substantially in the quality of their supporting evidence.

The empirical study includes substantial and complementary validation. The evaluation covers 22 models and 13,511 human-validated questions from six datasets. The authors distinguish schema-blind human judgments of evidential support from a contract-aware audit of the automated pipeline, and report 89.4% agreement between the schema-blind judgments and automated decisions. The appendix further examines extractor choice, prompt format, a common requested evidence budget, and open-ended answering. In answer-matched comparisons, human raters prefer faithfully supported responses over missing-support responses in 69.0% of pairs. Together, these results provide meaningful support for the benchmark’s usefulness and the practical relevance of evaluating stated evidence.

Critical weaknesses
  1. The interpretation of missing support remains sensitive to how evidence is elicited. Missing support accounts for 63% of the mean answer–evidence gap, making its interpretation central to the paper. I appreciate the answer-first/open-format, fixed five-claim, and open-ended controls in Tables A15–A17. However, these controls do not establish how much missing support can be recovered by explicitly requesting complete answer-critical evidence or by an answer-preserving follow-up that does not reveal gold evidence. The baseline requests relevant visual evidence, which may elicit an incomplete selection of facts even when a model could provide additional correct support. The findings therefore establish insufficient stated evidence under the tested protocols, while the distinction between an elicitation-dependent omission and a persistent difficulty in supplying the required facts remains unresolved. This limits the broader interpretation of the model-comparison and capability-level conclusions.

  2. Contract completion needs more targeted validation as an operational measure of evidence sufficiency. The scoring rule requires supported claims to fill a predefined contract. Although compatible slot matching and count aggregation are supported, it remains unclear how broadly the framework accepts alternative sufficient justifications that do not reconstruct the question program’s canonical inputs. Verification coverage adds a related concern: the median proportion of decidable claims is 68%, and 74.6% of uncertain claims in Table A14 fall into annotation- or vocabulary-related categories. These statistics do not establish that M-CUR labels are incorrect, but they leave the contribution of relevant yet unverifiable evidence unquantified. The schema-blind audit provides valuable validation; however, aggregate agreement does not reveal how frequently M-CUR reflects genuine omissions versus sufficient evidence that the representation or verifier fails to recognize. A targeted analysis of this category would strengthen the interpretation of the paper’s dominant failure mode.

Questions for authors
  1. Requested control for the current evaluation: Could you compare pure answer-only generation with the current evidence-first protocol on the same questions, using the same models and decoding settings? Please report answer accuracy and paired answer changes. The existing answer-first and open-ended conditions still request evidence, so they do not isolate how requiring evidence affects answer generation. FA would not be an informative comparison for the answer-only condition, which intentionally provides no evidence.
  2. Requested diagnostic experiment for the main finding: How much of M-CUR is recoverable through explicit, task-type-specific completeness instructions or an answer-preserving follow-up requesting additional supporting facts? The prompts should not reveal gold evidence values. Reporting transitions from M-CUR to FA, persistent missing support, and newly introduced contradictions would help determine whether the dominant gap is readily reduced by evidence elicitation.
  3. Clarification and targeted validation of the metric: How does the verifier handle a sufficient justification that differs from the canonical evidence contract? Could you provide representative M-CUR examples and a schema-blind audit focused on this category, separating genuine omissions, alternative valid justifications, and extraction/normalization/coverage failures? A confusion matrix and breakdown by reasoning subtype would help establish how much these issues affect the reported missing-support rate.
4 comments
AuthorsOpenReview ↗

Official Response to Reviewer 2xmG (1/2)

Thank you for the careful assessment and concrete diagnostic suggestions. The new controls quantify how much missing support can be recovered, and targeted human audits examine the causes of M-CUR labels.

Q1: Answer-only control

Mean AA is 52.4% with evidence-first generation and 53.3% with answer-only generation, with answers differing in 9.2% of cases. We compare eight models on the same 500 questions, keeping images, options, and decoding settings fixed. The table reports AA (%) for both conditions. ΔAA is evidence-first minus answer-only, in percentage points (pp); Changed is the percentage of answers that differ between conditions.

ModelAnswer-onlyEvidence-firstΔAAChanged
GPT-5.159.657.4−2.28.2
Gemini 3.1 Pro61.664.2+2.67.6
Qwen3.5-27B59.460.8+1.48.2
Qwen3-VL-32B57.456.6−0.88.6
InternVL3.5-38B57.855.6−2.26.2
RSThinker45.442.0−3.410.0
GeoChat34.833.6−1.212.6
RemoteReasoner50.648.6−2.012.2
Mean53.352.4−1.09.2

The paired mean change, computed before rounding, is −0.975 points (95% CI [−2.00, 0.03]). Across 4,000 responses, switching from answer-only to evidence-first generation changes 2.6% of answers from wrong to right and 3.6% from right to wrong. Another 3.0% change from one wrong answer to another. 95.0% of answer-only responses follow the requested format. Appendix G.7 now reports this control and the paired answer changes (Table A22).

W1 & Q2: Evidence elicitation and model comparisons

With task-specific completeness instructions, 23.4% of baseline M-CUR responses become FA and 59.5% remain M-CUR. The instructions specify fact types, while the model supplies the values. For directional questions, the prompt requests object identities, locations, and relative direction. The table tracks label transitions for 667 baseline M-CUR responses (%), using the first scoring pass for Evidence-first and Completeness.

LabelShare
FA23.4
M-CUR59.5
C-CUR13.9
Wrong answer3.1

The 13.9% that move to C-CUR now contain a refuted claim. In total, 490 of 667 responses (73.5%) remain correct but unfaithful under the completeness prompt. For model comparisons, the next table gives Evidence-first / Completeness results on the same 500 questions for each model. Evidence scores are averaged over two scoring passes. Scores are percentages; FA ranks are within these eight models.

ModelAAFAC-CURFA rank
GPT-5.157.4 / 59.627.0 / 31.417.2 / 17.45 / 4
Gemini 3.1 Pro64.2 / 65.636.0 / 40.513.0 / 15.71 / 1
Qwen3.5-27B60.8 / 60.427.9 / 31.515.4 / 17.94 / 3
Qwen3-VL-32B56.6 / 57.828.9 / 30.119.6 / 19.93 / 5
InternVL3.5-38B55.6 / 55.029.5 / 33.38.0 / 8.82 / 2
RSThinker42.0 / 41.622.4 / 26.17.4 / 8.86 / 6
GeoChat33.6 / 38.08.6 / 15.63.8 / 8.58 / 8
RemoteReasoner48.6 / 50.612.9 / 18.88.0 / 12.47 / 7
Mean52.4 / 53.624.2 / 28.411.6 / 13.7—

FA improves for all eight models, with a mean gain of 4.3 points (95% CI [3.34, 5.20], computed before rounding). M-CUR decreases from 16.6% to 11.5%, and the fidelity gap (AA − FA) narrows from 28.2 to 25.2 points. The FA ranking remains similar across the two prompts (Spearman ρ = 0.93). Section 4.2 now specifies that the 63% M-CUR share refers to the standard evidence request. Appendix G.7 adds the completeness control and per-model results (Table A22).

Across the six within-family upgrades in Table 2, mean C-CUR rises by 3.6 points while M-CUR changes by −0.4 points. Refuted claims account for the widening gap. We have added these paired comparisons to Appendix G.2.

AuthorsOpenReview ↗

Official Response to Reviewer 2xmG (2/2)

W2 & Q3: Evidence sufficiency and M-CUR validation

Genuine omissions account for 67% of responses in the targeted M-CUR audit. Two annotators independently classified 100 randomly sampled M-CUR responses (κ = 0.66). The verifier accepts equivalent facts, including inverse spatial relations and counts summed over disjoint regions covering the queried scope (Appendix B.3). The audit also identifies valid alternatives: responses that establish the answer through a different set of facts from those specified by the contract. For example, stating the locations of the building and all candidate fields can establish which field lies below the building. Shares and 95% CIs below are percentages (Table A24).

CauseShare95% CI
Genuine omission67[57.3, 75.4]
Valid alternative8[4.1, 15.0]
Extraction / normalization / coverage17[10.9, 25.5]
Unclear8[4.1, 15.0]

Across the benchmark, 17.3% of the 70,255 unfilled contract slots in M-CUR responses have a corresponding claim that the source annotations cannot verify. The remaining slots either have no matching extracted claim or have only partial support, such as a subregion count or a broader category label. In the table, n counts slots and Share is a percentage; percentages are rounded independently.

EvidencenShare
No matched claim46,89666.8
Partial support11,21916.0
Unverifiable12,14017.3

Human sufficiency judgments. Raters judge 73.6% of M-CUR responses partially or not supported and 89.1% of FA responses fully supported. In this larger audit of 2,500 responses from five models, raters inspect images and responses without seeing contracts or automated labels. Table A23 compares their judgments with the automatic labels; entries are counts.

LabelFullPartialNoneTotal
FA4763325534
C-CUR6414386293
M-CUR9617296364
Wrong answer473788841,309
Total6837261,0912,500

Using these human sufficiency ratings while retaining the automatic contradiction check gives a 24.8-point gap, compared with 26.3 under automatic scoring. For the 364 M-CUR responses, the next table gives the ratings by reasoning level and subtype (Table A24). Bold rows summarize each level; n counts responses. Full, Partial, and None are human support ratings (%), and brackets give 95% Wilson CIs for Full.

Level / subtypenFullPartialNone
Perception10918.3 [12.2, 26.6]45.935.8
Object presence4814.6 [7.2, 27.2]43.841.7
Object counting2321.7 [9.7, 41.9]47.830.4
Object localization1915.8 [5.5, 37.6]52.631.6
Fine-grained recognition1926.3 [11.8, 48.8]42.131.6
Relational Reasoning18033.9 [27.4, 41.1]45.620.6
Directional3930.8 [18.6, 46.4]48.720.5
Topological4833.3 [21.7, 47.5]54.212.5
Proximity3125.8 [13.7, 43.2]41.932.3
Projective ordering2128.6 [13.8, 50.0]38.133.3
Aggregate distribution4146.3 [32.1, 61.3]39.014.6
Temporal Reasoning7520.0 [12.5, 30.4]53.326.7
Category turnover119.1 [1.6, 37.7]45.545.5
Semantic transition3528.6 [16.3, 45.1]51.420.0
Net change2913.8 [5.5, 30.6]58.627.6
All36426.4 [22.1, 31.1]47.326.4

By reasoning level, 81.7% of M-CUR responses in Perception, 66.1% in Relational Reasoning, and 80.0% in Temporal Reasoning are rated partially or not supported. Appendix G.8 now includes these ratings and the analyses of M-CUR causes and verification coverage.

Representative cases. The three added examples show why the missing fact is needed to distinguish the answer options. In Perception, the stated row places the roundabout in the lower third; its horizontal position distinguishes bottom-center from bottom-right. In the Relational Reasoning case, the evidence identifies two fields on the left side of the image without stating which lies below the building. For Temporal Reasoning, the evidence describes buildings at the later date but omits whether the same area previously contained low vegetation or trees. Appendix E now includes these cases and identifies the unfilled slots.

The authors' responses have adequately addressed my concerns, and the additional experiments further strengthen the paper. I appreciate the authors' efforts in clarifying the methodology and providing further empirical evidence. I have no remaining concerns and will maintain my original rating.

Thank you for your positive assessment and for maintaining your rating. We sincerely appreciate your constructive feedback.

Reviewer_8KeVOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper evaluates MLLMs in remote sensing, highlighting that correct answers may be accompanied by visual evidence that is either contradicted by the image or insufficient to support the answers. To address this issue, the authors introduce RSFaith-Bench, which uses evidence contracts to evaluate the faithfulness of models' stated visual evidence. The benchmark covers three reasoning levels: Perception, Relational Reasoning, and Temporal Reasoning, and introduces three metrics: FA, C-CUR, and M-CUR. Evaluations across 22 MLLMs reveal a substantial gap between AA and FA. Furthermore, the authors conduct extensive analyses, including human judgment, visual dependence, pipeline robustness, extractor quality, prompt sensitivity, and answer format, to further validate and investigate their findings.

Critical strengths

This paper extends the evaluation of MLLMs beyond final-answer correctness to the consistency and sufficiency of stated visual evidence. By introducing the Grounded Fact Layer, evidence contracts, and a three-valued verifier, the authors establish a structured and traceable faithfulness evaluation framework rather than directly relying on an LLM to judge entire explanations. Furthermore, RSFaith-Bench features a well-structured design in terms of dataset scale, task diversity, and construction procedures. The experiments cover a diverse range of MLLMs and include extensive reliability analyses, further strengthening the credibility of the findings.

Critical weaknesses
  1. The paper primarily examines whether stated visual evidence is faithful and sufficient for correct answers, but provides limited analysis of the relationship between evidence quality and answer correctness, particularly whether incorrect answers are associated with unreliable visual evidence. Furthermore, evidence contracts map each question to predefined evidence slots (RqR_q), where satisfying these slots is considered sufficient support. However, it remains unclear whether this formalized verification truly captures the explanatory value of visual evidence or merely evaluates factual correctness and schema-level sufficiency.
  2. Although the benchmark covers Perception, Relational Reasoning, and Temporal Reasoning, its tasks primarily focus on object recognition, spatial relations, and temporal changes, with limited coverage of complex multi-hop reasoning and decision-making tasks. This restricts its applicability to more challenging remote sensing scenarios. Moreover, the reliability of closed scopes heavily depends on the completeness of source annotations. In particular, a predefined relation predicate inventory does not necessarily guarantee exhaustive annotation of all valid relation instances. Further validation of annotation completeness across source datasets is needed to avoid incorrectly labeling valid visual claims as contradictions.
Questions for authors
  1. For counting and fine-grained recognition tasks, simply restating the final answer as a visual fact may satisfy the evidence contract. How do the authors distinguish such answer restatements from genuinely explanatory visual evidence?
  2. The paper mainly evaluates evidence faithfulness for correct answers, with limited analysis of the evidence accompanying incorrect answers. Could the authors further investigate the relationship between evidence faithfulness and answer correctness, particularly whether more faithful evidence can improve AA?
  3. Section 4.3 shows that answer options increase AA much more than FA. Could the authors further analyze the relationship between AA and FA under different answer formats and explain the implications of this difference for FA evaluation?
  4. The paper describes question generation through executable question programs, but the details of natural-language question and answer-option generation remain unclear. Could the authors clarify the roles of rule-based templates, human involvement, and LLMs in this process?
  5. Appendix B.2 indicates that atomic claims that cannot be correctly extracted or normalized may fail to fill the required evidence slots. Despite the extractor robustness analysis, could the authors further quantify the impact of claim extraction errors on FA and M-CUR?
  6. Missing support is one of the paper's key findings, yet Appendix E mainly presents FA and C-CUR cases. Could the authors provide representative M-CUR cases across Perception, Relational Reasoning, and Temporal Reasoning, highlighting the unfilled evidence slots?
  7. In the case studies, even when correct visual evidence sufficiently supports the answer, an additional irrelevant contradicted claim causes the entire response to be classified as C-CUR. Is this criterion overly strict? Could the authors distinguish answer-critical from ancillary contradictions and analyze their impact on FA?
4 comments
AuthorsOpenReview ↗

official Response to Reviewer 8KeV (1/2)

We appreciate your positive assessment of the framework and its validation. The additional analyses connect evidence quality to answer correctness and human judgments, while testing the findings under alternative scoring rules.

W1 & Q1: Explanatory value and answer restatements

Across presence, counting, and fine-grained questions, 75.5% of faithful responses add verified facts about the queried object beyond the question and selected option, such as instance locations. Pure restatements account for 20.1% of faithful counting responses and 29.7% of faithful fine-grained responses. Excluding pure restatements on the full benchmark gives FA = 22.5%, compared with 24.3% under standard scoring, leaving a 27.7-point fidelity gap (AA − FA).

Raters prefer the FA response in 69.0% of 200 FA–M-CUR pairs with correct answers and similar claim counts and evidence lengths. Both responses answer the same question and contain no automatically detected contradiction. In a further 200 FA–C-CUR pairs, the preference is 81.5% (Appendix D.3). In the 2,500-response audit, raters assess evidence sufficiency without seeing contracts or automated labels. Using their ratings while retaining the automatic contradiction check leaves a 24.8-point gap, compared with 26.3 under automatic scoring. The revised paper adds the restatement analysis in Appendix G.4 and the comparison with human sufficiency ratings in Appendix G.8.

W1 & Q2: Evidence quality and answer accuracy

Relative to the standard evidence-first prompt, completeness instructions improve mean AA by 1.2 points and FA by 4.3 points. Responses with correct answers also have higher claim precision (CP) and are less likely to contain a refuted claim. Both measures below are percentages.

AnswerCPContradiction rate
Correct83.219.1
Wrong67.262.4

The evidence contract is filled in 1.8% of wrong responses. To examine whether better evidence elicitation also benefits accuracy, we compare prompts for eight models on 500 matched questions, with images, options, and decoding settings fixed. Completeness instructions specify the fact types needed for each task; the model supplies the values. FA, C-CUR, and M-CUR are reported for the two evidence prompts and averaged over two scoring passes. Scores are percentages; Gap = AA − FA, in percentage points (pp). CIs denote 95% confidence intervals.

PromptAAFAC-CURM-CURGap
Answer-only53.3————
Evidence-first52.424.211.616.628.2
Completeness53.628.413.711.525.2

The standard evidence request changes AA by −1.0 points relative to answer-only generation (95% CI [−2.00, 0.03]). The completeness gains over the standard prompt have paired 95% CIs of [0.53, 1.95] points for AA and [3.34, 5.20] for FA, using unrounded scores. In this comparison, completeness instructions change 2.5% of answers from wrong to right and 1.3% from right to wrong. Explicit completeness instructions improve both answer accuracy and verified support under the tested protocol. We have added the evidence-quality comparison to Appendix G.5 and the prompt controls to Appendix G.7.

Q3: Answer format

Answer options increase mean AA by 13.9 points and FA by 3.5 points. A 17.7-point fidelity gap remains under open-ended answering. We use 500 matched questions with the same images, gold answers, contracts, and evidence request. Predefined rules map open answers to the same answer space. Each cell reports multiple-choice / open-ended results. AA and FA are percentages; Gap is in pp.

ModelAAFAGap
GPT-5.157.4 / 42.827.0 / 22.730.4 / 20.1
Gemini 3.1 Pro64.2 / 49.836.0 / 30.828.2 / 19.0
Qwen3.5-27B60.8 / 47.827.9 / 25.832.9 / 22.0
Qwen3-VL-32B56.6 / 43.528.9 / 26.127.7 / 17.4
InternVL3.5-38B55.6 / 39.029.5 / 24.226.1 / 14.8
RSThinker42.0 / 29.422.4 / 17.319.6 / 12.1
GeoChat33.6 / 21.88.6 / 7.225.0 / 14.6
RemoteReasoner48.6 / 33.212.9 / 11.335.7 / 21.9
Mean52.4 / 38.424.2 / 20.728.2 / 17.7

Of the 13.9-point AA gain from answer options, 10.5 points (75%) correspond to a wider gap. Among correct open-ended responses, 46.2% are C-CUR or M-CUR. Missing support accounts for similar shares of the gap in the two formats, 59.0% and 61.5%, respectively. Comparing AA with FA distinguishes gains in answer accuracy from gains in verified support across both formats. Section 4.3 now clarifies how answer options affect the gap; Appendix D.10 gives the full comparison (Table A18).

AuthorsOpenReview ↗

Official Response to Reviewer 8KeV (2/2)

W2 & Q4: Benchmark design and validation

Task coverage. The benchmark tests whether models provide the visual facts needed to justify spatial and temporal answers. Directional questions require object identities and their spatial relation; semantic-transition questions need states at both dates. These explicit requirements make answer support directly checkable and expose missing facts in correct responses. Relational and temporal questions make up 66.1% of the benchmark (8,931 questions). Their mean fidelity gaps are 27.5 and 24.0 points, respectively (Section 4.2). For change monitoring, evidence from both dates makes a claimed land-cover transition directly checkable.

Question generation. Executable programs compute gold answers, evidence contracts, and answer options from source annotations. Rule-based templates render the question text; generation uses no LLM. Annotators check each question, gold answer, and required contract fact against the image and confirm that exactly one option is valid. For example, a ship-ordering question lists permutations of the same three ships; their annotated positions determine the correct order. Appendix A.2 now clarifies question and option generation; Appendix A.4 describes validation.

Annotation validation. The independent image-only audit confirms 93.2% of contradiction labels for relation claims and overturns 4.7%. We sampled C-CUR responses and audited every contradicted claim in each sampled response. Two annotators independently reviewed the images without seeing annotations or automated labels; a third adjudicated disagreements (κ = 0.64). In the table below, n counts claims, rates are percentages, and brackets give 95% CIs.

GroupnConfirmedOverturnedUnclear
Objects15397.4 [94.5, 99.4]0.72.0
Relations14893.2 [89.2, 96.8]4.72.0
Temporal changes15484.4 [77.8, 90.4]5.89.7

“Confirmed” means the image shows that the claim is false; “overturned” means the image shows that it is true. Rescoring the sampled responses with the audit corrections gives an estimated 0.36-point decrease in mean C-CUR. Verification uses relation records, decisive geometry, and absence within declared closed scopes. We have clarified these rules in Section 3.1 and added the image-only audit to Appendix G.1 (Table A19).

Q5: Claim extraction errors

Using human gold claims changes FA by +1.0 percentage point and M-CUR by −0.8 points. We rescore 500 responses with answers, contracts, and verification rules unchanged. Δ denotes gold-claim minus automatic-extraction scores; changes and CIs are in pp.

MetricΔ95% CI
FA+1.0[−0.63, 2.79]
M-CUR−0.8[−2.58, 0.82]

Labels change for 23 responses (4.6%): 13 move from M-CUR to FA, 9 from FA to M-CUR, and 1 from C-CUR to FA. The gold-claim rerun supports the stability of aggregate FA and M-CUR. Appendix G.6 now reports this rescoring analysis.

Q6: Representative M-CUR cases

The three added M-CUR cases identify the missing fact needed to distinguish the answer options. In Perception, the evidence places the roundabout in the lower third, but its horizontal position is needed to distinguish bottom-center from bottom-right. The Relational Reasoning evidence identifies two fields on the left side of the image but leaves their positions relative to the building unstated. For Temporal Reasoning, the evidence describes buildings at the later date but omits whether the same area previously contained low vegetation or trees. These cases and their unfilled slots are now shown in Appendix E.

Q7: Ancillary contradictions

Counting only answer-critical contradictions raises FA from 24.3% to 25.8%, leaving a 24.4-point fidelity gap. Standard FA evaluates all stated visual claims. The answer-critical variant counts a contradiction only if it conflicts with a contract slot or an input to the answer function. The comparison varies one scoring rule at a time on the full benchmark, with AA fixed at 50.2%. Scores are percentages; Gap is in pp.

RuleFAC-CURM-CURGap
Standard24.39.616.325.9
Restatement excluded22.59.618.127.7
Answer-critical only25.87.117.224.4

Among C-CUR responses, 74.5% contain an answer-critical contradiction and 25.5% contain only ancillary contradictions. Under the answer-critical rule, 63.0% of the ancillary-only group move to FA; 37.0% move to M-CUR because support remains incomplete. Removing the contradiction penalty gives an FA upper bound of 33.9%, with the support check fixed. The resulting gap is at least 16.3 points. Appendix G.4 now defines the answer-critical criterion and compares it with standard FA (Table A21).

Reviewer_8KeVOpenReview ↗

Response to Authors

The authors have addressed most of my concerns; accordingly, I lean toward raising my previous score to 3.

Thank you for taking the time to review our work and consider our responses. We sincerely appreciate your constructive feedback, which has helped us improve the paper.

Reviewer_EHKNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether multimodal LLMs give correct evidence when they answer questions about satellite images, not just correct answers. The authors build RSFaith-Bench, with 13,511 questions from six remote sensing datasets, covering perception, relational and temporal reasoning. Each question has an “evidence contract” that lists the facts needed to support the answer. The model’s stated claims are checked against the dataset labels using rules, without an LLM judge, which works because these datasets label every object of their categories. A response counts as faithful only if the answer is correct, no claim is contradicted, and the supported claims cover the contract. Moreover, by testing 22 models, the authors find that average answer accuracy is 50.2% but faithful accuracy is only 24.3%. Most of this gap comes from missing support rather than false claims. The bigger or newer models improve answer accuracy much more than faithful accuracy, and ranking by faithful accuracy changes the model order.

Critical strengths

Following are some of the key strengths of the paper:

  • The paper separates two things that current benchmarks mix up: getting the answer right, and giving evidence that actually supports it. Splitting unfaithful correct answers into contradiction-bearing (C-CUR) and missing-support (M-CUR) is simple but useful.
  • The study covers 22 models across proprietary, open-source and remote-sensing groups so the experiments are large and the main findings are interesting
  • Defining the required facts first and then generating the question from them means the gold answer and its required evidence always come from the same source. This makes sufficiency checkable, which claim-level methods like FaithScore cannot do.
  • The evaluation design is principled and does not depend on an LLM judge
Critical weaknesses

Following are some of the weaknesses of the paper:

  • The idea that a model can give the correct answer with wrong or incomplete visual evidence is already studied. For example, On the Faithfulness of Visual Thinking [1] already checks if visual evidence is reliable and sufficient, and shows that correct answers can still come with bad evidence. CLEVR-X generates explanations using scene graphs and question programs, which seems close to the evidence contract idea. VISTAQA and ViEBench also evaluate answers along with their evidence. Except for ViEBench, these papers are not discussed. I think the authors should clearly explain what evidence contracts add over these works.
  • The paper assumes the labels are complete, but what if an object or relation was missed in the annotation and the model is actually right? The 95.2% audit helps, but it only checks a sample using the same annotations, so I think the authors should explain how they confirm completeness for each dataset and report separate error rates for objects, relations and temporal changes.
  • FA checks whether the evidence is true and complete, not whether it actually led to the answer, and there are no confidence intervals even though many gaps between models are only 1 - 3 points. An intervention test, like editing a key fact in the evidence or changing the image and checking if the answer changes, plus bootstrap confidence intervals on FA and on the rankings, would make the main claims much stronger.
  • For counting, presence and fine-grained questions, just repeating the answer as a sentence (“there are three courts”) fills the slot, and false claims marked uncertain don’t affect FA at all so i think FA is too easy to pass in some cases.

[1] On the Faithfulness of Visual Thinking: Measurement and Enhancement

Questions for authors
  • Which exact splits did the authors use for each dataset? As far as I know, the test labels for DOTA-v2.0 and FAIR1M are not public. So are these validation splits, and could any of the remote sensing models have seen these images during training?
  • For the proprietary models, was thinking or reasoning mode turned on for GPT-5.1, Claude and Gemini 3.1 Pro, and with what budget?
  • Qwen3-32B is the claim extractor, but it is also the language backbone of Qwen3-VL-32B and InternVL3.5-38B. Did the authors check whether the extractor works better on responses from models with the same backbone?
  • what was estimated time spent on each QA pair validation?
5 comments
AuthorsOpenReview ↗

Official Response to Reviewer EHKN (1/3)

Thank you for recognizing the value of evaluating answers together with their supporting evidence. We clarify what evidence contracts measure and provide independent audits, an evidence intervention, and confidence intervals.

W1: Contribution of evidence contracts

Evidence contracts make answer-specific evidence sufficiency directly checkable against source annotations. We verify each claim against source annotations, then check whether the supported claims fill the contract generated with the question and gold answer. Related benchmarks evaluate different aspects of evidence: On the Faithfulness of Visual Thinking [1] assesses crop sufficiency with a judge model. CLEVR-X generates reference explanations from scene graphs and question programs, then scores generated explanations by text overlap. VISTAQA and ViEBench evaluate answers and evidence regions or masks.

The distinction matters when a correct response contains no refuted claim but omits a fact needed to establish its answer. Under the standard evidence request, such responses account for 63% of the fidelity gap (AA − FA). Equivalent statements, including inverse spatial relations and counts summed over disjoint regions, can satisfy the same contract. We have expanded the comparison with prior work in Section 2 and Table A1.

W2: Annotation completeness and contradiction audit

We check contradiction labels directly against the images, with separate results for objects, relations, and temporal changes. Two annotators independently reviewed all 455 contradicted claims in sampled C-CUR responses without seeing source annotations or automated labels; a third adjudicated disagreements (κ = 0.64). After correcting the claim labels, we rescore each sampled response for remaining contradictions. This yields an estimated 0.36-point decrease in mean C-CUR. “Confirmed” means the image shows that the claim is false; “overturned” means the image shows that it is true. Rates are percentages, n counts claims, and brackets give 95% confidence intervals (CIs).

GroupnConfirmedOverturnedUnclear
Objects15397.4 [94.5, 99.4]0.72.0
Relations14893.2 [89.2, 96.8]4.72.0
Temporal changes15484.4 [77.8, 90.4]5.89.7

For benchmark construction, each question's gold answer and required facts are checked against the image. Object datasets provide instance annotations, ReCon1M relation records, SECOND changed-region labels, and xBD building damage labels (Appendix A.1). A closed scope specifies the predicate, objects, region, and time of a check. Refutation uses incompatible records, decisive geometry, or absence within the declared closed scope; unresolved claims remain uncertain. Section 3.1 now clarifies these rules, and Appendix G.1 adds the independent image-only audit (Table A19).

AuthorsOpenReview ↗

Official Response to Reviewer EHKN (2/3)

W3: Evidence intervention and confidence intervals

Evidence intervention. Editing one visual fact increases selection of the option indicated by the edit by 36.6 points. We test 555 faithful responses from six models, changing one fact while leaving the remaining evidence unchanged. We exclude evidence containing explicit answers, option selections, or concluding statements. Each model receives either the original or edited evidence as its own previous response and then answers the question. The image is present in both conditions. Entries give the percentages selecting the target option, the original correct option, or another option.

EvidenceTargetOriginalOther
Original1.496.62.0
Edited38.057.84.1

The intervention establishes an effect on subsequent answer generation, complementing FA's assessment of verifiable support. We have added the intervention protocol and results to Appendix G.3.

Confidence intervals. Scene-level bootstrap intervals support the main rank reversals. We resample scenes 10,000 times and recompute each model's FA and rank on each resample. A scene is one image or one bi-temporal pair. Table A20 reports all 22 models; AA and FA are percentages, brackets give 95% CIs, and ranks refer to this full set.

ModelAAAA rankFAFA rank
GPT-4o54.5930.0 [29.2, 30.8]7 [6, 8]
GPT-5.155.8631.3 [30.5, 32.1]5 [4, 6]
Claude Haiku 4.546.41828.3 [27.5, 29.1]10 [9, 10]
Claude Sonnet 4.657.6333.1 [32.3, 33.9]3 [2, 3]
Gemini 3.1 Pro63.8135.3 [34.5, 36.2]1 [1, 1]
Qwen3-VL-8B52.41127.3 [26.5, 28.0]11 [11, 12]
Qwen3-VL-32B55.5729.9 [29.1, 30.6]8 [6, 8]
Qwen3.5-9B53.41028.4 [27.6, 29.2]9 [9, 10]
Qwen3.5-27B57.8227.1 [26.4, 27.9]12 [11, 12]
Qwen2.5-VL-7B47.61622.4 [21.6, 23.1]14 [14, 15]
Qwen2.5-VL-72B50.51221.0 [20.3, 21.7]16 [16, 16]
InternVL3.5-8B55.2831.9 [31.1, 32.7]4 [4, 5]
InternVL3.5-38B55.9530.3 [29.5, 31.1]6 [6, 8]
LLaVA-OneVision-7B47.61612.7 [12.2, 13.3]20 [20, 20]
GLM-4.5V56.2433.8 [33.0, 34.6]2 [2, 3]
RSThinker42.01924.2 [23.5, 24.9]13 [13, 13]
GeoChat35.9219.2 [8.8, 9.7]21 [21, 22]
VHM39.22014.8 [14.2, 15.4]18 [18, 19]
GeoReason49.01417.8 [17.2, 18.5]17 [17, 17]
GeoZero-8B49.31322.2 [21.5, 22.9]15 [14, 15]
RemoteReasoner47.91514.4 [13.8, 15.0]19 [18, 19]
RS-EoT-7B30.1228.7 [8.3, 9.2]22 [21, 22]
Mean50.2—24.3 [23.9, 24.7]—

All 12 adjacent model pairs separated by at least 1.0 FA point have paired 95% intervals excluding zero. Paired comparisons also substantiate the main rank reversals. Claude Haiku 4.5 trails Qwen3.5-27B in AA but leads in FA by 1.2 points [0.27, 2.08]. RSThinker also has lower AA than Qwen2.5-VL-72B, yet its FA is 3.2 points higher [2.35, 4.07]. In Relational Reasoning, Claude Sonnet 4.6 similarly leads Gemini 3.1 Pro in FA by 4.8 points [3.10, 6.32] despite lower AA.

The six within-family upgrades give the following paired changes in percentage points (pp). Brackets give 95% CIs for ΔFA.

UpgradeΔAAΔFAΔGap
Qwen2.5-VL 7B → 72B+2.9−1.4 [−2.23, −0.58]+4.3
Qwen3-VL 8B → 32B+3.1+2.6 [1.68, 3.50]+0.5
Qwen3.5 9B → 27B+4.4−1.3 [−2.21, −0.39]+5.7
InternVL3.5 8B → 38B+0.7−1.6 [−2.53, −0.67]+2.3
GPT-4o → GPT-5.1+1.3+1.3 [0.30, 2.19]0.0
Claude Haiku 4.5 → Sonnet 4.6+11.2+4.8 [3.87, 5.70]+6.4
Mean+3.9+0.7 [0.37, 1.08]+3.2

The mean AA gain has a 95% interval of [3.51, 4.30] points, and all three FA declines are significant. Across these upgrades, mean C-CUR rises by 3.6 points while M-CUR changes by −0.4 points; refuted claims account for the widening gap. The paired comparisons and bootstrap intervals are now reported in Appendix G.2.

AuthorsOpenReview ↗

Official Response to Reviewer EHKN (3/3)

W4: Answer restatements and uncertain claims

Excluding pure answer restatements lowers FA from 24.3% to 22.5%, leaving a 27.7-point fidelity gap. We define a pure restatement as adding no supported fact about the queried object beyond the question and selected option. Such responses account for 24.5% of faithful responses on presence, counting, and fine-grained questions; the other 75.5% provide additional verified facts, such as instance locations. In the schema-blind audit, raters judge 89.1% of FA responses fully supported without seeing the contracts (Table A23). Appendix G.4 now includes the restatement analysis (Table A21).

Only supported claims can fill contract slots. Claims that the annotations neither support nor refute are labeled uncertain and kept separate from contradictions.

Q1: Source splits and training overlap

We use the DOTA-v2.0 training and validation partitions and the FAIR1M validation partition. Test annotations are not public for either dataset. Six of the seven remote sensing models report using datasets that overlap with our sources, so some benchmark images may have appeared in their training data. The table gives question counts for all six sources, with Val denoting validation.

DatasetTrainValTestTotal
DIOR001,0331,033
DOTA-v2.04352060641
FAIR1M02,27302,273
ReCon1M005,5175,517
SECOND2,21701,2153,432
xBD00615615
Total2,6522,4798,38013,511

Our questions and evidence contracts are newly constructed. Among the 15 general-purpose models, mean AA is 54.0% and FA is 28.2%, leaving a 25.8-point gap. Appendix A.1 now reports the source partitions and question counts in Table A2.

Q2: Reasoning modes and budgets

Thinking/reasoning mode was disabled for all evaluated proprietary models, with no additional reasoning-token budget specified. Each model used the same reasoning settings throughout all experiments. We have clarified these settings in Appendix B.1.

Q3: Claim extraction for shared-backbone models

Qwen3-32B achieves comparable extraction quality on the original gold set and on responses from Qwen3-VL-32B and InternVL3.5-38B. Here n counts responses and Claims counts gold claims; precision and recall are percentages, and F1 is on a 0–1 scale.

Gold setnClaimsPrecisionRecallF1
Original5005,77491.689.80.907
Qwen3-VL-32B1001,53391.188.80.899
InternVL3.5-38B10076792.691.40.920

For the questions in the shared-backbone comparison, the original gold set gives F1 = 0.908. The paired F1 difference is −0.008 [−0.022, 0.005] for Qwen3-VL-32B and +0.012 [−0.006, 0.030] for InternVL3.5-38B (95% intervals). With GPT-4.1 as the extractor, FA changes by +1.6 and +1.8 points for these models, respectively. Both changes fall within the −2.0 to +2.4 range for the other 20 models (Table A14). The added shared-backbone results appear in Table A13, with paired F1 intervals in Appendix G.6.

Q4: Question validation time

Validation took about 90 seconds per question, totaling approximately 337 annotator-hours. Annotators checked the generated question, gold answer, evidence slots, and fact records against the source image and annotation. Appendix A.4 details the validation procedure and total annotation time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.