acceptodds
Under review as a conference paper at ICLR 2027

MMSciFact: A Benchmark for Multimodal Dependency-Aware Scientific Fact-Checking

Abstract

Multimodal large language models (MLLMs) are increasingly used to answer questions about scientific papers with long-form responses grounded in figures, tables, and text. Yet factuality evaluation typically decomposes these answers into atomic claims and verifies them independently. This independence assumption overlooks how observations serve as premises for downstream interpretations, forming a dependency graph within each answer. An interpretation can be supported in isolation yet depend on a premise contradicted by or unverifiable from the source—a failure we call Invalid Support. We introduce MMSciFact, a benchmark for dependency-aware multimodal scientific fact-checking comprising 257 question-answer pairs and 2,737 sentences from 45 papers. Model answers are preserved without post-hoc corruption and annotated with expert grounding judgments, sentence roles, and dependencies. Across 12 MLLM fact-checkers and single-page, cross-page, and full-paper evidence, models struggle to identify statements contradicted by or unverifiable from the source, as well as interpretations based on invalid premises. Providing sentence roles and dependency graphs improves the detection of Invalid Support, but scores remain low even with this information, indicating that dependency structure alone is insufficient for reliable verification. These findings suggest that dependency awareness complements claim-level verification in supporting reliable scientific fact-checking.

Primary area:
datasets and benchmarks
Keywords:
factuality, scientific question answering, long-form generation, fact-checking, claim verification, dependency-aware evaluation, scientific documents, multimodal large language models, benchmark

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_QLqiOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper identifies a gap in current factuality evaluation for long-form answers. The dominant paradigm, established by FActScore and followed by FACTS Grounding and subsequent work, decomposes an answer into atomic claims and verifies each claim in isolation. This "independence assumption" overlooks the reasoning structure internal to the answer: later interpretations typically take earlier observations as premises, forming a dependency graph over the answer. The paper's Figure 1 illustrates the consequence: when an observation is contradicted by the source, interpretations built on it are still labeled "supported" by claim-level verification, because in isolation each interpretation can indeed be found in the source.

The authors name this failure INVALIDSUPPORT and build a benchmark, MMSciFact, to make it measurable: 45 papers, 257 question–answer pairs, and 2,737 answer sentences, spanning three evidence scopes (single-page, cross-page, full-paper). Each instance pairs a paper and a question with an MLLM-generated long-form answer whose errors occur naturally and are neither injected nor corrected. Expert annotators label sentence roles, cross-sentence dependencies (forming a DAG), and grounding labels (SUPPORTED / CONTRADICTED / NEI / INVALIDSUPPORT). In total, 148 sentences (5.4% of all sentences) are labeled INVALIDSUPPORT, affecting 23% of answers. Evaluation is conducted under three settings: Independent (dependencies ignored), Holistic (the judge must infer dependencies itself), and Oracle-graph (annotated roles and dependency graph are provided).

Key findings: across 12 MLLM judges, current models remain weak scientific fact-checkers, strongly over-predicting SUPPORTED, struggling most with NEI, and showing limited evidence-localization ability. INVALIDSUPPORT is hard to detect when judges must infer dependencies themselves; providing the annotated dependency graph improves its detection substantially (significant for 9 of 12 judges), yet absolute scores remain low even then.

Critical strengths
  1. The problem is well identified and targets a genuine blind spot in the prevailing evaluation paradigm. The observation that claim-level verification misses conclusions built on false premises is conceptually clear, is illustrated convincingly by Figure 1 and the constructed examples in the three judge prompts, and is indeed not covered by existing scientific fact-checking benchmarks (Table 1).

  2. INVALIDSUPPORT is a label of real value, and it is formally defined. The paper gives an explicit decision rule (an interpretation that would be SUPPORTED on its own, at least one of whose direct or transitive premises is CONTRADICTED or NEI) and separates "whether the conclusion itself holds" from "whether the premises supporting it hold" — a distinction of practical importance in scientific settings.

  3. The three verification settings constitute a sound diagnostic design. The progression Independent / Holistic / Oracle-graph separates three distinct situations — not modeling dependencies, having to recover them, and having them given but still needing to use them correctly — which is the most methodologically valuable aspect of the paper.

Critical weaknesses

W1 INVALIDSUPPORT is the paper's central contribution, yet its two causes are reported jointly and the alternative explanation — that the premises themselves are misjudged — is not ruled out

INVALIDSUPPORT is the only new label relative to prior benchmarks (Table 1) and the anchor of every conclusion in the paper. Yet it is reported as a single quantity:

  • Table 3 gives one INV column, with no breakdown by cause.
  • Appendix F defines a complete error-code taxonomy (C1–C3 for factual misread, visual misinterpretation, and relational error; N1–N3 for fabricated evidence, unsupported inference, and overclaiming conclusion), together with two dedicated propagation codes: C4 (false-premise conclusion), for a conclusion built on premises contradicted by the source, and N4 (unverifiable-premise conclusion), for one built on unverifiable premises. However, neither the main text nor the appendix reports the distribution of these codes — I could not find sentence counts for C1–C3, N1–N3, C4, or N4 anywhere in the paper.

C4 and N4 are tasks of a different nature:

  • C4 requires the judge to locate an explicit contradiction in the document; there is a locatable erroneous element to point at.
  • N4 requires the judge to determine that the document never says this at all, which is detection of absence with no location to point to.

Table 3 already shows that among the three labels shared across settings, NEI has the lowest F1 under every setting. The central question is therefore: does the difficulty of detecting INVALIDSUPPORT come mainly from premises that are falsified (failure of dependency reasoning), or mainly from premises that cannot be verified (NEI-detection failure amplified along the dependency chain)? If the latter predominates, the paper's central finding is closer to "models cannot recognize missing evidence," with INVALIDSUPPORT merely amplifying a known weakness; only if the former predominates is it the new bottleneck, independent of grounding, that the paper claims. The current reporting cannot distinguish these two explanations, and they carry very different implications for the paper's conclusions.

A related argumentative gap appears in Section 5. The paper claims that because INVALIDSUPPORT remains low even when the annotated dependency graph is provided, recovering and reasoning over sentence dependencies constitutes a bottleneck distinct from evidence grounding. That inference skips an intermediate step: to label a sentence INVALIDSUPPORT correctly, the premise it depends on must first be judged CONTRADICTED or NEI. Yet CON F1 is only 0.1–0.6 and NEI only 0–0.4, so when premises themselves are misjudged, the judge never reaches INVALIDSUPPORT at all. "Low INV under Oracle-graph" therefore admits at least two explanations: (a) the judge cannot use the given dependency structure, or (b) the judge cannot even ground the premise sentences correctly. The "distinct bottleneck" claim requires (b) to be ruled out first, and no such conditional analysis appears in Appendix E (E.1–E.8).

W2 Domain coverage of the sources, and the validity confound introduced by pretraining memorization

Domain coverage. The introduction states that the sources are "drawn primarily from computer science," so the wording does not itself claim cross-domain generality. Per Appendix Figure 4, however, the topic distribution of QA pairs is: multimodal / vision-language 28%, biology & life sciences 13%, factuality, grounding & RAG 12%, summarization & text clustering 10%, LLM evaluation & generalization 8%, ML methods & architecture 7%, biomedical & clinical NLP 5%, with all remaining topics below 5%. Roughly three quarters of the instances therefore come from within NLP/ML, while physics, chemistry, materials science, earth sciences, and the social sciences are essentially absent. In addition, the 45 source papers were nominated by 12 authors with NLP/ML backgrounds (each paper contributing only about 6 QA pairs), and the venues cluster in ICLR / *ACL / NeurIPS (about 50%).

This raises a direct question: are the failure modes characterized here domain-general? The observation–interpretation distinction and dependency structure are common across the sciences, but the present data provide no evidence for cross-domain applicability. Appendix A already concedes "limiting source diversity and potentially introducing selection bias," yet the framing in the abstract and main text ("scientific fact-checking") is broader than the data support. I would suggest the authors either narrow the corresponding claims or add a discussion of domain coverage.

Pretraining memorization. The paper's treatment of leakage risk is confined to one sentence in Appendix A: public source papers may appear in pretraining data, but the questions, answers, dependency structures, and labels are newly created, which reduces the risk of instance-level contamination.

I do not think this argument covers the main risk in this setting. The task here is to judge whether a sentence is grounded in a given document, not whether it holds in the open world. The primary risk is therefore not that the model has seen this QA pair, but that the model has seen the source paper itself — if so, it can answer from memory rather than actually reading the few pages of evidence it is given. What is then measured is closer to recall than to grounding ability, which is precisely what the paper sets out to measure. The confound is further amplified by the paper's design: the single-page / cross-page / full-paper scopes presuppose that the judge really is restricted to the supplied pages.

The paper's own data hint that this confound may be present. Appendix E.4 specifically examines whether judges favor answers produced by models from their own family, which shows that the memorization/familiarity dimension was recognized — but only the model-family half was examined, not the source-paper half.

One concrete lead. Figure 1 and the worked examples in all three judge prompts are built on highly prominent public benchmarks such as MedQA / MedMCQA, and assume that "swapping the two benchmarks' scores" is a mistake models readily make. Those benchmarks are themselves in the training corpora of the models under evaluation. Does the authors' actual data contain instances of the same kind — errors a model would not make precisely because it is too familiar with the material? This bears directly on how the data were constructed: answers were generated by relatively recent models with naturally occurring errors, so if the generators are overly familiar with such background knowledge, the error distribution may not represent failure modes in real use. I would ask the authors to characterize the relationship between error types and how well known the source papers are.

W3 Further disclosure on annotation quality, and the scalability of the pipeline

The annotation quality is on the better end for work of this kind (role κ = 0.792–0.810, dependency F1 = 0.818–0.878, grounding pairwise κ = 0.839; grounding labels assigned entirely by humans without LLM pre-annotation; at least two independent annotators per paper, with rationales and evidence boxes; roughly 165 person-hours). The points below are requests for additional disclosure rather than a challenge to data quality.

First, the extent of disagreement and how it was resolved are under-reported. The paper reports agreement metrics but not the disagreement rate (how many of the 2,737 sentences reached the resolver interface), nor the proportion that remained unresolved after discussion, nor how often annotators used an override. Appendix G (Figures 15–17) describes disagreement as resolved by two annotators discussing and then selecting or overriding a label, with no third-party adjudication, so the reliability of the final labels on disputed cases cannot be assessed from the material provided.

Second, and more importantly, agreement on the INVALIDSUPPORT subset is not reported separately. The overall κ = 0.839 covers all sentences, but INVALIDSUPPORT has the longest decision chain — the role, the dependency edges, and the premise sentence's label must all be correct, and a difference in any one of them changes the final label. Appendix A already describes the reference dependency graph as "validated silver annotations" (pre-annotated by two Gemini runs, with human involvement only where the two runs disagree; its agreement with humans, κ = 0.770, is below the human–human 0.81). If the 148 INVALIDSUPPORT sentences had notably lower initial agreement between annotators than the corpus average, a substantial share of the paper's central claim — that models struggle to identify INVALIDSUPPORT — would be attributable to annotation noise.

Third, on scalability. The annotation budget is roughly 165 person-hours for 45 papers and 257 QA pairs, and grounding labels depend heavily on expert human judgment, with LLMs involved only in structural pre-annotation. This cost structure makes it hard to expand data scale and domain coverage together, which compounds the coverage issue raised in W2. This is not a defect of the paper so much as a request that the authors discuss more explicitly in Limitations what the scaling path would be if the benchmark were to cover additional disciplines.

Questions for authors
  1. Could the authors report the sentence-level distribution of all error codes (counts for C1–C3, N1–N3, C4, and N4, broken down by the three evidence scopes), and re-report INVALIDSUPPORT F1 split by C4 versus N4? This requires no re-inference; the existing annotations and predictions suffice.
  2. Could the authors provide a 2×2 decomposition: for each gold INVALIDSUPPORT sentence, cross-tabulate whether the model judged the sentence's premises (depends_on) correctly against whether the model correctly labeled the conclusion as INV. This separates two distinct failures directly — not recognizing a bad premise at all, versus recognizing it but failing to propagate it to the conclusion. If the latter accounts for only a small share, the paper's claim that dependency reasoning constitutes a distinct bottleneck would have direct support; otherwise the claim needs revision.
  3. Could the authors release the full list of the 45 source papers (with titles and publication years), and recompute the main metrics grouped by whether the source paper is likely to appear in pretraining data (for instance by publication year: older conference papers vs. newer 2025–2026 papers)? This requires no new annotation, but it would directly test whether the current numbers are inflated by memorization.
  4. Could the authors report the disagreement rate (number and share of sentences reaching the resolver), the frequency of overrides, and inter-annotator agreement specifically on the INVALIDSUPPORT subset (κ or raw agreement rate)?
6 comments
AuthorsOpenReview ↗

Thank you for your review!

Thank you for the careful feedback. We have completed the additional analyses described below. We are also updating the manuscript to include these results and clarify the main findings. We will upload the revised version separately.

Neither the main text nor the appendix reports the distribution of these codes — I could not find sentence counts for C1–C3, N1–N3, C4, or N4 anywhere in the paper.

Of the 148 INVALID SUPPORT sentences, 124 (83.8%) are C4 (a contradicted premise) and 24 (16.2%) are N4 (a premise that cannot be verified). Across the 12 judges, recall is 35.3% for C4 and 13.4% for N4. We report recall because judges predict only INVALID SUPPORT, not C4 or N4 separately. This means that false positives cannot be assigned to either type. N4 is harder to detect, but C4 accounts for about 80% of missed INVALID SUPPORT cases. The low scores therefore cannot be explained mainly by NEI premises.

"Low INV under Oracle-graph" therefore admits at least two explanations: (a) the judge cannot use the given dependency structure, or (b) the judge cannot even ground the premise sentences correctly.

We checked whether each judge correctly identified at least one incorrect premise (CON or NEI) in a conclusion's full reference dependency chain, and whether it correctly labeled the conclusion INVALID SUPPORT. This covers 1,776 judge–sentence pairs (12 × 148).

INV detectedINV missed
At least one incorrect premise correctly labeled436 (24.5%)106 (6.0%)
No incorrect premise correctly labeled130 (7.3%)1,104 (62.2%)

In 91.2% of missed INVALID SUPPORT cases (1,104/1,210), the judge also failed to correctly label any incorrect premise. This shows that checking the premises is a major difficulty. Even when a judge correctly labeled at least one incorrect premise, it still missed INVALID SUPPORT in 19.6% of cases (106/542). Finding an incorrect premise is therefore not enough. These 106 errors have two possible causes: the judge did not carry the premise error over to the conclusion, or it checked the conclusion itself incorrectly and labeled it CONTRADICTED or NEI, although the conclusion is supported when checked in isolation.

The "distinct bottleneck" claim requires (b) to be ruled out first, and no such conditional analysis appears in Appendix E (E.1–E.8).

For the four judges tested with self-predicted graphs, the dependency linking an INVALID SUPPORT conclusion to its faulty premise was correctly identified in only 51–61% of cases. Some INVALID SUPPORT errors remained even when the dependency and an incorrect premise label were correct, although the number of such cases was small.

These analyses show that many missed INVALID SUPPORT cases come from incorrect premise labels, but two other errors also play a role: the judge does not find the dependency between the conclusion and its premise, or it finds the premise error but still gives the conclusion a wrong label. We took these findings into account when interpreting the Oracle-graph results. To know which sentences are the premises of each conclusion, we always used the human-annotated graph, also for judges that did not receive it in their prompt.

Domain coverage of the sources, and the validity confound introduced by pretraining memorization.

Domain coverage. MMSciFact mainly covers computer science. We do not claim that our findings apply equally to all scientific fields. Expanding the benchmark would require annotators who understand those fields. We are updating the abstract, main text, and limitations to make this scope clearer.

Pretraining familiarity. We agree that a judge may have seen a source paper during training and could rely on memory instead of the pages provided. This could also make the difference between single-page, cross-page, and full-paper evidence less clear. A judge that remembers the paper can use content from pages it was not given, so a single-page instance may in practice be checked with knowledge of the whole paper. We are adding this limitation to the manuscript. Our questions, answers, and labels are new, but that does not mean the source papers are unknown to the models. The MedQA/MedMCQA error in Figure 1, for example, appeared naturally in a generated answer. We did not introduce it ourselves.

Additional analysis. Most of MMSciFact comes from recent work: 24 of the 45 source papers were published in 2025–2026, and they account for 157 of the 257 QA pairs (61%). To check whether familiarity with older papers affects the results, we split the benchmark by publication year: 21 papers published before 2025 (100 QA pairs) and 24 published in 2025–2026 (157 QA pairs). The table reports macro-F1 and INVALID SUPPORT F1 for each group across the same 12 judges (higher is better). Independent verification has no INVALID SUPPORT label, so its INVALID SUPPORT F1 is not defined.

SettingMacro-F1: older / newerINV F1: older / newer
Independent0.506 / 0.473–
Holistic0.384 / 0.3570.164 / 0.119
Oracle-assisted0.400 / 0.3730.247 / 0.207

Scores are slightly higher for older papers, but detecting INVALID SUPPORT remains difficult in both groups, and the reference graph improves it by a similar amount (+0.083 for older and +0.088 for newer papers). The pretraining data of the judges are not public, so no one can verify whether a paper was seen during training, and methods that test whether a text was in the training data perform close to random guessing on LLMs in most settings (Duan et al., 2024). Differences in topics or questions could also explain the gap, so we cannot attribute it to memorization. Like other benchmarks built on public scientific papers (Li et al., 2024; Wang et al., 2025a), MMSciFact limits this risk with newly created instances and mostly recent papers. We are adding this analysis and its limitations to the manuscript, along with the full list of source papers.

Further disclosure on annotation quality, and the scalability of the pipeline.

Disagreement resolution. At least two human experts independently labeled each QA pair. Of the 2,737 sentences, 301 (11.0%) received different initial grounding labels. All disagreements were resolved through discussion. No cases remained unresolved, and annotators did not use the interface's override function. Before adjudication, mean pairwise agreement was κ = 0.839. Agreement specifically for INVALID SUPPORT was κ = 0.72. We are adding these results to the annotation section.

Structural annotations. Two independent Gemini runs proposed sentence roles and dependencies. We automatically accepted the annotations when both runs agreed on the role and the complete dependency set (86%). Human annotators resolved the remaining 14%. When compared with human annotations, automatically accepted structures achieved κ = 0.770 for roles and F1 = 0.840 for dependencies. These results are close to human–human agreement (role κ = 0.792–0.810, dependency F1 = 0.818–0.878, Table 2). Human annotators also checked the dependency paths of all 148 INVALID SUPPORT sentences.

Scalability. Annotating grounding, sentence roles, dependencies, and evidence locations required about 165 person-hours from 12 ML/NLP researchers. We used LLMs to help with structural pre-annotation, but all grounding labels were assigned by humans. Expanding MMSciFact to other scientific fields would require experts in those fields, possibly using the same two-run pre-annotation approach. We are making this limitation clearer in the manuscript. The annotation guidelines and interface are also available to support future extensions.

References

  • Duan, M. et al. (2024). Do Membership Inference Attacks Work on Large Language Models? COLM 2024.

  • Li, C. et al. (2024). M3SciQA: A Multi-Modal Multi-Document Scientific QA Benchmark for Evaluating Foundation Models. Findings of EMNLP 2024.

  • Wang, C. et al. (2025). SciVer: Evaluating Foundation Models for Multimodal Scientific Claim Verification. ACL 2025.

Thank you for your response; I have no further questions at this stage. Regarding the dataset, have you considered making it open-source? This might influence how I adjust my rating later on.

Reviewer_LYpTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a new benchmark for evaluating the scientific claims agasints documents and its supporing evidences or premises. In addition to the previous benchmarks, which annotates claims as supported, contradiction, and not enough information, the proposed benchmark also includes a new label of invalid support. The idea of invalid supports is that, event if some claims if are supported by the original text, some of its extracted/oracle dependent claims are not supported. The newlly proposed benchmark wants to measure from this perspective.

Critical strengths
  1. The problem of multimodal fact-checking of long-form answers about scientific papers is important.

  2. The annotation process is carefully conducted with trained expert annotators, double grounding annotation with high agreements, and error codes and evidence boxes.

  3. The evaluation is thorough, convering many judges, three verification settings, and three different scopes.

Critical weaknesses
  1. However, I have some concerns about the necessity of the INVALID SUPPORT labels. Moreover, I also have the concerns about the dependency-aware verification for fact-checking itself. INVALID SUPPORT sentence is one that is supported by the source when verified on its own. INVALID SUPPORT therefore does not reveal any factual error that independent verification misses, so the central motivation, that break-then-verify "misses" a failure mode, is not well justified. What the label really captures is whether a conclusion was reached through valid reasoning, not a fact-checking one. That can be a legitimate target, but the paper should then frame itself accordingly and position itself against work on reasoning faithfulness and process-level evaluation.

  2. Given its definition, INVALID SUPPORT can be computed mechanically: take each judge's independent SUP/CON/NEI predictions and propagate them over the oracle (or a predicted) graph. Without this baseline, it is unclear whether judges fail at dependency reasoning or simply at grounding the premises.

  3. The benchmark is small, the key label is rare, and the oracle graphs are largely LLM-generated.

Questions for authors
  1. What is INVALID SUPPORT F1 if you take each judge's independent-setting predictions and deterministically apply the rule over the oracle graph and the LLM-predicted graph?

  2. Can you give concrete evidence that INVALID SUPPORT matters to users beyond flagging the faulty premise itself? Why is a dependency graph preferable to decontextualizing each claim so it can be verified independently?

4 comments
AuthorsOpenReview ↗

Thank you for your review!

Thank you for the thoughtful comments. We have completed the additional experiments and checks described below. We are also updating the manuscript to include these results and clarify the role of INVALID SUPPORT. We will upload the revised version separately.

I have some concerns about the necessity of the INVALID SUPPORT labels.

INVALID SUPPORT identifies conclusions that rely on incorrect premises, even when the conclusions themselves are correct. In Figure 1, independent verification finds the error in S2 but still labels S3 and S4 as SUPPORTED. It does not show that both conclusions depend on S2. INVALID SUPPORT makes this problem visible.

I also have the concerns about the dependency-aware verification for fact-checking itself.

Checking dependencies matters because a conclusion can be correct on its own but based on incorrect evidence. Previous studies have shown the importance of relationships between claims, including argument structure in fact verification (Si et al., 2024), discourse relations in factual consistency checking (Zhang et al., 2025), and relational errors that independent fact verification can miss (Liu et al., 2026). MMSciFact builds on this work by checking whether conclusions in scientific answers rely on premises supported by the source document. This adds to independent claim verification by checking not only individual statements but also how they are justified.

INVALID SUPPORT therefore does not reveal any factual error that independent verification misses ...

The reviewer is right that independent verification can find the incorrect premise. What it misses is that other conclusions, which may be SUPPORTED on their own, depend on that premise. INVALID SUPPORT captures the difference between a correct conclusion and a correct justification.

Given its definition, INVALID SUPPORT can be computed mechanically.

What is INVALID SUPPORT F1 if you take each judge's independent-setting predictions and deterministically apply the rule over the oracle graph and the LLM-predicted graph?

We agree. We applied the rule in Section 3.1 to each judge's independent SUP/CON/NEI predictions using the reference dependency graph. This gives us a baseline without errors from graph prediction.

INVALID SUPPORT F1, averaged across the three evidence scopes:

JudgeOracle-assistedDeterministic
Qwen3.5-9B0.1980.251
Qwen3.8-27B0.2990.304
Qwen3.6-35B-A3B0.2230.319
Qwen3.5-122B-A10B0.2580.303
Qwen3.5-397B-A17B0.2650.295
Gemma4-E4B0.1900.219
Gemma4-26B-A4B0.1950.274
Gemma4-31B0.1920.297
GLM-4.6V-Flash0.1000.199
GLM-4.6V0.1460.133
GPT-5.6-Luna0.2690.285
Gemini-3.6-Flash0.2390.325
Mean0.2150.267

Deterministic propagation performs better than Oracle-assisted verification for 11 of 12 judges, increasing mean F1 from 0.215 to 0.267. This shows that asking models to use dependencies directly does not always work better than applying a fixed rule to independent grounding predictions.

The two approaches are different. Deterministic propagation first checks each sentence independently and then applies a fixed rule. Oracle-assisted verification instead asks the model to check both the sentences and how they support one another in a single pass. Our goal is to move toward models that can do both on their own. MMSciFact is a first step in understanding how well current models can interpret and use dependency graphs to check not only whether statements are correct, but also whether their conclusions are properly justified.

The benchmark is small

MMSciFact contains 2,737 annotated sentences. This is similar in size to other benchmarks such as SciFact (1,409 claims), SciClaimEval (1,664 samples), and SciVer (3,000 examples), when comparing the number of items to verify. MMSciFact also includes sentence roles, dependencies, grounding labels, and visual evidence regions. Collecting these annotations and training the annotators took about 165 person-hours.

The key label is rare

We agree that INVALID SUPPORT is relatively rare at the sentence level, with 148 instances (5.4%). However, it appears in 23% of the answers. This means that nearly one in four answers contains a conclusion based on an incorrect premise.

These errors occurred naturally in model-generated answers. We did not add or change errors to make INVALID SUPPORT more common. We consider this a strength of the benchmark because it preserves a less frequent but real type of error that can affect the justification of otherwise correct conclusions.

The oracle graphs are largely LLM-generated

The dependency graphs were checked by humans and are not simply unchecked LLM outputs. Two independent Gemini runs suggested sentence roles and dependencies. We automatically accepted their annotations when they matched exactly (86%). Human annotators resolved the remaining cases.

We also compared the automatically accepted annotations with human annotations. Agreement was κ = 0.770 for roles and F1 = 0.840 for dependencies, close to human–human agreement (Table 2). Experts also checked the dependency paths of all 148 INVALID SUPPORT cases.

We ran an additional blind human audit of 50 automatically accepted SUPPORTED interpretations. The annotators sometimes chose different dependencies, but none of these differences changed the final grounding labels. All 50 interpretations remained SUPPORTED. This gives us further evidence that differences in dependency edges do not always lead to errors in the final labels.

Can you give concrete evidence that INVALID SUPPORT matters to users beyond flagging the faulty premise itself? Why is a dependency graph preferable to decontextualizing each claim so it can be verified independently?

INVALID SUPPORT gives useful information about the errors in an answer. In Figure 1, finding the error in S2 does not tell the reader that S3 and S4 also depend on it. Dependency graphs make these links clear and help readers see which conclusions need further checking. This also follows fact-checking practices that consider the assumptions behind claims (Full Fact).

Decontextualization helps verify individual claims, but it can lose the links between premises and conclusions. Adding the full context to each claim could keep these links, but would repeat the same information across claims, especially in longer chains. The two approaches can work together. Decontextualization helps check individual claims, while dependency graphs help check how those claims support one another. Previous work has also discussed the trade-offs of decontextualization (Wanner et al., 2025).

References

  • Si, J. et al. (2024). CHECKWHY: Causal Fact Verification via Argument Structure. ACL 2024.

  • Zhang, K. et al. (2025). Structured Discourse Representation for Factual Consistency Verification. Findings of ACL 2025.

  • Wanner, M. et al. (2025). DnDScore: Decontextualization and Decomposition for Factuality Verification in Long-Form Text Generation. EMNLP 2025.

  • Liu, Z. et al. (2026). BeyondAtoms: Benchmarking and Verifying Relational Factuality Beyond Atomic Facts. EMNLP 2026.

  • Full Fact. Frequently asked questions. https://fullfact.org/about/frequently-asked-questions/

Thank you for the additional experiments and clarifications. The deterministic baseline and the annotation audits address some of my concerns. However, I remain unconvinced that the rebuttal establishes the central motivation for INVALID SUPPORT as a distinct fact-checking target.

Thank you for your follow-up. We respectfully disagree that INVALID SUPPORT does not capture a distinct fact-checking target. It evaluates whether a conclusion is justified by the premises used to support it, which means verifying not only whether a claim is grounded in the source, but also whether the reasoning presented to justify it rests on valid premises. As also recognized in prior work, both aspects are essential for reliable fact-checking.

Moreover, dependency-aware fact-checking is not the same as independently verifying sentences, identifying their dependencies, and then applying an algorithm to propagate errors. The former evaluates whether MLLMs can perform this verification directly in a single end-to-end step, while the latter decomposes the task into three separate steps. The success of this alternative approach does not undermine the motivation for studying whether models can perform such verification directly, as human fact-checkers do.

As there is still time for discussion, we would appreciate it if you could clarify which specific aspects of this motivation remain unconvincing.

Comments on the paper

5 comments
AuthorsOpenReview ↗

Thank you for your review!

Thank you for the careful review. We have completed the additional experiments and quality checks described below. We are also updating the manuscript to include these results and clarify the main label and evaluation settings. We will upload the revised version separately.

I think a simple rule-based propagation baseline is missing. Once the sentence roles, dependency graph, and local grounding labels are available, Invalid Support follows directly from the rule in Section 3.1.

We agree. We applied the rule in Section 3.1 to each judge's independent SUP/CON/NEI predictions using the reference dependency graph. This gives us a baseline without errors from graph prediction.

INVALID SUPPORT F1, averaged across the three evidence scopes:

JudgeOracle-assistedDeterministic
Qwen3.5-9B0.1980.251
Qwen3.8-27B0.2990.304
Qwen3.6-35B-A3B0.2230.319
Qwen3.5-122B-A10B0.2580.303
Qwen3.5-397B-A17B0.2650.295
Gemma4-E4B0.1900.219
Gemma4-26B-A4B0.1950.274
Gemma4-31B0.1920.297
GLM-4.6V-Flash0.1000.199
GLM-4.6V0.1460.133
GPT-5.6-Luna0.2690.285
Gemini-3.6-Flash0.2390.325
Mean0.2150.267

Deterministic propagation performs better than Oracle-assisted verification for 11 of 12 judges, increasing mean F1 from 0.215 to 0.267. This shows that asking models to use dependencies directly does not always work better than applying a fixed rule to independent grounding predictions.

The two approaches are different. Deterministic propagation first checks each sentence independently and then applies a fixed rule. Oracle-assisted verification instead asks the model to check both the sentences and how they support one another in a single pass. Our goal is to move toward models that can do both on their own. MMSciFact is a first step in understanding how well current models can interpret and use dependency graphs to check not only whether statements are correct, but also whether their conclusions are properly justified.

The “oracle-graph” condition is not consistent across the reported models.

The reviewer is right. The role_only prompt provides reference sentence roles but no dependency edges. It was selected for GPT-5.6-Luna because it achieved a validation macro-F1 of 0.612, compared with 0.593 for the full-graph baseline prompt. We used the same selection rule for all judges.

GPT-5.6-Luna therefore had to infer the dependencies itself, unlike the other 11 judges, which received the complete reference graph. We are updating the tables and discussion to make this distinction clear.

I would like the authors to explain the mapping and confirm that all 148 Invalid Support sentences satisfy the local-Supported requirement.

Figure 14 shows intermediate error codes rather than final grounding labels. C4 marks a conclusion based on a CONTRADICTED premise, while N4 marks one based on an NEI premise. An Interpretation receives INVALID SUPPORT only when it has no independent grounding error (C1–C3 or N1–N3). Otherwise, it keeps its CONTRADICTED or NEI label.

We confirm that human annotators checked all 148 INVALID SUPPORT sentences and verified that each was locally SUPPORTED when evaluated on its own against the source. This is why independent verification also treats them as SUPPORTED.

I have a related concern about using whole sentences as dependency nodes. A premise sentence can contain several claims, but a conclusion may use only one of them.

Annotators were trained to link conclusions only to claims actually used to justify them, rather than to sentences that were simply nearby or related. They rarely found premise sentences containing multiple unrelated claims. When a sentence contained several claims, these usually concerned related findings. Annotators checked that the conclusion depended on the incorrect claim rather than an unrelated correct one. They also checked the dependency paths of all INVALID SUPPORT cases.

Sentence-level graphs cannot show exactly which individual claims within a sentence are used. We are adding this limitation to the manuscript.

The structural annotations need a more targeted quality check.

We agree that checking existing INVALID SUPPORT paths may miss dependencies that were not annotated in sentences currently labeled SUPPORTED. We therefore carried out a blind audit of 50 automatically accepted SUPPORTED interpretations, including cases from answers containing CONTRADICTED or NEI statements. Two experts independently annotated the dependencies without seeing the original graphs or final labels.

The experts sometimes chose different dependencies, showing that exact agreement on dependency graphs is not always easy. However, these differences did not change any final grounding labels. All 50 interpretations remained SUPPORTED. This suggests that missing dependencies did not lead to missed INVALID SUPPORT cases in the audited sample, although it does not guarantee that every graph is correct.

We also measured agreement specifically for INVALID SUPPORT before adjudication (κ = 0.72), compared with overall grounding agreement (κ = 0.839). We are adding the audit results and agreement scores to the annotation section.

The filtering is appropriate for a diagnostic benchmark, but its effect should stay visible.

Our preliminary tests on fully SUPPORTED answers helped us decide which answers to keep. Judges correctly recognized answers without errors in about 98% of cases. In contrast, they had much more difficulty detecting errors when errors were present. We therefore kept answers containing at least one grounding error to focus on these harder cases.

We did not remove SUPPORTED sentences from these answers. The benchmark still contains 1,445 SUPPORTED sentences (52.8%), so we can measure how often judges incorrectly flag correct sentences. Our preliminary tests also give an initial measure of false alarms on fully correct answers. We are adding this result and clarifying that MMSciFact is designed to study grounding errors, not to estimate how often they occur in real-world use.

Dear Reviewers and Area Chair,

We have uploaded a revised version of the manuscript incorporating the clarifications and additional analyses discussed in our rebuttal.

The revision includes a deterministic propagation baseline, further analyses of INVALID SUPPORT errors and their underlying causes, additional annotation-quality checks, and clarifications of the experimental protocol and limitations.

All changes are highlighted in blue for ease of review.

We thank the reviewers for their constructive feedback, which has helped us strengthen the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.