acceptodds
Withdrawn from ICLR 2027

From Structured Facts to Logical Conflicts Visual-Text Conflict

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large multimodal models are typically evaluated on image-text inputs that are semantically consistent, leaving their ability to handle visual-textual contradictions underexplored. We introduce LogicCon, a benchmark for visual-textual logical conflict reasoning that requires models to detect conflicts, localize the conflicting fact pair, and diagnose the conflict type. LogicCon constructs controllable contradiction samples by converting visually grounded question-answer pairs into consistent declarative statements, applying structured fact mutations, and verifying each mutation against symbolic visual evidence. We evaluate 13 representative open-source and closed-source multimodal models under a unified zero-shot protocol. Results show that current models can often detect whether a conflict exists, but perform substantially worse when required to recover the complete conflicting fact pair or reason over compositional samples. A large gap between the strongest models and human performance further indicates that visual-textual logical conflict reasoning remains a challenging and unresolved capability. Comparative experiments further reveal that hallucination mitigation methods mainly improve basic conflict detection but provide limited gains in recovering complete conflicting fact pairs, while general reasoning methods still struggle with precise visual evidence grounding. By explicitly parsing target-aware claims, verifying visual evidence, and aggregating conflicting facts, our method achieves more consistent improvements on higher-level logical conflict reasoning tasks. Code and benchmark artifacts are available at https://anonymous.4open.science/r/LogicCon-ICLR2027/.

Primary area:
datasets and benchmarks
Keywords:
logic conflict, Visual-Text Conflict

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_zc8eOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LogicCon, a multimodal logical conflict reasoning benchmark that requires models to (1) detect conflicts, (2) localize the conflicting fact pair, and (3) diagnose the conflict type. This benchmark covers five types of conflicts, including attribute, object, relation, spatial, and quantity conflicts. The main contribution is a verifiable data generation pipeline and a plug-in module to enhance the conflict reasoning ability of multimodal models.

Critical strengths
  • The authors released their code with detailed instructions.

  • This paper includes the latest models into their benchmark, e.g., Qwen3.5 in 9B and 27B.

  • The representation is clear to solve an essential task in multimodal models.

Critical weaknesses
  • Lack of novelty: Compared with previous work (MMIR, CLASH and so on), this benchmark differs mainly on extending the conflict localization. It’s helpful to highlight the core contribution in a few words.

  • The four-module reasoning framework is a cascaded band-aid, with limited theoretical support.

  • Formatting-1: Fig.1 and Fig.2 have very narrow space to the main text. It’s hard to distinguish the figure caption from main text.

  • Formatting-2: Line 111, the citation for “cross” (original name “CLASH”) is incomplete, should be CVPRF’26.

Questions for authors
  • Will CoT models gain benefits in conflict detection? E.g. compare LLaVA with LLaVA-CoT.

  • Will your plug-in framework introduce substantial latency in test time?

Reviewer_h7ZZOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces LogicCon, a benchmark for detecting and explaining conflicts between images and text. It contains 20,000 image-text pairs built from GQA through controlled changes to facts and scene-graph checks. It covers five conflict types and evaluates conflict detection, conflicting details, visual evidence, and conflict types. The paper also proposes a framework that breaks statements into claims, checks each claim against the image, and combines the results. Experiments include model comparisons, component studies, a human audit of 500 samples, and transfer tests on MMIR.

Critical strengths
  1. The benchmark provides more detail than a simple conflict label. It records the target object, the conflicting text, and the relevant visual fact. These annotations can help identify where a model fails. The dataset includes a useful human quality check. Three annotators reviewed 500 samples across conflict types and difficulty levels. The paper reports agreement, validity rates, and common errors.
Critical weaknesses
  1. Some key results are inconsistent, and the scoring rules are not fully defined. In Table 1, Qwen3.5-27B has a conflict detection score of 69.1%. In Table 3, the same model has a base-model score of 86.6%. The paper does not explain this difference. Weighting the scores in Table 9 by the published MMIR category sizes gives an overall open-ended accuracy of about 34.89% for the proposed method with Qwen2.5-VL-72B. However, the paper reports 34.00%, and the reason for this difference is unclear. The paper also reports an average score without stating which metrics it includes or how they are weighted. The released evaluation documentation states that this score is left as null. It is therefore unclear how the Avg scores in the paper were calculated.

  2. The comparisons do not clearly show the value of the proposed framework. Table 2 applies LLaVA-CoT to several backbones, including GPT-4o-mini. However, the original LLaVA-CoT uses a trained model and a specific search procedure. The paper does not explain which parts were used or how they were adapted. This makes the comparison difficult to assess. LLaVA-CoT. The ablations also add reasoning stages one at a time. They do not test whether simpler repeated checking would produce similar gains. As a result, it remains unclear how much the gains come from the proposed design.

Questions for authors
  1. Why does Qwen3.5-27B score 69.1% on Conf in Table 1 but 86.6% in Table 3? Were different evaluation settings used?

  2. Table 9 reports 34.00% for Qwen2.5-VL-72B on open-ended questions. Weighting the category scores by the published MMIR sample counts gives about 34.89%. What explains this difference?

  3. How is Avg calculated? Which metrics and weights are used? The evaluation documentation leaves Avg as null, so how were the table values obtained?

  4. How was LLaVA-CoT adapted to GPT-4o-mini and the other backbones? Were its training and search steps used, or was only its prompting format used?

Reviewer_5T8POpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LogicCon evaluates visual-textual conflict reasoning using 20,000 image-text pairs derived from GQA. Its construction pipeline aligns question-answer pairs with scene graphs and functional programs, generates constrained factual mutations, and retains structured records of the modified facts. Evaluation covers conflict detection, target-object identification, textual conflicting details, visual evidence, and conflict types, with a compositional subset requiring more involved target resolution. Conflict Point accuracy (CP) requires the target, textual detail, and visual fact to be jointly correct. A prompt-based framework decomposes claims, verifies evidence, and aggregates conflicts using the same backbone. Experiments include 12 models, method comparisons, component ablations, a 500-sample human quality audit, and transfer to MMIR.

Critical strengths
  1. The benchmark makes conflict diagnosis more informative than binary detection. Separately evaluating the target, textual claim, visual fact, and their joint recovery provides a useful way to distinguish recognizing an inconsistency from explaining it correctly (§3 and Appendix G). Retaining grounded frames and mutation records also makes the intended conflict traceable (§4). This is a concrete resource contribution for studying multimodal reliability.

  2. The construction pipeline has meaningful human quality validation. Appendix D audits 500 conflicting examples across five types and two complexity levels, with three annotators assessing seven validity criteria. The reported 473/500 fully valid samples, confidence intervals, and residual-error breakdown provide useful evidence that most generated examples are valid. Reporting lower validity for compositional cases also exposes an important limitation. This supports the dataset's potential utility and gives subsequent users a basis for targeted quality checks.

Critical weaknesses
  1. The central quantitative comparisons cannot be reconstructed from a consistent reported setup. Qwen3.5-27B has All/Conf/CP scores of 73.8/69.1/41.2 in Table 1, but its Base MLLM row in Table 3 gives 78.4/86.6/45.2. With the full framework's CP of 55.5, the improvement is therefore either 14.3 or 10.3 percentage points, without an explanation of changed configurations or evaluation sets. Avg. is also not defined in Appendix G. The MMIR aggregation raises a related concern. For Qwen2.5-VL-72B, open-ended domain gains are 7.27, 7.90, and 8.12 points, while Overall improves by only 6.76 (Table 9). This is incompatible with a fixed nonnegative weighted average if the same samples and scoring rule are used. An alternative aggregation or sample-selection rule could explain it, but must be specified. These unresolved reporting differences prevent a reliable interpretation of the main improvement claims.

  2. The new diagnostic metrics lack validation on the actual response-scoring task. GPT-4o judges whether model responses match the reference target, textual claim, and visual fact, with CP requiring all three judgments to be correct (§6.1 and Appendices G/M). The paper does not report agreement with independent human grading of actual model outputs. Appendix D validates examples and annotations, and explicitly distinguishes that audit from human task performance; it therefore does not calibrate the response judge. This matters because the proposed framework explicitly generates the fields the judge expects, whereas other methods can express equivalent information differently. Without calibration, the contribution of response format or semantic-adjudication error to method differences remains unknown. Table 1's human baseline is also insufficiently specified beyond five experts, particularly its sample coverage and aggregation. These omissions do not demonstrate judge bias or eliminate the observed human-model gap, but leave the main diagnostic comparisons without an established measurement basis.

Questions for authors
  1. Which evaluation configurations, sample sets, and aggregation rules explain the Table 1/Table 3 baseline differences and Table 9 Overall values, and can these results be regenerated from consistent prediction records?

  2. What validation against independent human grading of actual model responses supports TO/TC/VE/CP and the method comparisons, and what sampling and scoring protocol produced the human baseline?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.