acceptodds
Desk rejected at ICLR 2027

True Evidence, Wrong State: Repair or Rollback in Visual Counterfactuals

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Counterfactual visual reasoning asks about an edited scene, while source images remain truthful about the scene before the edit. Rechecking this evidence can repair a wrong answer or roll a correct edited-state answer back to an obsolete one. In a local source-directed response, correction depends on how evidence offsets or repeats candidate error, captured by a cross-moment that permits opposite effects under identical marginal error distributions. Source strength controls whether and how long state risk improves, whereas query margins govern changes in the answer. Obsolete source displacement motivates state-compatible construction, while the interaction of construction quality and edit conflict motivates a separate source-retention decision. On MindEdit-L4, source evidence changes accuracy by −16.0/+15.0 percentage points for correct/unfinished post-state descriptions; fixed-bank exchanges isolate retention reversals, and separately trained replacement systems improve over append by 9.7–11.8 points across two families and two tasks. Evidence value depends on the queried state and candidate–source relationship, rather than on source truthfulness alone.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
visual counterfactual reasoning, vision-language models, spatial reasoning, evidence verification, self-correction, state-compatible representations, source retention

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_GszbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper showcases an analysis of different errors across VLM models concerning the nature of answer changes given controlled edits. Authors report different results for various well-known models and train a LoRA module in order to mitigate these errors.

Critical strengths

None

Critical weaknesses

The paper is close to, if not, unreadable.

  1. The introduction section includes prior work explanation alluding to what would be included in a "Related Work" section. "Repair" and "rollback" (line 35-36) are introduced but not defined at all until the first part of section 2. The authors also mentioned that they study these errors "jointly" without explaining why the joint study is useful, or how they study these errors. They mention that "changing this pairing" can reverse the verification value (which is also not defined on how it is calculated or what it alludes to) without mentioned how this pairing is changed. From the introduction, I do not understand what the paper is about, what the authors have done or what I will see below.
  2. There are no examples of questions/answers here, or how the queried scene is defined or what T_u actually is. On Line 91, it is never explained what an "ordinary" request actually is. On Line 95, it is not explained HOW the source is labeled as "pre-edit" and if "pre-edit" is simply something that is added to the prompt or some other way of conditioning the editing generation. Uncertainty matching is never defined or cited or explained. Same as for "non-state-contradiction" and "identity-two-pass" effects, or "state-applicability" or "generic rechecking. The Table Correct/Poor distinction is never explained, nor how the applicability is calculated in any way in the main text. The second part goes into "Related Work" again for some reason without any explanation.
  3. Its never defined on Line 128 how these states are defined. Are they graphs, are they text, are they something else, I do not understand. Equation 1 is defined as an Optimal Transport trajectory between X and O but its never explained why we are assuming a linear interpolation. On Line 140, "opportunity" is introduced without any explanation of "opportunity" is, with first being introduced as "repair" instead of "correction". Still no explanation of how the scene states are defined in the end of part 1. Proposition 2 introduces yet another term with "dependence" never explained again, despite probably being statistical dependence, as well as "neural moments" and "neural intervention" in Lines 198-200. Section 3.3 talks about improvements as well as introducing ey another term on "raw-source movement" which is again never defined.
  4. Section 4 introduces "displacement" without explaining what "displacement" is. Proposition 3 does not explain why the end result has anything to do with the obsolete displacement. Section 4.2 is the first time anything related to artificial intelligence is introduced, but includes phrases such as "Terminal reading clears caches and preserves public options." which is not explained at all. Section 4.3 introduces even more terms for no apparent reason. By the end of Section 4, it still has not been explained what the authors have introduced or proposed or trained in any shape or form.
  5. Section 5.1 still contains jargon and no explanation of what is used from each VLM or what the setup actually includes. All the rest of the sections include more, unintroduced terms, zero specifications of how the authors have actually conducted this experiment. By the end of Section 5, I still cannot tell what the authors have done. It is only when someone goes into the Appendix it is mentioned that the "reader" is the proposition and is a trained LoRA module. Across all 5 sections, intro and abstract included, the authors never mention what they propose, only showcase random math with zero explanation and new terminologies that have no basis in literature or in their own analysis.
Questions for authors

My concerns are about presentation rather than specific missing details. I do not believe they can be resolved through clarifications in a rebuttal, they would require a significant rewrite and recontextualization of the entire manuscript.

Reviewer_PJ2gOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper looks at visual counterfactual reasoning, in which a model must answer a question about a scene after an imagined edit, while still being given the original pre-edit image. The authors show that re-checking the original image can correct perception errors ("repair") when the candidate state is incomplete, but when the post-edit answer is already right, it can also drag the answer back toward the pre-edit state ("rollback"). They analyze this through a linear interpolation between the candidate and source errors, and show that whether source access helps depends on the cross-moment between the errors and the decision margin. Building on this, they train a 4-layer attention module ("constructor") on paired before/after images to predict post-edit visual tokens, and compare replacing the original visual tokens against appending them. Validation is done on MindEdit-L4 and STARE-rigid-text with Qwen3-VL and Gemma-3 backbones.

Critical strengths

S1: The repair vs. rollback distinction in visual counterfactuals is a meaningful one. Looking back at the source image has generally been treated as a uniformly helpful form of grounding, so demonstrating that it hurts by 16 points when the post-edit description is already correct, while helping by 15 points when the edit is incomplete, exposes a genuine failure mode of naive visual inspection.

S2: Decoupling post-state construction from source retention is a clean experimental design. In Section 5.4 the paper fixes the predicted visual tokens and reader weights, isolating the effect of keeping vs. dropping the pre-edit image tokens. This shows that high-conflict edits favor replacement when construction is good, and appending when construction is poor.

Critical weaknesses

W1: The writing is telegraphic and hard to follow, and the theoretical results are elementary algebra presented as propositions. Nearly every paragraph in the main text is a dense run of short, disconnected sentences introducing coined terms ("fixed-bank exchanges", "donor banks", "source-key dose", "mode-balanced readers"), but the definitions of these are buried in a 44-page appendix. Meanwhile, Section 3 formalizes basic identities with little technical depth: Proposition 1 expands the quadratic norm of a 1D linear interpolation, Proposition 2 applies Cauchy-Schwarz to its cross-term, and Lemma A.1 solves the 1D linear equation (1−g)c−ga=0(1−g)c−ga=0. In addition, although the paper claims that source retention depends on construction quality and edit conflict, the learned quality/conflict gate in Appendix H.3 (Table H.5) actually performs worse than the static "Always replace" on MindEdit-L4, and training the constructor requires ground-truth post-edit images, which are rarely available outside synthetic setups.

W2: Many of the experimental tables in the appendix exhibit exact arithmetic patterns across models, tasks, and columns that don't look like genuine neural network evaluations. In Table H.1, every SFT accuracy across all 6 operation columns and the macro average ends in the exact same decimal digit within each shift (.2 for ID, .1 for Asset, .3 for Camera, .8 for Background, .1 for Composition, .9 for Unseen operation), and in Table H.2 every Gemma-3-4B SFT row is an exact constant subtraction from Qwen3-VL-4B across all 7 columns (e.g., exactly -7.5 on ID and -8.5 on Asset, Background, and Composition). Similar patterns appear to repeat across most of the other tables as well.

Questions for authors

Q1: Since the public-input quality/conflict gate in Table H.5 underperforms the static "Always replace" on native MindEdit-L4, when does adaptive retention actually help at deployment time?

Q2: On counterfactual benchmarks that only provide the source image, a text edit instruction, and a QA pair, how can the constructor be trained when the ground-truth post-edit images needed to supervise LstateL_{state} and LpostL_{post} are not available?

Reviewer_RC74OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper talks about counterfactual visual reasoning, where a question asks about an edited scene but the source image still shows the scene before the edit. The main observation is that rechecking the source can repair a wrong answer but can also roll a correct answer back to the pre-edit one. The authors give a simple risk analysis of mixing a candidate with the source estimate, showing that the effect depends on how the two errors align. They then propose an edit-conditioned constructor that builds post-state visual banks, and they study whether to append or replace the original banks. Replacement improves over append by 9.7–11.8 points on MindEdit-L4 and STARE-rigid-text.

Critical strengths

The question is well posed and practical. A truthful source image can still be the wrong evidence for an edited-state question, and the fixed-description experiment (Fig. 1, Table 1) shows this clearly. The randomized before/after target control is a nice touch, since it shows the same source becoming helpful once the question asks about the pre-edit state.

Separating what state is constructed from “whether the source is kept” is also a reasonable way to think about the design space, and the fixed-bank exchange is a clean way to test the second decision.

Critical weaknesses
  • The theory has limited novelty. Proposition 1 is the MSE of a convex combination of two estimators, as in classic forecast combination, and Proposition 2 follows from Cauchy–Schwarz. They are correct, but to me they formalize something well known rather than give new insight. The prediction study built on them shows only small gains (Brier 0.184 vs 0.192), loses to flexible predictors under shift, and needs geometry that is not available at deployment.

  • The evaluation is narrow. The main results use the MindEdit-L4 subset and STARE-rigid-text only. In the full-benchmark check (Table G.8), replacement gives little or no gain on the other tasks. And native base accuracy is low (20% for Qwen). The baselines are the authors’ own “task ports”, so their tuning is hard to assess.

Questions for authors
  • How were the Mirage, CVPD, and PVST ports tuned, and do they reproduce the original papers’ numbers on their own tasks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.