- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduces \textbf{CARVE}, a solver-certified counterfactual benchmark whose edits change one piece of visual evidence (a printed value, or the element that value refers to), certify the new answer symbolically, and record the image region carrying the change. By using programmatic visual edit approach, authors propose \textbf{Causal Evidence Following (CEF)}, \text{BEF} and other metrics, carries experiments based on that, this paper claims that RL post-training raises \emph{behavioral} evidence following on three of four backbones but (i) leaves a persistent \emph{prior lock} on referent edits (10.9--39.5% of such edits keep the original answer vs.\ on numeral edits in all 16 configurations), (ii) does not reliably route the gain through the edited region ($\sim$43% gain share at 3B; wide interval at 7B), with the frozen-tower pattern implicating the \emph{decoder} rather than the visual encoder, and (iii) decouples attention-to-image from causal use, cautioning against reading grounding from accuracy or attention alone.
Critical strengths
- Great cerfification pipeline. Answers are solved from scene parameters before rendering, so they never depend on pixels; the mask is a verified pixel difference with nothing changed outside it. Crucially, the filters audit the post-edit answer distribution, not just the question text, and model correctness is deliberately excluded as a filter. This is a measurement instrument, not another paired-image suite.
- "Prior lock" is a genuinely new diagnostic that survives the obvious objections. Models follow a changed numeral but keep the old answer when the same value moves to another element. The gap persists within scenes that carry both edit types. Failed referent edits return the original answer rather than a different printed value. Readout probes show that most locked edits are read and located correctly.
Critical weaknesses
- Limitation in Causal Evidence Following experiments. The patched sequence mixes two global summaries of the image, so it corresponds to no image that could be rendered, and the transferred region carries only a quarter of the edit's effect. A post-training drop in CEF can therefore mean either that the model uses the edit less or that it resists an internally inconsistent patch, which the paper's controls cannot tell apart. The complement region and a consistency-preserving donor would.
- The gain share is a ratio of two differences of a few points each, and it is not identified at that precision. At 7B the estimate exceeds 100% ([66, 139]), which cannot be read as a share, and the 54 point gap over 3B ([16, 101]) rests on two sizes of one family with one seed. At 3B the frozen-tower value (54.1%) sits above the trainable one (42.7%), so the ordering a localization story would need is not stable. The four underlying accuracies, per objective, would be more informative than the ratio, and the 7B PRPO run, which collapses on Geo3K (40.6 to 24.1, one fixed answer on 11% of items), should be justified before it enters a pooled headline.
Questions for authors
- For the complement region: what is transfer accuracy at layer 0 when you copy instead of , per configuration? If the complement alone recovers most of edited-image accuracy, then the gap between seeing the edit and transferring it reflects how much of the image the patch covers rather than where the evidence sits. Either outcome would change how I read Section 6.3, so I would like the number even if it is unflattering.
- For the consistency problem: can you rerun CEF with a donor that keeps the figure self-consistent, for instance a MODE-L item where the numeral glyph is rewritten but the geometry it annotates is left unchanged? The current donor differs from the recipient both at the edited tokens and in its global summary, so a smaller CEF after training admits two readings. If the base-to-trained gap survives a donor that is closer to renderable, the weaker reading is ruled out and my main objection goes with it.
- For the gain share: could you report the four accuracies behind it (edited-image and transfer, base and trained, per objective) rather than the pooled ratio, and state whether the 7B PRPO run is included? With a ratio of two differences of a few points, the per-objective accuracies are what I can actually check against the claim, and I would rather see them than the quotient.
2 comments
Part 2/2
W2/Q3. What are the four accuracies behind the gain share, and is 7B PRPO included?
Table C gives the four accuracies behind the gain share for every objective, on the same 1,545 numeral pairs. The 7B PRPO run is included in the paper's pooled 7B gain share and is shown here as its own row.
Table C. The four accuracies per objective, on the paper's 1,545 numeral pairs, layer 0. Accuracy = % of pairs on which the run gives the edited answer y'. Edited image: the model sees the edited figure. Transfer: only the edited region is patched into the original figure (the paper's CEF). The frozen-tower runs start from the same base model.
| Objective | 3B trainable tower: edited image | 3B trainable tower: transfer | 3B frozen tower: edited image | 3B frozen tower: transfer | 7B: edited image | 7B: transfer |
|---|---|---|---|---|---|---|
| Base | 52.4 | 40.7 | 52.4 | 40.7 | 62.7 | 46.0 |
| GRPO | 61.4 | 45.0 | 61.7 | 46.5 | 69.1 | 52.4 |
| VPPO | 61.4 | 43.9 | 56.4 | 41.4 | 67.5 | 50.9 |
| PRPO | 61.0 | 44.6 | 61.8 | 46.5 | 71.7 | 54.1 |
7B PRPO. The run does not collapse on CARVE: it has the highest edited-image accuracy of the 7B runs (71.7). The Geo3K drop the reviewer notes is real, and we will report it next to the CARVE numbers.
W2. Frozen tower. The paper does not claim an ordering between frozen and trainable towers; it claims that freezing the tower leaves the same gap between seeing the edit and transferring it (Figure 4: −25% to −27% frozen, −27% to −28% trainable). The pooled shares agree within error (54.1% vs 42.7%, difference +11.4, 95% CI [−5.8, +30.5]). With a frozen tower the edited region's states stay exactly the base model's, so the gap cannot come from the encoder, which is why we locate the change in the decoder.
We will add these controls and the four-accuracy table to Section 6.3, together with a notation table, one running example, and boxed definitions of BEF and CEF. We thank the reviewer again; the complement and consistency questions led directly to a sharper statement of the CEF result.
Part 1/2
We thank the reviewer for the careful review and for recognizing CARVE as “a measurement instrument, not another paired-image suite” and prior lock as “a genuinely new diagnostic that survives the obvious objections.” We ran the three experiments the reviewer asked for; each supports the paper's reading of Section 6.3, which we will sharpen.
W1/Q1. What is transfer accuracy when the complement is patched?
Transfer accuracy when the complement is patched is low in every configuration: copying the complement at layer 0 gives the edited answer on only 12–15% of pairs, against 52–69% for the edited image, although the complement holds 91% of the visual tokens and 71–74% of the edit's representation change. The evidence the model uses sits at and around the edit, not across the image, so the gap reflects where the evidence sits, not how much the patch covers.
Table A. Accuracy (% of the paper's 1,545 numeral pairs on which the run gives the edited answer y'), layer 0, all 11 configurations (published CEF values reproduce exactly). Representation change: edited minus original, squared, summed over visual tokens; range over 3B and 7B. RL = mean of GRPO, VPPO, PRPO.
W1/Q2. Does the base-to-trained gap survive a self-consistent donor?
Yes. The smaller CEF after training is the 3B pattern in Figure 4: transferring only the edited region loses 27–28% of edited-image accuracy after RL, against 22% for the base model. In 9 of our 26 numeral templates (tables, Venn counts, path weights, number puzzles, sector, number train) nothing else in the figure moves when the numeral changes (pixel difference under 0.3% of the image), so the donor is already the self-consistent figure the reviewer describes: the original with only the numeral glyph rewritten. Rerunning CEF on these 538 pairs reproduces the same pattern: the region loses 24% of the base model's accuracy and 31% of the RL models' with a trainable tower (28% with a frozen tower). The larger drop after training therefore survives a donor with nothing inconsistent in it, which rules out the inconsistency reading. At 7B there is no larger drop to explain, in Figure 4 or here.
Table B. Glyph-only numeral pairs (538), layer 0. Drop = (edited region − edited image) / edited image. Last column: the same drop with the published donor (Figure 4).