- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces RegionTrace, a toolbox for measuring how image regions shape an answer throughout multimodal generation, and demonstrates its use through three empirical studies. Study I contrasts questions about target, elsewhere, and absent colors, using regional perturbations to identify visual evidence supporting incorrect attribute answers. Study II varies regional visibility while holding recorded reasoning prefixes and the eventual answer fixed at selected checkpoints to trace changes in visual support. Study III compares explicit Look-Back spans with equal-length ordinary reasoning continuations to assess changes in regional support and answer accuracy. The results show stronger contributions from distractor regions than from queried regions to color errors, declining incremental visual support during explanations, and no consistent Look-Back gains in regional support or accuracy.
Critical strengths
-
Clear and systematic study design. The three studies connect spatial attribution of attribute errors with changes in conditional visual support during explanation generation and explicit visual revisits, demonstrating the toolbox’s diagnostic value.
-
Broad quantitative evaluation. The experiments cover multiple models and four benchmarks, complemented by alternative masking, scoring-scale, and matched-continuation analyses that strengthen the empirical assessment.
Critical weaknesses
The proposed toolbox offers limited methodological novelty over existing attribution and reasoning-analysis methods. EAGLE already performs perturbation-based visual-region attribution for selected generated tokens conditioned on preceding text. Similar reasoning-analysis techniques have also been used in prior work, such as PDCR [1]. PDCR defines a Visual Dependence Score as the log-likelihood ratio of a reasoning step under real versus blank images, conditioned on the same preceding text. RegionTrace’s main distinction is tracking signed, context-dependent regional effects on the same candidate answer across generation checkpoints, with Shapley averaging as a summary. While useful, the methodological advance provided by this integration over existing fixed-prefix perturbation and attribution approaches remains insufficiently established.
[1] PDCR: Perception-Decomposed Confidence Reward for Vision-Language Reasoning, CVPR 2026.
Questions for authors
Could the authors more clearly articulate RegionTrace’s core incremental contribution over existing work and its implications for future research on visual grounding in VLMs?
How well does its fixed-target analysis generalize to practical VLM applications, particularly open-ended generation beyond the evaluated Yes/No and multiple-choice settings?
In Study I, how often does the elsewhere region have a positive signed contribution among cases where it outranks the queried region? Separating positive support from weaker suppression would clarify the interpretation that distractor regions support incorrect answers.
6 comments
[2/2]
Q2: Application to open-ended generation
In fact, we had already explored open-ended generation while developing RegionTrace. We examined which image regions influence a freely generated description and where their effects are strongest. We used 500 AMBER images and InternVL3-8B, which produced descriptions averaging 114 tokens without predefined answer options.
We first record each description, then hide or restore two selected image regions and rescore the recorded tokens. Each token and its preceding text stay unchanged during the comparison. To measure the strength of regional influence, we add the absolute effects of the two regions on each token. The table reports the average for the first and last quarters of each description, in log-probability units.
| First quarter | Last quarter | Change (95% CI) | Lower in last quarter |
|---|---|---|---|
| 1.4339 | 0.3781 | −1.0558 [−1.1201, −0.9909] | 470/500 (94.0%) |
We computed the confidence interval by resampling the 500 images. This measure was about 74% lower in the last quarter, with a decrease in 94% of descriptions. A plot across ten parts of each description shows a sharp early drop, followed by generally weaker effects. We have added this plot, with 95% confidence bands, to the revised manuscript (Figure 16, p. 40). The plot is also provided in the supplemental material, with the corresponding summary statistics on p. 3 (Table 24).
In these descriptions, the selected regions had a larger average effect on early tokens than on later ones. This illustrates a practical use of RegionTrace in open-ended generation: locating the parts of a response most affected by particular image regions.
Q3: Positive support from distractor regions
Thank you for pointing out this distinction. We checked the sign of the elsewhere region’s contribution among false-Yes cases where it outranks the queried region. Its contribution is positive in 97.9–99.6% of PTR cases and 91.0–94.9% of PACO cases across models, as detailed below.
| Dataset | Model | Positive elsewhere contribution | Non-positive elsewhere contribution |
|---|---|---|---|
| PTR | Qwen2.5-VL-7B | 265/266 (99.6%) | 1/266 (0.4%) |
| PTR | InternVL3-8B | 746/752 (99.2%) | 6/752 (0.8%) |
| PTR | InternVL3-14B | 428/437 (97.9%) | 9/437 (2.1%) |
| PACO | Qwen2.5-VL-7B | 75/79 (94.9%) | 4/79 (5.1%) |
| PACO | InternVL3-8B | 201/221 (91.0%) | 20/221 (9.0%) |
| PACO | InternVL3-14B | 126/136 (92.6%) | 10/136 (7.4%) |
The table shows that the elsewhere region usually supports the wrong answer, rather than simply suppressing it less than the queried region.
Hiding only the elsewhere region also switches the constrained Yes–No decision to No in 76.0–83.6% of PTR errors and 35.5–53.9% of PACO errors (Section 3.2, Table 2, p. 6). These rates cover all elsewhere-color false-Yes cases, not just those where elsewhere ranks higher. The sign breakdown is included in Appendix A.3, Table 5 (p. 20) of the revised manuscript and in the supplemental material, with the minority of zero or negative contributions explicitly identified.
Thank you for the additional analyses. My main concern remains that the paper does not establish a substantial methodological advance over existing perturbation and fixed-prefix analyses. As P8vJ points out, keeping the reasoning prefix fixed measures how visual evidence affects the answer given the recorded reasoning, leaving open how changes to the image would alter the reasoning itself. I therefore see RegionTrace as a useful diagnostic extension, but my overall rating remains unchanged.
[1/2]
W1 & Q1: Core contribution
Thank you for raising this point. RegionTrace connects regional attribution to a central question in visual reasoning: what does reasoning actually change about the visual evidence supporting an answer? A wrong answer can draw support from a real visual cue in the wrong region. As an explanation grows, the generated text can sustain the same answer while regional support weakens. Even an explicit second look does not reliably restore that support.
RegionTrace makes these questions accessible within one framework by following the same answer and image regions across reasoning stages, while retaining whether a region supports or opposes the answer in different visual contexts. Its core contribution is to connect where an answer’s visual support comes from with what happens to that support during reasoning. This gives researchers a way to examine whether explanations and visual revisits actually change the evidence behind a decision, alongside whether they improve accuracy. We compare RegionTrace with EAGLE and PDCR below.
Comparison with EAGLE
EAGLE combines sufficiency and necessity to rank image regions for selected generated tokens. A ranking alone, however, does not tell us whether a region consistently supports the answer. The same region may support an answer when shown alone but oppose it when other regions are visible. RegionTrace preserves these opposing effects and tracks how support for the same answer changes across visual contexts and reasoning stages (Table 20, p. 36).
Across six Study I settings, 63.0–67.2% of false-Yes cases contain a region that supports the wrong answer in one visual context but suppresses it in another. EAGLE’s combined score can hide this reversal: opposing effects can cancel. RegionTrace retains the signed effects for each context (Appendix F.2, Table 22, p. 38).
Using the same regions, target answer, probability scores, and image masking, we compared RegionTrace with a matched EAGLE-style ranking on 366 PACO cases where the complete-image answer was Yes. RegionTrace increased Gold Top-1 from 76.5% to 78.4% and Drop AUC from 0.379 to 0.387 (Appendix F.1, Table 21, p. 37).
We timed each method’s regional pipeline on the same 37 images, with Qwen2.5-VL-7B on one NVIDIA A40 (Appendix F.3, Table 23, p. 38). RegionTrace takes 4.11 seconds per image, including automatic region construction, versus 168.04 seconds for efficient EAGLE: approximately 41× faster, with comparable peak memory.
Comparison with PDCR
PDCR replaces the image with a blank one and measures how the probability of a reasoning step changes, while keeping the preceding text fixed. It uses increases in confidence in the correct answer to guide training. RegionTrace addresses a complementary question: which image regions support or oppose the same answer, and how does that support change during reasoning? At each checkpoint, it measures signed regional effects under different visible contexts, keeping the answer target fixed whether correct or incorrect.
In Figure 2 (p. 5), the black mug body supports the incorrect answer by +5.87 log-score units, while the queried white rim opposes it by −0.44. Separating these contributions localizes the misleading support. In Study II, regional support decreases across all six settings from the first sentence to the final checkpoint, even as the text increasingly sustains the same answer (Appendix B.3, Table 9, p. 24). We added the PDCR comparison and citation to Section 6 (p. 10).
Core contribution and implications
RegionTrace traces an answer’s visual support through reasoning and revisits. It could also test whether a GUI agent’s action depends on the intended interface element. See our response to reviewer P8vJ, Q2 for further implications.