acceptodds
Under review as a conference paper at ICLR 2027

Tracing the Evidence: Faithful Token Attribution Through Vision-Language Reasoning

10 views

Abstract

Large vision-language models (LVLMs) exhibit strong reasoning capabilities, yet the visual and textual evidence supporting the generated responses remains difficult to identify. Faithful token attribution explains an LVLM's response by assigning scores that rank image and prompt tokens by how much the model relies on them, such that removing higher-ranked tokens causes the likelihood of the generated response to drop more rapidly. However, existing token-attribution methods have been developed mainly for text-based language models, and our empirical study reveals two challenges when complex multimodal sources are involved. First, the joint image-text attribution can underrepresent visual evidence relative to text, obscuring the image regions supporting the response. Second, visual evidence may influence the generated response through multiple intermediate reasoning paths, while existing methods trace only a limited subset of these paths, causing important visual contributions to be underestimated. Motivated by these insights, we introduce **VTrace**, a multimodal token-attribution framework that traces input contributions through intermediate reasoning and calibrates attribution scores across modalities. **VTrace** constructs pairwise attributions that highlight token-specific contributions and aggregates all forward attribution paths in closed form to account for both direct and indirect contributions. Cross-modal calibration then rescales image and text attribution scores using modality contributions estimated from response-likelihood changes, enabling a unified ranking of input tokens. Evaluations against seven baselines across six visual reasoning benchmarks demonstrate the superior attribution faithfulness. A demo and a reference code are available.

Primary area:
interpretability and explainable AI
Keywords:
Large Vision Language Models, LVLM Interpretability, Token Attribution

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_QDRzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces VTRACE, a token-attribution framework for explaining the visual and textual evidence supporting responses generated by large vision-language models. The method combines three components: modality-centered pairwise attribution, closed-form aggregation of direct and indirect paths through preceding tokens, and cross-modal calibration using two-modality Shapley contributions estimated from response-likelihood changes.

Critical strengths

S1. A meaningful problem with a clear multimodal motivation. Identifying which image regions and prompt tokens support a generated reasoning trace is an important interpretability problem. The paper identifies two relevant challenges: visual evidence can be under-ranked in joint attribution, and its influence may pass through intermediate generated tokens. The proposed method directly addresses both issues.

S2. A coherent design supported by useful ablations. The three components have distinct roles, and Table 2 examines their individual contributions. The substantial degradation without multi-hop aggregation supports the importance of this component under the chosen evaluation. Appendix H.2 further applies multi-hop aggregation to IFR, suggesting that the idea may be useful beyond the specific VTRACE construction. Reusing the attribution matrix across target tokens is also a practical advantage, supported by the reported efficiency measurements.

S3. Broad and consistent empirical evidence under the stated evaluation protocol. The experiments include seven baselines, six benchmarks, image-only and joint perturbations, and multiple model families and sizes. Results are reported for both correct and incorrect predictions, avoiding evaluation restricted to successful responses. The GRPO experiment provides an additional, promising indication of downstream utility beyond attribution visualizations.

Critical weaknesses

W1. The scope of the faithfulness evidence should be distinguished more clearly from answer-level and mechanistic interpretations. The paper explicitly defines faithfulness through changes in the likelihood of a fixed generated response, and the evaluation is appropriate for that definition. However, scoring the complete response under teacher forcing does not directly establish which evidence determines the final answer when reasoning is regenerated. In particular, information removed from the image may remain available through the original reasoning prefix. The results therefore support attribution of a fixed response more directly than the broader interpretation of recovering the evidence responsible for the model’s answer and reasoning process. This distinction matters when interpreting the path visualizations and the paper’s conclusions. Relevant sections: Section 5; Appendices F and G.

W2. The interpretation of multi-hop scores as evidence-propagation paths needs more direct validation. Equation 6 correctly aggregates paths in the constructed attribution matrix, but the matrix itself averages layer-level scores after modality centering and positive clipping. Paths in this collapsed graph need not correspond to layer-consistent paths in the model’s computation. Moreover, the incoming-flow factor in the score can favor tokens lying on many paths. The ablation establishes that aggregation improves the perturbation metrics, but does not by itself determine whether the recovered intermediate paths faithfully represent how evidence is used. A targeted validation would strengthen this central interpretation. Relevant sections: Sections 4.1–4.2; Appendix C.

W3. Robustness to the perturbation and metric-processing choices is insufficiently established. Appendix G applies cumulative minima or maxima to enforce monotonic perturbation curves and clips the resulting curves before computing AUC. These transformations can affect method rankings when model responses are nonmonotonic, making results without these transformations important. In addition, modality calibration uses padding embeddings, whereas visual evaluation uses blurred image regions followed by re-encoding. This does not invalidate the evaluation, but leaves uncertainty about how dependent the gains are on the selected interventions. Applying the same calibration to strong baselines would also help isolate the contribution of the underlying token rankings. Relevant sections: Section 4.3; Appendices D and G.

W4. The attribution-guided GRPO experiment does not yet isolate the benefit of VTRACE’s token selection. The average improvement of 1.28 percentage points over GRPO is encouraging. However, the experiment assigns weight 1.5 to the top 40% of response tokens without comparing against alternative token-selection rules under the same weighting budget. Consequently, the improvement cannot yet be specifically attributed to VTRACE’s attribution quality. Table 4 also does not provide uncertainty across repeated training runs, and the gains are not uniform across tasks, with small decreases on MathVista and MMMU-Pro. Relevant sections: Table 4; Appendix F.1.

Questions for authors
  1. Can the authors clarify which conclusions concern support for a fixed response and which concern the final answer decision? On a representative subset, do the advantages persist when attributing only the final-answer span and when regenerating the reasoning and answer after perturbation? Please distinguish final-answer scoring with the original reasoning prefix from evaluation with regenerated reasoning.

  2. Can the authors validate selected high-scoring intermediate paths through targeted activation interventions or a controlled task with known evidence dependencies, comparing them with matched low-scoring or random paths? An ablation using outgoing attribution alone would also help determine what the incoming-flow factor adds beyond path aggregation.

  3. Do the main rankings persist without monotonic enforcement and clipping, and under an alternative perturbation strategy? Please explain the handling of near-zero or reversed clean-to-perturbed endpoint differences and provide paired uncertainty estimates for the main AUC comparisons. How much of the joint-setting advantage remains if a strong baseline receives the same modality calibration?

  4. Under matched training and token-weighting budgets, how does VTRACE-guided GRPO compare with random token selection and selection using a strong existing attribution method? Can repeated training runs establish the uncertainty of the reported gain and clarify which tasks benefit consistently?

6 comments
AuthorsOpenReview ↗

Response to Reviewer QDRz Part 1

We thank the reviewer for the careful reading and concrete suggestions. We clarify your concerns below.

W1 & Q1a & Q1c. The scope of the faithfulness evidence should be distinguished more clearly from answer-level and mechanistic interpretations. ... Can the authors clarify which conclusions concern support for a fixed response and which concern the final answer decision? ... Please distinguish final-answer scoring with the original reasoning prefix from evaluation with regenerated reasoning.

Scoring only the final answer under teacher forcing still conditions on the original reasoning tokens, which may already contain the answer or the visual evidence needed to derive it. Following prior work, we evaluate the complete generated response to assess support for both the reasoning and the answer, not just the answer given that reasoning. This protocol measures faithfulness to a fixed response; it does not establish answer-level dependence when reasoning is regenerated.

Q1b. Do the advantages persist when regenerating the reasoning and answer after perturbation?

Yes. With regenerated reasoning, VTrace maintains the superiority against baselines. We kindly invite the reviewer to review our response to Reviewer rsBr's "W2b & Q2b" for detailed experiment results, where we evaluate VTrace under two complementary interventions of regeneration.

W2 & Q2a. The interpretation of multi-hop scores as evidence-propagation paths needs more direct validation. ... Paths in this collapsed graph need not correspond to layer-consistent paths in the model's computation.

Since WW aggregates contributions across layers, its paths represent composed attribution relationships among tokens. The purpose is to identify intermediate reasoning tokens through which input evidence supports the answer. To directly assess their behavioral relevance, we replace five reasoning tokens with the highest image-to-answer path scores: the answer changes on 10.1% of MMStar and 20.3% of MathVista traces, compared with 5.7% and 9.5% for the strongest of six baselines (Q2b). These interventions support the effectiveness of the path scores in identifying intermediate tokens that are relevant to the answer. We also isolate the incoming-flow factor: using outgoing attribution alone changes input-attribution RISE/MAS scores by at most 0.006 (Q2c), showing that the main gains are not driven by this factor. We will clarify that these paths capture token-level attribution relationships, and our interventions show that the identified intermediate tokens influence the final answer.

Q2b. Can the authors validate selected high-scoring intermediate paths through targeted activation interventions or a controlled task with known evidence dependencies, comparing them with matched low-scoring or random paths?

Following the reviewer's suggestion, we intervened on the intermediate reasoning tokens that VTrace scores highest, and compared them with matched,low scoring and random controls.

Which tokens? For each trace, we take the five reasoning tokens with the highest relay score, i.e., the image evidence flowing into the token times the evidence flowing from it to the answer.

Setup: We keep the first half of the original reasoning and blur the image, so the model must rely on what the reasoning has written. We replace five tokens in that half with the model's next most likely alternatives, and let the model regenerate the rest of the reasoning and the answer. We report how often the final answer changes, on the traces where the unmodified first half still gives the original answer.

Five tokens replacedMMStar (n=476)MathVista (n=610)MMMU (n=452)
VTrace top relays10.1%20.3%2.7%
Highest outflow to the answer only5.5%9.0%2.7%
Random neighbours of each relay5.7%7.4%2.0%
Most image attending neighbours5.5%9.3%1.5%
Most image attending tokens5.3%9.5%1.5%
Random tokens4.0%6.6%1.3%
VTrace lowest relays3.2%3.4%0.7%

What does each control rule out?

  • Random and lowest relays: that replacing any five tokens changes the answer equally.
  • Neighbours of each relay: that the effect comes from the position of the tokens in the reasoning.
  • Most image attending tokens: that plain attention to the image finds the same tokens.
  • Highest outflow only: the necessity of the incoming factor (see also Q2c).

Results: On MMStar and MathVista, replacing VTrace's top relays changes the answer about twice as often as the strongest control, with MMMU showing a scenario where the answer rarely changes for any selection (at most 2.7%).

Thus, the high scoring intermediate tokens identified by VTrace are the ones the final answer depends on.

AuthorsOpenReview ↗

Response to Reviewer QDRz Part 2

Q2c. An ablation using outgoing attribution alone would also help determine what the incoming-flow factor adds beyond path aggregation.

Following the reviewer's suggestion, we evaluated VTrace using outgoing attribution alone. The gains over baselines come from path aggregation, while the incoming factor removes a position bias that matters for reasoning tokens.

VTrace scores token xrx_r by (1+In(xr))⋅Out(xr)(1+\mathrm{In}(x_r))\cdot\mathrm{Out}(x_r), where Out\mathrm{Out} sums paths from xrx_r to the answer and In\mathrm{In} sums paths arriving at xrx_r. Outgoing attribution alone favours early tokens regardless of content because more tokens follow them and more paths originate from them. On a complete causal graph with equal edge weights and the answer at position aa, Out(xr)=γ(1+γ)a−r−1\mathrm{Out}(x_r)=\gamma(1+\gamma)^{a-r-1} grows exponentially towards the sequence start. By contrast, (1+In)⋅Out=γ(1+γ)a−1(1+\mathrm{In})\cdot\mathrm{Out}=\gamma(1+\gamma)^{a-1} at every position, cancelling this bias.

Input tokens. Joint-setting results are reported as insertion ↑ / deletion ↓:

ReadoutMetricMMStarMathVistaMMMU
Outgoing onlyRISE0.583 / 0.1820.647 / 0.1770.660 / 0.160
MAS0.398 / 0.2470.496 / 0.2440.500 / 0.215
In & OutRISE0.581 / 0.1800.647 / 0.1780.660 / 0.159
MAS0.397 / 0.2460.497 / 0.2460.503 / 0.215

The readouts differ by at most 0.006, as expected: input tokens occur near the sequence start, with few preceding tokens and little inflow. The gains in Table 1 therefore come from path aggregation.

Reasoning tokens. Inflow matters here because reasoning tokens differ in how much visual evidence reaches them. In the Q2b intervention, replacing tokens selected by outflow alone changes the answer about half as often as replacing tokens selected with the incoming factor:

Five tokens replacedMMStarMathVista
Highest outflow only5.5%9.0%
Inflow × outflow10.1%20.3%

Thus, the incoming factor identifies intermediate tokens that carry visual evidence to the answer.

W3 & Q3a. Robustness to the perturbation and metric-processing choices is insufficiently established. ... Do the main rankings persist without monotonic enforcement and clipping?

VTrace remains the best-performing method without monotonic enforcement or clipping.

These transformations reduce noise: adding patches can temporarily lower response likelihood, so the monotonic envelope smooths fluctuations, while clipping keeps normalised curves within [0,1][0,1]. Since both can affect rankings, we removed all three transformations: (1) response-curve clipping to [0,1][0,1], (2) monotonic enforcement, and (3) MAS-penalised curve clipping to [0,1][0,1].

Each cell reports VTrace / strongest baseline; raw results also include the paired 95% confidence interval for their difference.

RISE:

BenchmarkProtocolImage del ↓Image ins ↑Joint del ↓Joint ins ↑
MMStarOriginal0.357 / 0.3880.596 / 0.5610.182 / 0.2040.576 / 0.550
Raw0.369 / 0.399 [−0.035, −0.025]0.583 / 0.547 [+0.030, +0.042]0.184 / 0.206 [−0.024, −0.020]0.571 / 0.545 [+0.023, +0.030]
MathVistaOriginal0.312 / 0.3540.658 / 0.5980.179 / 0.2430.643 / 0.570
Raw0.327 / 0.368 [−0.048, −0.035]0.644 / 0.583 [+0.054, +0.068]0.183 / 0.243 [−0.065, −0.057]0.637 / 0.564 [+0.068, +0.079]
MMMUOriginal0.389 / 0.4240.657 / 0.6150.165 / 0.1850.643 / 0.612
Raw0.408 / 0.444 [−0.044, −0.027]0.639 / 0.597 [+0.032, +0.054]0.168 / 0.186 [−0.022, −0.015]0.639 / 0.608 [+0.026, +0.037]

MAS:

BenchmarkProtocolImage del ↓Image ins ↑Joint del ↓Joint ins ↑
MMStarOriginal0.494 / 0.5320.455 / 0.4080.248 / 0.3330.390 / 0.345
Raw0.603 / 0.667 [−0.073, −0.055]0.330 / 0.244 [+0.075, +0.096]0.254 / 0.353 [−0.104, −0.094]0.299 / 0.150 [+0.141, +0.158]
MathVistaOriginal0.447 / 0.4820.530 / 0.4600.247 / 0.3810.492 / 0.373
Raw0.528 / 0.587 [−0.072, −0.046]0.450 / 0.330 [+0.106, +0.134]0.256 / 0.413 [−0.164, −0.149]0.446 / 0.206 [+0.229, +0.252]
MMMUOriginal0.534 / 0.5710.527 / 0.4870.223 / 0.3020.480 / 0.420
Raw0.697 / 0.761 [−0.079, −0.050]0.423 / 0.362 [+0.039, +0.083]0.230 / 0.313 [−0.090, −0.076]0.418 / 0.288 [+0.113, +0.147]

Removing the transformations changes individual RISE AUC scores by at most 0.02. MAS changes more because it penalises disagreement between retained attribution mass and model response. This penalty can push curves outside [0,1][0,1]; removing clipping exposes larger penalties, particularly for baselines. VTrace's advantage remains under both metrics.

AuthorsOpenReview ↗

Response to Reviewer QDRz Part 3

Q3b. Do the main rankings persist under an alternative perturbation strategy? ... Modality calibration uses padding embeddings, whereas visual evaluation uses blurred image regions.

Yes. With an alternative perturbation, VTrace still ranks first on every score.

Following the reviewer's suggestion, we replaced the blur with a mean colour fill: each removed patch is filled with the image's mean colour, and the image is encoded again. This removes all content of a patch, including the coarse structure that blur keeps. Attribution and calibration are unchanged.

Due to tight turnaround, we evaluated on a subset of MMStar traces (ins$\uparrow$ / del$\downarrow$):

SettingMetricIFRFlowTracerFlashTraceAttnLRPVTrace
ImageRISE0.557 / 0.4140.548 / 0.4200.573 / 0.3980.582 / 0.3930.617 / 0.361
MAS0.399 / 0.5560.381 / 0.5720.427 / 0.5320.411 / 0.5590.476 / 0.499
JointRISE0.515 / 0.2240.514 / 0.2300.561 / 0.2070.538 / 0.2220.591 / 0.182
MAS0.315 / 0.3680.308 / 0.3790.358 / 0.3380.326 / 0.3470.410 / 0.250

Results: VTrace is first in all eight comparisons, with margins similar to those under blur (e.g., Image RISE insertion 0.617 vs. 0.582 here, and 0.600 vs. 0.563 under blur).

Why does calibration use padding embeddings while the evaluation uses blur?

  1. Calibration removes a whole modality to measure its total contribution, padding embeddings do this for image and text in the same way, within three forward passes and without encoding the image again.
  2. The evaluation removes individual patches, which is done in pixel space so that the vision encoder processes the perturbed image.

Does this mismatch matter? We recomputed the calibration from the blurred image instead of the padding embeddings. The resulting image share ranks traces almost identically:

MMStarMathVistaMMMU
Spearman correlation of the image share (padding vs. blur)0.920.950.92

Thus, VTrace still ranks first under a different evaluation perturbation, and this shows that calibration is not tuned to the perturbation used in the evaluation.

Q3c. Please explain the handling of near-zero or reversed clean-to-perturbed endpoint differences.

Thank you for your attention to detail. We highlight why this is not a problem for VTrace as follows:

  1. They are rare (0.16% of the samples measured).
  2. The endpoints are the clean and the fully perturbed input, which are the same for every method on a trace. No method receives a more favourable denominator.
  3. VTrace performs best no matter if these curves are kept (Table 1) or excluded (Q3a).

How often? We counted the curves with ∣f0−fK∣<0.01|f_0-f_K|<0.01 on 2,736 samples:

Count
All curves (traces x settings x insertion and deletion)10942
Near zero endpoint difference18 (0.16%)
of which reversed3

How do we handle them?

  • Main Table 1 protocol: these curves are kept. A small constant in the denominator avoids division by zero, and clipping to [0,1][0,1] keeps their normalised values bounded.
  • Raw curve evaluation (Q3a): there is no clipping, so a near zero denominator would produce extreme values. We exclude these curves for every method.
Reviewer_bDVDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents VTRACE, a token attribution framework for large vision-language models (LVLMs). The authors identify two limitations of existing attribution methods in multimodal settings: (1) visual evidence is systematically under-ranked relative to textual tokens in joint image-text attribution, and (2) attribution methods that trace only direct or limited-hop paths miss contributions that propagate through intermediate reasoning chains. VTRACE addresses these with three components: modality-aware pairwise attribution (centering writes within each modality), compositional multi-hop attribution via a closed-form Katz-based path aggregation, and cross-modal calibration via two-player Shapley values estimated from response-likelihood changes. Experiments across six visual reasoning benchmarks against seven baselines demonstrate consistent improvements in RISE and MAS insertion/deletion faithfulness metrics, with additional results on efficiency and attribution-guided post-training.

Critical strengths
  1. Each component has a clear mathematical motivation. The modality-centering operation connects naturally to residual-stream interpretability (Elhage et al., 2021). The Katz-based closed-form aggregation is well-known in graph theory and its application here is clean, with a formal proof provided (Appendix C). The Shapley-based calibration is a principled and theoretically grounded choice for two-modality credit assignment.

  2. The evaluation covers six diverse benchmarks, seven baselines, two model families, multiple model sizes, both correct and incorrect predictions, and includes efficiency profiling. The consistent improvements across this range of conditions substantially strengthen the empirical case.

Critical weaknesses
  1. Practical Utility Remains Underexplored.

The paper demonstrates consistent improvements on perturbation-based faithfulness metrics, but the practical utility of the proposed attribution method remains underexplored. The GRPO experiment (Table 4) is the most compelling demonstration of downstream value, yet it is treated as secondary. A stronger paper would either (a) provide a concrete use case where better attribution leads to measurably better outcomes, or (b) conduct a human evaluation confirming that VTRACE's attributions align with human judgment about which evidence matters. Without this, the evaluation risks becoming circular: the method is designed to find tokens whose removal degrades model output, and it is then evaluated by checking whether removing those tokens degrades model output. A method that scores well on this metric has not necessarily identified tokens that are semantically meaningful or that reflect genuine visual reasoning — it may simply be better at concentrating attribution mass on tokens that the perturbation protocol happens to be sensitive to.

  1. The GRPO Experiment Is Underdeveloped.

The attribution-guided GRPO experiment is the most practically meaningful contribution in the paper, as it moves VTRACE beyond interpretability into a concrete downstream application. However, the experiment is presented as a secondary result and leaves several important questions unanswered. First, the reported gain of 1.28 points on average (63.99 → 65.27) is not accompanied by any statistical significance test, making it difficult to assess whether the improvement is reliable or within noise. Second, the mechanism is underexplained: VTRACE scores are used to upweight the top 40% of response tokens during training, but there is no analysis of why this particular weighting strategy is effective, whether the benefit comes from the attribution ranking itself or simply from the upweighting scheme, or how sensitive the result is to the 40% threshold and the 1.5 weight factor. Without these details, the experiment reads more like a proof-of-concept appendix than a substantiated contribution.

  1. The Calibration Step Lacks an Isolated Baseline Comparison.

VTRACE applies three additional perturbation-based forward passes to estimate the relative contribution of each modality, then uses these estimates to rescale image and text attribution scores. The ablation in Table 2 shows that removing calibration hurts joint-setting performance, confirming that calibration helps VTRACE. However, this ablation does not establish how much of the joint-setting improvement over existing baselines is attributable to the calibration step itself, as opposed to the pairwise attribution or multi-hop aggregation components. The most informative experiment would be to apply the same calibration procedure to existing baselines — specifically FlashTrace, IFR, and FlowTracer — and compare their joint-setting scores before and after calibration. If calibration alone closes a substantial portion of the gap between VTRACE and these baselines, then the contribution of the other components would need to be reassessed. Without this comparison, it is unclear whether the proposed calibration is a general-purpose improvement that benefits any attribution method, or whether it is specifically effective in combination with VTRACE's other components.

Questions for authors

Q1. For the post-training experiment (Table 4), the improvement over plain GRPO is 1.28 points on average. Is this difference statistically significant? How sensitive is it to the 40% threshold and the 1.5 weight used to select important tokens?

Q2. The paper evaluates on single-image reasoning. For multi-image inputs (common in MMMU-Pro's vision-only setting), how does modality centering behave? Are all image tokens from all images pooled into a single modality mean?

3 comments
AuthorsOpenReview ↗

Response to Reviewer bDVD Part 1

We thank the reviewer for the careful review, and for recognising the motivation of each component and the breadth of the evaluation. We provide detailed clarification below.

W1. Practical utility remains underexplored. ... A stronger paper would either (a) provide a concrete use case where better attribution leads to measurably better outcomes, or (b) conduct a human evaluation confirming that VTRACE's attributions align with human judgment about which evidence matters. Without this, the evaluation risks becoming circular ...

Thank you for highlighting the need for stronger evidence of practical utility. To complement the perturbation-based evaluation, we conducted a human evaluation with two human experts to annotate image and text tokens that provide relevant evidence for answering the question. We measure alignment with these annotations using R-Precision, defined as the proportion of a method’s top RR tokens that are annotated as evidence, where RR is the number of evidence tokens annotated for each sample. The results are shown below.

SettingMetric ↑HETAFlowTracerIFRAttn RolloutAttnLRPFlashTraceVTRACE
ImageR-Precision0.0960.2610.0990.1300.1970.1000.401
JointR-Precision0.0790.2050.0850.1020.1060.0970.236

VTRACE achieves the highest scores in both settings, and these results show better alignment with human-identified evidence.

W2 & Q1. The GRPO experiment is underdeveloped ... Is this difference statistically significant? How sensitive is it to the 40% threshold and the 1.5 weight used to select important tokens?

Our primary contribution is a token-attribution framework for tracing evidence through multimodal reasoning. The GRPO experiment is intended as a demonstration of its potential use in downstream training. Our GRPO implementation follows the setup of FlowTracer (Dong et al., 2026), where we upweight the top 40% of reasoning tokens by a factor of 1.5 according to their attribution to the final answer, excluding final-answer tokens from selection. This directs more training emphasis toward reasoning tokens estimated to support the answer. We normalize weights to mean 1 per response to preserve the average token weight. The 40% threshold and 1.5 factor follow the prior setup. We are conducting supplementary GRPO experiments, including controls based on random token selection and baseline rankings. Given the computational cost of the training and limited rebuttal period, this evaluation remains ongoing. We will incorporate the supplementary results into the final version, and we will soften the wording in the main paper to reflect the exploratory scope. We appreciate your suggestions that help us better clarify the scope of our contributions and strengthen the evaluation.

AuthorsOpenReview ↗

Response to Reviewer bDVD Part 2

W3. The calibration step lacks an isolated baseline comparison. ... The most informative experiment would be to apply the same calibration procedure to existing baselines, specifically FlashTrace, IFR, and FlowTracer, and compare their joint-setting scores before and after calibration.

Thank you for the thoughtful suggestion; we have extended our evaluation to isolate the effect of calibration across existing baselines (IFR, FlashTrace, FlowTracer and AttnLRP, as well as IFR + multihop). The results show that calibration alone does not consistently improve attribution faithfulness for baselines. Its effectiveness depends on the quality of the underlying attribution rankings. We report the Joint results below, where each cell is (RISE ins$\uparrow$ / del$\downarrow$), (MAS ins$\uparrow$ / del$\downarrow$):

MethodMetricMMStarMathVistaMMMU
IFRRISE0.506 / 0.2210.513 / 0.2700.607 / 0.195
MAS0.305 / 0.3630.322 / 0.4280.427 / 0.324
IFR + calRISE0.453 / 0.2520.477 / 0.2860.509 / 0.237
MAS0.249 / 0.3770.270 / 0.4130.309 / 0.372
FlashTraceRISE0.553 / 0.2030.572 / 0.2420.628 / 0.178
MAS0.348 / 0.3310.375 / 0.3800.435 / 0.293
FlashTrace + calRISE0.500 / 0.2310.530 / 0.2560.551 / 0.216
MAS0.300 / 0.3350.329 / 0.3620.349 / 0.332
FlowTracerRISE0.518 / 0.2200.525 / 0.2680.605 / 0.196
MAS0.314 / 0.3430.329 / 0.4080.417 / 0.312
FlowTracer + calRISE0.464 / 0.2540.493 / 0.2840.508 / 0.241
MAS0.261 / 0.3870.287 / 0.4200.305 / 0.396
AttnLRPRISE0.529 / 0.2180.512 / 0.2740.615 / 0.191
MAS0.315 / 0.3400.304 / 0.4220.421 / 0.294
AttnLRP + calRISE0.522 / 0.2240.508 / 0.2700.599 / 0.195
MAS0.308 / 0.3300.295 / 0.3990.408 / 0.286
IFR + multihop + calRISE0.559 / 0.1930.612 / 0.1950.639 / 0.174
MAS0.399 / 0.2600.492 / 0.2860.509 / 0.231
VTrace w/o calRISE0.574 / 0.1820.622 / 0.1980.659 / 0.159
MAS0.375 / 0.2670.437 / 0.2840.512 / 0.219
VTraceRISE0.579 / 0.1800.645 / 0.1780.676 / 0.155
MAS0.394 / 0.2460.495 / 0.2470.501 / 0.215

Notably, calibration has different effects across attribution methods.

  • Raw baselines generally perform worse after calibration.
  • VTrace remains ahead of calibrated baselines.

Why does calibration help some methods but not others? Calibration adjusts the relative importance assigned to image and text tokens without changing their rankings within each modality. It therefore promotes the image patches that a method already considers important but underranked due to dominance of text attribution.

  1. Raw baselines. IFR, FlashTrace, FlowTracer and AttnLRP initially assign little attribution share to the image, which is what we illustrate in Figure 1a) and 7. Calibration increases the prominence of their highest-ranked image patches. However, when these patches do not align well with the measured perturbation effects, they displace more informative text tokens and reduce faithfulness.

  2. IFR + multihop. Although IFR + multihop uses the same edge weights as raw IFR, our multi-hop aggregation produces more informative attribution rankings. Aggregation alone improves IFR's attribution (RISE and MAS), calibration then provides further improvements. This is particularly informative: with the same underlying edges, calibration generally hurts raw IFR but benefits IFR + multihop, highlighting the importance of our attribution aggregation.

Thus, calibration is most effective when the underlying method already identifies informative tokens within each modality. It cannot correct poor rankings by itself. This also explains why VTrace's improvements primarily come from its attribution construction rather than calibration alone. The ablation in Table 2 supports this interpretation: multi-hop aggregation provides the largest improvement, while centering and calibration contribute additional gains.

Q2. The paper evaluates on single-image reasoning. For multi-image inputs (common in MMMU-Pro's vision-only setting), how does modality centering behave? Are all image tokens from all images pooled into a single modality mean?

Thank you for raising this question. When extending to multi-image cases, each image token is centered using the mean write of its own image. Text tokens remain centered using the mean of input text tokens as before. Specifically, the centering groups thus become {I1,…,IK,T}\{\mathcal I_1,\ldots,\mathcal I_K,\mathcal T\}. The centered tokens still form a modality-aware attribution matrix, so multi-hop aggregation remains unchanged. We will include these clarifications in the paper.

Dear Reviewer bDVD,

We sincerely apologize for the gentle reminder and greatly appreciate your thoughtful review. We have provided thorough responses to your questions and will incorporate all the feedback received during the discussion into the final version of our paper.

As the discussion deadline is approaching, we would be grateful if you could share your feedback to our responses. Should you have any concerns, we are happy to provide further clarification. Thank you again for your consideration!

Authors 2281

Reviewer_rsBrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VTRACE is a token-level attribution framework for vision-language reasoning built around modality centering, Katz path aggregation, and Shapley-based cross-modal calibration. It evaluates attribution through input perturbations on fixed generation traces and reports efficiency, ablations, and an attribution-guided GRPO experiment.

Critical strengths

VTRACE combines modality-aware scoring, multi-hop aggregation, and calibration to address cross-modal score imbalance and evidence propagation through reasoning. Its gains are broadly consistent across baselines and model families, with competitive computational cost. The ablations support all three components, and the GRPO experiment provides preliminary evidence of downstream utility.

Critical weaknesses
  1. Appendix H.2 shows that Katz aggregation substantially improves IFR and can approach VTRACE in some settings. The full six-benchmark comparison should therefore include tuned IFR + Katz, with and without the same calibration. The paper should also explain the omission of GLIMPSE and justify both multiplicative path composition and the summation of correlated paths, which may overcount redundant evidence.

  2. Calibration and evaluation both use likelihood changes of the same fixed trace, despite using different perturbations. This alignment may favor the calibrated method. The quantitative evaluation targets the complete trace; final-answer-only results are qualitative, and no intervention tests whether an intermediate step mediates the answer. The paper should evaluate the final answer with a target or metric not based on fixed-trace likelihood and report how often calibration is skipped because a required quantity is non-positive.

  3. The GRPO experiment compares standard GRPO only with VTRACE-based weighting. It shows that token weighting may help, but does not isolate VTRACE as the cause. Another attribution signal and a random-weighting control are needed.

Questions for authors
  1. In the full evaluation, how does VTRACE compare with tuned IFR + Katz and IFR + Katz with the same calibration? Is GLIMPSE applicable? How does the method avoid overweighting correlated paths?

  2. Does the advantage remain when the final answer is evaluated quantitatively using a target or metric not based on fixed-trace likelihood? How often is calibration skipped?

  3. With data, weighting, and budget matched, does VTRACE outperform FlowTracer, another attribution signal, and random weighting as a GRPO training signal?

5 comments
AuthorsOpenReview ↗

Response to Reviewer rsBr Part 1

We thank the reviewer for your positive rating and the thoughtful questions. We carefully clarify the concerns below.

W1/Q1a. Appendix H.2 shows that Katz aggregation substantially improves IFR and can approach VTRACE in some settings. The full comparison should include tuned IFR + Katz, with and without the same calibration.

We thank the reviewer for this suggestion. We clarify that IFR + Katz is our multi-hop aggregation applied to IFR's edges. App. H.2 therefore shows that the aggregation transfers to other methods, as Challenge 2 predicts.

Following the reviewer's suggestion, we compared VTrace with IFR + multihop at its best value (γ=0.5\gamma = 0.5, App. H.2), with and without VTrace's calibration. Due to tight turnaround, we compared on MMStar, MathVista and MMMU under the Joint setting. The results are shown below, where each cell is (RISE ins$\uparrow$ / del$\downarrow$), (MAS ins$\uparrow$ / del$\downarrow$):

MethodMetricMMStarMathVistaMMMU
IFRRISE0.506 / 0.2210.513 / 0.2700.607 / 0.195
MAS0.305 / 0.3630.322 / 0.4280.427 / 0.324
IFR + multihopRISE0.555 / 0.1980.581 / 0.2270.648 / 0.175
MAS0.369 / 0.2700.414 / 0.3050.497 / 0.230
IFR + multihop + calRISE0.559 / 0.1930.612 / 0.1950.639 / 0.174
MAS0.399 / 0.2600.492 / 0.2860.509 / 0.231
VTraceRISE0.579 / 0.1800.645 / 0.1780.676 / 0.155
MAS0.394 / 0.2460.495 / 0.2470.501 / 0.215

From the results, we identify that:

  • VTrace is consistently better even compared with a fully equipped IFR, and our proposed modules are transferable.
  • We highlight that the calibration module is not applicable under Image perturbation setting as it does not rank the text tokens.

We attribute the remaining Joint gap to differences in edge weighting. VTrace centres each token's write within its modality, whereas IFR scores each source by its share of the receiver's proximity. On 201 MMStar traces, IFR assigns nearly twice the total incoming edge weight per token (2.42 vs. 1.28). Consequently, contributions along longer paths accumulate more strongly, requiring a lower γ\gamma of 0.5 to prevent them from dominating the sum, while VTrace finds best at γ=1\gamma = 1.

W1b/Q1b. Explain the omission of GLIMPSE. Is GLIMPSE applicable?

We clarify that while GLIMPSE is relevant, we could not find any official or publicly available implementation code. Therefore, to ensure fairness in comparison, we have omitted GLIMPSE in the main tables, while still acknowledging the work appropriately.

Nevertheless, to provide a more comprehensive comparison, we made our best effort to reproduce GLIMPSE based on the paper and adapt it to our evaluation setting. For Joint evaluation, we combine its image and prompt attribution scores into a single ranking using their original scales.

Based on our reproduced runs, VTrace consistently outperforms our reproduction of GLIMPSE.

MethodMetricMMStarMathVistaMMMU
GLIMPSERISE0.535 / 0.2050.542 / 0.2500.627 / 0.181
MAS0.323 / 0.3040.339 / 0.3690.443 / 0.270
VTraceRISE0.579 / 0.1800.645 / 0.1780.659 / 0.159
MAS0.394 / 0.2460.495 / 0.2470.501 / 0.215

GLIMPSE nevertheless remains a competitive baseline.

AuthorsOpenReview ↗

Response to Reviewer rsBr Part 2

W1c/Q1c. Justify multiplicative path composition and the summation of correlated paths, which may overcount redundant evidence.

Thank you for the question. The path sum captures all weighted paths in our constructed graph. While VTrace does not explicitly remove correlations between paths, edge weighting and modality centering help limit redundant contributions. Our experiments further show that the full path sum improves attribution faithfulness over partial paths.

1. Why multiply edge weights?

W^ij\widehat W_{ij} measures the contribution of token xix_i to a later token xjx_j (Eq. 4).

  • For a path through an intermediate token xkx_k, the contribution is W^ikW^kj\widehat W_{ik}\widehat W_{kj}.

  • Summing over intermediate tokens gives (W^2)ij(\widehat W^2)_{ij} (2-hop).

  • More generally, (W^τ)ij(\widehat W^\tau)_{ij} sums the weighted contributions of all paths with τ\tau edges.

2. Why sum over paths?

Information may reach a token directly or through intermediate tokens, including reasoning steps. Summing these paths captures their combined contributions in the constructed graph.

Since generation is causal, W^\widehat W is strictly upper triangular, with W^T=0\widehat W^T=\mathbf 0. The sum is therefore finite and equals (I−γW^)−1−I(\mathbf I-\gamma\widehat W)^{-1}-\mathbf I (Eq. 6, App. C).

Importantly, these paths describe the attribution computation, not actual information routes through the network, because WW averages edges across layers (App. H.6).

3. How does VTrace limit redundant contributions?

  1. Edge weighting: Each source token receives credit proportional to its contribution to the receiving token's update (Eq. 4). The contributions sum to ∣Δj∣|\Delta_j|. Identical writes therefore share the receiving token's credit rather than each receiving full credit.

  2. Modality centering: Eq. 3 removes the average write within each modality. For example, if fi=c+uif_i=c+u_i, centering gives f~i=ui−uˉ\widetilde f_i=u_i-\bar u. The shared component cc is therefore removed before computing edge weights.

While not guaranteeing that the paths are counted independently, these mechanisms limit local redundancy.

4. Does path redundancy affect attribution faithfulness?

We evaluate this empirically through two experiments:

  1. Perturbation faithfulness (Table 1): VTrace achieves the best MAS across all benchmarks. This indicates that its attribution rankings agree well with measured perturbation effects, despite possible path redundancy.

  2. Path-length ablation (Fig. 12c): Restricting the sum to paths of at most one, two or three edges consistently reduces faithfulness across all four evaluations. Retaining the full path sum performs best.

Hence, while these experiments do not directly measure correlations between paths, they show that retaining the full path sum improves attribution faithfulness in practice, rather than introducing harmful redundancy that outweighs its benefits.

W2a. Calibration and evaluation both use likelihood changes of the same fixed trace. This alignment may favor the calibrated method.

We thank the reviewer for this careful observation. The calibration step measures how much the answer depends on the image as a whole and on the text as a whole, and sets one scale factor per sample from that. It does not access the per token perturbation results used for evaluation or alter token rankings within each modality (e.g., among image tokens). Its only effect is to adjust how image and text tokens are interleaved in the Joint ranking, so the Image results are identical with or without it. Calibration therefore does not introduce bias from the evaluation perturbations. Importantly, VTrace remains competitive without calibration and performs strongly under evaluation metrics that do not rely on likelihood. We kindly refer the reviewer to our response to W2b below and our response to Reviewer bDVD's W3 for further results.

AuthorsOpenReview ↗

Response to Reviewer rsBr Part 3

W2b & Q2a. Does VTRACE’s advantage persist under final-answer evaluation without fixed-trace likelihood, and do intermediate reasoning steps mediate the answer?

We conducted two additional experiments: evaluation with fully regenerated reasoning and answers, and interventions on intermediate reasoning tokens.

1. Final-answer evaluation without fixed-trace likelihood

After image perturbation, the model generates both its reasoning and final answer from scratch. We compare the regenerated answer with the original answer obtained from the clean image under two interventions:

  • Sufficiency: Retain only the top-ranked image patches and blur the rest; measure recovery of the original answer.
  • Necessity: Blur only the top-ranked patches; measure failure to recover the original answer.

Given the limited rebuttal period, we pooled dataset subsets into two groups. We report sufficiency AUC and both metrics at a 5% patch budget (higher is better).

Metric ↑RandomIFRFlowTracerFlashTraceAttnLRPVTRACE
MMStar / MathVista / MMMU (n=934)
Sufficiency AUC.615.641.630.648.663.697
Sufficiency @ 5% (%)32.831.832.133.138.942.9
Necessity @ 5% (%)18.221.219.121.226.431.0
MathVerse / MMMU-Pro / VisualPuzzles (n=480)
Sufficiency AUC.500.573.551.575.572.602
Sufficiency @ 5% (%)27.527.527.529.029.830.8
Necessity @ 5% (%)33.537.431.536.139.847.2

VTRACE achieves the highest scores across all reported metrics in both groups, supporting its advantage beyond fixed-trace likelihood under fully regenerated reasoning and final answers.

2. Interventions on intermediate reasoning tokens

We removed the entire image while preserving the first half of the original reasoning. The model reproduced its original answer in 77% of traces, showing that answer recovery often persists with the retained reasoning prefix alone.

We then replaced five of VTRACE’s highest-ranked relay tokens in this prefix with the model’s next most likely alternatives and regenerated the remaining reasoning and final answer. The answer-change rates were:

Token selectionMMStarMathVistaMMMU
VTRACE relays10.1%20.3%2.7%
Six controls3.2–5.7%3.4–9.5%0.7–2.7%

Replacing VTRACE’s relay tokens changes the answer more often than any of the six controls on MMStar and MathVista; changes are rare and comparable on MMMU. These interventions support the influence of the identified intermediate tokens on the final answer, although they do not establish full causal mediation of visual evidence. Full results appear in our response to Reviewer QDRz’s Q2b.

W2c & Q2b. How often is calibration skipped?

Calibration was skipped on 3 of 2,736 traces (0.1%). We apply the scaling factor λ=(ϕI/ϕT)⋅(∑text scores/∑image scores)\lambda=(\phi_I/\phi_T)\cdot(\sum \text{text scores}/\sum \text{image scores}) only when both modality Shapley values and both attribution-score sums are positive, ensuring a well-defined, positive factor.

In all three skipped cases, ϕI\phi_I was slightly negative: removing the image marginally increased the model’s confidence in the fixed response. Calibration was therefore applicable to 99.9% of the evaluated traces.

W3/Q3. The GRPO experiment does not isolate the benefit of VTRACE’s attribution ranking. Comparisons with FlowTracer, another attribution signal, and random weighting are needed under matched data, weighting, and training budgets.

Thank you for highlighting the need for additional controlled experiments. Our primary contribution is a token-attribution framework for tracing evidence through multimodal reasoning. The GRPO experiment provides an initial demonstration of its potential use in downstream training. Our GRPO implementation follows the setup of FlowTracer (Dong et al., 2026): we upweight the top 40% of reasoning tokens, ranked by their attribution to the final answer, by a factor of 1.5, excluding final-answer tokens from selection. Both hyperparameters are adopted from the prior setup rather than tuned for VTRACE. We are conducting supplementary GRPO experiments, including controls based on random token selection and baseline rankings. Given the computational cost of the training and limited rebuttal period, this evaluation remains ongoing. We will incorporate the supplementary results into the final version, and we will soften the wording in the main paper to reflect the exploratory scope. We appreciate your suggestions that help us strengthen the evaluation and clarify the scope of our claims.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.