acceptodds
Withdrawn from ICLR 2027

Rethinking Post-Pruned Evidence: Counterfactual Decision Reasoning for Fixed-Budget Visual Token Refinement

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual token pruning improves the efficiency of vision-language large models (VLLMs) but often discards query-relevant evidence under aggressive compression, leading to degraded model decisions. Existing post-pruning recovery methods restore discarded information according to predefined relevance or representation criteria, without explicitly evaluating how the resulting token composition affects the current decision. Consequently, restoration introduces a recovery–risk trade-off: useful evidence is recovered, while unfavorable interventions disrupt otherwise reliable predictions. To address this issue, we propose Counterfactual-Aware Restoration through Token Exchange (CARE), a training-free and fixed-budget post-pruning recovery framework based on query-adaptive token exchange and counterfactual decision-guided selection. Ours first identifies replaceable retained tokens and retrieves complementary discarded evidence to construct diverse one-to-one exchange proposals without increasing the visual-token budget. Ours then treats alternative token compositions as counterfactual interventions and evaluates their decision-level effects against the original pruned representation using the frozen VLLM. An alternative composition is adopted only when it provides stronger decision support than the no-change reference. Experiments across multiple benchmarks, pruning methods, and token budgets demonstrate consistent performance improvements, substantially fewer recovery-induced errors, and preserved pruning efficiency without additional training or token-count increase.

Primary area:
interpretability and explainable AI
Keywords:
post-pruning recovery, counterfactual, vision-language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TgFKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CARE, a training-free framework for post-pruning visual token refinement in vision-language models. CARE constructs one-to-one token exchange proposals using query relevance and token redundancy, followed by counterfactual decision-guided selection based on the frozen model's prediction confidence. Experiments across multiple benchmarks, pruning methods, and model backbones show improved accuracy and reduced recovery-induced errors without increasing the visual token budget.

Critical strengths
  1. The paper identifies an important limitation of existing token recovery methods: restoring relevant visual information can also disrupt originally correct predictions. This provides a meaningful motivation for decision-aware recovery.

  2. The fixed-budget token exchange strategy is training-free and applicable to different pruning methods. The reported reductions in recovery-induced errors are encouraging.

Critical weaknesses
  1. Several reported numbers are difficult to reconcile. Section 5.3 mentions 500 evaluated samples, but Table 2(b) reports 13 damaged samples with an RIE of 2.03%, implying approximately 640 originally correct samples. Similarly, the 1,500-sample setting in Section 5.4 is inconsistent with Table 3 (e.g., 221 damaged samples at 4.05% RIE). Moreover, Figure 4 shows 11–20 token exchanges, exceeding the stated maximum of eight. These inconsistencies undermine confidence in the experimental results.

  2. Counterfactual beam search requires multiple VLLM evaluations, yet Table 4 reports nearly identical latency for CARE and the base pruner. It is unclear whether the complete search overhead is included. Furthermore, higher self-generated answer likelihood does not necessarily indicate greater correctness. The gains over RESTORE are relatively small, and stronger compute-matched comparisons are needed to establish the benefit of the proposed selection strategy.

Questions for authors
  1. Can the authors reconcile the sample counts, RIE values, and token exchange numbers in Tables 2–3 and Figure 4 with the stated experimental settings?

  2. Does the latency in Table 4 include all counterfactual model evaluations? Please report end-to-end inference cost and the average number of VLLM evaluations per sample.

  3. How does CARE compare with simpler fixed-budget exchange strategies using confidence-based filtering under the same computational budget?

Reviewer_qeF6OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the recovery–risk trade-off in post-pruning visual token recovery, where restoring discarded evidence may improve incorrect predictions but also disrupt originally correct ones. The authors propose CARE, a training-free, fixed-budget framework that generates alternative token compositions through one-to-one exchanges and selects them using decision-support scores from a frozen VLLM. Experiments on four datasets with two base pruners show improved accuracy and fewer recovery-induced errors.

Critical strengths
  • Recovery–risk perspective: The paper considers the potential risks of post-pruning recovery, noting that restoring discarded tokens may introduce errors in originally correct predictions. This provides an additional perspective for evaluating recovery methods.

  • Guaranteed fixed visual-token budget: The one-to-one, non-overlapping token exchange mechanism explicitly guarantees that every candidate composition preserves the original visual-token budget without additional training.

Critical weaknesses

W1. CARE uses the frozen VLLM’s length-normalized log-likelihood to evaluate alternative token compositions. However, higher model confidence does not necessarily indicate better prediction accuracy, particularly when comparing different answers generated under different visual inputs. Length normalization mitigates answer-length bias but does not resolve differences in language priors or confidence calibration. The paper explicitly acknowledges that a positive decision-support score, Δ(𝒜), does not imply correctness. The remaining concern is whether this score reliably distinguishes beneficial exchanges from harmful ones. The positive-score criterion in Eq. (9) selects an alternative with stronger model-internal support, but does not ensure that an initially correct prediction is preserved. The ablation results support the usefulness of counterfactual selection, but do not directly validate the relationship between Δ(𝒜) and actual recovery outcomes. An analysis of score distributions for beneficial, harmful, and neutral exchanges would help establish whether the score captures recovery utility beyond model confidence.

W2. CARE preserves the visual-token count through one-to-one exchanges, but this does not establish its overall computational efficiency. The counterfactual beam search requires additional VLLM evaluations, with a reported complexity of O(MB C_VLLM) in Appendix B.7. Yet Table 4 reports almost identical latency for CARE (105.16 ms) and the base pruner (105.43 ms), with only a 1 MB increase in peak memory. It is unclear whether these measurements include all candidate evaluations, answer generation, likelihood computation, and search overhead. A complete end-to-end breakdown is needed to support the claim that CARE preserves pruning efficiency. The intervention statistics also require clarification. On GQA at the 64-token budget, Table 3 reports an RIR of 12.43%. Under Eq. (11) and the default maximum of eight candidate exchanges, the upper bound is 12.5%. The reported value therefore corresponds to approximately 7.96 exchanges per sample, nearly the maximum of eight. This suggests limited selectivity in the number of exchanges, although it does not by itself establish how often predictions change. Please clarify whether the averaging set includes no-change samples and how these statistics relate to the claim of conservative intervention. The authors should report the number of candidate evaluations per sample, total inference latency, adoption and no-change rates, and the distribution of exchange counts. These measurements would clarify both the computational overhead and how selectively CARE intervenes.

Questions for authors

Q1. How well does the proposal generator identify beneficial token exchanges? On a small candidate set, could the authors exhaustively evaluate all valid exchanges and report what fraction of beneficial exchanges the proposal generator retains? This would help assess whether the proposed strategy achieves high recall.

Q2. How reliably does the decision-support score Δ(𝒜) distinguish beneficial exchanges from harmful ones? The paper acknowledges that a positive score does not imply correctness. Could the authors compare score distributions across beneficial, harmful, and neutral exchanges, and report how often positive-score exchanges introduce new errors?

Q3. Are the accuracy gains over RESTORE statistically reliable? Please report variability across repeated runs or confidence intervals. Comparisons using additional recent base pruners and recovery methods, where compatible, would also help establish whether the gains extend beyond the current baselines.

Q4. How well does CARE extend beyond single-image understanding? Could the authors discuss or evaluate its applicability to multi-image reasoning and video understanding? In particular, how would the cost of evaluating alternative token compositions scale with longer visual inputs?

Q5. Figure 3(c) already examines sensitivity to M_max. Could the authors also report sensitivity to beam width B and proposal threshold κ, including their effects on accuracy and inference cost? Comparisons with simpler exchange and selection strategies under matched computational budgets would help assess the benefit of counterfactual beam search.

Reviewer_Nhh7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CARE is a training-free recovery procedure applied after visual-token pruning. It proposes non-overlapping exchanges between retained and discarded tokens using query relevance, redundancy, and an excluded-evidence direction. A bounded subset search evaluates alternative token compositions with the frozen VLLM and adopts a composition only when its generated answer has higher length-normalized model probability than the original answer under the original composition. The final token count stays fixed. Experiments report task accuracy, recovery-induced errors, recovery benefits, intervention ratios, and efficiency across pruners, budgets, and several VLLM backbones.

Critical strengths
  1. The recovery–damage trade-off is worth studying. Recovery can fix base-pruner errors while damaging correct predictions, so separating these transitions is more informative than reporting net accuracy alone. Appendix A's recovered/damaged counts and net gains address the important question of whether conservative intervention sacrifices all useful recovery.

  2. Fixed-size exchange with an explicit no-change option is a practical design. The method can reuse existing pruners and frozen models, and its one-to-one construction genuinely preserves the final token count. Table 1 reports consistent, albeit small, improvements over RESTORE across both pruners and all four tasks; the backbone and proposal/selection ablations extend the evidence beyond one configuration.

Critical weaknesses
  1. The recovery statistics and selection description contain unresolved inconsistencies that affect the central claim. Eq. (10) defines RIE as #Dmg divided by the number of base-correct examples. Table 2(b)'s 4 damaged examples at 0.63% imply roughly 635 base-correct examples, although the text describes recovery on 500 samples. Table 3's GQA CARE result of 47 at 0.77% implies roughly 6,104 base-correct examples, whereas Section 5.4 describes 1,500 recovery samples. A larger evaluation population may explain this, but the denominator and intervention population must be specified. Figure 3(c) labels normalized RIE as lower-is-better yet peaks at the marked 'best trade-off' M=8. In addition, Section 4 promises prioritization of prediction-preserving compositions, but Eq. (9) and Appendix B rank solely by probability gain, with no explicit preservation priority. These discrepancies prevent a reliable reading of the safety/recovery mechanism and its measured effect.

  2. The efficiency evidence does not reconcile full-model search with the small reported overhead. Appendix B evaluates O(MB) compositions using the frozen VLLM, with default M=8 and B=4, but Table 4 reports only 11.7 ms restoration overhead and nearly unchanged latency/memory. That is not impossible with strong reuse, filtering, or batching, but the necessary implementation and timing boundaries are not given. A fixed final token count is not a fixed inference-compute budget. Since Table 1's advantage over RESTORE is generally about 0.25–0.37 points, paired uncertainty and an accuracy-versus-total-compute comparison are needed to establish a meaningful practical gain.

Questions for authors
  1. Numerical clarification: Please give total evaluated, recovery-attempted, base-correct, recovered, and damaged counts for every reliability setting. Reconcile Table 2(b)'s 500-sample description and Table 3's 1,500-sample description with Eq. (10), and explain or correct the direction of Figure 3(c).
  2. Algorithm clarification: Is prediction preservation explicitly prioritized anywhere in the released selection rule? If so, please state the rule; if not, please align the description with Eq. (9) and distinguish increased self-confidence from verified correctness.
  3. Efficiency validation: How many distinct VLLM compositions are actually scored per query, what is cached/batched, and does Table 4 include all candidate generation/scoring and answer decoding? Please provide full-pipeline timing and paired accuracy intervals against RESTORE at comparable total compute.

If you can address the questions mentioned above, I wouldn't mind raising my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.