acceptodds
Withdrawn from ICLR 2027

Efficient Multimodal Contrastive Decoding via Budgeted Activation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal contrastive decoding (MCD) has been widely adopted to mitigate hallucination in multimodal large language models (MLLMs). It is performed by contrasting predictions from an original input with those from an auxiliary branch. The inference latency thus increases to at least *2×* that of vanilla decoding. However, our preliminary experiments show that MCD sparsely alters the output token (e.g.,  7% on POPE). This mismatch between computational cost and the rate of altered tokens motivates a token-level analysis of MCD's activation. Accordingly, we introduce **FlipTrace**, a theoretical framework that characterizes MCD's effect as *flips*. We then prove a *flip-immunity condition* that bounds how many tokens MCD can affect and show that these effects concentrate where the bound predicts. Together, these findings lead to **flip susceptibility**, a statistic computed from the original logits alone that predicts token flips. Using this measure, we develop a **training-free budgeted activation** method. Within this budget, our method prioritizes MCD activation on decoding steps with high flip susceptibility. We evaluate our method across diverse combinations of *six MCD methods, four MLLM backbones, and five benchmarks*, spanning visual and audio-visual tasks. Across these settings, budgeted activation integrates with existing MCD methods to achieve a maximum reduction of **87% in inference latency**, while preserving their performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
MLLM, multimodal contrastive decoding, inference efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_LvE3OpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes an efficient inference strategy for multimodal contrastive decoding. The authors propose analyzing token-level changes based on the top-2 logit margin. Based on the margin, they selectively activate the auxiliary forward pass under a user-specified budget.

Critical strengths
  1. The design is reasonable, as tokens with a large vanilla top-2 margin are unlikely to be changed by MCD.
  2. The proposed budget is useful for practical systems.
  3. The paper honestly acknowledges that MCD does not always improve performance and that the proposed method only determines whether to activate MCD.
Critical weaknesses
  1. The proposed method is essentially a more budget-friendly version of MCD, without improving performance. Moreover, it relies only on the budget, and without a clear performance preference, it is difficult to determine the appropriate budget in practice. Therefore, the contribution is limited.
  2. The intuition behind the method is simply that low-confidence (small top-2 margin) tokens are more likely to change. This simple idea is expressed through complex mathematical definitions without theoretical contribution.
Questions for authors
  1. Can the method be improved to provide an indication of potential accuracy improvement, rather than only controlling the computational budget?
  2. Can this method be applied only to MCD, or can it be extended to other settings?
Reviewer_9RdCOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a training-free budgeted activation method to reduce the inference cost of multimodal contrastive decoding (MCD). The authors observe that MCD only changes the selected token at a small fraction of decoding steps. Based on this observation, they introduce FLIPTRACE to analyze token flips and derive a flip-immunity condition. They then use the vanilla top-2 logit margin to estimate flip susceptibility and selectively activate the auxiliary branch under a user-specified budget. An online calibration strategy is used to control the activation rate. Experiments show that the proposed method can reduce inference latency while largely preserving the performance of full MCD.

Critical strengths
  1. The paper addresses a practical efficiency problem in multimodal contrastive decoding. Reducing unnecessary auxiliary computation is meaningful for improving inference speed.

  2. The proposed method is training-free and can be easily integrated with existing MCD methods without changing their original designs.

  3. The experiments cover multiple MCD methods and different multimodal tasks. The reported results show promising latency reductions while largely preserving the performance of full MCD.

Critical weaknesses
  1. The theoretical contribution is trivial. The theoretical analysis is quite simple and provides limited new insight. Proposition 1 basically states that a token cannot be flipped if its original margin is larger than the maximum relative gain of other tokens. Corollary 1 directly follows from this observation. I think these results are straightforward and the theoretical contribution is limited.

  2. Limited connection between the theory and the actual method. The theoretical analysis relies on both vanilla and auxiliary logits, but the actual method only uses the vanilla top-2 margin. Although the authors provide some intuition and empirical evidence for this choice, the theory does not show that selecting tokens with smaller margins can reliably identify the steps where MCD would change the prediction. Therefore, the theoretical results provide limited support for the actual activation strategy.

  3. Lack of sequence-level analysis. The paper mainly analyzes token-level flips, especially through teacher-forced replay. However, missing a flip during autoregressive generation may change the following tokens and lead to a different output sequence. Although the authors acknowledge this limitation, the paper lacks a direct analysis of how closely the generated outputs match those of full MCD. I would like to see more sequence-level comparisons.

  4. Writing suggestion: The experimental setup should be presented more clearly in Section 5.1. For example, the authors mention four MLLM backbones but only provide citations without explicitly listing their names. I suggest including the specific model names and benchmark names in this section.

  5. Incomplete evaluation on POPE. The authors only report F1 scores for POPE, while Accuracy is also an important metric for evaluating hallucination mitigation. More importantly, Table A18 shows that negative flips account for 63.5% of all flips, resulting in a negative net effect on correct predictions under greedy decoding. This raises concerns about whether the reported F1 improvements come at the cost of lower Accuracy. I suggest the authors report both Accuracy and F1 for all POPE experiments and clarify the relationship between Table A18 and the main results. Additionally, since the proposed method aims to recover as many MCD flips as possible, I wonder whether maximizing flip recall is an appropriate objective when a large portion of these flips are negative. The authors should further explain how flip recall relates to actual performance improvement.

  6. Evaluation on outdated models. The experiments mainly rely on very old MLLMs, such as LLaVA-1.5 and Qwen2.5-VL, and all four backbones are limited to the 7B scale. For an ICLR 2027 submission, I would expect evaluations on more recent models, such as Qwen3-VL-Instruct or Qwen3.5. More importantly, it remains unclear whether the original MCD methods are still effective on these newer and stronger models. If MCD itself provides limited benefits on modern MLLMs, the practical value of accelerating it also becomes questionable.

Questions for authors

Please refer to the weaknesses.

Reviewer_yepnOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a training-free budgeted activation method to reduce the inference latency of executing the auxiliary branch at every generation step in Multimodal Contrastive Decoding (MCD). The authors observe that MCD changes vanilla token selections at only a subset of decoding steps. The authors introduce the theoretical framework FLIPTRACE and use the vanilla model’s top-2 logit margin and the auxiliary branch’s relative contrastive gain to derive a flip-immunity condition under which MCD cannot change token selection. Based on this analysis, flip susceptibility is computed using only vanilla logits to estimate token-flip likelihood and select activation steps. The auxiliary branch is selectively activated according to a user-specified budget, with warmup and online threshold calibration adjusting the actual activation rate. Incremental prefill efficiently updates the auxiliary KV cache for skipped steps. Experiments span six MCD methods, four MLLM backbones, and five multimodal benchmarks. The authors report up to an 87% reduction in inference latency relative to full-activation MCD while largely maintaining or achieving some performance improvements across various settings. Comparisons with random, entropy-based, and confidence-based activation strategies evaluate the usefulness of flip susceptibility.

Critical strengths
  1. The proposed method has the advantage of requiring no additional training and being easy to apply without modifying the auxiliary branch structure of existing MCD methods. In addition, experiments across 6 MCD methods, 4 MLLM backbones, and various visual and audio-visual tasks demonstrate the generality and practicality of the proposed method.

  2. Experiments across various benchmarks and activation budgets show that the method can substantially reduce inference latency while largely maintaining or achieving some improvements in the performance of existing MCD. In particular, the authors measure actual wall-clock latency and compare the method with random, entropy-based, and confidence-based activation methods under the same budget, empirically supporting the effectiveness of the proposed method.

Critical weaknesses
  1. FLIPTRACE’s flip-immunity condition depends on both the vanilla margin and the auxiliary branch’s relative contrastive gain, but the latter is unavailable when making activation decisions. The actual flip susceptibility score is simply a monotonic transformation of the top-2 margin, producing the same ranking as conventional margin-based selection. Although Appendix C.1.4 and Figure 2 provide supporting analyses, they do not fully establish whether this ranking reliably predicts flips across different MCD methods and contrastive strengths. Therefore, despite the reported efficiency improvements, I am not fully convinced that FLIPTRACE provides a sufficiently novel theoretical or methodological contribution beyond existing uncertainty-based selective decoding.

  2. The proposed method performs full MCD during the first 200 generated tokens on POPE and maintains the calibration state across the benchmark stream. However, Table 1 reports latency for 40 short POPE responses alongside a realized activation rate of 10.3% at B=10%, without clearly specifying whether these measurements were conducted after warmup or from a fresh calibration state. If the 40 responses were processed from a fresh state with fewer than 200 generated tokens in total, the activation rate should be 100%. Although Appendix D.1 discusses benchmark-level budget tracking, how the warmup is reflected in the activation-rate and latency measurements in Table 1 remains unclear. This makes it difficult to assess the reproducibility and practical significance of the reported efficiency improvements, particularly for short or independently processed requests.

Questions for authors

Could you explain to what extent the assumptions used in Appendix C.1.4 to justify the margin-based activation score hold across different MCD methods and contrastive strengths? In particular, I would like to know how the flip recall of margin-based rankings changes when the magnitude of the changes induced by the auxiliary branch varies, and how FLIPTRACE’s analysis explains the scope of validity and limitations of the actual activation policy. In addition, regarding the setting in which full MCD is executed for the first 200 generated tokens on POPE, could you clarify whether the latency in Table 1 and the actual activation rate of 10.3% at B=10% were measured on the same 40 responses, or whether the activation rate was aggregated over the entire benchmark stream? I would also like to know the total number of generated tokens in those responses, the calibration state at the start of measurement, and whether the warmup tokens and time were included in the activation rate and latency calculations, respectively.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.