acceptodds
Under review as a conference paper at ICLR 2027

Your Filter Picks the Winner: Predictable Selection Bias in Difficulty-Filtered Evaluation of Visual Token Compression

Abstract

A visual-token-compression benchmark scores methods only on items its "difficulty filter" (downsampling) fails. Phang et al. (2022) found adversarially filtered rankings unstable; we derive the distortion. In a threshold model, one-dimensional difficulty makes filtering the rescaling : rankings above the filter's accuracy survive, its not unfair. With an operator-specific component it decreases strictly, at fixed accuracies, in the tetrachoric correlation of the two failure patterns: a filter measures dissimilarity from itself, invertibly given the marginals. On 1.1M generations from six VLMs, difficulty is not one-dimensional (–; – net of cell-level noise on the open models), and that correlation spreads by –, net of sampling noise, over a filter's competitors, which, by the theorem, can reorder them. The hard-subset winner departs from the unfiltered order in 11–17 of 45 cells per model against 6–8 by sampling, significantly on both LLaVA models (Qwen2-VL: ). On VTC-Bench's model and filter, its conclusion that designed methods beat random selection holds at four of five budgets, filtered or not; a compressor filter can reverse it. The bias is choosable: a method's most flattering filter, picked on half the items, lifts its held-out rank by +0.9 to +1.0 (open models). It buys nothing: a matched-size random subset tracks the unfiltered ranking better on all six models (– against –) and creates at most 1.5% unsupported orderings, inverting almost none, versus the failure set's 2–11%. Sample uniformly; for hard items, cross-fit a panel. Code, raw generations and analysis scripts will be released at https://github.com/xxx/xxx upon acceptance.

Primary area:
datasets and benchmarks
Keywords:
visual token compression, vision-language models, difficulty filtering, evaluation protocol, selection bias, benchmark design, multimodal evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_FGCzOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper models visual-token-compressor correctness with a two-component threshold model. Theorem 1 shows that when visual difficulty is one-dimensional, evaluating on a filter's failure set is a monotone rescaling that preserves the ranking. Theorem 2 adds an operator-specific component and expresses the filtered accuracy as a closed form in the two marginals and the tetrachoric correlation of the failure patterns, strictly decreasing in that correlation. The authors estimate a one-dimensionality ratio κ on roughly 1.1M generations from six VLMs, report κ between 0.51 and 0.66, show that the hard-subset winner depends on which operator is used as the filter, audit VTC-Bench on its own model, and recommend either uniform subsampling or a cross-fitted panel.

Critical strengths

None.

Critical weaknesses
  1. No empirical content: given fixed marginals, filtered accuracy is just an invertible re-encoding of the unfiltered table, so the model is unfalsifiable and its only real claim is already known.

  2. The challenged conclusion survives: best-method-minus-random is +0.08→+0.01 filtered versus +0.07→0.00 unfiltered, differing by ≤0.01 and flipping sign in just 1 of 45 cells.

  3. The existing remedy already suffices: ensemble filtering cuts wrong significant orderings to 0.00–0.38% (versus 0–6%), and cross-fitting improves only one of the two estimands.

  4. The headline quantity is ill-defined: κ̂ ranges 0.417–0.866 with pool choice alone, yet the abstract reports only the lowest variant and still recommends reporting it.

  5. The headline findings are weak: key p-values are 0.11, 0.38 and 0.99, only a minority of cells yield a significant winner, and genuine inversions number 1, 3 and 1.

  6. No estimand is declared: the criterion is agreement on the very contaminated pool filtering exists to correct, and the recommendation reverses between the marginal and hard-item orderings.

  7. Scope and setup are inadequate: heterogeneous 7B arms on different tasks, a nominal budget axis, conceded-false assumptions, clipped estimates, and a placeholder repository.

Questions for authors

What observation would falsify M2, given Proposition 1? Which κ̂ should a benchmark report, and who chooses the pool definition when it moves the number by 0.4? Could you rerun Sections 5.2 and 5.3 with the four published implementations, on all three models?

1 comment

The authors provided no further explanation or response, so I am maintaining my original score.

Reviewer_QbgoOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This work aims to study a framework for adversarially testing visual compression methods by selecting subsets of evaluation datasets that are challenging. Challenging can be defined in various ways. One particular way they study it is through models that do filtering of the dataset.

Critical strengths

None

Critical weaknesses

The paper is unclear, does not set up the problem, nor define key terms.

Writing is unprofessional. See below for examples:

  • “(Theorem 1): it preserves the ranking of every method above AF , ties everything below at zero, and awards the filter 0%, so its 0.0 row is correct, not a scandal.”

  • “The threshold structure is real and, on LLaVA-1.5, the aggregate curves align under a per-operator rescaling, yet the item-level failure sets disagree by enough to reorder a leaderboard: at a fixed filter the correlation ρOF spreads by 0.06–0.15 over the competitors it ranks, net of sampling noise, a spread that survives removing the cell-specific residual, which leaves κδ = 0.78–0.88.”

  • The filtered protocol of Liao et al. (2026) has three steps, all in its released code. 5.3 THE PROTOCOL AS PUBLISHED, ON THE ARCHITECTURE IT WAS PUBLISHED FOR

  • "The contribution is a measurement framework and an audit, not new mathematics."

Moreover, The contributions are not novel and obvious. --> “ A threshold model of visual sufficiency with a measurable one-dimensionality ratio κ, and a theorem that under one-dimensionality difficulty filtering rescales without reordering the methods it does not tie, so “circular” is not “unfair” “

Why is CLS-attn in Figure 1 left negative?

Questions for authors

Why is CLS-attn in Figure 1 left negative?

Reviewer_GCHAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates selection bias in difficulty-filtered evaluation of visual token compression, where methods are tested only on examples that a model answers incorrectly after applying a chosen filter, such as downsampling. It introduces a two-component threshold model that separates shared item difficulty from operator-specific effects. The model explains how accuracy on this filtered subset depends on the method’s and filter’s accuracies on the unfiltered pool, as well as how closely their failure patterns overlap, measured by tetrachoric correlation. Experiments with three open VLMs and three API models show that compression-method rankings depend on which operator selects the evaluation examples. Random subsets of the same size better preserve the unfiltered ranking. For evaluating methods on difficult examples, the paper proposes XFilter, which selects these examples using a panel of operators, excluding the evaluated operator and others from the same mechanism family

Critical strengths
  1. The paper provides convincing evidence of filter dependence. The authors compare filtered method rankings with rankings from random subsets of the same size, which helps to separate the impact of filtering from ordinary sampling variation. They also show that each filter produces reasonably consistent rankings across two halves of the data, while different filters produce different rankings.
  2. The paper also explains when filtering can preserve or change rankings. Theorem 1 shows that under one-dimensional difficulty, methods whose unfiltered accuracy exceeds the filter’s unfiltered accuracy retain their order, while methods at or below that level are tied at zero. Theorem 2 shows how differences in the correlation between each method’s failure pattern and the filter’s failure pattern can change rankings, even when their marginal accuracies are fixed.
Critical weaknesses
  1. The benefit for ranking prediction is limited. In Section 5.4, the held-out-operator fit improves filtered-accuracy prediction but does not predict method rankings better than a constant correlation, ρ = κ̂. This leaves its additional practical value for ranking prediction unclear.
  2. Random selection significantly outperforms the best designed method in only one of 45 tested filter-budget settings on Qwen2-VL. This result uses stand-ins and should be checked with the published forms of the compressors. The published-form experiments on LLaVA-1.5 already show that some ranking inversions disappear, although filter instability remains
Questions for authors
  1. In Section 5.4, the held-out-operator fit improves filtered-accuracy prediction but does not improve ranking prediction over a constant ρ = κ̂. Can you identify settings where predicting correlations from mechanism family and budget gives a useful improvement in ranking prediction?
  2. Could you repeat the Qwen2-VL comparison using the published forms of FastV, VisionZip, PruMerge+ and DART? In particular, does random selection still significantly outperform the best designed method with DivPrune as the filter at 99% token reduction, and how does this compare with the downsampling filter?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.