- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper models visual-token-compressor correctness with a two-component threshold model. Theorem 1 shows that when visual difficulty is one-dimensional, evaluating on a filter's failure set is a monotone rescaling that preserves the ranking. Theorem 2 adds an operator-specific component and expresses the filtered accuracy as a closed form in the two marginals and the tetrachoric correlation of the failure patterns, strictly decreasing in that correlation. The authors estimate a one-dimensionality ratio κ on roughly 1.1M generations from six VLMs, report κ between 0.51 and 0.66, show that the hard-subset winner depends on which operator is used as the filter, audit VTC-Bench on its own model, and recommend either uniform subsampling or a cross-fitted panel.
Critical strengths
None.
Critical weaknesses
-
No empirical content: given fixed marginals, filtered accuracy is just an invertible re-encoding of the unfiltered table, so the model is unfalsifiable and its only real claim is already known.
-
The challenged conclusion survives: best-method-minus-random is +0.08→+0.01 filtered versus +0.07→0.00 unfiltered, differing by ≤0.01 and flipping sign in just 1 of 45 cells.
-
The existing remedy already suffices: ensemble filtering cuts wrong significant orderings to 0.00–0.38% (versus 0–6%), and cross-fitting improves only one of the two estimands.
-
The headline quantity is ill-defined: κ̂ ranges 0.417–0.866 with pool choice alone, yet the abstract reports only the lowest variant and still recommends reporting it.
-
The headline findings are weak: key p-values are 0.11, 0.38 and 0.99, only a minority of cells yield a significant winner, and genuine inversions number 1, 3 and 1.
-
No estimand is declared: the criterion is agreement on the very contaminated pool filtering exists to correct, and the recommendation reverses between the marginal and hard-item orderings.
-
Scope and setup are inadequate: heterogeneous 7B arms on different tasks, a nominal budget axis, conceded-false assumptions, clipped estimates, and a placeholder repository.
Questions for authors
What observation would falsify M2, given Proposition 1? Which κ̂ should a benchmark report, and who chooses the pool definition when it moves the number by 0.4? Could you rerun Sections 5.2 and 5.3 with the four published implementations, on all three models?
The authors provided no further explanation or response, so I am maintaining my original score.