- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies how visual-token pruning comparisons depend on the intervention and the rule used to score answers. It fixes each selector's retained positions and compares actual outputs with choices restricted to an allowed answer set. An excluded-correct event occurs when the restricted choice is correct but the actual answer falls outside the allowed set. J compares the selectors’ zero-minus-delete changes in these events, while L averages those changes. LLaVA shows substantial readout sensitivity, while Qwen often shows shared failures despite small changes in the selector comparison. A separate analysis of receipt traces compares free generation, token-by-token selection within a candidate trie, and whole-string probability ranking.
The experiments are carefully done, and I find the reporting approach useful. I lean toward rejection because the clearest single-token results come from cases where zeroing causes the model to produce mostly invalid answers. Outside this setting, I am less convinced that the choice of readout meaningfully changes how we compare selectors. I could move toward acceptance if the receipt analysis shows a reliable, practically meaningful change in the selector comparison, or the chance comparison clarifies what the exclusion measures add beyond accuracy and valid-output rates.
Critical strengths
-
The authors explain their experimental controls clearly. They keep the same token positions across interventions, include invalid outputs in the scores, and preserve the paired comparisons when estimating uncertainty. They also explain when masking and deletion should give the same result and report differences caused by numerical precision. I appreciate that they disclose both the TextVQA task being easier than planned and the implementation fix in the native-choice study. These checks and disclosures give me more confidence in the reported comparisons.
-
The breakdown helps explain why the scores change. For LLaVA, Table 2 shows how the comparison between selectors depends on how answers are chosen. The receipt analysis also distinguishes choosing an allowed answer one token at a time from choosing the most likely complete answer. The authors check these reconstructed decisions against saved traces and make clear that a correct restricted choice does not fix the answer the model originally generated. This makes it easier to interpret what the readout differences actually show.
Critical weaknesses
-
I am not yet convinced how much the audit adds in practice for single-token answers. On the OCRBench-v2 confirmation subset, zeroing gives 1.40% accuracy for actual outputs and 20.75% when choices are restricted to allowed answers, giving the reported 19.35-point shared shift (Table 29). Section 5.6 shows that the restricted accuracy is compatible with the 21.80% expected from uniform guessing. The gap is measured correctly, but both ways of choosing answers still perform poorly. What does reporting this gap tell us about the selectors beyond their accuracy and how often they produce valid answers? I would also make the chance comparison more prominent when presenting the headline result.
-
The evidence is less clear when we look beyond zeroing. The historical LLaVA panel has no excluded-correct events under mean replacement (Table 5; Appendix L.2), but the newer Qwen single-token experiments do not include this operator. The older Qwen TextVQA experiments do, but they intervene at a different point in the model and score complete answers (Appendix AF). The receipt study helps extend the analysis beyond single-token answers, although both planned interaction tests have intervals that include zero (Table 31). One result worth looking at more closely is MonkeyOCR under deletion. High-norm selection scores about 9.80 points above DivPrune with exact-match scoring, but 0.51 points below it when answers are chosen through the trie (Table 32). Without uncertainty estimates for these gaps and their difference, it is hard to tell how much weight to put on this change.
Questions for authors
Questions 1 and 2 could change my rating. Question 3 concerns a limited additional experiment that would help assess how broadly the findings apply.
-
How do the observed exclusion effects compare with what would be expected from uniform guessing among the allowed answers when the actual output is invalid? Would this comparison change how you interpret J and L?
-
How reliable is the roughly 10.31-point change in the high-norm versus DivPrune gap between exact match and trie decisions under deletion (Table 32)? A paired bootstrap interval for this change, resampling receipt groups, would help assess whether the readout meaningfully changes the MonkeyOCR comparison.
-
Does the absence of excluded-correct events under mean replacement in LLaVA also hold in the newer Qwen single-token setting?
Response to reviewer o2ft (Part 2/2)
(Part 2/2 — Continued Response to Reviewer o2ft)
3. Why Candidate-Decision Loss Is Large in Qwen Deletion-vs.-Zero Comparisons (Addressing Q2)
On the 429 OCRBench-v2 confirmation questions at Site 2, the 66.55 pp actual accuracy drop from deletion (67.95) to zeroing (1.40) partitions into 47.20 pp of candidate-decision loss (Q drops from 67.95 to 20.75) plus 19.35 pp of excluded-correct loss (R) (and on A-OKVQA, 86.88 pp=61.63 pp+25.25 pp). Our probability-mass and chance-decomposition diagnostics explain why both components occur together:
Cis chosen on 48.95 / 47.09 of questions vs. 25.41 / 25.87 under deletion). As a result, restricted candidate accuracy Q=20.75 falls right to the uniform-guessing expectation (21.80), decomposing R=19.35 pp into 20.98 pp of uniform-chance credit and −1.63 pp of above-chance excess. Thus, zeroing simultaneously destroys within-set visual discrimination (producing the 47.20 pp candidate loss) and valid-token emission (producing the 19.35 pp exclusion shift).4. Scope for Open-Ended Generation Tasks Beyond Receipts (Addressing Q1)
Whenever an open-ended or semi-structured VLM task is evaluated by comparing unconstrained generation against a finite candidate set, rubric, or constrained vocabulary (such as entity extraction, key-value OCR, classification-by-generation, or multiple-choice reasoning), the exact prefix-trie identity A=EQ applies and shared/differential exclusion shifts can arise under physical deletion—as demonstrated by the 19.07 exclusion rates on WildReceipt. For unconstrained free-form description tasks scored by continuous similarity without a discrete candidate set, single-step exclusion does not apply directly; instead, FIRA's broader principle—reporting absolute task outcomes and valid-format compliance alongside relative selector differences—remains essential. We have made this boundary explicit in Section 6.
We thank you again for your supportive review and hope these new native experiments and analyses further reinforce your confidence in the paper.