acceptodds
Under review as a conference paper at ICLR 2027

Where You Score Matters More Than What You Score

Abstract

Token reduction in vision transformers decides which tokens to keep, merge, or drop. Most existing methods rank tokens by a score computed once, and assume the ranking is independent of the reference set. But the same scoring rule produces different rankings when the reference set changes. We call this the reference-set shift. Changing the reference changes the outcome: relative to full-reference scoring, scoring on the candidate set improves Top-1 on ImageNet-1K by points, with a mean normalized loss improvement of . Because the task loss is a set function, a token's marginal effect depends on interactions with tokens that will be removed, and this dependence is not recoverable from the token's own features. Recovering it requires re-evaluating the candidate set, and matching-based methods do exactly that: they re-evaluate as the set shrinks, so they succeed where scorers computed once cannot. The reference set is therefore an integral part of the method.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Token reduction, Vision transformers, Token pruning and merging, Token importance, Candidate set, Reference-set shift

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_BbhKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies reference-set dependence in token reduction for ViTs. The authors define token importance as the marginal loss induced by deleting a token from a reference set, distinguishing between di(V)d_i(V), computed on the full token set, and di(C)d_i(C), computed on the candidate set. The paper shows that these two reference sets can yield different rankings, thereby affecting the quality of the retained subset. It further argues that re-evaluating the surviving tokens during reduction is more effective than scoring them only at the initial state.

The experiments primarily cover DeiT-III Small and DINOv2 ViT-S/14, comparing candidate-set construction strategies, loss definitions, and token budgets on the full ImageNet-1K validation set. They also include overlap interventions with a fixed reference-set size, learned-scorer experiments, and controlled comparisons with ToMe. Additional validation is conducted on ImageNet-V2, segmentation, depth estimation, and a larger backbone. Overall, the results support the conclusion that reference choice affects token selection quality. However, the evidence remains insufficient to establish the general limitations of per-token scoring or to identify re-evaluation as the primary source of ToMe’s advantage.

Critical strengths

1. The paper explicitly identifies reference choice as a decision variable in token reduction and provides targeted experiments.

Beyond comparing d(V)d(V) and d(C)d(C) the paper varies the overlap between the reference set and the candidate set while holding the reference-set size fixed. This is more convincing than directly comparing two sets of different sizes. The finding has practical implications for understanding existing token importance measures.

2. The main empirical findings are supported by broad validation and reasonably rigorous statistical analysis.

The experiments use all 50,000 ImageNet-1K validation images and cover two backbones, two loss definitions, four candidate-set construction strategies, and three token budgets. Candidate-set scoring is favored in 45 of the 48 configurations, with an aggregate normalized advantage of 0.305. The paper also reports confidence intervals and validation across distributions and tasks. These results make it difficult to attribute reference-set shift to an isolated model or configuration, providing reasonably reliable support for the empirical findings.

Critical weaknesses

1. The current proofs and learning experiments do not support the theoretical claims about scorer expressivity.

Propositions 2 and 3 are important components of the theoretical argument, but the main text primarily provides construction sketches, while the appendix does not sufficiently detail the corresponding loss constructions or proofs of the key inequalities. More importantly, under conditions such as a specified feature map and identical features for certain candidate tokens, Proposition 3 shows that a shared per-token scoring rule of the form si=g(hi)s_i=g(h_i) cannot guarantee correct selection across all constructed instances. This worst-case result does not directly establish that learned scorers operating on actual ViT features generally cannot recover di(C)d_i(C), nor does it directly constrain a set-attention scorer with access to the entire candidate set.

Nevertheless, the paper also interprets the poor performance of the latter as an empirical manifestation of this proposition. The learning experiments use only approximately 300 images per backbone, which is insufficient to rule out the effects of training data, model capacity, or optimization. Therefore, the paper cannot move from “reference-set dependence” to the stronger conclusion that “improving the scorer cannot solve the problem in principle.”

2. The existing controls are insufficient to attribute ToMe’s advantage primarily to re-evaluation while discounting the contribution of merged content.

In Table 1, ToMe merges the full patch sequence, whereas the other methods prune only within a candidate set of 96 patches. An equal final token count does not imply equal information sources, reduction procedures, or computational budgets. This system-level comparison therefore establishes a performance difference but cannot, by itself, identify re-evaluation as its cause.

The merged-content ablation also fixes contextual keys, matching edges, and reduction rounds. This tests the direct effect of preserving merged content under a fixed matching process, but it simultaneously blocks the potential pathway through which merged content changes subsequent features and matching decisions. The conclusions therefore overlook the importance of preserving merged content in this setting. The conclusions drawn from the audit of EViT attention scoring should also be limited in scope: the original EViT includes token reorganization within layers and fusion of inattentive tokens, rather than simply performing one-shot attention-based selection.

Questions for authors
  1. How are the set losses, feature maps, and interaction terms defined explicitly? How can approximate additivity, ranking reversal, and a non-vanishing excess loss be shown to hold simultaneously?

  2. What is the scope of Proposition 3 with respect to practical learned scorers? Can hih_i contain contextual information, candidate-set identity, or survival status? Does a set-attention scorer with access to the entire candidate set belong to the function class constrained by this proposition? If not, how should its failure be connected to the proposition?

  3. Could the negative results for learned scorers be limited by the data scale or training configuration? Please clarify the training objectives, data splits, hyperparameter selection, and convergence behavior, and provide evidence sufficient to assess whether data availability or model capacity is a primary limitation. Please also explain the discrepancy between the ridge correlation coefficients reported in the two sections.

  4. When merged content is allowed to affect subsequent features and matching decisions, how does merging perform relative to dropping?

  5. Which experimental configurations and aggregation procedure produce the 10.6% Top-1 improvement reported in the abstract? Please report the selection results for teacher-KL and ground-truth-based label-NLL separately, together with the corresponding unnormalized accuracy values.

2 comments

We sincerely appreciate your time and valuable comments, which help us significantly improve our paper.

Explicit Constructions and Scope of Propositions 2 and 3

Appendix C provides full proofs and exact definitions for Propositions 2 and 3. The revision defines the set losses, feature representations, and token interactions directly. It presents the simultaneous derivation of near-additivity, ranking inversion, and non-vanishing excess loss. Proposition 2 constructs a mirror pair of candidate-conditional edges. Proposition 3 constructs two candidates with identical features that produce opposite ranking orders under the two references.

Proposition 3 applies only to scorers of the form si=g(hi)s_i = g(h_i) that use fixed per-token features. It says nothing about scorers that process candidate-set context, survival status, or dynamic features. The set-attention scorer runs as a separate test outside this function class. The text now clarifies this exact boundary.

Scale Analysis and Discrepancy Clarification for Learned Scorers

We evaluated learned scorers on 300, 1,000, 3,000, 10,000, and 30,000 images per model to measure the effect of training volume. Rank correlation with exact d(C)d(C) stays near 0.05 across all training sizes on DINOv2. At 30,000 images, the learned scorer scores below full-reference scoring, with normalized differences of −0.016-0.016 [−0.036-0.036, 0.0030.003] on teacher-KL and −0.319-0.319 [−0.355-0.355, −0.282-0.282] on label-NLL. Appendix A lists the training objectives, splits, seeds, and hyperparameters for every model scale.

The two ridge correlation values reported earlier reflect different aggregation levels. The numbers 0.0130.013 and 0.0110.011 measure the mean per-candidate Spearman correlation. The numbers 0.0300.030 and 0.0170.017 measure the Spearman correlation evaluated over the full 96-token ranking. The revised manuscript defines both metrics and uses the candidate-set-level statistic consistently.

Dynamic Feature Updates and Content Preservation in ToMe

The new experiment allows merged features to update subsequent keys and matching decisions directly. It tests four conditions under one matched layer-wise schedule. When both conditions use dynamic re-matching, merging beats dropping by 0.0960.096 at k=32k=32, 0.0670.067 at k=64k=64, and 0.0630.063 at k=80k=80 on DINOv2. Dynamic re-matching produces larger differences over the frozen baseline, reaching 0.2160.216, 0.1560.156, and 0.1500.150 across the same budgets. The revision reports all four conditions across both backbones.

Clarification of the Top-1 Claim in the Abstract

The +10.6+10.6 Top-1 claim in the abstract came from an early proof-of-concept experiment and was included by mistake. We removed this claim from the revised abstract and main text. The revision reports teacher-KL and label-NLL results separately, listing unnormalized Top-1 accuracy for all methods. In Table 1, ToMe outperforms d(C)d(C) on teacher-KL Top-1 at k=32k=32, scoring 0.7770.777 against 0.6770.677 on DeiT-III and 0.7430.743 against 0.5900.590 on DINOv2. The new label experiment adds full validation results for label-NLL, reporting unnormalized Top-1 across both loss functions.

Sorry for the late response. We hope our response and the revised manuscript address your concerns. We remain open to any further discussion.

Thank the authors for their detailed response and supplementary experiments. Overall, these additions strengthen the paper.

Reviewer_oXeVOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how the reference set affects the marginal-loss scores used for token reduction in vision transformers. It compares scoring on the full token set V with scoring on the candidate set C while holding the candidate pool and selection budget fixed. Candidate-set scoring generally yields better subsets on ImageNet-1K, and iterative re-evaluation further improves fidelity. The paper also presents worst-case results for independent per-token scorers and uses ToMe comparisons to argue that the reduction process, rather than preservation of merged content, explains the benefit of matching-based reduction.

Critical strengths
  1. The controlled reference-set comparison is interesting. Holding the scoring rule, candidate pool, and budget fixed makes a convincing case that the reference itself can change token selection and model performance.

  2. The empirical study goes beyond a single setting. The authors test multiple losses, candidate-set constructions, and budgets, report confidence intervals, and include ImageNet-V2 and dense-prediction probes. I also appreciate that they report the GLOBAL-KL cases where candidate-set scoring does not help.

Critical weaknesses
  1. The ToMe experiments do not isolate the claimed cause of its advantage. In Table 1, the one-shot methods prune from an attention-selected pool of 96 tokens, whereas ToMe merges the full sequence across transformer blocks. Matching the final token count does not control for these differences. The content-preservation ablation also freezes the matching edges and schedule, excluding the possibility that merged features change subsequent matching decisions. Its near-zero measured difference therefore supports a narrower conclusion than the claim that preserving merged content adds nothing. The original ToMe paper reports a substantial gap between weighted averaging and keeping one token (84.25% vs. 81.01%, Table 1(d)). Although the settings differ, this deserves discussion.

  2. The broader claims are not yet established by the theory or evaluation scope. Proposition 3 concerns scorers of the form (s_i = g(h_i)) with fixed per-token features; it does not rule out candidate-conditioned scoring or features recomputed in context. The contextual-scorer experiment uses only 300 training images per model, one candidate-set construction, and one budget. Further, the evaluation covers two ViT model families (DeiT-III and DINOv2), with the larger-model check remaining within DINOv2. This makes it difficult to judge how far the conclusions extend. Recent visual-token reduction work increasingly targets VLMs, including FastV, VisionZip, and MetaCompress. In VLMs, token utility may depend on the text query, cross-modal processing, and later dialogue turns. The paper acknowledges extensions beyond ViTs as open, but its broader conclusions about token reduction would benefit from a clearer qualification of this scope.

Questions for authors
  1. Could you separate dynamic matching from content aggregation under a matched layer-wise reduction schedule, allowing future matching decisions to respond to updated token features? How does this compare with the content-aggregation ablation in the original ToMe paper?

  2. Is Proposition 3 intended only for fixed-feature independent scorers, or also for candidate-conditioned scorers? Does the set-attention result persist with more training data and different candidate sets?

  3. Do you expect reference-set shift to behave similarly in VLMs, where token importance can depend on the text query? Even a small diagnostic across different queries, or a more explicit discussion of where the current analysis might fail, would help establish the significance of the finding beyond standalone vision backbones.

1 comment

Thank you for your careful reading and encouraging comments.

Controlled ToMe Comparison and Content Preservation

The new experiment matches the candidate pool and the layer-wise schedule across four conditions. It tests token merging and token dropping, both with dynamic re-matching and with keys frozen at block zero. This setup lets merged features update subsequent matching decisions directly. On DINOv2, dynamic matching outperforms the frozen baseline by 0.2160.216 [0.2050.205, 0.2260.226] at k=32k=32, 0.1560.156 [0.1460.146, 0.1670.167] at k=64k=64, and 0.1500.150 [0.1400.140, 0.1610.161] at k=80k=80. Merging features exceeds dropping under dynamic matching by 0.0960.096 at k=32k=32, 0.0670.067 at k=64k=64, and 0.0630.063 at k=80k=80. Re-evaluation accounts for the larger share of the gain.

The original ToMe paper reports an 84.25%84.25\% versus 81.01%81.01\% difference between weighted averaging and keeping one token. That evaluation pruned an uncompressed backbone without fixing candidate pools. Under our matched schedule, content preservation produces a positive effect, but dynamic re-matching drives the larger difference on DINOv2. The revised text discusses this comparison directly.

Scope of Proposition 3 and Data Scale in Learned Scorers

Proposition 3 applies only to fixed-feature per-token scoring rules of the form si=g(hi)s_i = g(h_i). It says nothing about candidate-conditioned scorers. The revision states this exact scope in the text.

The set-attention scorer runs as a separate test outside Proposition 3. We evaluated learned scorers across an increasing ladder of 300, 1,000, 3,000, 10,000, and 30,000 images per model. Rank correlation with exact d(C)d(C) stays near 0.05 across all training sizes on DINOv2. At 30,000 images, the learned scorer yields normalized differences of −0.016-0.016 [−0.036-0.036, 0.0030.003] on teacher-KL and −0.319-0.319 [−0.355-0.355, −0.282-0.282] on label-NLL over full-reference scoring. Additional training data does not change the set-attention result.

Reference-Set Shift in Vision-Language Models

The reference-set shift persists in multimodal models. We evaluated LLaVA-1.5-7B, InternVL2.5-8B, and Qwen2.5-VL-7B across 1,000 images with five text queries per image. Cross-query rank correlation of token importance averages 0.097 [0.093, 0.100] for LLaVA, 0.048 [0.045, 0.051] for InternVL, and 0.103 [0.100, 0.107] for Qwen. Same-category queries produce higher correlations than cross-category queries. Changing the text query reorders visual token importance across all three architectures.

Sorry for the late response. Thank you once again for your insightful comments that helped improve our paper. We look forward to hearing your thoughts on our response.

Reviewer_A6D8OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates how the refeerence set used to evaluate token importance affects token pruning in VITs. It compares marginal contributions computed on the full token set and the candidate set and studies iterative re-evaluations. While the underlying idea is intuitive, the presentation is difficult to follow, and the novelty and the practical significance of the contribution are unclear.

Critical strengths

The idea is intuitive and the experiments cover multiple models and datasets.

Critical weaknesses
  • Limited novelty and insuffiecient related work: The central ideas of context-dependent importance and iterative subset selection seem well established. DynamicVIT performs progressive pruning, Beyond Attentive Tokens (Long et al., 2023) considers token redundancy diversity, and V-Pruner (Yao et al. 2026) explicitly models dependencies between pruning decisions. While the specific reference-set analysis might be novel, the paper does not convincingly differentiate its contributions from these existing approaches. The related work section is surprisingly short and misses several related works and a substantive discussion of new contributions by this paper.

  • Incomplete experimental validation and questionable practical relevance: The evaluation lacks comparisions against established token-reduction methods and a meaningful accuracy-efficiency trade-off analysis. This would be of high relevance to compare methods and support the claim of existing methods being not optimal. Additionally, the reported timings of 60ms for evaluating token marginals versus 11ms for a standard forward pass raises the question of the benefit. If efficiency is the main objective it seems that a standard forward pass would clearly outperform a token marginal evaluation by far.

  • Insufficient substantiated theoretical claims: Propositions 2 and 3 are presented as central contributions, but neither is accompanied by a formal proof (main text nor appendix). Moreover, Proposition 3 concerns context-independent per-token scorers, but the paper extends its implications to contextual learned scorers and matching-based methods without justification.

Questions for authors
  • Where are the detailed results supporting the claimed "+10.6 points Top-1" improvement of the abstract?

  • In Table 5 why are d(V) and d(C) combined in one column?

  • see weaknesses

3 comments

We sincerely appreciate your time and valuable comments, which help us improve our paper.

Novelty and Distinction from Prior Token Reduction Work

DynamicViT, Beyond Attentive Tokens, and V-Pruner optimize the pruning schedule, but they evaluate token importance against the full token sequence. Our study treats the reference set itself as a decision variable. It measures how shifting this reference changes token rankings and the resulting subsets. The revised text discusses each method and clarifies this distinction.

Comparisons with Established Token Reducers

The revised evaluation includes official checkpoints of DynamicViT and EViT. We evaluated these baselines under matched token budgets on 10,000 ImageNet validation images, alongside ToMe, class attention, d(V)d(V), and d(C)d(C).

Practical Relevance and Runtime of Deployed Methods

This paper presents an empirical analysis of reference dependence, not a practical inference algorithm. Exact candidate-set scoring d(C)d(C) serves as an analytical reference that requires multiple forward passes. Deployed reducers such as ToMe and DynamicViT run within a single forward pass of roughly 11 ms. Figure 5 plots this exact trade-off between scoring cost and selection quality. We updated the introduction and the discussion to state this distinction plainly.

Proofs for Propositions 2 and 3

Appendix C provides complete proofs for Propositions 2 and 3. The appendix includes explicit set losses, feature maps, and interaction terms.

Clarification of the Top-1 Claim in the Abstract

The +10.6+10.6 Top-1 claim in the abstract came from an early proof-of-concept experiment and was included by mistake. We removed this claim from the revised abstract and main text. The revision reports teacher-KL and label-NLL results separately, with unnormalized Top-1 accuracy for all methods. Table 1 reports teacher-KL Top-1 for attention, d(V)d(V), d(C)d(C), and ToMe across each budget. The new experiment adds full validation results for label-NLL.

Separation of Columns in Table 5

Table 5 previously combined d(V)d(V) and d(C)d(C) due to minor differences between the two fixed-reference rules. The revision splits them into separate columns and reports each rule individually.

Sorry for the late response. We hope this clarification and the additional results resolve your questions. We are happy to provide any further details if needed.

Thank you for the response and the revision. Several of my concerns are addressed: the proofs are now included, the unsupported Top-1 claim was removed, and the paper is framed more clearly as a analysis and not as a practical method. I appreciate these changes. Some concerns remain.

Novelty: I am not convinced by the distinction from prior work. DynamicViT and EViT both prune at three stages and score, at each stage, only the tokens that survived the previous one (DynamicViT’s prediction module takes the current decision mask as input while EViT uses the class attention of the current layer). They therefore do not evaluate importance against the full token set once, as Table 14 and the related work state

Significance: With the analysis framing, the main finding is that d(C) improves over d(F) by 1 to 4 points Top-1 at about 97 forward passes, while ToMe is better than both in a single pass. I find it hard to see what a practitioner should take from this.

Theory: Both construction use k = |C| - 1, where d(C) coincides with exhaustive search, so they are small counterexamples and say little about realistic settings. The extension of Proposition 3 to contextual scorers is still not justified.

Minor The +15.5 pp Top-1 in Section 3.3 refers to Table 10, which has no Top-1 values. Table 8 still combines d(F) and d(C) in one column and there are unresolved references ("Table eftab:anatomy", "eftab:official") in the main text (page 6).

In light of the improvements I raise my rating to 2 but the remaining concerns on novelty and significance keep me from recommending acceptance.

Thank you for the careful follow-up and for raising your rating. We address each remaining concern below.

Novelty and the Description of DynamicViT and EViT

We agree that DynamicViT and EViT score the surviving tokens at each stage, not the full set once. We corrected the taxonomy table, the related work, and the discussion.

Our contribution is about design, not a new method. When you build a reducer, you pick a reference to score against. That pick changes which tokens survive. The paper measures the shift, with a mean normalized loss improvement of 0.305. It also proves a per-token scorer cannot represent the shift. So a builder makes two design choices. What to score against, and how often to re-score.

Significance and Practical Takeaway

The paper is an analysis, not a method. Exact d(C)d(C) costs about 97 forward passes, and that cost only pays for measuring the shift exactly. Nobody should run it in production.

The point is the design view. ToMe, DynamicViT, and EViT already re-score on the shrinking set, and the paper explains why that helps them. A practitioner leaves with a way to think about any reducer, not a new reducer to run.

Theory and the Degenerate Budget

We note that in the minimal counterexample constructions of Propositions 2 and 3, setting k=∣C∣−1k = |C| - 1 allows exact correspondence between marginal scoring under CC and optimal subset selection, formally proving the existence of an irreducible constant gap (Ω(1)\Omega(1)) between d(F)d(F) and d(C)d(C). For realistic scenarios where k≪∣C∣−1k \ll |C| - 1, d(C)d(C) computed once becomes a greedy heuristic rather than an exact search, which theoretically motivates why sequential dynamic re-evaluation (Section 3.3) is further required in practice.

Proposition 3 does not extend to contextual scorers. It covers only scorers of the form si=g(hi)s_i = g(h_i) with fixed per-token features. The set-attention scorer is a separate empirical test outside this class. The revision states this scope, and the scale analysis shows the failure persists from 300 to 30,000 training images.

Minor Corrections

The +15.5+15.5,pp Top-1 number was a leftover from an early draft and was left in by mistake during revision. The revision removes it. The group-probe table now reports d(F)d(F) and d(C)d(C) in separate columns. The unresolved references \texttt{eftab:anatomy} and \texttt{eftab:official} are corrected to \texttt{tab:anatomy} and \texttt{tab:official}.

We hope this addresses the remaining concerns, and we are happy to provide further detail.

Reviewer_5D5WOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how changing the reference token set changes marginal-loss rankings. Candidate-set scoring generally improves over full-reference scoring, and further experiments examine iterative selection, learned scorers, and token merging. The main contribution is an analysis of reference dependence in token reduction.

Critical strengths
  1. The main intervention is well controlled: candidates and final token count stay fixed while the scoring reference changes. The full ImageNet evaluation, confidence intervals, and disclosure of the three negative conditions make the empirical finding convincing.
  2. The overlap and iterative-selection experiments help explain when the reference matters. They provide useful guidance for evaluating token importance, rather than treating a score computed on the original sequence as equally meaningful after pruning.
Critical weaknesses
  1. The impossibility claim is too broad. Proposition 3 assumes indistinguishable per-token features and does not cover a scorer given candidate-set context. Failure of the tested learners, trained on 300 images per backbone, does not establish that such context-dependent scoring cannot be learned.
  2. The ToMe comparison does not isolate re-evaluation: it starts from the full sequence, while pruning is restricted to 96 candidates. The separate fixed-path experiment helps, but its confidence interval crossing zero does not establish that preserving merged content has no benefit.
Questions for authors
  1. Can you provide the explicit constructions for Propositions 2–3 and clarify which conclusions apply to contextual scorers?
  2. Does the claimed re-evaluation advantage remain when initial candidates and reduction stages are matched between methods?
1 comment

We sincerely appreciate your time and valuable comments, which help us significantly improve our paper.

Scope of Proposition 3 and Scale Analysis of Learned Scorers

We admit that the impossibility claim was stated too broadly in the initial draft. Proposition 3 applies only to fixed per-token scorers of the form si=g(hi)s_i = g(h_i). We updated the statement and the surrounding text to clarify this exact scope.

Controlled ToMe Comparison

Table 1 compared full-sequence merging in ToMe against pruning within 96 candidates under different reduction schedules. The new experiment matches the candidate pool and the layer-wise schedule across four conditions. It tests token merging and token dropping, both with dynamic re-matching and with keys frozen at block zero. On DINOv2, dynamic matching outperforms the frozen baseline by 0.2160.216 [0.2050.205, 0.2260.226] at k=32k=32, 0.1560.156 [0.1460.146, 0.1670.167] at k=64k=64, and 0.1500.150 [0.1400.140, 0.1610.161] at k=80k=80. Under dynamic matching, merging features exceeds dropping by 0.0960.096 at k=32k=32, 0.0670.067 at k=64k=64, and 0.0630.063 at k=80k=80.

Explicit Constructions and Contextual Scorers

Appendix C now provides explicit loss and feature constructions for Propositions 2 and 3 with complete proofs. The construction uses symmetric patch pairs that share identical features but have opposite marginal values across different sets. Proposition 3 applies only to fixed-feature per-token scoring rules. It says nothing about contextual scorers like set-attention modules or bipartite matching.

Re-evaluation Under Matched Candidates and Stages

The re-evaluation advantage holds when initial candidates and reduction schedules match between methods. In the matched setting, dynamic candidate-set re-scoring beats static single-pass scoring. Dynamic matching consistently outperforms fixed-path baselines across all tested budgets under identical pool sizes and stage allocations.

Sorry for the late response. We hope our response and the revised manuscript address your concerns. We remain open to any further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.