acceptodds
Under review as a conference paper at ICLR 2027

SheafPrune: Sheaf-Guided Graph Propagation for Visual Token Pruning

Abstract

Multimodal large language models process the complete visual token sequence, increasing prefill cost even when many tokens are irrelevant to the instruction. Token-wise saliency captures limited inter-token dependence; graph-based selection instead depends on how relations govern importance propagation and ranking. We study support-constrained propagation with node-degree correction. Inspired by sheaf geometry, SheafPrune uses modality-specific maps and a sparse text–visual graph to transfer relevance, then propagates scores on a separate visual graph and corrects them for degree. It retains a hard top- subset once before the frozen language model. The selector has about M parameters and is trained end-to-end through the frozen LLaVA-NeXT-8B backbone using only the language-modeling objective. Explicit retention-ratio conditioning lets one checkpoint serve every budget. Across four backbones, ten benchmarks and seven pruning ratios, with all baselines re-implemented under one protocol, SheafPrune achieves the best retention at of backbone–budget settings, with monotonically growing margins in the moderate-to-aggressive regime. On LLaVA-NeXT, retaining of visual tokens yields a net prefill speedup and an end-to-end FLOPs reduction. Together with component ablations, these results support graph-constrained importance propagation and degree correction as a design for visual token selection across the evaluated backbones and tokenization schemes, improving the trade-off between accuracy retention and inference cost.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Visual Token Pruning, Multimodal Large Language Models, Sheaf Theory, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jf5qOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

SheafPrune studies how to select visual tokens before they enter a multimodal language model. The idea is that a token’s importance should depend not only on its own score, but also on how relevance moves through the graph of visual tokens. The method first combines learned text–image relevance with a visual-feature prior. It then propagates these scores over a sparse visual graph, applies a degree correction, and keeps a hard top-kk subset. A small selector is trained through a frozen LLaVA-NeXT backbone, and the authors report reusing the same checkpoint across different pruning budgets and backbones. The experiments cover four backbones and ten benchmarks. The results suggest that the graph operator is useful, especially under aggressive pruning. However, the headline comparisons select the best degree-correction parameter at each evaluation point, and the ablations leave open how much of the benefit requires the learned, question-conditioned part of the method.

Critical strengths
  1. The paper makes a useful distinction between normalizing propagation and correcting the final ranking. I find this a worthwhile observation for graph-based token selection.

  2. The paper evaluates methods on the same splits, with the same answer extraction and metric implementations. It also documents differences in pruning location and budget accounting. This is much more convincing than comparing new results against numbers copied from unrelated evaluation setups.

  3. The authors distinguish training-time gating from inference-time removal, preserve original position identifiers, include selector overhead in prefill timing, and acknowledge the quadratic similarity workspace. They also separate prefill speedup from full-answer latency. These details make the reported results easier to interpret and give a clearer picture of what the method actually does.

Critical weaknesses
  1. My main concern is that the headline comparison uses the best parameter choice at each evaluation point. The “22 out of 28 settings” result comes from selecting the best δ\delta from four options. Appendix M explicitly says that held-out selection has not been established. I appreciate the transparency, but this still means the main curve is an optimistic envelope rather than the performance of a configuration chosen before evaluation.

  2. The results make the graph contribution clearer than the need for the learned, sheaf-guided components. At 90% pruning, uniform initialization reaches 88.19% RA, only 1.20 points below the full fixed-configuration method. From the equations, this path appears to depend on frozen visual features and fixed graph settings, while bypassing the learned text encoder and cross-modal relevance branch. If that reading is correct, it should be possible to implement this as a training-free selector.

  3. The practical speed comparison needs a more direct measurement. Appendix K uses different profiling paths for one-shot and decoder-internal methods, then applies backbone-specific timing corrections. Those corrections may be justified, but some are several milliseconds, which is substantial relative to the differences between competing methods.

Questions for authors
  1. Is it chosen separately for each benchmark, or once per backbone–budget pair? Please report the full fixed-δ\delta curves and clarify how the 22/28 result changes under a held-out selection policy.

  2. Can the uniform-initialization version run without any trained selector parameters? If so, how does it compare with the full method across all backbones in accuracy, selector latency, and memory? This would help quantify the practical value of training.

2 comments

Dear Reviewer,

Thank you again for your constructive feedback. Following your suggestions, we have added fixed-δ results and a training-free Uniform + Graph comparison, including accuracy, latency, and memory measurements.

We would appreciate it if you could revisit our responses and consider whether these additions address your concerns. We warmly welcome further questions and discussion, and we are happy to provide any clarification.

Best regards,
The Authors

AuthorsOpenReview ↗

Reponse to Reviewer

Dear Reviewer,

Thank you very much for your careful evaluation and constructive comments on our manuscript.

In response to your suggestions, we have added the requested experiments and incorporated the corresponding results and discussion into the revised manuscript. However, we noticed that the comments appear to remain unchanged after your review.

We would therefore be grateful if you could clarify whether the additional experiments were insufficient or whether there are any other concerns that we may have overlooked. If further improvements or additional analyses are needed, we would sincerely appreciate your guidance. We will be happy to address any remaining issues carefully.

Thank you again for your time and valuable suggestions.

Best regards,

Reviewer_6njjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SheafPrune, a method that removes visual tokens once, before they enter the language decoder. The method first computes text-visual relevance through modality-specific maps and combines this signal with visual-feature norms to obtain initial scores. It then propagates the scores for a fixed number of steps on a sparse visual-similarity graph, applies a power of neighborhood degree as a correction, and retains the top-k tokens under a fixed budget. The selector has approximately 4.5M trainable parameters, while the backbone remains frozen; retention-ratio conditioning allows one checkpoint to serve multiple budgets. The paper evaluates four backbones, ten benchmarks, and seven pruning ratios. Its main comparison selects the best degree exponent separately at each backbone-budget point.

Critical strengths
  1. The evaluation is broad and comprehensive across backbones, tasks, and pruning budgets settings. In addition to accuracy retention, the paper reports prefill latency including selector overhead, decoding throughput, and FLOPs, making the accuracy-efficiency trade-off relatively transparent.

  2. The graph propagation and degree correction are defined clearly. In particular, the authors openly report that uniform initialization is close to the full method rather than overstating the contribution of the learned relevance signal.

  3. Implementation boundaries and failure cases are disclosed carefully. It also documents insufficient spatial coverage at aggressive pruning rates and exceptions on particular backbones. These details facilitate reproduction and discourage overgeneralization of the efficiency claims.

Critical weaknesses
  1. The main results use a test-point-wise hyperparameter envelope, which can introduce substantial selection bias. For each backbone and budget, the paper reports the best test result over delta in {0, 0.5, 1, 2}. This is the main issue affecting my recommendation.

  2. The technical role of “sheaf-guided” is weaker than the title and framing may suggest. The method uses modality-specific projections and edge comparison energy as inspiration from sheaf geometry, but the visual stage is essentially scalar-score propagation on a similarity graph followed by heuristic degree correction. The authors explicitly state that it is not a learned sheaf Laplacian. The authors should either narrow the claims or demonstrate a property that specifically follows from the sheaf construction.

  3. The comparison mixes a supervised selector with several training-free baselines, leaving the effects of training data and operator design entangled.

Questions for authors
  1. If delta must be selected once per backbone on a disjoint development set and then fixed across all seven budgets, at how many of the 28 backbone-budget points does SheafPrune still lead? Does the ranking change when every baseline receives the same development-set tuning budget?

  2. Could the authors provide a training-free version, such as uniform initialization with the current graph propagation and degree correction, as well as a non-sheaf graph or MLP ranker trained on the same 779K data with a comparable parameter count? This would clarify whether the gains primarily come from supervision, graph structure, degree correction, or the sheaf-inspired design.

4 comments

Dear Reviewer,

Thank you again for your constructive feedback. In response to your concerns, we have added fixed-δ results and a training-free Uniform + Graph comparison. We have also clarified the scope of the sheaf-inspired formulation and acknowledged the comparisons that remain to be completed.

We would be grateful if you could revisit our response and consider whether it addresses your concerns. We warmly welcome any further questions or discussion and would be happy to clarify any points.

Best regards, The Authors

AuthorsOpenReview ↗

Reponse to Reviewer

Dear Reviewer,

Thank you very much for your careful evaluation and constructive comments on our manuscript.

In response to your suggestions, we have added the requested experiments and incorporated the corresponding results and discussion into the revised manuscript. However, we noticed that the comments appear to remain unchanged after your review.

We would therefore be grateful if you could clarify whether the additional experiments were insufficient or whether there are any other concerns that we may have overlooked. If further improvements or additional analyses are needed, we would sincerely appreciate your guidance. We will be happy to address any remaining issues carefully.

Thank you again for your time and valuable suggestions.

Best regards

Thank you for your detailed response. However, the training-free version exhibits a significant performance gap compared to the version that has undergone training; therefore, I have decided to maintain my original rating.

AuthorsOpenReview ↗

Response for reviewer

Dear Reviewer,

Thank you for your thoughtful response. We would like to gently clarify that the trained version is our proposed method, while the training-free version serves as a comparison. The observed performance gap indicates that the proposed training method contributes to the improvement in performance.

Thank you again for your time and consideration.

Best regards, The authors

Reviewer_JWu9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SheafPrune, a sheaf inspired visual token pruning method that combines cross modal relevance with graph based ranking. Experiments across four MLLM backbones and ten benchmarks demonstrate competitive accuracy retention and inference efficiency, supported by component ablation studies.

Critical strengths

S1.

The proposed graph-based ranking mechanism substantially improves accuracy retention through support-constrained propagation and degree correction, as demonstrated by the ablation studies. This highlights the effectiveness of the proposed approach beyond simple token importance estimation.

Critical weaknesses

W1.

While the paper provides a sheaf theoretic interpretation of cross modal comparison, the resulting operations closely resemble conventional learned projections and graph based score propagation. It remains unclear what methodological benefits arise specifically from the sheaf perspective.

W2.

SheafPrune requires training a 4.5M parameter selector, yet its performance gains over competitive training-free pruning methods are relatively modest. The benefits of additional training remain unclear, particularly when considering inference efficiency.

Questions for authors

Could the authors compare the learned modality-specific projections against frozen pretrained vision-language representations (e.g., CLIP) or similarities computed directly from existing backbone features? This would clarify whether the learned cross-modal alignment provides meaningful benefits over simpler, training-free alternatives, particularly given its central role in the sheaf-inspired formulation.

2 comments
AuthorsOpenReview ↗

Reponse to Reviewer

Dear Reviewer,

Thank you again for your constructive feedback. Following your suggestions, we have added experiments comparing our method with frozen CLIP and native backbone features, and further clarified the methodology and accuracy–efficiency trade-off in our rebuttal.

We would appreciate it if you could revisit our responses and consider whether these additions address your concerns. We warmly welcome further questions and discussion and are happy to respond throughout the discussion period.

Best regards, The Authors

AuthorsOpenReview ↗

Reponse to Reviewer

Dear Reviewer,

Thank you very much for your careful evaluation and constructive comments on our manuscript.

In response to your suggestions, we have added the requested experiments and incorporated the corresponding results and discussion into the revised manuscript. However, we noticed that the comments appear to remain unchanged after your review.

We would therefore be grateful if you could clarify whether the additional experiments were insufficient or whether there are any other concerns that we may have overlooked. If further improvements or additional analyses are needed, we would sincerely appreciate your guidance. We will be happy to address any remaining issues carefully.

Thank you again for your time and valuable suggestions.

Best regards,

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised version of the manuscript

Dear Area Chair and Reviewers,

Along with our author response, we are submitting a revised version of the manuscript, which incorporates additional experiments and results conducted in light of your feedback. The new content is highlighted in blue for your convenience.

Thank you once again for your constructive comments. We welcome any further questions you may have regarding our response or the additional experiments during the discussion phase, and we would be happy to provide further clarification.

Best regards,

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.