- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
SheafPrune studies how to select visual tokens before they enter a multimodal language model. The idea is that a token’s importance should depend not only on its own score, but also on how relevance moves through the graph of visual tokens. The method first combines learned text–image relevance with a visual-feature prior. It then propagates these scores over a sparse visual graph, applies a degree correction, and keeps a hard top- subset. A small selector is trained through a frozen LLaVA-NeXT backbone, and the authors report reusing the same checkpoint across different pruning budgets and backbones. The experiments cover four backbones and ten benchmarks. The results suggest that the graph operator is useful, especially under aggressive pruning. However, the headline comparisons select the best degree-correction parameter at each evaluation point, and the ablations leave open how much of the benefit requires the learned, question-conditioned part of the method.
Critical strengths
-
The paper makes a useful distinction between normalizing propagation and correcting the final ranking. I find this a worthwhile observation for graph-based token selection.
-
The paper evaluates methods on the same splits, with the same answer extraction and metric implementations. It also documents differences in pruning location and budget accounting. This is much more convincing than comparing new results against numbers copied from unrelated evaluation setups.
-
The authors distinguish training-time gating from inference-time removal, preserve original position identifiers, include selector overhead in prefill timing, and acknowledge the quadratic similarity workspace. They also separate prefill speedup from full-answer latency. These details make the reported results easier to interpret and give a clearer picture of what the method actually does.
Critical weaknesses
-
My main concern is that the headline comparison uses the best parameter choice at each evaluation point. The “22 out of 28 settings” result comes from selecting the best from four options. Appendix M explicitly says that held-out selection has not been established. I appreciate the transparency, but this still means the main curve is an optimistic envelope rather than the performance of a configuration chosen before evaluation.
-
The results make the graph contribution clearer than the need for the learned, sheaf-guided components. At 90% pruning, uniform initialization reaches 88.19% RA, only 1.20 points below the full fixed-configuration method. From the equations, this path appears to depend on frozen visual features and fixed graph settings, while bypassing the learned text encoder and cross-modal relevance branch. If that reading is correct, it should be possible to implement this as a training-free selector.
-
The practical speed comparison needs a more direct measurement. Appendix K uses different profiling paths for one-shot and decoder-internal methods, then applies backbone-specific timing corrections. Those corrections may be justified, but some are several milliseconds, which is substantial relative to the differences between competing methods.
Questions for authors
-
Is it chosen separately for each benchmark, or once per backbone–budget pair? Please report the full fixed- curves and clarify how the 22/28 result changes under a held-out selection policy.
-
Can the uniform-initialization version run without any trained selector parameters? If so, how does it compare with the full method across all backbones in accuracy, selector latency, and memory? This would help quantify the practical value of training.
2 comments
Reponse to Reviewer
Dear Reviewer,
Thank you very much for your careful evaluation and constructive comments on our manuscript.
In response to your suggestions, we have added the requested experiments and incorporated the corresponding results and discussion into the revised manuscript. However, we noticed that the comments appear to remain unchanged after your review.
We would therefore be grateful if you could clarify whether the additional experiments were insufficient or whether there are any other concerns that we may have overlooked. If further improvements or additional analyses are needed, we would sincerely appreciate your guidance. We will be happy to address any remaining issues carefully.
Thank you again for your time and valuable suggestions.
Best regards,
Dear Reviewer,
Thank you again for your constructive feedback. Following your suggestions, we have added fixed-δ results and a training-free Uniform + Graph comparison, including accuracy, latency, and memory measurements.
We would appreciate it if you could revisit our responses and consider whether these additions address your concerns. We warmly welcome further questions and discussion, and we are happy to provide any clarification.
Best regards,
The Authors