- Rating
- 2(−1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper has a clear motivation, which originally comes from a mechanism discovery in Figure 1. The paper shows that attention routing stabilizes before the representation refinement is complete. That means we can reduce the attention calculation in later refinement steps, as the routing converges earlier. This paper proposes WISE, the working set inference with supporting exploitation. It balances performance and efficiency based on the observation from the above paragraph. The experiments support both the mechanism and the computational implications. The proposed WISE yields up to1.76x attention speedup over native Flash Attention at 4K context, indicating the effectiveness of the mechanism dis-observation and the proposed method.
Critical strengths
- The motivation is substantially clear in this paper. The observation acts as a very important foundation for the proposed method. The experiment shows that the proposed method has a rational, reasonable speedup for the recurrent transformer models. The contribution may be important for the recurrent-style models.
- The experiment is well designed in the direction of proving the performance. Table 1, Table 2, and Figure 2 show the efficiency improvement gained by the proposed method compared to the other reasonable baselines.
Critical weaknesses
- It is necessary to prove that the approach can be applied to diverse model series, instead of only using one model to be responsible for all the results. Because one model may have a coincidental combination gain for the result, different model architecture families should be tested.
- The paragraph's length is a bit long for reading, although it is clear to understand. I think we can condense the explanations and shorten the repeated claims and proofs. Save more space for extensive experiments. For example, we can test more model families and benchmarks. That will be more useful and make the paper's novelty and contribution solid.
Questions for authors
I don't have any other questions for this paper. If the experimental flaws can be addressed, I would consider this paper a significant contribution.
3 comments
Dear Reviewer,
Thank you again for your valuable feedback. As the discussion period nears its end, we would appreciate knowing whether our rebuttal has addressed your concerns or if you have any remaining questions.
Thank you for your time and consideration.
Best regards,
The Authors
Response to author's rebuttal.
Thanks to the author for their effort in working on the weaknesses.
Only one model was added for the experiments that do not match the requirements of more model families.
Although I appreciate the general direction and the novelty of this work, the experimental scope concern is still not fully addressed. And I agree with reviewer JnR1 on the concern they found. Based on these, I will show my final assessment on the rating.
Dear Reviewer,
We would like to kindly follow up on our rebuttal and the additional experimental results we provided to address your concerns.
We would greatly appreciate it if you could take another look at our responses and let us know whether any questions or concerns remain.
We are happy to provide further clarification and look forward to your feedback.
Thank you for your time and consideration.
Best regards,
The Authors