- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper aims to accelerate the infernce of WAMs using Mix-of-Transformers architectures. The proposed methods include TACD and CEWP. TACD improves the consistency between the student model and the endpoint while reducing denoising steps. CEWP mitigates the inter-expert waiting through a KV pipeline.
Critical strengths
- Accelerating WAMs is definitely a very important research topic.
- The paper has deeply analyzed the time comsumption during WAMs' inference, and further proposed two corresponding methods.
- The experimental results are promising, showing substantial speed-up while maintaining the accuracy.
Critical weaknesses
- To my opinion, the contribution and novelty of this paper is limited. (1) TACD seems to add an item of teacher anchor to the loss function of vanilla consistency distillation. (2) CEWP is the natural extension of the idea of pipeline operation.
- This paper provides no new ideas about the inference acceleration of WAMs, compared with strong baselines such as FastWAM and FasterWAM.
- The two main methods on this paper, TACD and CEWP, seems to be independent from each other, which makes this paper a combination of two irrelevant papers. If there is no other links between the two main parts, this paper should be treated as two submissions.
Questions for authors
- Please justify the contribution of this paper, addressing my concern in weakness #1 (1) and (2).
- What is the logical links between TACD and CWEP? I think these two methods seem to be independent, respectively accelerating the WAMs from two differenct points.
Thank you for your clarification and additional experiments. I agree with reviewer B9Xd that two main contributions are not related to each other and indeed CEWP contributes little to the final performance. This is a critical point. Considering all factors, I would like to adjust my rating to weak rejection.