- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper aims to target two challenges in fine-grained audio-text retrieval which are prototype collapse and the lack of informative negatives. For these two challenges, this paper propose SARA to address these two issues using stable anchirs and more rigorous alignment. The experiments conducted on AudioCaps and Clotho show the performance improvement on the proposed methods.
Critical strengths
1.big perfomance gain is achieved on AudioCaps SARA ourperforms the strongest baseline by 5.31, which is a relatively big improvement.
2. compreshensive ablation studies and analysis. This paper provides ablation studies on orthogonal regularization, OT, hard negative mining, and prototype count. The analysis shows that reducing prototype collapse alone does not necessarily improve retrieval performance. The appendix also examine how model resonds to the changes in event order and content.
Critical weaknesses
1. The contribution of Stable Anchors is unclear.. Equation (1) introduce the basic linear-projection attention pooling, Equation (5) uses shared learnable queries. However, under the single-head formulation, the difference between them are not clear. The author should clarify this difference, and it would be better to see the comprison of using the original GPA pooling with the same orthognal loss, OT and negative queen. Moreover, as shown in Table 2, GPA+HN still suffers from collapse but achieves the same T2A R@1 as full SARA, and SARA without the queue make little improvement compared with GPA baseline. These make it unclear whether the gains come from better alignment or simply stronger constrasive learning.
2. The fine-grained semantic claims are not supported by sufficiently evidence. APCS and effective rank measure protoype geometry, but do not show whether the prototype capture the different events or relations. It is better to see more systematic evalutions, i.e. the order and attribute tests. Furthermore, the current analysis are mostly conducted on Audiocaps but the improvement on Clotho are very limited.
Questions for authors
- what is the main difference between Equation (1) and Equation (5)? Could the author clarify whether additional projections or attention head are used?
- Could the author provide more systematic test on the event order and/or attribute discrimination? it would be useful to compare SARA with other baselines under the same experimental setting. It would be interesting to see if reducing prototype improves semantic discrimination beyond the hard negative mining alone.
- the evluation settings, such as data splits, retrieval candidate pools and how checkpoints are selcted etc. are not well clarified in the paper. It would better to see the resutls on random seed given the limited improvements on Clotho.