- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper aims to address the challenge that in AVQA, when the audio or visual modality is missing, the information needed to answer the question is lost, and existing retrieval-based methods rely on semantic similarity and therefore may not preserve information such as the number of sound sources or the temporal configuration. To solve this challenge, the authors propose QRePair. QRePair clusters the question-conditioned audio and visual features of the training data to construct paired prototypes, and at inference, it retrieves the prototype most similar to the remaining modality and uses its paired prototype as the information of the missing modality. The authors also adapt the retrieved prototype to the current question through QGR, which refines it together with the question. This was demonstrated through comparisons with existing missing-modality methods on the MUSIC-AVQA and AVQA datasets, and full-modality evaluations on MUSIC-AVQA-R and MUSIC-AVQA 2.0 are also provided.
Critical strengths
-
The authors propose a method that constructs paired audio and visual prototypes from the training data and retrieves and uses information from the missing modality with the remaining modality. The performance improvement is shown through ablation experiments.
-
The proposed method demonstrates superior performance on MUSIC-AVQA compared to existing missing-modality methods in both the missing-modality and complete-modality settings.
Critical weaknesses
-
Novelty and motivation. The main contribution and the motivation of this paper are unclear. Compared to the existing missing-modality AVQA works, Missing-AVQA and R$^2$ScP, it is not clear what makes the proposed method different, especially in the way it retrieves and uses the information of the missing modality from the training data. The authors need to clarify what advantage the proposed method provides over prior work. It is also unclear how the cluster-based paired prototypes are related to the structured evidence claimed by the authors (such as the number of sound sources and the temporal configuration), and what relationship between the prototypes and the question the model exploits to effectively compensate for the missing modality.
-
The experiments and evaluation rely on limited settings.
2-1. The prototypes are constructed only from the training set of each dataset, and it is not clear whether they can be applied to other datasets or domains. MUSIC-AVQA-R and MUSIC-AVQA 2.0, which have different distributions, are also evaluated only in the full-modality setting. This raises the question of whether the proposed method is valid only in environments similar to the training data, or whether it can be used in general.
2-2. All experiments rely only on the setting where the entire input feature of the missing modality is replaced with zeros. Partial missing, noisy inputs, and various missing ratios, which correspond to transient failures or transmission errors, are not evaluated.
2-3. The evaluation is limited to MUSIC-AVQA. The analysis of the proposed method is mainly conducted on MUSIC-AVQA, and there are no qualitative results on general audio-visual video pairs or sufficient evaluations in other generalized data environments. Therefore, it is difficult to confirm whether the proposed method is valid in general environments.
-
It is not clear whether the performance improvement of the proposed method comes from the paired prototypes. In Table 2, using the prototypes without QGR results in lower audio-missing performance than the backbone. In addition, reconstruction from the question alone without prototypes also shows a large improvement over the backbone, and the gap to the full model is small, especially in the audio-missing setting. Furthermore, in Table 1, the performance on Audio-Visual questions barely drops even when the audio is missing. Therefore, it is unclear whether the proposed method actually recovers the information of the missing modality, or exploits the bias of the question or the dataset.
-
The authors design paired prototype retrieval to compensate for the missing modality. However, what information the retrieved prototypes actually provide is only shown through the final performance, and qualitative analyses such as visualization of the prototype distribution with t-SNE, examples of the retrieved prototypes, and attention maps between the reconstructed features and the other modality and the question are missing.
-
There is no handling of cases where the retrieved prototype does not match the current sample. The single most similar prototype is always used regardless of the similarity, and there is no mechanism to filter out or weaken incorrect prototypes, nor a loss to guide this. Figure 3 also shows some cases where prototypes with low similarity to the original are retrieved, but there is no discussion of this.
Questions for authors
-
What is the motivation of the proposed method, and what is its key difference compared to prior work? A clear explanation of the contribution would help clarify the novelty.
-
Can the proposed framework be extended to real-world data?
-
To verify what information the retrieved prototypes actually provide, qualitative analyses such as visualization of the prototype distribution with t-SNE, examples of the retrieved prototypes, and attention maps between the reconstructed features and the other modality and the question are needed.
-
How are the cases handled where prototypes with low similarity are retrieved? Results of applying a method to filter out or weaken incorrect prototypes, or a performance analysis with respect to the retrieval similarity, would be helpful.
Please refer to the weaknesses.