acceptodds
Withdrawn from ICLR 2027

QRePair: Question-Guided Reconstruction via Paired Prototype Retrieval for Missing-Modality Audio-Visual Question Answering

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Audio-Visual Question Answering (AVQA) requires reasoning over audio, visual, and linguistic cues, yet one modality may be unavailable at inference time, leaving question-relevant evidence unobserved. Semantic retrieval can provide useful priors, but may not preserve structured evidence such as source count or temporal configuration. We propose Question-Guided Reconstruction via Paired Prototype Retrieval (QRePair), which reconstructs task-relevant missing-modality representations from paired temporal prototypes. Because relevant evidence depends on the question, Question-Guided Modality Encoding first produces question-conditioned audio and visual representations. Paired Prototype Construction organizes complete training representations into temporally resolved audio-visual prototype pairs, and Paired Prototype Retrieval uses the available representation to retrieve the corresponding missing-modality prior. Question-Guided Reconstruction then adapts this prior to the current question and the feature space of a frozen answering backbone. On 5,096 audio-visual test questions from MUSIC-AVQA, QRePair achieves 71.59% and 73.49% accuracy with visual and audio inputs missing, respectively, representing gains of 2.22 and 1.43 percentage points over the reported R^2ScP results. Ablations further show that disrupting audio-visual correspondence reduces average missing-modality accuracy by 4.25 points, while removing explicit question conditioning from reconstruction reduces it by 8.96 points. QRePair also shows strong performance on AVQA and fully observed MUSIC-AVQA-R and MUSIC-AVQA 2.0 benchmarks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Audio–visual question answering, Missing modality

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LZW8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper aims to address the challenge that in AVQA, when the audio or visual modality is missing, the information needed to answer the question is lost, and existing retrieval-based methods rely on semantic similarity and therefore may not preserve information such as the number of sound sources or the temporal configuration. To solve this challenge, the authors propose QRePair. QRePair clusters the question-conditioned audio and visual features of the training data to construct paired prototypes, and at inference, it retrieves the prototype most similar to the remaining modality and uses its paired prototype as the information of the missing modality. The authors also adapt the retrieved prototype to the current question through QGR, which refines it together with the question. This was demonstrated through comparisons with existing missing-modality methods on the MUSIC-AVQA and AVQA datasets, and full-modality evaluations on MUSIC-AVQA-R and MUSIC-AVQA 2.0 are also provided.

Critical strengths
  1. The authors propose a method that constructs paired audio and visual prototypes from the training data and retrieves and uses information from the missing modality with the remaining modality. The performance improvement is shown through ablation experiments.

  2. The proposed method demonstrates superior performance on MUSIC-AVQA compared to existing missing-modality methods in both the missing-modality and complete-modality settings.

Critical weaknesses
  1. Novelty and motivation. The main contribution and the motivation of this paper are unclear. Compared to the existing missing-modality AVQA works, Missing-AVQA and R$^2$ScP, it is not clear what makes the proposed method different, especially in the way it retrieves and uses the information of the missing modality from the training data. The authors need to clarify what advantage the proposed method provides over prior work. It is also unclear how the cluster-based paired prototypes are related to the structured evidence claimed by the authors (such as the number of sound sources and the temporal configuration), and what relationship between the prototypes and the question the model exploits to effectively compensate for the missing modality.

  2. The experiments and evaluation rely on limited settings.

    2-1. The prototypes are constructed only from the training set of each dataset, and it is not clear whether they can be applied to other datasets or domains. MUSIC-AVQA-R and MUSIC-AVQA 2.0, which have different distributions, are also evaluated only in the full-modality setting. This raises the question of whether the proposed method is valid only in environments similar to the training data, or whether it can be used in general.

    2-2. All experiments rely only on the setting where the entire input feature of the missing modality is replaced with zeros. Partial missing, noisy inputs, and various missing ratios, which correspond to transient failures or transmission errors, are not evaluated.

    2-3. The evaluation is limited to MUSIC-AVQA. The analysis of the proposed method is mainly conducted on MUSIC-AVQA, and there are no qualitative results on general audio-visual video pairs or sufficient evaluations in other generalized data environments. Therefore, it is difficult to confirm whether the proposed method is valid in general environments.

  3. It is not clear whether the performance improvement of the proposed method comes from the paired prototypes. In Table 2, using the prototypes without QGR results in lower audio-missing performance than the backbone. In addition, reconstruction from the question alone without prototypes also shows a large improvement over the backbone, and the gap to the full model is small, especially in the audio-missing setting. Furthermore, in Table 1, the performance on Audio-Visual questions barely drops even when the audio is missing. Therefore, it is unclear whether the proposed method actually recovers the information of the missing modality, or exploits the bias of the question or the dataset.

  4. The authors design paired prototype retrieval to compensate for the missing modality. However, what information the retrieved prototypes actually provide is only shown through the final performance, and qualitative analyses such as visualization of the prototype distribution with t-SNE, examples of the retrieved prototypes, and attention maps between the reconstructed features and the other modality and the question are missing.

  5. There is no handling of cases where the retrieved prototype does not match the current sample. The single most similar prototype is always used regardless of the similarity, and there is no mechanism to filter out or weaken incorrect prototypes, nor a loss to guide this. Figure 3 also shows some cases where prototypes with low similarity to the original are retrieved, but there is no discussion of this.

Questions for authors
  1. What is the motivation of the proposed method, and what is its key difference compared to prior work? A clear explanation of the contribution would help clarify the novelty.

  2. Can the proposed framework be extended to real-world data?

  3. To verify what information the retrieved prototypes actually provide, qualitative analyses such as visualization of the prototype distribution with t-SNE, examples of the retrieved prototypes, and attention maps between the reconstructed features and the other modality and the question are needed.

  4. How are the cases handled where prototypes with low similarity are retrieved? Results of applying a method to filter out or weaken incorrect prototypes, or a performance analysis with respect to the retrieval similarity, would be helpful.

Please refer to the weaknesses.

Reviewer_NJbkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates missing-modality audio-visual question answering (AVQA) and proposes QRePair, a framework that reconstructs missing-modality representations through question-guided paired prototype retrieval. The method first employs Question-Guided Modality Encoding (QG-ME) to extract question-relevant audio and visual features. Paired Prototype Construction (PPC) then organizes complete training representations into temporally structured audio-visual prototype pairs. During inference, Paired Prototype Retrieval (PPR) identifies a matching prototype using the available modality, while Question-Guided Reconstruction (QGR) adapts the corresponding missing-modality representation for subsequent multimodal reasoning and answer prediction through QG-MoE. The framework follows a two-stage training strategy, first learning an AVQA backbone with complete inputs and then optimizing the reconstruction module through feature alignment and knowledge distillation while keeping the backbone frozen.

The authors evaluate QRePair on MUSIC-AVQA and AVQA under both complete- and missing-modality settings, with additional full-modality evaluations on MUSIC-AVQA-R and MUSIC-AVQA 2.0 to assess performance under distribution shifts. On MUSIC-AVQA, QRePair achieves 71.59% and 73.49% accuracy under visual-missing and audio-missing conditions, respectively, outperforming reported baselines. Ablation studies further demonstrate the contributions of paired prototype retrieval and question-guided reconstruction, while prototype compression experiments examine the trade-off between storage efficiency and prediction accuracy.

Critical strengths

Strength 1: Well-Motivated Paired Prototype Retrieval and Question-Guided Reconstruction.

This paper combines cross-modal paired prototype retrieval with question-guided feature reconstruction, providing a meaningful approach to missing-modality AVQA. Unlike methods that primarily rely on semantic similarity for retrieval, the proposed framework explicitly maintains audio-visual correspondence between prototypes through shared sample groupings and adapts the retrieved missing-modality priors to the current question. Ablation results show that disrupting audio-visual prototype pairing or removing question conditioning from reconstruction reduces the average accuracy across the two missing-modality settings by 4.25 and 8.96 percentage points, respectively, supporting the effectiveness of these key components.

Strength 2: Strong Missing-Modality Performance and Storage Efficiency.

The proposed method outperforms reported baselines on both MUSIC-AVQA and AVQA under missing-modality conditions. Prototype compression experiments further show that retaining only 28.8% of the prototype pairs from the unclustered setting achieves comparable accuracy, with a decrease of just 0.23 percentage points. These results demonstrate a favorable trade-off between prediction performance and prototype storage requirements.

Critical weaknesses

(1) Unclear Attribution of Performance Gains to the Proposed Retrieval Mechanism.

The performance improvements over existing methods may not be entirely attributable to the proposed paired prototype retrieval mechanism. Table 1 adopts previously published baseline results, while QRePair employs a different feature extraction pipeline (CLIP with Token Merging and BEATs) and introduces additional backbone components, including QG-ME, GFI, and QG-MoE. On MUSIC-AVQA, the improvements over R²ScP are relatively modest (+2.22 and +1.43), making it difficult to assess the specific contribution of paired prototype retrieval relative to other architectural differences. This concern is particularly relevant on AVQA, where QRePair improves over R²ScP by only 1.23 percentage points under complete-modality inputs (91.87% vs. 90.64%), but achieves a substantially larger gain of 13.79 points when audio is missing (88.91% vs. 75.12%). While these results suggest improved performance under missing-modality conditions, it remains unclear to what extent the gains arise from the proposed retrieval mechanism rather than differences in feature extraction and missing-modality handling. Furthermore, the component ablations in Table 2 are conducted only on MUSIC-AVQA, making it difficult to explain the substantially larger improvements observed on AVQA. Although the existing ablations provide evidence for the effectiveness of paired retrieval within the proposed framework, additional controlled comparisons using the same backbone and feature representations would help clarify the specific contribution of the proposed retrieval mechanism.

(2) Limited Evidence for Preserving Temporal Structure in Retrieved Prototypes.

A key motivation for the proposed paired temporal prototypes is to preserve structured evidence that conventional semantic retrieval may overlook, including event multiplicity, temporal occurrence, and event ordering. However, it remains unclear to what extent the proposed framework effectively preserves such temporal information. In Eq. (4), temporal prototypes are constructed by averaging feature sequences across samples within the same group without explicit temporal alignment of events, as acknowledged in Section 3.2 and Appendix D.2. When grouped samples contain events occurring at different time positions, this averaging process may blur event timing and ordering. Furthermore, Eq. (6) reconstructs each time step independently, without an explicit mechanism for instance-specific temporal realignment. It is therefore unclear whether temporal information potentially lost during prototype averaging can be reliably recovered. This concern is also reflected in Table 1. While the model achieves relatively substantial improvements on counting questions (+5.12 and +4.53), the gains on temporal questions are considerably smaller (+0.01 and +1.83). These results may not fully establish the claimed advantage of preserving temporal structure. Additional analyses or controlled experiments examining temporal alignment and ordering would help clarify whether the proposed temporal prototypes provide meaningful benefits for temporal reasoning.

Questions for authors
  1. Independent Contribution of the Proposed Retrieval Mechanism. The baseline results in Table 1 are taken from previously published work, while QRePair introduces a different feature extraction pipeline and several additional backbone components. Could the authors compare different retrieval strategies under the same backbone and feature representations to better isolate the contribution of paired prototype retrieval?

  2. Effectiveness of Temporal Structure Preservation. Eq. (4) constructs temporal prototypes by directly averaging feature sequences across samples. Given that the paper acknowledges no explicit onset alignment, and that QGR in Eq. (6) operates independently at each time step, what temporal information do the authors expect the prototypes to retain beyond the sequence format itself? Considering the relatively limited improvements on temporal questions in Table 1, could the authors provide controlled experiments to directly validate whether the proposed temporal representations preserve information beneficial for temporal reasoning?

  3. Reliability of Missing-Modality State Prediction. The current evaluation simulates missing modalities through zero-masking, where modality availability can be readily inferred from the input features. Could the authors report the performance gap between oracle and predicted modality states, as well as the false-trigger rate of the state predictor under complete-modality inputs? These results would help clarify the impact of state prediction errors on the reliability of the overall framework.

Reviewer_Wr5WOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes QRePair for missing-modality AVQA, combining paired audio-visual prototype retrieval with question-guided reconstruction. It reports improved missing-modality accuracy and reduced prototype storage.

Critical strengths

The paired retrieval design is reasonable, and the shuffled-pair ablation supports the usefulness of cross-modal correspondence. Prototype compression also retains competitive accuracy.

Critical weaknesses

The claimed preservation of temporal structure is insufficiently established. Equation (4) averages sequences across clips without aligning event onsets. Retaining temporal indices does not necessarily preserve event order or multiplicity when constituent clips have different temporal configurations. Furthermore, the reconstruction MLP in Eq. (6) operates independently at each time step and provides no explicit temporal realignment. The paper therefore does not establish that its prototypes retain the structured evidence motivating the method. A temporal-shuffling or temporal-pooling control would directly test this claim.

The reconstruction baseline does not adequately isolate the benefit of retrieval. In Table 2, “direct reconstruction” receives only the question, whereas QRePair additionally uses the observed modality to select a prototype. The comparison therefore changes both the reconstruction mechanism and the information available to it. A direct predictor conditioned on both the observed modality and the question, trained with the same backbone and losses, is the essential missing comparison. Shuffling prototype pairs supports correct pairing within the proposed architecture, but does not establish its advantage over this simpler alternative.

The distribution-shift results do not validate the proposed reconstruction mechanism. MUSIC-AVQA-R and MUSIC-AVQA 2.0 are evaluated only with complete inputs, where reconstruction is bypassed. These results therefore cannot establish whether retrieved priors remain useful when training-set correlations change. This matters because question-only reconstruction already achieves 71.35% audio-missing accuracy versus 73.49% for the full model in Table 2.

Questions for authors

Does QRePair outperform reconstruction conditioned on both the observed modality and the question under identical training conditions? How much performance is lost when temporal structure in the prototypes is removed?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.