acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Audio Visual Segmentation with Asymmetric Visual Entity Construction and Audio Querying

Abstract

Audio-Visual Segmentation (AVS) requires associating unlocalised audio cues with spatially dense visual entities. Prevailing methods largely rely on symmetric modality fusion, which can entangle modality-specific representations and induce spurious audio-visual correlations. Such dense coupling overlooks a crucial inductive bias of AVS: vision determines what entities are present and where they are located, whereas audio primarily provides evidence about which visible entities are acoustically active. Motivated by this asymmetry, we reformulate AVS as an asymmetric entity-querying problem and propose AEQNet that separates visual entity construction from candidate-grounded acoustic verification through four complementary components. Firstly, we propose the Hierarchical Visual Candidate Constructor (HVCC), which uses adaptive aggregation of different visual features and mask-conditioned candidate abstraction to construct compact object-centric visual candidates, addressing the spatial ambiguity of direct dense audio-visual interaction. Secondly, we propose the Coarse-to-Fine Audio Query Router (CF-AQR), which uses coarse semantic matching and candidate-constrained fine verification to progressively determine the audio activation of visual candidates, addressing visual ambiguity and simultaneous multi-source sounding. Thirdly, we propose the Factorised Semantic Composer (FSC), which explicitly combines spatial membership, semantic identity, and audio activation to generate structured semantic predictions, avoiding entangled multimodal decoding. Finally, we propose Progressive Candidate-Grounded Counterfactual Query Learning (PCQL), which uses progressive audio-query learning and hard audio counterfactuals to enforce genuine audio-dependent candidate decisions, reducing visual shortcuts and spurious audio-visual correlations. Extensive experiments on standard AVS benchmarks demonstrate the effectiveness and generality of our framework and show superior performance over existing methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Audio-Visual Segmentation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_owoROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors reformulate AVS as an asymmetric entity-querying problem and propose AEQNet that separates visual entity construction from candidate-grounded acoustic verification through four complementary components.

Critical strengths
  1. The method seems feasible.
  2. The grammar and words are proper.
  3. The assumption is interesting.
Critical weaknesses
  1. I find the motivation and contributions of this paper rather confusing. The abstract claims that “This overlooks a fundamental inductive bias: vision delineates entity locations, whereas audio merely verifies their acoustic activation.” However, the paper provides neither empirical observations nor experiments to support this claim. More importantly, I do not see a clear connection between this motivation and the proposed method. The method does not seem to explicitly exploit or investigate this inductive bias.

  2. The novelty of the proposed method is also unclear to me. Although the abstract emphasizes the aforementioned inductive bias as the main motivation, the subsequent methodological discussion does not seem to be built around it. As a result, I find it difficult to identify the key technical contributions of this work.

  3. The related work discussion is insufficient. Section 2.1 discusses only five related papers, despite AVS having been studied for approximately four years. I believe a more comprehensive discussion of existing methods is necessary to properly position the proposed approach within the literature.

  4. The paper contains many equations, but I find most of them unnecessary. For example, Eq. 1, Eq. 2, and Eq. 3 mainly describe standard and widely adopted operations. At least from my understanding, the paper does not explain what is novel about these formulations. Some equations also appear to lack careful verification. For instance, j∗j^* (which is defined) appears without a proper definition of $ j $, and the superscript $ * $ seems to be used only in this particular instance. These issues give me the impression that some mathematical descriptions may have been automatically generated or summarized using AI without sufficient human verification of their correctness and relevance to the paper.

  5. I have serious concerns about the qualitative results in Figure 4. In particular, I encourage the authors and reviewers to zoom in on the second, fourth, and fifth rows of the rightmost example. The masks produced by AVSegFormer, ICF, and SELM exhibit noticeable white outlines, whereas the masks from other methods do not. Upon closer inspection, some of these outlines are purely white, while others contain yellow, blue, or green pixels. At a minimum, this indicates a lack of consistency in the visualization process. More concerningly, these artifacts raise the possibility that some masks may have been manually edited or postprocessed, potentially using image editing tools such as the lasso tool in Photoshop. I cannot determine whether this is actually the case, but the unusual visual artifacts make me question the reliability of the presented results. Furthermore, apart from this example, the authors do not provide additional qualitative visualizations for AVSS. Overall, I have substantial concerns about the authenticity and reliability of the qualitative analysis.

  6. The paper also appears to lack careful proofreading. For example, I do not understand what “Progressive loss weights” means in Figure 3. I searched through the paper but could not find a corresponding explanation. Similarly, Table 1 is first referenced at the bottom of page 8, even though the table itself appears near the top of page 7. This makes the paper unnecessarily difficult to follow.

  7. The paper introduces too many abbreviations, including AEQNet, HVCC, CF-AQR, FSC, and PCQL. It is difficult for readers to remember all these abbreviations while following the technical discussion. I suggest reducing the number of abbreviations or using more descriptive terminology wherever possible to improve readability.

  8. Line 377 states that all input frames are resized to 224×224224 \times 224. However, to my knowledge, although some methods in Table 1 use this resolution, others adopt different input resolutions. I believe the authors should explicitly report the input resolution of each compared method in Table 1, as these differences may affect the fairness and interpretation of the comparisons.

Overall, my main reason for assigning a score of 1 is my concern about the authenticity of the qualitative analysis, particularly the unusual mask boundaries in Figure 4. I believe these visual artifacts require a convincing explanation from the authors before the experimental results can be considered reliable. If I disregard the concerns regarding Figure 4, I would assign a score of 2.

Questions for authors

NA

3 comments
AuthorsOpenReview ↗

Author Responses (continued)

Q7. The large number of abbreviations makes the method difficult to follow, and more descriptive terminology would improve readability.

We sincerely thank the reviewer for this helpful suggestion. We agree that frequent use of abbreviations can make the technical discussion less accessible. To improve readability, we have expanded the overview of AEQNet in Appendix A.1 (Methodology) to provide the full names and functional descriptions of its four components.

The revised overview explains their roles as follows:

  • Visual candidate construction (HVCC): constructs visual candidates independently of audio.
  • Candidate-level acoustic verification (CF-AQR): estimates the acoustic support for each visual candidate.
  • Structured composition (FSC): integrates spatial membership, visual identity, and acoustic support.
  • Training supervision (PCQL): introduces factual and counterfactual supervision to mitigate reliance on visual shortcuts.

We also explicitly clarify that PCQL is an optimization strategy used only during training and does not introduce an additional inference module. This distinction helps readers understand how the prediction components and training strategy fit into the overall framework.

These expanded explanations make each component's role explicit and provide a clearer reference for following the terminology throughout the technical discussion.

Q8. The input resolution of each compared method should be reported in Table 1 to help readers assess the fairness and interpret the results of the comparisons.

We sincerely thank the reviewer for highlighting this important aspect of experimental comparison. We agree that input resolutions should be explicitly reported to provide the necessary context for interpreting performance differences.

Reporting input resolutions in Table 1. We have added an "Input Size" column to Table 1 to specify the input resolution used by each compared method.

Clarifying the scope of Section 4.1. We have revised Section 4.1 to explicitly state that all AEQNet experiments use a fixed input resolution of 224×224224 \times 224. This statement refers specifically to AEQNet's experimental setup, while the input resolutions of the other compared methods are documented in Table 1.

These revisions make the experimental settings more transparent and allow readers to interpret the reported performance in light of each method's input resolution.

Reviewer_owoROpenReview ↗

Feedback

Q5. Why were color replacement and pixel resampling necessary? Generating visualizations from an integer-valued mask tensor should be straightforward. What exactly does “color replacement” mean in this context, why is it needed for an integer-valued mask tensor, and how could it introduce artifacts at mask boundaries?

Furthermore, the visualizations in the previous version were inconsistent across rows, columns, and cases. I find these inconsistencies difficult to reconcile with the authors’ statement that “all reported metrics were computed directly from raw model outputs before any visualization processing.” I remain unconvinced that the reported results are reliable.

The manuscript also appears to have received insufficient proofreading, and parts of the text read as though they may have been AI-generated.

I therefore maintain that the manuscript does not meet the standard for acceptance.

I also encourage the AC to examine the visualization inconsistencies in the previous version of the manuscript when assessing the reliability of the reported results.

Thank you for the comment. Here, "colour replacement” refers to harmonising previously saved RGB visualisations from different export versions. The historical pipeline was RGB image → nearest VOC colour matching → HSV recolouring → enlargement for display. This processing operated on saved visualisation images, rather than the original integer-valued mask tensors.

Some source images already contained mixed boundary colours. During nearest-colour matching, these pixels could receive different inferred labels and consequently appear as coloured fringes after recolouring. For example, an ICF boundary pixel (93, 0, 0) was mapped to “boat” and displayed in blue. These reconstructed labels should not be interpreted as the model’s original predictions. Enlargement was a separate presentation step, not a requirement for palette mapping.

Using the archived pipeline, we reproduced all 20 audited historical visualisations exactly, pixel for pixel, providing a reproducible explanation for the verified coloured fringes. We respectfully ask that this evidence be considered when assessing the visualisation concern and welcome a careful comparison of the corresponding images.

Reviewer_i8eVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AEQNet, an asymmetric entity-querying framework for Audio-Visual Segmentation (AVS). The central idea is to decouple visual entity construction from acoustic verification: visual features are first used to construct candidate object representations, and audio information is subsequently used to determine which candidates are acoustically active. The proposed framework consists of four components: (a) Hierarchical Visual Candidate Constructor (HVCC) generates visual candidate masks and entity embeddings. (b) Coarse-to-Fine Audio Query Router (CF-AQR) estimates candidate-level acoustic support through global semantic matching and local feature verification. (c) Factorised Semantic Composer (FSC) combines candidate membership, visual identity, and audio support to generate segmentation predictions. And (d) Progressive Candidate-Grounded Counterfactual Query Learning (PCQL) introduces hard negative audio samples to encourage audio-sensitive candidate predictions. Experiments on S4, M3, and AVSS demonstrate strong and competitive segmentation performance across different visual backbones.

Critical strengths
  1. Strong empirical performance on standard AVS benchmarks. The proposed method achieves competitive results across S4, M3, and AVSS, with particularly strong performance using the ResNet-50 backbone.

  2. The four components form a logical framework, from visual candidate construction to audio verification and final segmentation.

  3. The paper provides detailed investigations of the individual architectural choices, training strategies, and hyperparameters.

Critical weaknesses
  1. The idea of first identifying visual objects and then using audio to determine which objects are sounding is not new. For example, Bhosale et al. (AAAI 2025) explored visual object proposals with audio-visual matching, Huang et al. (CVPR 2025) proposed vision-derived queries for audio-visual segmentation, and Lee and Seo (Interspeech 2025) explored zero-shot AVS using pretrained models and cross-modal verification. Although AEQNet introduces different architectural components, the paper should better clarify what is fundamentally new compared to these existing approaches.

  2. HVCC uses AVS ground-truth masks to supervise visual candidate generation. However, these masks only cover sounding objects, rather than all visible objects. This may introduce a bias toward objects that are likely to produce sound. Also, the model generates 8 candidates, but it is unclear whether these candidates are complete, meaningful, and unique. Could some candidates overlap or represent the same object? Add some evaluations of candidate quality might helps.

  3. Follow the 2nd question, the audio-support labels are based on Hungarian matching. If two candidates overlap with the same sounding object, one may be labeled positive while the other is labeled negative? Then this could introduce incorrect negative supervision for audio verification.

  4. A major motivation of this work is to reduce visual shortcuts, but the experiments do not directly show whether the final predictions truly depend on audio. Previous work, Do Audio-Visual Segmentation Models Truly Segment Sounding Objects? (Li et al., AAAI 2026), has already demonstrated that many AVS models are dominated by visual information and introduced AVSBench-Robust to evaluate their audio sensitivity. Interestingly, removing audio support only decreases performance by 0.51 on S4 and 0.18 on M3 (Table 4). I would like to see evaluations on AVSBench-Robust or similar scenarios, including silence, noise, and off-screen audio

  5. Although the paper provides extensive ablations, some results do not strongly support the claimed benefits. For example, factorised composition only slightly outperforms dense fusion (0.48 on S4 and 0.15 on M3). It is also unclear what complementary information coarse and fine audio matching provide beyond feature similarity. Regarding efficiency, the method is not consistently more computationally efficient than existing approaches, and no inference latency is reported. Similarly, the generality and robustness claims need stronger experimental evidence.

Questions for authors
  1. Could the authors clarify how AEQNet differs from previous visual-first AVS methods and compare it with a simple visual-proposal-plus-audio-matching baseline?

  2. Could the authors visualize all eight predicted visual candidates and evaluate their completeness, uniqueness, and redundancy, including unmatched candidates?

  3. If two candidates overlap with the same sounding object, could Hungarian matching assign one a positive label and the other a negative label, and how would this affect audio verification?

  4. Could the authors evaluate audio sensitivity using AVSBench-Robust and other mismatched audio cases, and explain why removing audio support causes such a small performance drop in Table 4?

  5. Could the authors clarify what complementary information coarse and fine matching provide, and give stronger evidence that FSC reduces representation entanglement compared to dense fusion?

Reviewer_NHaeOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper revisits Audio-Visual Segmentation (AVS) from the perspective of asymmetric modality roles. The authors argue that the visual modality should primarily be responsible for identifying entities and determining their spatial locations, whereas the audio modality is better suited for verifying which visual entities are acoustically active. Based on this motivation, the authors propose AEQNet, which reformulates conventional dense audio-visual fusion into a pipeline consisting of visual candidate construction, audio-based candidate verification, and structured semantic composition. The proposed method is evaluated on the S4, M3, and AVSS datasets. Its effectiveness is further examined through overall and component-level ablation studies, hyperparameter analyses, qualitative visualizations, and efficiency comparisons.

Critical strengths

Instead of following the commonly adopted paradigm of symmetric and dense audio-visual interaction, this work explicitly decouples visual entity construction from acoustic verification according to the different roles of the two modalities. This perspective is well motivated and provides an interesting alternative to conventional feature-fusion and query-based AVS frameworks.

Critical weaknesses

The current experiments demonstrate that the proposed modules lead to performance improvements, but the evidence is still insufficient to verify whether each module actually performs the specific function claimed in the paper. The ablation studies in Tables 2–4 mainly evaluate the contributions of HVCC, CF-AQR, FSC, and PCQL using overall segmentation metrics such as J, F, and J&F. The progressive performance gains after introducing these modules indicate that they are beneficial to the final segmentation results. However, based solely on these overall metrics, it is difficult to determine whether the improvements truly arise from the specific mechanisms claimed by the authors, rather than from increased model capacity or more general feature enhancement. The current qualitative results do not fully address this concern either. Figure 5 mainly illustrates the gradual improvement of the final segmentation masks as different modules are introduced, but the specific effect of each module is not sufficiently evident from these examples. Therefore, there remains a gap between demonstrating that the modules are effective and verifying that they function as intended.

Questions for authors
  1. Could the authors provide key module-level ablation studies on the AVSS dataset? The detailed ablation experiments are currently conducted mainly on S4 and M3, while AVSS differs substantially from these two settings. S4 and M3 focus on class-agnostic binary audio-visual segmentation, whereas AVSS requires semantic segmentation of sounding objects with explicit category labels. Since FSC explicitly performs structured semantic composition involving visual identity, spatial membership, and acoustic activation, and CF-AQR and PCQL may also behave differently under more challenging semantic discrimination scenarios, ablations limited to S4 and M3 are insufficient to fully demonstrate the contribution of these modules in semantic AVS. I therefore suggest that the authors provide at least key module-level ablations on AVSS to further verify the consistent effectiveness of the proposed components across different AVS settings.

  2. Could the authors provide more targeted experiments or visualizations to directly validate the claimed functionality of each module? The current performance improvements and final-mask visualizations mainly demonstrate that the proposed modules are effective, but they do not sufficiently establish whether the observed gains are produced by the mechanisms described in the paper. It would be helpful to include analyses that directly correspond to the intended role of each module, or to provide more informative qualitative examples. For example, for HVCC, the authors could report metrics such as the coverage of ground-truth objects by visual candidates and the overlap among different candidates, in order to examine whether the constructed candidates indeed form discriminative entity-level representations and reduce spatial ambiguity.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers (NHae, i8eV, and owoR) for their constructive feedback and for recognizing our well-motivated asymmetric perspective (NHae), coherent framework (i8eV), and underlying modeling assumptions (owoR). Their suggestions have significantly strengthened AEQNet's evaluation and presentation.

We summarize our main updates—including new experiments, analyses, and manuscript revisions—below:

  • Clarified motivation and strengthened comparisons with visual-first methods (i8eV, owoR): We revised the Abstract and Introduction to explicitly connect our asymmetric modeling strategy to audio-independent candidates, acoustic verification, and score-based routing. We expanded the related work to contextualize AEQNet against MoCA and VCT. We also added direct proposal-matching baselines: AEQNet improves J&F by 1.28 points on S4 and 1.89 points on M3 over global matching, and by 1.26 and 1.58 points over coarse matching.

  • Added progressive module-level ablations on AVSS for semantic evaluation (NHae): Progressively introducing HVCC, CF-AQR, FSC, and PCQL increases AVSS J&F from 43.51 to 48.12 (+4.61 points). Specifically, adding FSC improves J by 1.40 and F by 1.05 points, demonstrating the effectiveness of our structured composition in the semantic AVS setting.

  • Provided direct candidate analyses and clarified supervision (NHae, i8eV): We added visualizations of all eight candidates (matched and unmatched) in Figure 8 and reported quantitative diagnostics (e.g., Average Best IoU is 79.92% on S4 and 55.80% on M3). We also detailed the Hungarian assignment procedure and the distinct supervision paths for candidate masks and acoustic support.

  • Added controlled audio-substitution experiments and clarified acoustic-support ablations (i8eV): With visual inputs fixed, we evaluated silence, noise, and off-screen audio. The resulting J&F performance drops (28.34–50.34 points on S4 and 11.28–28.59 points on M3) confirm the model's responsiveness to audio cues. We also clarified the roles of explicit support weighting, FiLM modulation, and query residuals, identifying this variant as AS→AF (Alternative Audio Fusion).

  • Clarified complementary roles of coarse and fine matching and detailed the FSC comparison (NHae, i8eV): We explained that coarse matching evaluates pooled candidate compatibility, while fine verification aggregates local similarities within candidate regions. We also specified the internal operations of the dense-fusion variant, connecting our architectural choices directly to the empirical evidence.

  • Enhanced qualitative visualizations and evaluation transparency (owoR): We updated Figure 4 with original predictions from all compared methods, added further AVSS examples in Figure 7, and confirmed that all metrics were computed directly from raw model outputs prior to visualization processing. Code, scripts, and raw results will be released upon acceptance to ensure reproducibility.

  • Expanded efficiency reporting and improved presentation (i8eV, owoR): We reported AEQNet's throughput (49.31 FPS, the highest among compared methods) and added an Input Size column to Table 1. We also streamlined mathematical notations, detailed the progressive loss-weight schedule, and clarified that PCQL is used strictly during training without adding inference overhead.

We thank the reviewers for their valuable feedback. We have detailed the additional experiments in our response and updated the manuscript, highlighting the main revisions in red. These additions strengthen AEQNet's empirical support and clarify our contributions. We appreciate your time and welcome any further questions.

Best regards,

All #6221 Authors

AuthorsOpenReview ↗

Author Responses (continued)

Thank you for raising this concern. We would like to clarify that AI writing tools were used only to polish the language and improve readability. These tools were not used to generate or modify experimental data, reported results, or visualisation images. We take full responsibility for the manuscript’s content, the reported findings, and the conclusions drawn, and we remain firmly committed to research integrity.

Finally, we would like to express our sincere gratitude to the reviewers for their comments on this paper, which have helped us improve its quality; however, we have done our best to address some of the concerns raised.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.