acceptodds
Withdrawn from ICLR 2027

STAR-AVL: Structured Temporal-Aware Reasoning for Training-Free Video-Centric Audio-Visual Localization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video-centric Audio-Visual Localization (AVL) aims to identify the spatial location of sound-emitting objects by jointly interpreting audio and temporally changing visual information. Existing methods mainly rely on feature-level similarity between audio and visual embeddings, which limits their ability to explicitly reason about the audio-visual context and re-evaluate predictions using temporal information. To address these limitations, we propose STAR-AVL (Structured Temporal-Aware Reasoning for Audio-Visual Localization), a training-free structured reasoning framework built on Multimodal Large Language Models (MLLMs). STAR-AVL consists of two complementary components, Audio-Visual Reasoning and Temporal Context Calibration, which are together organized into four reasoning steps. Audio-Visual Reasoning identifies the actual sound-emitting objects by verifying the audio-visual evidence, while Temporal Context Calibration resolves the remaining localization ambiguity by leveraging the surrounding temporal context. On the AVATAR dataset, STAR-AVL consistently outperforms existing video-centric AVL and CoT reasoning methods. Additional experiments on VGGSound-Duet demonstrate its generalization to image-centric AVL settings. The source code will be publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Audio-Visual Localization, Multimodal Large Language Models, Multimodal Reasoning, Temporal Reasoning, Training-Free

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_n8vsOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper is motivated by the observation that existing sound source localization methods rely on audio-visual feature similarity, which only establishes category-level correspondence. The model cannot tell whether an object is actually sounding, so silent objects of the same category get localized too. Generic CoT prompting lets an MLLM reason in steps, but it does not choose the right modality for each step, and it never revisits earlier predictions with temporal context. The paper proposes STAR-AVL, a training-free structured prompting pipeline that turns video-centric audio-visual localization into four reasoning steps, and claims three contributions: the framework itself; two complementary components, one deciding which object is sounding and the other resolving the remaining localization ambiguity; and experiments on AVATAR and VGGSound-Duet against trained methods, single-step MLLMs on the same backbones, and three CoT approaches.

The presentation and soundness of is paper is OK, but my main concern is methodological. This is a routine combination of CoT and localization. The recipe of turning audio into a textual reference and then localizing the sounding object from it is already done by OpenAVS, and training-free multi-step MLLM reasoning for this task by GAR. The only new element is the temporal dimension: history conditioning and a neighboring-frame re-check, and the re-check is essentially smoothing. The experiments do not establish much beyond "multi-step prompting with temporal smoothing compensates for a weak backbone". The control that would decide the question, per-frame GAR plus simple smoothing, is missing, and the central hypothesis is never tested directly.

Critical strengths

The problem framing is reasonable. Feature similarity alone cannot tell a sounding object from a silent one of the same category, and the paper says so. This is a real reason why AVGN-style trained methods (Mo & Tian, 2023) have long underperformed on AVATAR.

The step decomposition hangs together. AA sees only audio, AVM sees audio-visual input, LOC sees vision plus history. Each step gets the modality its decision actually needs. That said, coherence is not novelty (Weakness 1).

TR is cheap. Triggering only on suspicious frames keeps latency below every CoT baseline, and Figure 7 shows two kinds of genuine fixes: box-scale corrections and recovery of missed entities.

Critical weaknesses
  1. The method is a routine CoT-plus-localization combination, and a nearly identical paper exists. The core chain, turning audio into a textual reference (Step 1, audio analysis) and then localizing the sounding object from that reference (Step 2, audio-visual matching, and Step 3, localization), has direct precedents. OpenAVS [1] does the same thing with a training-free pipeline of audio-to-text prompt, LLM translation, and text-to-sounding-object segmentation, and GAR [2] applies training-free multi-step MLLM reasoning to sound source localization. Task decomposition with reflective correction is standard practice from Least-to-Most and Step-Back prompting. What is left is the temporal dimension, and it is thin: history conditioning is ordinary context conditioning, and the temporal-aware re-evaluation (Step 4) is essentially smoothing. The method section separates itself from prior work in a single sentence. So what the paper adds over these methods is small, and this is the main reason for my score.

  2. The experiments do not isolate what the framework adds. GAR appears only in Table 6 on VGGSound-Duet, with cited numbers, and not at all on the main benchmark AVATAR. The obvious control for the temporal claim, running GAR per frame on AVATAR and then adding simple smoothing, was never done; if it recovers most of the gain, the framework adds little. Table 3 also confounds structure with call count, since the full pipeline takes about three times as long per frame as the single-step baseline and there is no control matched on the number of calls. The central hypothesis is never tested directly either: there is no muted or swapped audio test, no count of boxes on silent distractors, and no accuracy for the Step 1 labels or Step 2 visibility decisions. Without these controls, the results only show that multi-step prompting with temporal smoothing makes up for a weak backbone. Table 4 points the same way: the gain shrinks on the strongest backbone, and most of the headline lead comes from the backbone itself.

References

[1] Chen, Yin, Cao, Xiang, Liu & Zimmermann. OpenAVS: Training-free open-vocabulary audio visual segmentation with foundational models. ICME 2026 (arXiv:2505.01448).

[2] Park & Kim. Generate, analyze, and refine: Training-free sound source localization via MLLM meta-reasoning. CVPR 2026.

Questions for authors
  1. How does GAR [1] perform when run per frame on AVATAR, with and without simple temporal smoothing?
  2. How does the method compare with a control matched on the number of MLLM calls?
  3. Could you add a muted or swapped audio test, report how often boxes land on silent distractors, and give the accuracy of the Step 1 labels and Step 2 visibility decisions?
  4. On the cross-event subset (Table 5), STAR-AVL loses much more from Total to Cross-event than TAVLO, and its gain over its own baseline shrinks. Sound-source switching is where temporal reasoning should help most. How do you explain this?
  5. The AUC and CIoU columns of the MLLM rows in Table 1 are nearly identical digit for digit. How was the threshold sweep configured? Also, how are the low-resolution saliency maps of the trained methods scored against box outputs? With this format mismatch, "outperforms all audio-visual models" says very little.
  6. How are multi-entity matches scored, and how are empty outputs handled?
  7. AVATAR is described only as a test set. Were the prompts designed on it or on a held-out split? Please also report decoding configurations.
  8. Do the results hold on a second video-centric benchmark? All video-centric results come from AVATAR, and VGGSound-Duet is a stitched image-centric setting.

References

[1] Park & Kim. Generate, analyze, and refine: Training-free sound source localization via MLLM meta-reasoning. CVPR 2026.

Reviewer_hgQsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies audio-visual localization, aiming to locate entities that are actively producing sound in video frames using audio and visual information. The authors propose STAR-AVL, a training-free structured reasoning pipeline built on existing multimodal large language models. The pipeline sequentially performs audio analysis, audio-visual matching, and spatial localization, followed by selective re-evaluation of predictions deemed temporally inconsistent using the initial predictions of neighboring frames. The authors evaluate localization performance and adaptability to changing sound sources on AVATAR and assess applicability to image-centric multi-source localization on VGGSound-Duet. The reported results show improvements over direct-prediction baselines across multiple backbone models and better performance than the methods compared in the evaluated settings.

Critical strengths
  • The method requires no additional training while delivering substantial performance gains over direct-prediction baselines across multiple backbone models.
Critical weaknesses
  • The methodological novelty is limited, and the necessity of the proposed task decomposition is not sufficiently established. The cumulative ablation study demonstrates incremental gains as stages are added but does not show whether each stage is necessary or whether a simpler pipeline could achieve comparable performance. Moreover, the pipeline requires more model calls than direct prediction, increasing inference cost and reducing the practical appeal of its performance gains. Comparisons of matched inference budgets would help clarify this trade-off. Finally, the paper lacks a systematic analysis of failure cases in the direct-prediction and CoT baselines that would explain why the proposed decomposition addresses their key limitations.
Questions for authors
  1. How does the method compare with simpler baselines under the same inference budget?
  2. Following on question 1, could you add ablations to show whether each stage is necessary?
  3. Could you analyze baseline errors to explain why the proposed decomposition is appropriate?
Reviewer_jKbiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes STAR-AVL, a training-free framework based on Multimodal Large Language Models (MLLMs) for video-centric Audio-Visual Localization (AVL). It consists of two components: Audio-Visual Reasoning, which includes audio analysis (AA), audio-visual matching (AVM), and localization (LOC), and Temporal Context Calibration, which selectively re-evaluates predictions using neighboring frames (TR). The framework aims to identify actual sound sources and improve localization through structured reasoning and temporal context. Experiments on AVATAR show improvements over existing AVL methods and general CoT reasoning approaches. Additional experiments on VGGSound-Duet are conducted to evaluate its generalization.

Critical strengths
  1. The paper identifies important challenges in video-centric AVL. It points out that an object matching the sound category is not necessarily the actual sound source. It also considers the temporal consistency of localization results across frames.

  2. The method has a clear multi-stage design. STAR-AVL consists of four steps: AA, AVM, LOC, and TR, covering sound classification, visibility verification, localization, and temporal re-evaluation. Verifying the sound source before localization and re-evaluating inconsistent predictions are reasonable design choices.

  3. The experimental evaluation is reasonably adequate. The paper reports results on AVATAR and VGGSound-Duet, along with experiments using different backbones and ablation studies.

Critical weaknesses
  1. The novelty over existing works is unclear. The paper proposes two main components: Audio-Visual Reasoning and Temporal Context Calibration. However, their differences from existing methods need to be better explained. GAR [1] has already explored multi-stage MLLM reasoning, including sound classification, audio-visual consistency checking, and prediction refinement. TAVLO [2] has also studied temporal modeling for video-centric AVL. The authors should clarify what is new in STAR-AVL compared with these methods and what additional benefits the proposed components bring.

  2. Potential error accumulation from historical predictions. If an early frame is incorrectly predicted, this error may affect subsequent frames. How does the framework prevent such error propagation? In addition, the framework uses initial predictions YAALY^{AAL} as historical context P1:t−1P_{1:t-1}, rather than the final predictions after TR refinement. Why are the initial predictions used instead of the refined predictions as historical contex

  3. The role of raw audio AtA_t in AVM is unclear. The AA stage has already extracted the sound label ltl_t and confidence score sts_t. Why does AVM need the raw audio$A_t$ again? What additional information does it provide, and how much does it contribute to the final performance? An ablation study would help clarify its necessity.

  4. Insufficient comparison on AVATAR. Table 1 mainly compares STAR-AVL with conventional feature-based AVL methods and general-purpose MLLMs using single-step inference, while Table 2 evaluates general CoT strategies adapted to AVL. However, existing MLLM-based sound source localization methods, such as GAR [1], are not sufficiently discussed or compared. Are there existing MLLM-based methods specifically designed for video-centric AVL? The authors should clarify this and better position STAR-AVL with respect to prior work.

  5. Fairness of the GAR comparison on VGGSound-Duet. In Table 6, the reported GAR results (CAP: 43.4, CIoU: 58.9, AUC: 38.1) correspond to the N=1 setting. However, the GAR paper [1] reports better results with N=5, achieving 47.2, 77.6, and 45.8, respectively. Why do the authors choose N=1 instead of the best reported N=5 setting? The authors should clarify this choice and provide a fairer comparison.

[1] Subin Park and Jung Uk Kim. Generate, Analyze, and Refine: Training-Free Sound Source Localization via MLLM Meta-Reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15752–15761, 2026.

[2]Hahyeon Choi, Junhoo Lee, and Nojun Kwak. What's Making That Sound Right Now? Video-centric Audio-Visual Localization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 20095–20104, 2025.

Minor Weaknesses:

  1. Inconsistent use of sts_t. Figure 2 indicates that sts_t is used in the AVM stage, but it is not included in Equation (3). The authors should clarify this and make the figure, equation, and prompt consistent.

  2. More implementation details in the main text. The functions gLOCg_{LOC} and fTRf_{TR} are introduced through equations, but their implementations are mainly described in the supplementary material. The authors are encouraged to briefly explain how gLOCg_{LOC} triggers temporal re-evaluation and how fTRf_{TR} uses neighboring-frame predictions to refine the results in the main text. This would make the method easier to understand.

Questions for authors

The author can respond to Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.