acceptodds
Under review as a conference paper at ICLR 2027

SCoPE: Sparse Cross-modal Prior Exchange for Training-Free Audio-Visual Event Perception

Abstract

Audio-Visual Event Perception (AVEP) identifies which events occur, when they occur, and whether they are audible, visible, or both. This distinction matters for sound sources outside the camera's field of view or visible soundless objects. Frozen text-aligned encoders enable training-free recognition with flexible event vocabularies, but events often co-occur and are selected by thresholding scores. When a related incorrect event label scores as high as a correct one, no single threshold can separate them. We analyze this false co-activation (FCA) through two types, shadows and rivals. To address this, we introduce SCoPE, a framework for Sparse Cross-modal Prior Exchange. SCoPE first makes event labels compete to explain the evidence within each modality. Across modalities, prior exchange adjusts event selection costs. Then, target re-selection fits each modality's own embedding under these costs. SCoPE improves event prediction across diverse AVEP benchmarks using the same hyperparameter settings. SCoPE also supports causal streaming for applications that need continuous predictions from incoming audio and video.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
audio-visual event perception, audio-visual video parsing, training-free, open-vocabulary, sparse coding, cross-modal prior, causal streaming, CLIP, CLAP

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_gxJbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SCoPE, a training-free framework for audio-visual event perception. It addresses false co-activation through competitive sparse selection and cross-modal prior exchange. Experiments on three benchmarks demonstrate improvements over existing training-free methods.

Critical strengths

The paper presents a clearly motivated and interpretable framework for training-free audio-visual event perception. It identifies the limitations of threshold-based event selection and introduces an intuitive categorization of false co-activations into shadows and rivals. The proposed cross-modal prior exchange mechanism adjusts class-specific selection costs while preserving target-modality evidence. Extensive experiments across multiple datasets and pretrained backbones demonstrate competitive performance, supported by ablation studies and efficiency analyses.

Critical weaknesses
  1. The manuscript is generally well written and organized. However, some annotations in Figures 1 and 3 are difficult to read. Figure 2 could more clearly illustrate reconstruction reliability estimation and video-level prior aggregation.
  2. Sparse concept representations and adaptive regularization have been explored in previous studies [1–3]. Although cross-modal prior exchange is meaningful, its advantages over simpler class-specific score calibration strategies remain insufficiently demonstrated.
  3. Eq. (6) uses reconstruction similarity to estimate source-modality reliability, but good reconstruction does not necessarily imply correct event recognition. Moreover, Table 3 shows only marginal improvements from reconstruction weighting.The video-level prior in Eq. (6) is shared across all temporal segments, which may limit its ability to capture short-duration events and temporally changing event compositions.
  4. Sparse competition may suppress genuinely co-occurring events with similar representations. Table 4 evaluates false candidate rejection but does not fully establish whether all ground-truth events are preserved.
  5. Table 1 shows substantial improvements in individual audio and visual predictions, but only 0.57 and 0.65 percentage-point gains in Audio-Visual@seg F1 over AV2A. The reasons for this discrepancy remain unclear.

References:

[1] Bhalla, U., Oesterling, A., Srinivas, S., Calmon, F. P., and Lakkaraju, H. Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE). Advances in Neural Information Processing Systems (NeurIPS), 37: 84298–84328, 2024. DOI: 10.52202/079017-2678.

[2] Zhang, A., Thomaz, E., and Lu, L. Transformation of Audio Embeddings into Interpretable, Concept-Based Representations. Proceedings of the 2025 International Joint Conference on Neural Networks (IJCNN), pp. 1–8, 2025. DOI: 10.1109/IJCNN64981.2025.11229318.

[3] Zou, H. The Adaptive Lasso and Its Oracle Properties. Journal of the American Statistical Association, 101(476): 1418–1429, 2006. DOI: 10.1198/016214506000000735.

Questions for authors
  1. What are the distinctive advantages of cross-modal selection-cost adaptation over existing sparse representation methods and simpler class-specific score calibration strategies?
  2. How strongly does reconstruction similarity correlate with event prediction correctness? Can the authors provide additional evidence supporting the effectiveness of reconstruction-based reliability weighting?
  3. How does sparse competition affect genuinely co-occurring events, particularly semantically related classes? Can the authors report ground-truth event recall on multi-label segments?
  4. How does the shared video-level prior perform on short-duration events and videos containing temporally separated or alternating events?
  5. Why are the improvements in joint audio-visual prediction substantially smaller than those in individual audio and visual predictions? Can the authors provide a more detailed analysis of this observation?
10 comments
AuthorsOpenReview ↗

Response to Reviewer gxJb (Part 1/5)

Thank you for your review. We address figure presentation, differences from existing methods, reliability and temporal priors, multi-label GT preservation, and joint AV results in turn. The additional LLP analyses use the paper’s CLIP+CLAP SCoPE configuration. Unless otherwise specified, all section, figure, table, and equation numbers below refer to the revised manuscript.

W1. Can the small annotations and stage explanations in Figures 1 and 4 be improved? Can Figure 2 show reconstruction reliability and video-level prior aggregation more clearly?

 

1. Improving readability, stage labels, and captions in Figures 1 and 4

We expanded Figures 1 and 4 to the full text width and enlarged the text within them.

We also enlarged and thickened the check marks in Figure 1 to make the selected events easier to identify.

We labeled Figure 1(b) “SCoPE (ours)” to identify it clearly as an illustration of our approach.

We grouped the audio and visual sparse selections under Stage 1 and clarified how their selections lead to prior exchange and re-selection in Stage 2. Red annotations now identify only the FCA types remaining at each stage.

We also revised the caption to explain the roles of the stages and the meaning of the annotations.

In Figure 4, we standardized the box titles as GT event / Non-GT event and revised the caption to describe the changes in event selection directly for the Vacuum cleaner–Blender and Dog–Cat examples.

2. Clarifying prior construction in Figure 2 with the new Figure 3

Figure 2 was designed to show the overall flow of prior exchange and re-selection between the two modalities.

To make it clearer how segment-level selections form a single video-level prior, we added Figure 3 directly below Figure 2.

The new Figure 3 shows how visual Stage 1 selections at three time steps are averaged using reconstruction reliability r as weights to construct Prior V→A in Figure 2. Its caption connects the two figures and refers to §3.4 for the calculation. Figure 4 illustrates the effect of the prior on event selection through the Vacuum cleaner–Blender and Dog–Cat examples. Appendix B.3 (Qualitative Examples) details how the bidirectional prior values and selection-cost changes lead to re-selection and final predictions in these examples.

We also added a reference to Appendix B.3 in the main-text discussion of Figure 4.

AuthorsOpenReview ↗

Response to Reviewer gxJb (Part 2/5)

W2 and Q1. How does SCoPE differ from existing sparse representations and adaptive regularization? Why is cross-modal selection-cost adjustment preferable to simpler class-specific score calibration?

 

1. Distinction from existing methods: intervention through the other modality

SCoPE is designed to apply to new data and events without task-specific training. It preserves the text–modality alignment of pretrained VLMs and ALMs while using evidence from the other modality to correct event selection.

The key idea is to extend human intervention in CBMs to cross-modal intervention, in which the other modality helps revise the selections in place of a human. Sparse selection in Stage 1 selects the events to pass from the source modality, and weighted regularization in Stage 2 incorporates these selections into the target modality’s selection costs.

The target receives this guidance while re-selecting events based on its own evidence. Classical formulations provide a simple, training-free implementation of this information exchange.

2. Why use target re-selection rather than simple score calibration?

Adding a prior directly to scores can strengthen an event in the target even when it is present only in the source.

SCoPE instead uses the prior to adjust target selection costs and reassesses whether each event explains the target evidence. Table 3 compares adding the same prior directly to target scores with using it to adjust selection costs for target re-selection.

SCoPE achieves higher Type@seg and Event@seg.

We further evaluated this difference by dividing LLP test videos into two groups. A video belongs to SYM if its video-level audio and visual GT label sets are identical, and to ASYM otherwise. ASYM includes labels present in only one modality, such as visible silent events or off-screen sound sources.

We compared Stage 1, direct fusion, and SCoPE within each subset. Values below are segment-level F1 (%).

MethodSYM Type / EventASYM Type / Event
Stage 141.54 / 37.91\mmlTokenmtext[fontfamily="NotoSans",mathsize="1em"]43.01/31.92‾\underline{\mmlToken{mtext}[fontfamily="Noto Sans",mathsize="1em"]{43.01 / 31.92}}
Direct fusion\mmlTokenmtext[fontfamily="NotoSans",mathsize="1em"]53.24/49.41‾\underline{\mmlToken{mtext}[fontfamily="Noto Sans",mathsize="1em"]{53.24 / 49.41}}36.45 / 31.67
SCoPE56.40 / 51.8049.08 / 35.15

* Bold denotes the best result for each metric, and underlining denotes the second best.

Direct fusion improves performance on SYM but reduces Type F1 below Stage 1 on ASYM.

SCoPE improves Type/Event F1 in both subsets.

We have added this comparison to Appendix B.4, Table 9.

This cross-modal re-selection can also be combined with dynamic thresholding. Appendix B.2 demonstrates this compatibility by applying a dynamic-threshold decoder to the same Stage 2 outputs.

AuthorsOpenReview ↗

Response to Reviewer gxJb (Part 3/5)

Reliability concern in W3 and Q2. What supports the use of reconstruction similarity as reliability?

 

1. How is reconstruction similarity r related to event prediction correctness?

r measures how well the embedding reconstructed from the selected labels matches the original input (Eq. 6).

For example, labels may still be selected when no candidate event is present or when the evidence is unclear due to camera shake or background noise. Segment-wise maximum normalization alone can give these selections a large contribution to the prior. When the selected labels do not reconstruct the input well, r reduces the segment’s contribution to the prior. Figure 3 illustrates how r weights segment-level selections when aggregating them into a video-level prior.

To examine the relationship between r and the correctness of the selections, we analyzed the fraction of total weight assigned to GT labels (GT weight fraction).

For each segment, we sum the Stage 1 weights assigned to GT labels and divide by the total weight.

For audio and visual separately, we ranked all segments with at least one label selected in Stage 1 by r, divided them into three equally sized groups, and averaged the per-segment fractions within each group.

* Fraction of total weight assigned to GT labels (%)

r rank groupAudioVisual
Lower third28.6070.48
Middle third43.7581.46
Upper third60.8084.69

Even when segments with no GT labels are included, groups with higher r assign a larger fraction of their total weight to GT labels on average in both modalities.

This supports assigning larger prior contributions to segments with higher r.

2. What additional evidence supports reconstruction weighting?

r adjusts the contributions of already-selected labels rather than selecting new candidates.

The small additional gain in Table 3 is consistent with this role, after average prior strength is matched.

We additionally found that performance decreases on OV-AVEBench and VGGSound-AVEL100k without r weighting, with all other settings kept unchanged.

DatasetMetricUnweightedSCoPE
OV-AVEBenchAvg47.4751.08
VGGSound-AVEL100kSegment F147.6250.09
VGGSound-AVEL100kEvent F132.9535.26

Temporal-prior concern in W3 and Q4. How does a shared video-level prior handle short events and temporally separated or alternating events?

 

Leveraging temporal and cross-modal context in AVEP requires balancing broader contextual information with evidence specific to each segment. As the reviewer points out, brief events and changes in event composition pose challenges for such context-based approaches.

To use video-level context while allowing event selections to change over time, SCoPE shares a prior across segments but re-selects events using the target evidence at each segment.

1. Does the shared prior interfere with distinguishing temporally separated or alternating events?

The existing training-free state of the art, AV²A, sequentially updates class-specific thresholds using previous predictions and similarity between neighboring segments, and re-evaluates contiguous positive predictions at the chunk level (Shaar et al., 2025, §3, Eqs. (7)–(9)).

SCoPE solves re-selection separately for each segment, given the shared prior. Re-selection can change the selected events based on the current segment’s target evidence, without a constraint to continue the previous segment’s selections.

2. Can temporal averaging dilute evidence from short events?

Preserving evidence from short events while using sufficient temporal context is a shared challenge in context-based AVEP design.

Under SCoPE’s current temporal averaging, evidence from brief source events can contribute less to the prior.

Prior exchange itself is not restricted to temporal averaging.

We have stated in Limitations that temporal averaging can dilute brief source evidence in the video-level prior.

In Future directions, we have proposed temporal-decay priors that give greater weight to temporally nearby source evidence.

Reviewer_2TmQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SCoPE, a training-free framework for audio-visual event perception. SCoPE adjusts the cost of event selection through intra-modal sparse selection and cross-modal prior exchange, and subsequently refines event selection by leveraging the target modality’s own features, thereby reducing the co-activation of irrelevant or incorrect event categories.

Critical strengths

The motivation behind the problem is clear. The proposed method accounts for modality mismatches—such as instances where a source is visible but silent, or audio occurs off-screen—and the design, which combines cross-modal guidance with autonomous decision-making by the target modality, is well-founded.

The experimental results are robust. The method demonstrates performance gains across multiple datasets and with frozen backbones, while maintaining consistent hyperparameters during transfer. For instance, on the LLP dataset, the Type@seg metric improved from 43.11 to 52.33 (Tables 1–2).

The core design is supported by ablation studies. Re-selecting the target modality proves significantly more effective than directly fusing priors, indicating that the manner of information interaction has a substantial impact on performance. Furthermore, the methodology and implementation details are clearly articulated (Table 3).

Critical weaknesses

The boundaries of innovation still need to be clarified. The method is mainly based on the existing sparse representation and weighted regularization ideas. The current ablation has not fully demonstrated the independent value of the sparse prior, and it is recommended to add a dense similarity prior control under the same reselection framework.

Insufficient validation of error propagation. There is a lack of systematic testing of audio and video mismatches, silent objects, and off-screen sound sources. The reconstructed similarity does not necessarily represent semantic correctness, and the gain brought by reliability weighting is also small, which is not enough to fully prove its robustness. (Table 3)

Mechanism metrics may be affected by true label loss. The determination of FCA relies on the true labels retained by each method, while Table 4 only requires that at least one true event be retained. It is recommended to report FCA under complete true event retention and matching recall simultaneously to avoid interpreting changes caused by missed detections as false suppression. (Definition 1; Table 4)

Questions for authors

N/A

4 comments
AuthorsOpenReview ↗

Response to Reviewer 2TmQ (Part 1/3)

Thank you for your review. We address novelty, error propagation from the source modality, and the potential effect of GT loss on FCA measurements through additional analyses. The additional LLP analyses use CLIP+CLAP, with the conditions specified in each response. Unless otherwise specified, all section, figure, table, and equation numbers below refer to the revised manuscript.

W1. How does SCoPE differ from existing sparse representations and weighted regularization? What is the independent benefit of a sparse prior over a dense similarity prior under the same re-selection procedure?

 

1. Core contributions beyond existing sparse representations and weighted regularization

SCoPE’s core contribution is identifying false co-activation that threshold adjustment alone cannot resolve in training-free AVEP, and presenting a framework to analyze and address it.

SCoPE is designed to apply to new data and events without task-specific training. It preserves the text–modality alignment of pretrained VLMs and ALMs while using evidence from the other modality to correct event selection.

The key idea is to extend human intervention in CBMs to cross-modal intervention, in which the other modality helps revise the selections in place of a human. Sparse selection in Stage 1 selects the events to pass from the source modality, and weighted regularization in Stage 2 incorporates these selections into the target modality’s selection costs.

The target receives this guidance while re-selecting events based on its own evidence. Classical formulations provide a simple, training-free implementation of this information exchange.

2. Why Stage 1 refines the prior: the shadow and rival examples in Figure 1

Stage 1’s sparse selection is also an important part of this mechanism.

In the visual example in Figure 1(a), the shadow Cat has high similarity alongside the GT events Dog and Car. Passing these similarities directly to the audio prior could also lower the selection cost of the incorrect label Cat. Stage 1 makes candidates compete to explain the visual evidence, suppresses Cat, and passes the selections for Dog and Car. Audio Stage 2 uses this prior together with its own evidence to distinguish Car from the rival Motorcycle.

In Figure 1(b), Car remains selected, while the visible but silent Dog is excluded from the audio output.

3. Dense and top-k similarity priors under the same target re-selection

We kept the total prior weight equal to SCoPE’s but constructed the prior from the top-k candidates by similarity instead of sparse selection.

We held target re-selection and the decoder constant.

Source priorType@segEvent@seg
Dense similarity, k=2545.7336.07
Similarity top-1046.7435.72
Similarity top-548.6437.20
Similarity top-349.7538.91
Similarity top-250.9240.45
Similarity top-151.3042.03
SCoPE sparse prior52.3342.54

With total prior weight and target re-selection held constant, the SCoPE sparse prior achieves higher Type@seg and Event@seg than all similarity-prior baselines in this comparison.

AuthorsOpenReview ↗

Response to Reviewer 2TmQ (Part 2/3)

W2. Can source mispredictions or cross-modal event mismatch propagate to the target? Why use reconstruction similarity as reliability, and why is the additional gain from weighting small?

 

1. Can visible silent objects or off-screen sound sources cause false detections in the other modality? What happens when the source itself makes an incorrect prediction?

SCoPE passes the source prior but re-selects the final events using target evidence.

In Figure 1(b), Dog is visible but silent. Although the visual prior is passed to audio, Dog is not added to the audio output because audio performs re-selection using its own evidence.

To examine re-selection under such mismatch, we divided LLP test videos into two groups. A video belongs to SYM if its video-level audio and visual GT label sets are identical, and to ASYM otherwise. ASYM includes labels present in only one modality, such as visible silent events or off-screen sound sources.

We compared Stage 1, direct fusion, and SCoPE within each subset. Values below are segment-level F1 (%).

MethodSYM Type / EventASYM Type / Event
Stage 141.54 / 37.91\mmlTokenmtext[fontfamily="NotoSans",mathsize="1em"]43.01/31.92‾\underline{\mmlToken{mtext}[fontfamily="Noto Sans",mathsize="1em"]{43.01 / 31.92}}
Direct fusion\mmlTokenmtext[fontfamily="NotoSans",mathsize="1em"]53.24/49.41‾\underline{\mmlToken{mtext}[fontfamily="Noto Sans",mathsize="1em"]{53.24 / 49.41}}36.45 / 31.67
SCoPE56.40 / 51.8049.08 / 35.15

* Bold denotes the best result for each metric, and underlining denotes the second best.

Direct fusion improves performance on SYM but reduces Type F1 below Stage 1 on ASYM.

SCoPE improves Type/Event F1 in both subsets.

We have added this comparison to Appendix B.4, Table 9.

The Blender example in Figure 4(a) illustrates a source misprediction. Blender is incorrectly selected in audio, and the resulting prior lowers its visual selection cost from 0.300 to 0.187.

Nevertheless, re-selection based on visual evidence assigns Blender a value of at most 0.022, which the sparse readout removes from the final output. Details of the prior and selection-cost changes are provided in Appendix B.3 (Qualitative Examples).

2. Does higher reconstruction reliability r correspond to more weight assigned to GT labels?

r measures how well the embedding reconstructed from the selected labels matches the original input (Eq. 6).

For example, labels may still be selected when no candidate event is present or when the evidence is unclear due to camera shake or background noise. Segment-wise maximum normalization alone can give these selections a large contribution to the prior. When the selected labels do not reconstruct the input well, r reduces the segment’s contribution to the prior. Figure 3 illustrates how r weights segment-level selections when aggregating them into a video-level prior.

To examine the relationship between r and the correctness of the selections, we analyzed the fraction of total weight assigned to GT labels (GT weight fraction).

For each segment, we sum the Stage 1 weights assigned to GT labels and divide by the total weight.

For audio and visual separately, we ranked all segments with at least one label selected in Stage 1 by r, divided them into three equally sized groups, and averaged the per-segment fractions within each group.

* Fraction of total weight assigned to GT labels (%)

r rank groupAudioVisual
Lower third28.6070.48
Middle third43.7581.46
Upper third60.8084.69

Even when segments with no GT labels are included, groups with higher r assign a larger fraction of their total weight to GT labels on average in both modalities.

This supports assigning larger prior contributions to segments with higher r.

3. Why is the additional gain from reconstruction weighting small in Table 3? Does it also play a role on other datasets?

r adjusts the contributions of already-selected labels rather than selecting new candidates.

The small additional gain in Table 3 is consistent with this role, after average prior strength is matched.

We additionally found that performance decreases on OV-AVEBench and VGGSound-AVEL100k without r weighting, with all other settings kept unchanged.

DatasetMetricUnweightedSCoPE
OV-AVEBenchAvg47.4751.08
VGGSound-AVEL100kSegment F147.6250.09
VGGSound-AVEL100kEvent F132.9535.26
AuthorsOpenReview ↗

Response to Reviewer 2TmQ (Part 3/3)

W3. Could the reduction in FCA result from true-label loss?

 

To examine whether the FCA rejection in Table 4 results from GT loss, we compared Stage 1 and Stage 2 over the full set of FCA candidates used in Table 4.

To address your concern, we provide two analyses below: candidate rejection while preserving all GT labels, and rejection at matched recall.

1. Complete-GT preservation: what fraction of FCA candidates can be rejected while preserving all GT labels?

For each FCA candidate in Table 4, requiring all GT labels in the same segment to be retained, we checked whether a threshold exists that rejects the candidate while preserving all GT labels under the same stage-specific processing conditions as Table 4.

The values below report the fraction of all FCA candidates from Table 4 that satisfy this condition. If a segment contains multiple FCA candidates, each is evaluated separately.

* Fraction of FCA candidates that can be rejected while preserving all GT labels (%)

ModalityTypeStage 1Stage 2
AudioShadow9.1822.30
AudioRival3.2837.45
VisualShadow0.009.88
VisualRival0.0043.85

In Stage 2, the fraction of FCA candidates that can be rejected while preserving all GT labels increases in all four cases.

This shows that more candidates can be rejected even under the requirement that all GT labels be preserved.

2. Matching recall: does rival rejection improve when the same number of GT labels is detected?

We kept the Stage 1 and Stage 2 scores unchanged and matched recall by varying a global threshold in place of the largest-gap readout. Recall was matched to the lower of the two stages’ original values, with interpolation between neighboring threshold results.

The table reports FCA rejection at matched recall: the fraction of all FCA candidates from Table 4 that are rejected while at least one GT label remains in the same segment.

* FCA rejection at matched recall (%)

ModalityTypeStage 1Stage 2
AudioShadow96.3984.92
AudioRival11.1733.02
VisualShadow100.0093.83
VisualRival1.7633.30

The same pattern as Table 4 holds at matched recall. Rival rejection increases in both modalities, while shadow rejection decreases somewhat.

Thank you for your questions about the role of each mechanism and GT preservation.

We would appreciate your feedback on whether our responses and the revised manuscript address your concerns.

We would be happy to provide any further clarification.

AuthorsOpenReview ↗

Additional stress test for error propagation

Thank you for raising the question of error propagation in W2. Our SYM/ASYM analysis groups videos by whether their annotated audio and visual event sets agree. In an ASYM video, even a correctly recognized source event can be absent from the target modality. To address your concern about errors originating in the source itself, we tested how target performance changes when the source supplies a misleading prior.

For each LLP test video ii, we randomly selected another video jj with no ground-truth event label in common, taking the union of audio and visual labels for each video. For each source modality ss and event class cc, we formed the mixed prior

P~i,s(c)=(1−ρ)Pi,s(c)+ρPj,s(c).\tilde{P}_{i,s}(c) = (1-\rho)P_{i,s}(c) + \rho P_{j,s}(c).

The mixing weight ρ is the weight assigned to the source prior from video jj. At ρ = 0.50, the two videos contribute equally for every class. We used the same video jj in both directions and kept the target inputs, Stage 1 predictions, decoder, and temporal post-processing fixed. The table reports Event@seg (%) on the full LLP test set.

Mixing weight ρSCoPEDirect fusionStage 1 (no cross-modal interaction)Dense
042.54\mmlTokenmtext[fontfamily="NotoSans",mathsize="1em"]39.54‾\underline{\mmlToken{mtext}[fontfamily="Noto Sans",mathsize="1em"]{39.54}}34.5818.26
0.2541.41\mmlTokenmtext[fontfamily="NotoSans",mathsize="1em"]37.29‾\underline{\mmlToken{mtext}[fontfamily="Noto Sans",mathsize="1em"]{37.29}}34.5818.26
0.5039.7231.48\mmlTokenmtext[fontfamily="NotoSans",mathsize="1em"]34.58‾\underline{\mmlToken{mtext}[fontfamily="Noto Sans",mathsize="1em"]{34.58}}18.26
0.7536.7425.83\mmlTokenmtext[fontfamily="NotoSans",mathsize="1em"]34.58‾\underline{\mmlToken{mtext}[fontfamily="Noto Sans",mathsize="1em"]{34.58}}18.26
1.00\mmlTokenmtext[fontfamily="NotoSans",mathsize="1em"]31.07‾\underline{\mmlToken{mtext}[fontfamily="Noto Sans",mathsize="1em"]{31.07}}22.8134.5818.26

* In each row, the best score is shown in bold and the second-best score is underlined.

At ρ = 0.50, SCoPE scores 39.72 and direct fusion scores 31.48, compared with 34.58 for Stage 1. SCoPE remains above Stage 1 at ρ = 0.75 (36.74).

Reviewer_Yyt7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SCoPE introduces a training-free framework for audio-visual event perception. It constructs sparse event representations from frozen embeddings and exchanges cross-modal priors through class-specific selection costs. The paper analyzes false co-activation and evaluates the approach on three benchmarks with multiple frozen encoder configurations.

Critical strengths
  • Practical and coherent approach with consistent empirical improvements. Results across three datasets and multiple encoder configurations support the method’s effectiveness.
  • Extensive ablation studies. The comparisons between independent sparse selection, direct fusion, and prior-guided re-selection help establish the value of the proposed interaction mechanism.
Critical weaknesses
  • Narrow comparative evaluation. Beyond dense scoring, the main comparisons primarily involve AV²A and OV-AVEL. These are relevant baselines, but the limited methodological coverage makes it difficult to assess SCoPE’s broader competitiveness in audio-visual event perception.
  • Missing comparison with multimodal large language models. Audio-video-capable models such as Gemini, Qwen2.5-Omni and Qwen3-Omni are neither evaluated nor adequately positioned as alternative approaches. Whether they can perform the same event recognition, temporal localization, and modality assignment through prompting remains unexplored, leaving SCoPE’s relative accuracy, generalization, and computational advantages unclear.
  • Uncertain generality of the inference rules. The framework combines sparse reconstruction, manually specified prior-to-cost mappings, and coefficient-gap decoding. Although the fixed configuration transfers across the tested benchmarks, robustness to changes in vocabulary composition, label granularity, and event co-occurrence remains insufficiently established.
Questions for authors
  • Table 1 uses AV²A as the only task-specific training-free comparator. Can you justify the coverage of the selected baselines and broaden the comparison to other relevant approaches, clearly distinguishing their training requirements and inference inputs?
  • Can you evaluate a representative audio-video multimodal language model with or without task-specific fine-tuning, using the same candidate labels and temporal evaluation protocol? Reporting both accuracy and inference cost would clarify SCoPE’s practical advantages.
  • With hyperparameters fixed, how does SCoPE perform when the candidate vocabulary is expanded with semantically related labels or changed in granularity? Can you also assess whether its gains persist for scenes with more simultaneously occurring events?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Yyt7 (Part 1/2)

Thank you for your review. We address the training and input requirements of the comparison methods, direct AV-LLM comparisons, and multi-event performance under fixed settings below. Unless otherwise specified, all section, figure, table, and equation numbers below refer to the revised manuscript.

W1 and Q1. What is the rationale for baseline selection, and can the comparison be broadened to other approaches? How do their training requirements and inference inputs differ?

 

1. Baseline selection and the training requirements of HSCHG and Cue2Rule

Our main comparisons focus on training-free AVEP methods.

We also considered HSCHG and Cue2Rule, which are discussed in Related Work. HSCHG trains a cross-modal graph model on task-specific data. Cue2Rule freezes its encoders but trains a policy network to predict video-specific decision-rule parameters. Both therefore differ from our training-free setting.

We have also been unable to obtain their official runnable code and trained checkpoints, so we have not included direct re-evaluations under our protocol.

2. Comparisons with trained methods: Appendix B.7, Table 11

To provide performance references for methods that use task-specific training, Appendix B.7, Table 11 also includes trained AVVP methods and temporally fine-tuned OV-AVEL.

MethodsTask-specific trainingLocation in the paper
Dense, AV²A, OV-AVEL, SCoPENoneTables 1–2
MGN-MA, JoMoLD, CMPAEWeak supervisionAppendix B.7, Table 11(a)
OV-AVEL, fine-tunedTemporal trainingAppendix B.7, Table 11(b)

W2 and Q2. Can AV-LLMs be evaluated with the same candidate labels and temporal evaluation protocol? What are their accuracy and inference costs?

 

1. LLP: multi-event and modality-specific predictions with Qwen2.5-Omni

We evaluated Qwen2.5-Omni-7B, one of the models suggested by the reviewer, without task-specific fine-tuning.

We evaluated Qwen on the same LLP test videos as SCoPE.

We used the same candidate labels and requested separate audio and visual event predictions for each second.

We evaluated five prompts specifying the same task and output schema, using the same temporal post-processing as SCoPE and the official F1 metrics, and report the mean of the per-prompt F1 scores.

We added these results to Table 1.

MethodType@seg (%)Event@seg (%)
Qwen2.5-Omni-7B24.4728.05
SCoPE — CLIP+CLAP52.3342.54

2. OV-AVEBench: VideoLLaMA2.1 localization performance

We also directly evaluated VideoLLaMA2.1-7B-AV without task-specific fine-tuning. On OV-AVEBench, we used the same candidate labels and complete test set as SCoPE.

We evaluated official segment/event F1 and applied the same temporal post-processing as SCoPE.

We added the VideoLLaMA2.1 results to Table 2.

We also include the paper’s results for the existing training-free OV-AVEL baseline. OV-AVEL and SCoPE both use CLIP+CLAP backbones.

MethodSeg F1 (%)Event F1 (%)
OV-AVEL42.9232.85
VideoLLaMA2.1-7B-AV40.0334.88
SCoPE50.1045.75

3. Inference cost comparison under the same conditions

We measured inference costs on the same OV-AVEBench videos.

We added these costs to Table 6 and described the evaluation and timing protocols in Appendix C.2.

MethodTime (ms/video)
OV-AVEL356.23
SCoPE383.66
Qwen2.5-Omni-7B2,510.45
VideoLLaMA2.1-7B-AV2,796.87
AuthorsOpenReview ↗

Response to Reviewer Yyt7 (Part 2/2)

W3 and Q3. With hyperparameters fixed, how does performance change when semantically related candidate labels are added or label granularity is changed? Do the gains persist with more simultaneous events?

 

1. Do the gains persist with more simultaneous events?

With the paper’s settings fixed, segment F1 improves over Stage 1 in all three groups: one, two, and three or more simultaneous events.

We partitioned LLP test segments by the number of GT events in each modality (one, two, or three or more) and, keeping each method’s predictions unchanged, evaluated the segments in each group with the official segment F1 function.

* Values below are segment F1 (%) averaged over videos containing such segments.

Number of GT eventsAudio: Stage 1 → SCoPEVisual: Stage 1 → SCoPE
141.05 → 52.0755.88 → 70.38
232.81 → 40.7131.24 → 38.43
3 or more23.43 → 26.4321.22 → 26.46

2. Is performance on existing events preserved when semantically related candidate labels are added?

Using dog-related events common to both datasets as the reference, we added the same five or ten related animal candidates to each existing vocabulary. Candidates were selected from a predefined animal pool using centered CLIP and CLAP text similarity, excluding classes already present in either dataset.

The +5 vocabulary adds Wolf, Bear, Pig, Fox, and Raccoon; +10 additionally includes Deer, Goat, Coyote, Rabbit, and Sheep.

We reran the full inference pipeline with the paper’s settings and hyperparameters unchanged and measured performance changes on the existing evaluation labels.

With five or ten related animal candidates added, the changes relative to the original vocabulary were within 1.0 percentage point for LLP Type@seg and Event@seg, and within 0.3 percentage points for OV-AVEBench Avg.

3. How do changes in label granularity affect SCoPE?

SCoPE follows the event vocabularies and GT definitions of the AVEP benchmarks used in this work. It evaluates the correspondence between each candidate event and the audio/visual evidence.

The setting does not require reasoning about parent–child concept hierarchies or jointly predicting labels at multiple hierarchical levels. There is also precedent for removing redundant concepts when constructing a vocabulary. SpLiCE prunes concepts with highly similar text embeddings [1]. A study of concept-based audio representations uses a pruned vocabulary that merges synonyms and expressions sharing the same root into representative concepts [2]. Likewise, SCoPE does not assume that synonyms for the same concept must be distinguished as separate events. Distinguishing finer-grained classes requires discriminative evidence from the frozen encoders.

Prior exchange can use such evidence when it is available in the other modality, but remains limited when neither encoder can distinguish the classes.

[1] SpLiCE, §4.1.

[2] Transformation of audio embeddings into interpretable, concept-based representations, §III-A.

Thank you for your specific suggestions, which helped us broaden the comparisons and strengthen the evaluation.

We would appreciate your feedback on whether our responses and the revised manuscript address your concerns.

We would be happy to provide any further clarification.

Thank you for evaluating Qwen2.5-Omni and VideoLLaMA under matched label and temporal evaluation protocols. The accuracy and latency results address my main concern about AV-LLM comparisons. The LLP breakdown also shows that cross-modal prior exchange still improves over Stage 1 when three or more events occur, although the gains are smaller.

My remaining concern is the scope of the vocabulary experiment. Could you report how F1 changes for the original dog-related classes when the animal candidates are added? This would show whether the stable aggregate scores also hold for the classes most exposed to confusion. I would also appreciate an explicit qualification that robustness to changes in label granularity remains untested.

I raise my rating to 3.

AuthorsOpenReview ↗

Vocabulary expansion and label granularity

Thank you for specifying these additional checks. We examined the original Dog class under the vocabulary expansions in our previous response and further evaluated the effect of candidate selection. Hyperparameters and ground-truth labels were kept unchanged throughout.

1. Candidates from our previous response

Using the same candidates as in our previous response, +5 adds wolf, bear, pig, fox, and raccoon; +10 additionally includes deer, goat, coyote, rabbit, and sheep. The table reports AV segment F1 for the original Dog class. All values are percentages.

Added candidatesLLP DogOV-AVEBench Dog
Base65.6582.26
+565.8682.45
+1066.6082.89

Dog segment F1 did not decrease under these vocabulary expansions on either dataset.

2. AudioSet candidates: similarity-based selection

To analyze vocabulary expansion using a dataset-defined vocabulary, we formed a pool of nine AudioSet candidates that do not duplicate existing classes: pig, goat, sheep, duck, pigeon, owl, housefly, snake, and whale. We compare selection by mean CLIP and CLAP text-embedding similarity to Dog with the mean over all combinations.

Added candidatesLLP DogOV-AVEBench Dog
Base65.6582.26
+165.0582.10
+365.0482.72
+566.2882.54
+966.1582.64

3. AudioSet candidates: mean over all combinations

We averaged all possible combinations for +1, +3, and +5 (9, 84, and 126 combinations, respectively), giving the exact mean under uniform random selection from the same pool. Both approaches coincide at +9.

Added candidatesLLP DogOV-AVEBench Dog
Base65.6582.26
+165.8582.23
+365.5482.40
+565.9182.50
+966.1582.64

Across the AudioSet similarity-based results and all-combination means, Dog segment F1 stays within 0.7 points of the baseline on LLP and 0.5 points on OV-AVEBench.

4. Overall benchmark performance

The table reports overall performance: Type@seg / Event@seg for LLP, and segment F1 / event F1 for OV-AVEBench. LLP metrics are evaluated on the original benchmark classes after decoding with the expanded vocabulary. On OV-AVEBench, predictions of added classes count as errors.

CandidatesAddedLLPOV-AVEBench
Base052.33 / 42.5450.10 / 45.75
Previous response+552.69 / 43.5349.93 / 45.56
Previous response+1052.53 / 43.3049.91 / 45.52
AudioSet similarity+152.38 / 42.8550.16 / 45.83
AudioSet similarity+352.62 / 43.3549.97 / 45.64
AudioSet similarity+552.52 / 43.5549.82 / 45.54
AudioSet mean+152.34 / 42.8250.06 / 45.73
AudioSet mean+352.38 / 43.2349.96 / 45.63
AudioSet mean+552.41 / 43.5549.87 / 45.55
All AudioSet candidates+952.51 / 44.1449.60 / 45.33

Across all expansion conditions in the table, LLP Type@seg and Event@seg exceed the baseline. OV-AVEBench segment and event F1 change by at most 0.50 and 0.42 points, respectively.

5. Label granularity

From a hierarchical perspective, when Dog is present, its parent category Animal also holds. Our evaluation uses the benchmark event labels without adding their parent categories as ground-truth labels. SCoPE does not explicitly model parent-child relations among candidate labels.

SCoPE defines candidates through text, so coarser or finer labels can be supplied without retraining. Measuring recognition at those granularities, however, requires ground truth annotated at the same level. The experiments above keep the benchmarks' original ground truth and therefore evaluate vocabulary expansion, not changes in label granularity. We have stated in the Limitations paragraph of Section 6 that hierarchy modeling and evaluation across label granularities are outside the scope of this study.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (Part 2/2)

3. Main changes in the revised manuscript

 

  • Added AV-LLM comparisons (Yyt7): Tables 1–2 report Qwen2.5-Omni and VideoLLaMA2.1 results; Table 6 reports inference costs. Evaluation and timing protocols are in Appendix C.2.

  • Added cross-modal mismatch analysis (2TmQ, gxJb): Appendix B.4, Table 9 presents the SYM/ASYM comparisons, referenced from §4.3.

  • Improved presentation (Yyt7, gxJb): We enlarged Figures 1 and 4 and refined their text and captions, clarified Figure 1(b)’s stages and check marks, and added Figure 3 below Figure 2 to illustrate reliability-weighted prior construction.

  • Added joint AV analysis (gxJb): Section 4.5 analyzes how AV²A and SCoPE form AV predictions and examines why the improvement is smaller for joint AV than for Audio and Visual.

We thank the reviewers again for their thoughtful feedback, which helped us clarify the presentation and strengthen the evaluation.

AuthorsOpenReview ↗

General Response (Part 1/2)

General Response to All Reviewers

 

We thank all three reviewers for their careful reviews and specific questions. Before the individual responses, we summarize the core contributions and the analyses added for this rebuttal.

We have uploaded a revised manuscript to accompany these responses. Unless otherwise specified, all section, figure, table, and equation numbers below refer to the revised manuscript.


1. Core contributions

 

We identify false co-activation (FCA) that threshold adjustment alone cannot resolve in training-free AVEP, and present a framework to analyze and correct it. When an incorrect event scores as high as or higher than a correct event, a threshold alone cannot reject that candidate while retaining every correct event. We analyze this problem through two types, shadows and rivals.

SCoPE is designed to apply to new data and event classes without task-specific training by using the text–modality alignment of pretrained vision–language models (VLMs) and audio–language models (ALMs). Preserving the pretrained text–modality alignment allows SCoPE to use cross-modal evidence without task-specific training.

We extend human intervention in CBMs to cross-modal intervention: the source modality provides a prior, and the target re-selects events using its own evidence. Classical formulations provide a simple, training-free implementation of this idea.


2. Additional analyses and experiments

 

Direct AV-LLM comparison (Yyt7 W2/Q2).

Following the reviewer’s suggestion, we compared Qwen2.5-Omni on LLP and VideoLLaMA2.1 on OV-AVEBench under the same evaluation protocol as SCoPE. SCoPE achieved higher F1 in both comparisons. Under our OV-AVEBench timing protocol, SCoPE also had lower measured runtime than both Qwen2.5-Omni and VideoLLaMA2.1.

GT retention and FCA reduction (2TmQ W3).

To avoid interpreting true label loss as FCA reduction, we examined complete true event retention and matching recall as requested by the reviewer. In Stage 2, thresholds that preserve all GT labels while rejecting an FCA candidate exist for a larger fraction of candidates. With matching recall, rival rejection increases while shadow rejection decreases somewhat, following the pattern in Table 4.

Multiple events and related candidate additions (Yyt7 W3/Q3).

Segment F1 improves in all three groups with one, two, or three or more co-occurring events. We also added five or ten related animal candidates around Dog, an event shared by both datasets, and evaluated how performance changes as related candidates are added.

Basis for reconstruction reliability (2TmQ W2; gxJb W3/Q2).

If the source selections do not adequately explain the input, their influence on the other modality should also be reduced. Segment-wise maximum normalization alone does not capture this explanatory quality. We therefore weight each segment’s prior contribution by r, the similarity between the input embedding and its reconstruction from the selected labels. In both modalities, higher-r groups have a higher average fraction of weight assigned to GT labels. Using weight r also consistently improved performance on OV-AVEBench and VGGSound-AVEL100k.

Roles of the sparse prior and target re-selection (2TmQ W1, W2; gxJb W2/Q1).

We compared the sparse prior with dense and top-k similarity priors to examine the effect of selecting which source candidates to pass. We also compared adding the same prior directly to the scores with target re-selection. To examine whether events present only in the other modality lead to false detections in the target, we analyzed the LLP test subset in which the audio and visual GT label sets differ. On this subset, direct fusion lowers Type/Event F1 relative to Stage 1, whereas SCoPE improves both metrics. This supports using target evidence to re-select events when incorporating information from the other modality.

Analysis of the joint AV performance gap (gxJb W5/Q5).

To examine why the performance improvement over AV²A is smaller for joint AV than for Audio and Visual, we analyzed how the two methods form AV predictions and the effect of score fusion.

Temporal prior design and limitations (gxJb W3/Q4).

We explained the design in which re-selection uses the target evidence at each time step despite sharing a prior, and the limitation that temporal averaging can dilute brief source evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.