- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes SCoPE, a training-free framework for audio-visual event perception. It addresses false co-activation through competitive sparse selection and cross-modal prior exchange. Experiments on three benchmarks demonstrate improvements over existing training-free methods.
Critical strengths
The paper presents a clearly motivated and interpretable framework for training-free audio-visual event perception. It identifies the limitations of threshold-based event selection and introduces an intuitive categorization of false co-activations into shadows and rivals. The proposed cross-modal prior exchange mechanism adjusts class-specific selection costs while preserving target-modality evidence. Extensive experiments across multiple datasets and pretrained backbones demonstrate competitive performance, supported by ablation studies and efficiency analyses.
Critical weaknesses
- The manuscript is generally well written and organized. However, some annotations in Figures 1 and 3 are difficult to read. Figure 2 could more clearly illustrate reconstruction reliability estimation and video-level prior aggregation.
- Sparse concept representations and adaptive regularization have been explored in previous studies [1–3]. Although cross-modal prior exchange is meaningful, its advantages over simpler class-specific score calibration strategies remain insufficiently demonstrated.
- Eq. (6) uses reconstruction similarity to estimate source-modality reliability, but good reconstruction does not necessarily imply correct event recognition. Moreover, Table 3 shows only marginal improvements from reconstruction weighting.The video-level prior in Eq. (6) is shared across all temporal segments, which may limit its ability to capture short-duration events and temporally changing event compositions.
- Sparse competition may suppress genuinely co-occurring events with similar representations. Table 4 evaluates false candidate rejection but does not fully establish whether all ground-truth events are preserved.
- Table 1 shows substantial improvements in individual audio and visual predictions, but only 0.57 and 0.65 percentage-point gains in Audio-Visual@seg F1 over AV2A. The reasons for this discrepancy remain unclear.
References:
[1] Bhalla, U., Oesterling, A., Srinivas, S., Calmon, F. P., and Lakkaraju, H. Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE). Advances in Neural Information Processing Systems (NeurIPS), 37: 84298–84328, 2024. DOI: 10.52202/079017-2678.
[2] Zhang, A., Thomaz, E., and Lu, L. Transformation of Audio Embeddings into Interpretable, Concept-Based Representations. Proceedings of the 2025 International Joint Conference on Neural Networks (IJCNN), pp. 1–8, 2025. DOI: 10.1109/IJCNN64981.2025.11229318.
[3] Zou, H. The Adaptive Lasso and Its Oracle Properties. Journal of the American Statistical Association, 101(476): 1418–1429, 2006. DOI: 10.1198/016214506000000735.
Questions for authors
- What are the distinctive advantages of cross-modal selection-cost adaptation over existing sparse representation methods and simpler class-specific score calibration strategies?
- How strongly does reconstruction similarity correlate with event prediction correctness? Can the authors provide additional evidence supporting the effectiveness of reconstruction-based reliability weighting?
- How does sparse competition affect genuinely co-occurring events, particularly semantically related classes? Can the authors report ground-truth event recall on multi-label segments?
- How does the shared video-level prior perform on short-duration events and videos containing temporally separated or alternating events?
- Why are the improvements in joint audio-visual prediction substantially smaller than those in individual audio and visual predictions? Can the authors provide a more detailed analysis of this observation?
10 comments
Response to Reviewer gxJb (Part 2/5)
W2 and Q1. How does SCoPE differ from existing sparse representations and adaptive regularization? Why is cross-modal selection-cost adjustment preferable to simpler class-specific score calibration?
1. Distinction from existing methods: intervention through the other modality
SCoPE is designed to apply to new data and events without task-specific training. It preserves the text–modality alignment of pretrained VLMs and ALMs while using evidence from the other modality to correct event selection.
The key idea is to extend human intervention in CBMs to cross-modal intervention, in which the other modality helps revise the selections in place of a human. Sparse selection in Stage 1 selects the events to pass from the source modality, and weighted regularization in Stage 2 incorporates these selections into the target modality’s selection costs.
The target receives this guidance while re-selecting events based on its own evidence. Classical formulations provide a simple, training-free implementation of this information exchange.
2. Why use target re-selection rather than simple score calibration?
Adding a prior directly to scores can strengthen an event in the target even when it is present only in the source.
SCoPE instead uses the prior to adjust target selection costs and reassesses whether each event explains the target evidence. Table 3 compares adding the same prior directly to target scores with using it to adjust selection costs for target re-selection.
SCoPE achieves higher Type@seg and Event@seg.
We further evaluated this difference by dividing LLP test videos into two groups. A video belongs to SYM if its video-level audio and visual GT label sets are identical, and to ASYM otherwise. ASYM includes labels present in only one modality, such as visible silent events or off-screen sound sources.
We compared Stage 1, direct fusion, and SCoPE within each subset. Values below are segment-level F1 (%).
| Method | SYM Type / Event | ASYM Type / Event |
|---|---|---|
| Stage 1 | 41.54 / 37.91 | |
| Direct fusion | 36.45 / 31.67 | |
| SCoPE | 56.40 / 51.80 | 49.08 / 35.15 |
* Bold denotes the best result for each metric, and underlining denotes the second best.
Direct fusion improves performance on SYM but reduces Type F1 below Stage 1 on ASYM.
SCoPE improves Type/Event F1 in both subsets.
We have added this comparison to Appendix B.4, Table 9.
This cross-modal re-selection can also be combined with dynamic thresholding. Appendix B.2 demonstrates this compatibility by applying a dynamic-threshold decoder to the same Stage 2 outputs.
Response to Reviewer gxJb (Part 3/5)
Reliability concern in W3 and Q2. What supports the use of reconstruction similarity as reliability?
1. How is reconstruction similarity r related to event prediction correctness?
r measures how well the embedding reconstructed from the selected labels matches the original input (Eq. 6).
For example, labels may still be selected when no candidate event is present or when the evidence is unclear due to camera shake or background noise. Segment-wise maximum normalization alone can give these selections a large contribution to the prior. When the selected labels do not reconstruct the input well, r reduces the segment’s contribution to the prior. Figure 3 illustrates how r weights segment-level selections when aggregating them into a video-level prior.
To examine the relationship between r and the correctness of the selections, we analyzed the fraction of total weight assigned to GT labels (GT weight fraction).
For each segment, we sum the Stage 1 weights assigned to GT labels and divide by the total weight.
For audio and visual separately, we ranked all segments with at least one label selected in Stage 1 by r, divided them into three equally sized groups, and averaged the per-segment fractions within each group.
* Fraction of total weight assigned to GT labels (%)
| r rank group | Audio | Visual |
|---|---|---|
| Lower third | 28.60 | 70.48 |
| Middle third | 43.75 | 81.46 |
| Upper third | 60.80 | 84.69 |
Even when segments with no GT labels are included, groups with higher r assign a larger fraction of their total weight to GT labels on average in both modalities.
This supports assigning larger prior contributions to segments with higher r.
2. What additional evidence supports reconstruction weighting?
r adjusts the contributions of already-selected labels rather than selecting new candidates.
The small additional gain in Table 3 is consistent with this role, after average prior strength is matched.
We additionally found that performance decreases on OV-AVEBench and VGGSound-AVEL100k without r weighting, with all other settings kept unchanged.
| Dataset | Metric | Unweighted | SCoPE |
|---|---|---|---|
| OV-AVEBench | Avg | 47.47 | 51.08 |
| VGGSound-AVEL100k | Segment F1 | 47.62 | 50.09 |
| VGGSound-AVEL100k | Event F1 | 32.95 | 35.26 |
Temporal-prior concern in W3 and Q4. How does a shared video-level prior handle short events and temporally separated or alternating events?
Leveraging temporal and cross-modal context in AVEP requires balancing broader contextual information with evidence specific to each segment. As the reviewer points out, brief events and changes in event composition pose challenges for such context-based approaches.
To use video-level context while allowing event selections to change over time, SCoPE shares a prior across segments but re-selects events using the target evidence at each segment.
1. Does the shared prior interfere with distinguishing temporally separated or alternating events?
The existing training-free state of the art, AV²A, sequentially updates class-specific thresholds using previous predictions and similarity between neighboring segments, and re-evaluates contiguous positive predictions at the chunk level (Shaar et al., 2025, §3, Eqs. (7)–(9)).
SCoPE solves re-selection separately for each segment, given the shared prior. Re-selection can change the selected events based on the current segment’s target evidence, without a constraint to continue the previous segment’s selections.
2. Can temporal averaging dilute evidence from short events?
Preserving evidence from short events while using sufficient temporal context is a shared challenge in context-based AVEP design.
Under SCoPE’s current temporal averaging, evidence from brief source events can contribute less to the prior.
Prior exchange itself is not restricted to temporal averaging.
We have stated in Limitations that temporal averaging can dilute brief source evidence in the video-level prior.
In Future directions, we have proposed temporal-decay priors that give greater weight to temporally nearby source evidence.
Response to Reviewer gxJb (Part 1/5)
Thank you for your review. We address figure presentation, differences from existing methods, reliability and temporal priors, multi-label GT preservation, and joint AV results in turn. The additional LLP analyses use the paper’s CLIP+CLAP SCoPE configuration. Unless otherwise specified, all section, figure, table, and equation numbers below refer to the revised manuscript.
W1. Can the small annotations and stage explanations in Figures 1 and 4 be improved? Can Figure 2 show reconstruction reliability and video-level prior aggregation more clearly?
1. Improving readability, stage labels, and captions in Figures 1 and 4
We expanded Figures 1 and 4 to the full text width and enlarged the text within them.
We also enlarged and thickened the check marks in Figure 1 to make the selected events easier to identify.
We labeled Figure 1(b) “SCoPE (ours)” to identify it clearly as an illustration of our approach.
We grouped the audio and visual sparse selections under Stage 1 and clarified how their selections lead to prior exchange and re-selection in Stage 2. Red annotations now identify only the FCA types remaining at each stage.
We also revised the caption to explain the roles of the stages and the meaning of the annotations.
In Figure 4, we standardized the box titles as GT event / Non-GT event and revised the caption to describe the changes in event selection directly for the Vacuum cleaner–Blender and Dog–Cat examples.
2. Clarifying prior construction in Figure 2 with the new Figure 3
Figure 2 was designed to show the overall flow of prior exchange and re-selection between the two modalities.
To make it clearer how segment-level selections form a single video-level prior, we added Figure 3 directly below Figure 2.
The new Figure 3 shows how visual Stage 1 selections at three time steps are averaged using reconstruction reliability r as weights to construct Prior V→A in Figure 2. Its caption connects the two figures and refers to §3.4 for the calculation. Figure 4 illustrates the effect of the prior on event selection through the Vacuum cleaner–Blender and Dog–Cat examples. Appendix B.3 (Qualitative Examples) details how the bidirectional prior values and selection-cost changes lead to re-selection and final predictions in these examples.
We also added a reference to Appendix B.3 in the main-text discussion of Figure 4.