acceptodds
Under review as a conference paper at ICLR 2027

Structuring Spatial Audio Representations via Invariance-Equivariance Learning

Abstract

A spatial audio representation must jointly capture scene semantics to recognize what is sounding and scene geometry to understand where sound events occur. Self-supervised objectives often favor semantic invariance, treating direction as a nuisance, making spatial geometry difficult to preserve. We propose a framework for structuring spatial audio representations through invariance–equivariance learning, using geometric transformations as a training signal. In first-order ambisonics (FOA), rotating a sound field is a matrix operation, providing rotated views and their relative rotations without labels. We train encoders to remain stable under spectral nuisances while predicting how their representations change under rotation. Within this framework, we compare representations explicitly partitioned into invariant and equivariant subspaces with an unsplit predictive latent. The latter, built on seq-JEPA, is to our knowledge the first joint-embedding predictive architecture (JEPA) for spatial audio that learns invariance and equivariance jointly. We show that this latent representation organizes direction as an ordered spherical map while preserving linearly separable sound classes without an architectural split. When frozen, it achieves a lower SELD score on real recordings than supervised baselines and improves spatial audio captioning and retrieval, with the largest gains in spatial accuracy. These results demonstrate a path toward robust, general-purpose spatial audio representations that jointly encode what is present and where it occurs in space.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
self-supervised learning, spatial audio, equivariance, invariance, joint-embedding predictive architectures, sound event localization and detection, ambisonics, captioning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7h83OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes an invariance–equivariance learning framework for spatial audio representations, using rotations of FOA recordings as a label-free training signal. The central idea is to encourage the representation to remain invariant to spectral nuisance transformations such as noise, gain, and time-frequency masking, while changing predictably under rotations of the acoustic scene. The authors instantiate this framework with three approaches: SIE, SimCLR-EquiMod, and a predictive seq-JEPA-based method that aggregates multiple rotated views into an unsplit latent representation. All methods use the same Conformer backbone and the same FOA-based input representation, allowing the authors to focus on the effect of the learning objectives. The learned representations are evaluated through linear probing, SELD, and spatial audio-language tasks. The proposed methods encode spatial information substantially better than several supervised and self-supervised baselines in linear probing. The paper further analyzes the latent geometry and shows that the seq-JEPA representation forms smooth trajectories under continuous rotations.

Critical strengths
  1. A clear and well-motivated self-supervised formulation for spatial representation learning.
    The use of analytically generated FOA rotations as a label-free signal is elegant and particularly well suited to spatial audio. Rather than treating rotation merely as data augmentation, the paper explicitly asks the representation to change predictably under scene rotation while remaining stable to spectral nuisance transformations. This gives a clear inductive bias for preserving spatial information without requiring DoA annotations.

  2. Strong transfer to realistic downstream spatial tasks.
    The learned representations show meaningful benefits beyond the synthetic single-source probing task. In particular, frozen seq-JEPA outperforms supervised EINV2, and it also performs well on ScaperSynth and TAU-NIGENS without retraining. The improvements in spatial audio-language transfer are also substantial. These results support the claim that the learned representation contains useful spatial structure that transfers to downstream tasks.

Critical weaknesses
  1. The demonstrated scope is primarily limited to FOA spatial audio.
    The core transformation mechanism relies on the exact Wigner-D rotation operation for FOA, and all reported representation learning experiments use FOA-based inputs. This is a particularly clean setting for studying the proposed idea, but the broader claim of a general-purpose spatial audio representation is therefore somewhat stronger than what is directly demonstrated. It remains unclear whether the same learned representation principle transfers to other spatial audio formats or microphone-array recordings where the corresponding transformation is not as direct.
Questions for authors

Q1. Potential generalization beyond FOA spatial audio. How much does the proposed framework depend on the FOA representation? Since the rotation operation is particularly clean and exact for FOA, could the authors clarify whether the same formulation can be applied to higher-order ambisonics or microphone-array recordings?

2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer 7h83

We are grateful for the reviewer's careful reading, and encouraged that the central idea came through as intended: rotation used not as an augmentation but as a signal the representation must respond to predictably (S1), with benefits that carry over to realistic downstream tasks (S2). The question of scope is a fair one, and answering it has helped us state more precisely what the paper demonstrates and what the framework permits.


W1, Q1. Concern: The demonstrated scope is limited to FOA; it is unclear whether the principle transfers to higher-order ambisonics or microphone-array recordings.

Action/Clarification. In our framework, only the input action DRD_R is format-specific. The objectives, the encoder, the rotation token, and the learned latent action ρR\rho_R make no assumption about the number of channels or the ambisonic order. We chose first-order ambisonics because it is the format of the available real datasets (STARSS23, TAU-NIGENS, Sphere360), not because the method requires it.

Higher-order ambisonics. The rotation we use is the first-order case of a general operator. An ambisonic signal of order NN has (N+1)2(N+1)^2 channels, and a rotation acts on it through a block-diagonal matrix with one (2ℓ+1)×(2ℓ+1)(2\ell+1)\times(2\ell+1) Wigner-DD block per order ℓ=0,…,N\ell = 0,\dots,N. Our DRD_R (Eq. 3) is exactly the N=1N=1 case: the 1×11\times1 order-0 block and the 3×33\times3 order-1 block MRM⊤MRM^\top. Second order appends a 5×55\times5 block, and the third order appends a further 7×77\times7 block. The operator remains an exact, invertible, label-free per-sample multiply, and the objective and training are unchanged. We state this as a property of the operator; we have not yet run higher-order experiments and leave them to future work.

Microphone-array recordings. We agree raw array signals admit no closed-form rotation; in practice they are encoded to the ambisonic domain, after which the same rotation and training apply. Part of our evaluation already takes this route: STARSS23 and TAU-NIGENS are captured with a 32-channel spherical array and converted to FOA, so the real-data SELD results in Tables 2 and 3 are on array recordings through an ambisonic front end. For arrays used without encoding, permuting the microphones of a symmetric array rotates the scene exactly, though only by the coarse angles the geometry allows. We will list arrays without such symmetry as a limitation and a direction for future work.

Why FOA is the right first setting. Our experiments are on FOA, which has become an industry standard for spatial audio: it is the format of 360° video and AR/VR audio, of commercial ambisonic microphones, and of the main public spatial audio datasets, and it is independent of the recording array. A representation learned on FOA therefore already serves a large share of practical spatial audio processing. The approach is not tied to this format, however. As outlined above, it carries over to higher orders and to array recordings, because what it relies on is rotation equivariance itself, which our experiments show to be a strong signal for self-supervised spatial audio representation learning. In the revised version, we will state the FOA scope of the experiments explicitly and incorporate this discussion.


We thank the reviewer again for the supportive assessment. The revised version will also include a direct measurement of equivariance on held-out rotations and ablations isolating rotation conditioning, which make the evidence for the predictable response to rotation noted in S1 quantitative. We are happy to expand on any of these points.

AuthorsOpenReview ↗

Follow up on our response.

Dear Reviewer 7h83, thank you for your feedback and the question on scope.

We clarified that our rotation is the first-order case of the Wigner-D action, extending to higher orders by adding blocks, and that array recordings can reach this pipeline through ambisonic encoding, as our STARSS23 and TAU-NIGENS results already do. We would welcome your view on whether this resolves the concern.

Reviewer_E9nuOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The authors design and train a spatial audio model training framework, capable of both semantic identification and location isolation. They do so by exploring invariance-equivariance losses that assert invariance of semantic labels on augmentation of input, and equivariance of the location(s) of sound. Specifically, methods that are explored: SIE, SimCLR-Equimod and seq-JEPA.

Critical strengths
  • The work puts forth a valuable exploration and discussion of invariance-equivariance class of loss functions for spatial audio encoder training.
  • Using seq-JEPA in this setting is a good idea, and the analysis done in Figure 2 was interesting.
Critical weaknesses
  • Results for semantic accuracy aren't strong enough to justify moving over to the class of loss functions discussed in the paper. (Table 1, column 1), at least in its current design. If the key idea is to introduce a joint encoding (spatial and semantic) loss, the semantic performance should be SOTA.
  • It doesn't seem entirely clear if one loss out of the 3 is consistently better(table 2,3 - SIE, seq-JEPA seem fairly competitive) - do the authors see a clear winner?
  • It is unclear how to compare the baselines discussed in table 1. Are all the methods trained on the dataset? What are the model sizes?

Smaller nits:

  • The discussion in the results section can be written in a more palatable manner - its quite dense and takeaways are hard to identify.
  • Missing Related Work/baseline - ELSA(Devnani et al)
  • 2 out of the 3 losses were not discussed in section 5.3
Questions for authors

See weaknesses above.

I believe the foundational line of exploration in this paper is interesting. However, the results don't seem convincing enough for the reasons discussed in the weakness section(also - it is not entirely clear to me that SSL methods aren't good enough). Perhaps the execution of this idea can be strengthened.

5 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer E9nu [1/2]

We are glad the reviewer sees value in studying invariance–equivariance losses for spatial audio encoders (S1) and found the seq-JEPA formulation and its analysis interesting (S2). The questions raised here pushed us to state the paper's conclusions more plainly and to support them with new experiments, several of which were also requested by Reviewer ZowE.


W1. Concern: Semantic accuracy is not strong enough to justify this class of losses; if the idea is a joint spatial-semantic loss, the semantic performance should be state-of-the-art.

Action. We do not claim state-of-the-art on semantics accuracy. GRAM-T (82.4) and EINV2 (79.3) exceed our encoders on the Table 1 classification probe; however, we note that the probe is single-source synthetic. The semantic performance on the real downstream tasks is already in Tables 2 and 5; we evaluated GRAM-T, the strongest semantic encoder, on every task and summarize the comparison below (to appear in Tables 2–5 of the revised version).

Encoder (frozen)Probe CLS ↑STARSS23 SELD ↓STARSS23 LE ↓STARSS23 LR ↑Capt. SPIDEr ↑Capt. Spatial Acc. ↑
EINV2 (supervised)79.30.57825.60.464.463.459
GRAM-T (SSL)82.40.56730.40.492.509.576
seq-JEPA (ours)67.70.57024.20.392.492.657

Condensed from Tables 1, 2 and 5 of the revised paper. STARSS23 is real recordings. EINV2 is trained end-to-end for SELD and is the Spatial-CLAP encoder for captioning. We compare GRAM-T and seq-JEPA as frozen encoders, the protocol under which GRAM-T was introduced. Fine-tuning a different backbone would reflect its architecture and capacity as much as its pretraining objective.

We fine-tune what we propose, as we have full control over the method. Fine-tuning GRAM-T can open more questions. None of the baseline models are fine-tuned.

Finding. The gap is confined to the probe. seq-JEPA scores 15 points below GRAM-T on the single-source CLS probe, but the gap does not persist on the real tasks: all three encoders achieve comparable SELD scores (0.567–0.578), seq-JEPA's SPIDEr (.492) is close to GRAM-T's (.509) and above EINV2's (.463), and frozen seq-JEPA attains the lowest localization error (24.2°) and the highest spatial captioning accuracy (.657); fine-tuned, it attains the best overall SELD score (0.540). Two factors account for the probe gap. (1) It reflects the backbone rather than the loss: a Conformer trained end-to-end on the same data attains 68.8%, and frozen seq-JEPA attains 67.7% without labels. (2) Classification is a tunable operating point rather than a ceiling: in the ablations requested by Reviewer ZowE, the full model attains 63.6%, the invariance-only variant 68.6%, and a single-context-view variant 72.7%, each at a cost in rotation and elevation error. The operating point we report prioritizes spatial structure. An important direction for future work is to improve the semantic discrimination capabilities of the backbone while preserving its spatial representations, potentially through more robust and expressive encoders.

AuthorsOpenReview ↗

Rebuttal to Reviewer E9nu [2/2]

W2, Q1. Concern: It is not clear whether one of the three losses is consistently better; do the authors see a clear winner?

Is there a clear winner? (W2) Yes: seq-JEPA. On STARSS23 it has the best SELD score under both protocols (frozen 0.570 vs 0.611 for SIE and 0.626 for SimCLR-EquiMod; fine-tuned 0.540 vs 0.564 and 0.601) and the best localization error, including on both cross-dataset sets when fine-tuned (Table 3). The reviewer is right that SIE is competitive: it matches seq-JEPA on localization recall when fine-tuned, and in the new direct equivariance measurement (Appendix A.7) it is the most exactly equivariant encoder. A SIE latent gives the cleanest rotation action; the seq-JEPA predictive latent transfers best, and needs only one forward pass and no rotation tokens at inference. We will state this recommendation explicitly in Section 5.2 of the revised version.

Are existing SSL methods already sufficient? (Q1) For source semantics and classification, yes, for localization and scene geometry, no. AVSA is at chance on direction. GRAM-T, the strongest existing SSL encoder, has a higher localization error than frozen seq-JEPA on all three SELD datasets, and the equivariance analysis shows why: its latent barely moves under rotation (displacement 0.04 vs 0.42, Table 10). Conversely, removing rotation conditioning from our own model triples its rotation error with data and views unchanged (Table 11). The spatial structure comes from the equivariance objective, and it is what current SSL encoders lack.


W3. Concern: It is unclear how to compare the Table 1 baselines: are all methods trained on the same data, and what are the model sizes?

Our three encoders share the backbone, the 512-dimensional latent, the pretraining data, and the training recipe, so differences among them come from the objective. No encoder is trained on the probe data: all are frozen and evaluated with the same probes on the same held-out set. To make the comparison transparent, we will add to Appendix A.4 of the revised version a comprehensive table listing the training data and parameter count of every encoder used in this work (our three encoders, GRAM-T, AVSA, SELDnet, and EINV2).


W4. Concern: (Miscellaneous) The results section is dense, ELSA (Devnani et al.) is missing, and two of the three losses are not discussed in Section 5.3.

Results section. In the revised version, we will restructure Section 5 so that each subsection opens with its takeaway, followed by the supporting numbers.

ELSA. We thank the reviewer for the pointer and will discuss ELSA (Devnani et al., 2024) in the related work. We looked into potentially using ELSA as a baseline but it is a closed-source model. In addition, ELSA learns spatially aware audio–text embeddings with caption supervision, whereas we study label-free pretraining of the audio encoder; our language experiments follow the more recent Spatial-CLAP protocol and replace only its spatial encoder.

Section 5.3. The language tasks use the recommended encoder, seq-JEPA, since each further encoder requires training new alignment and captioning heads; we will say so at the start of the section and add GRAM-T as a second encoder.


We appreciate the reviewer's view that this line of work is promising and that its execution could be strengthened. We hope the added baseline, the controls, and the direct equivariance measurements speak to that, and that the clearer account of what the objective does and does not deliver makes the contribution easier to assess.

AuthorsOpenReview ↗

Follow up on our response.

Dear Reviewer E9nu, thank you for the constructive feedback.

We clarified that the semantic gap is specific to the single-source synthetic probe and does not persist on the real tasks (Tables 2, 5), identified seq-JEPA as the recommended loss, clarified the Table 1 protocols, and added ELSA. We would be glad to know whether this addresses your concerns before the discussion period closes.

Reviewer_ZowEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies self-supervised spatial audio representations that preserve both sound semantics and source direction. It exploits analytically defined sound-field rotations in first-order Ambisonics (FOA), combining invariance to spectral augmentations with equivariance to rotation. Using a shared Conformer backbone, the authors compare SIE, SimCLR-EquiMod, and seq-JEPA. The first two organize invariant and equivariant information through a partition or separate projection heads, whereas seq-JEPA predicts a target representation from multiple views and their relative rotations. Evaluation covers representation probing on synthetic audio, sound event localization and detection (SELD) on real and cross-dataset recordings, and spatial audio retrieval and captioning. The results indicate that seq-JEPA retains decodable directional information and outperforms the evaluated supervised baselines on several downstream measures, while its sound classification accuracy remains below models such as GRAM-T.

Critical strengths
  1. The training signal has a clear physical basis and addresses an important spatial audio problem. FOA rotations provide precisely related views with known geometric transformations, offering a principled way to learn spatial information from recordings without direction annotations. Combining invariance to spectral perturbations with equivariance to rotation is also well motivated by the different responses of sound semantics and direction to these transformations. This makes the paper a valuable investigation of transferring established representation learning approaches to spatial audio.

  2. The downstream evaluation demonstrates practically meaningful transfer of spatial information. The experiments include real recordings, cross-dataset SELD, and audio–language tasks, extending beyond a single synthetic localization benchmark. On STARSS23, fine-tuned seq-JEPA achieves a SELD score of 0.540 compared with 0.578 for EINV2. With a frozen encoder, spatial captioning accuracy improves from 0.459 for Spatial-CLAP to 0.657. These results suggest that the learned spatial information is usable across downstream tasks and constitute the strongest evidence supporting acceptance.

Critical weaknesses
  1. The experiments do not sufficiently isolate the contribution of the central equivariance learning mechanism. Although the three methods share a backbone, they also differ in the number of views, aggregation architecture, objective, and collapse prevention mechanism. Relative to the supervised baselines, the proposed models additionally receive synthetic and real pretraining data. Consequently, the results do not adequately distinguish the effects of rotation-conditioned prediction, multi-view training, and richer data. The paper lacks key controls that remove or shuffle rotation conditioning while matching data, view count, and compute.

  2. GRAM-T, the most relevant self-supervised baseline in the probing experiments, is absent from the main SELD and language comparisons. Since the contribution is primarily empirical, these omissions materially limit the conclusions about the proposed mechanism and its relative advantages.

  3. The central representation is inconsistently defined, and its claimed geometric structure is insufficiently validated. Section 3.4.2 and Appendix A.1 state that downstream tasks use the predicted representation z^=ρR(s)\hat z=\rho_R(s), whereas Appendix A.3 describes the backbone output z=fθ(x)z=f_\theta(x) as the representation actually probed and transferred. These alternatives imply different inference procedures, input requirements, and computational costs.

  4. The smooth closed trajectories in Figure 2 do not establish equivariance: rotating an input by 360∘360^\circ restores the input and therefore also restores the output of a deterministic encoder. The spatial subspace is extracted using supervised direction regression, and the semantic residual is visualized using supervised LDA, making these plots exploratory evidence. Direct measurements of equivariance error on held-out rotations and transformation-composition consistency would provide stronger support for the central claim of an organized spatial representation.

Questions for authors
  1. Which representation is used in each downstream experiment: fθ(x)f_\theta(x), the aggregate ss, or the prediction z^\hat z? Please specify the complete inference procedure for one test recording, including the number of views LL, construction of rotation tokens, and extraction of frame-level SELD features.

  2. With data, view count, and compute held fixed, how does removing or shuffling rotation conditioning affect localization and SELD? Are comparisons with GRAM-T available under the same downstream decoder and training protocol? These results would directly affect my assessment of the contribution.

  3. Are direct equivariance-error or rotation-composition results available for held-out rotations? Are the spatial regression and projections in Figure 2 fitted and evaluated on separate data?

5 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer ZowE [1/3]

We thank the reviewer for the careful and thorough review, and for noting the importance of this work (S1) and the meaningful downstream evaluation methodology (S2).

We appreciate the constructive set of concrete requests, which have strengthened the paper. We address each below; all new results will be incorporated in the revised manuscript.


W1, Q2. Concern: The experiments do not sufficiently isolate the contribution of the central equivariance learning mechanism.

Action: Following the reviewer's concerns, we ran ablations varying one factor at a time with the encoder, pretraining data, and training budget fixed (key rows below). These use a 250-epoch schedule rather than the 1k epochs of Table 1 in the paper, so values are lower, but the patterns are already clear.

The final version will use fully trained models. We will add these ablations to the main body of the final manuscript.

MethodVariantCLS ↑Rot MAE ↓DoA azi ↓DoA ele ↓
seq-JEPA (L=3L=3)full (reference)63.67.428.413.4
no rotated views (invariance-only)68.614.829.217.6
no spectral aug. (equivariance-only)45.45.630.815.0
rotated views, no rotation conditioning66.223.948.130.7
seq-JEPA, viewsL=1L=172.719.129.520.6
L=2L=266.28.324.814.3
L=3L=363.67.428.413.4
L=5L=563.011.627.316.2
SIE (All)full (reference)57.612.629.017.9
invariance-only (no equivariance loss)61.932.870.539.3
equivariance-only (no invariance loss)38.610.727.817.1

Excerpt of Table 11 of the revised paper. Linear probes on the held-out single-source set; rotation and DoA errors in degrees. Bold: best within each block.

Findings:

Rotation conditioning induces spatial structure. The spatial structure comes from rotation-conditioned prediction, not from multi-view training or the data. With the same views, data, and compute, withholding the rotation token leaves classification intact (66.2 vs 63.6) but triples the rotation error (7.4° to 23.9°) and degrades DoA (28.4° to 48.1° in azimuth, 13.4° to 30.7° in elevation). Without the rotation, the predictor can only match the target by becoming invariant to it, which drops direction from the latent. SIE shows the same effect when its equivariance loss is removed (12.6° to 32.8° in rotation, 29.0° to 70.5° in azimuth) (SimCLR-Equimod results not added to this rebuttal table for brevity).

Each loss term has a distinct role. Semantics come from the invariance term, rotation structure from the equivariance term, and the full model needs both. Training seq-JEPA with spectral views only keeps azimuth decodable (29.2°) but doubles the rotation error (14.8°); removing spectral augmentation gives the best rotation error (5.6°) while classification collapses (45.4).

Views. By aggregating a few rotated views, rotation error falls from 19.1° (L=1L=1) to 7.4° (L=3L=3) and does not improve beyond that. We find that two or three context views are enough, as we see L=5L=5 is slightly worse. However, we acknowledge that this gain in spatial performance comes with a drop in classification performance (72.7 to 63.6). We will note this in the paper.

We would like to clarify that the remaining factors are controlled by design. All three methods in our experiments share the same backbone, data, view budget, and compute (Sec. 3.3), so the differences in Table 1 in the paper come only from the objective.

AuthorsOpenReview ↗

Rebuttal to Reviewer ZowE [2/3]

W2, Q2. Concern: GRAM-T, the most relevant self-supervised baseline, is absent from the main SELD and language comparisons.

Action. We agree GRAM-T is the most relevant self-supervised baseline, and have evaluated it on every downstream task; the results will appear in Tables 2–5 of the revised version. The protocol matches our encoders: for SELD, GRAM-T is frozen and paired with the same multi-ACCDOA decoder, training data, and three seeds; for the language tasks, it is frozen in place of the Spatial-CLAP spatial encoder.

SELD (ST = STARSS23, SS = ScaperSynth, TN = TAU-NIGENS)

EncoderST SELD ↓ST LE ↓ST LR ↑SS SELD ↓SS LE ↓SS LR ↑TN SELD ↓TN LE ↓TN LR ↑
GRAM-T0.56730.40.4920.56324.60.6020.59227.80.570
seq-JEPA (frozen)0.57024.20.3920.56922.70.4940.61726.60.331
seq-JEPA (fine-tuned)0.54023.60.4430.54119.80.5850.58925.30.467

Retrieval and captioning (all encoders frozen)

Encoder (frozen)T2A R@1 ↑A2T R@1 ↑Dir. Acc. ↑SPIDEr ↑DW-SBERT ↑Spatial Acc. ↑
S-CLAP (EINV2)15.814.767.9.463.188.459
GRAM-T17.518.474.8.509.219.576
seq-JEPA17.818.678.8.492.228.657

Condensed from Tables 2–5 of the revised paper. LE in degrees; SELD means over 3 seeds. Retrieval and Dir. Acc. are retrieval metrics; SPIDEr, DW-SBERT and Spatial Acc. are captioning metrics. Bold: best; italic: second best.

Finding. GRAM-T is a strong baseline. It has the highest localization recall on all three SELD sets and the highest SPIDEr, consistent with its classification lead in Table 1. Frozen seq-JEPA, however, achieves lower localization error (LE) than GRAM-T on every dataset (24.2° vs 30.4° on STARSS23, 22.7° vs 24.6° on ScaperSynth, 26.6° vs 27.8° on TAU-NIGENS). The two frozen encoders reach a comparable SELD score (0.570 vs 0.567); however, fine-tuned seq-JEPA achieves the best overall SELD score (0.540). The language tasks show the same division: recall and content are comparable, with GRAM-T ahead on SPIDEr, while seq-JEPA achieves higher directional accuracy (78.8 vs 74.8) and spatial accuracy (.657 vs .576). Reconstruction yields the stronger what, rotation equivariance the stronger where, neither using labels. We will state this in Sections 5.2 and 5.3 of the revised version.


W3. Concern: The central representation is inconsistently defined.

Action. We thank the reviewer for the close reading. This was an oversight in the writing and not a difference in the experiments: every downstream result uses the encoder output zz.

  • Representation used downstream. For seq-JEPA, all downstream experiments (probing, SELD, retrieval, captioning) use the online encoder output z=fθ(x)z = f_\theta(x), a single unsplit 512-dimensional vector. For SIE and SimCLR-EquiMod they use the projector outputs zinv=ginv(fθ(x))z_{\mathrm{inv}} = g_{\mathrm{inv}}(f_\theta(x)) and zequi=gequi(fθ(x))z_{\mathrm{equi}} = g_{\mathrm{equi}}(f_\theta(x)), with the "All" variant in the tables is their concatenation.
  • Components used only in pretraining. The prediction z^=ρR(s)\hat z = \rho_R(s) is used only in the pretraining loss. The aggregator, the predictor ρR\rho_R, the rotation tokens, and the EMA target encoder are discarded after pretraining, for all three methods.

In the revised manuscript we will make these corrections to Section 3.4.2, Section 4.2, Appendix A.1, and Appendix A.3.

AuthorsOpenReview ↗

Rebuttal to Reviewer ZowE [3/3]

W4, Q3. Concern: The smooth closed trajectories in Figure 2 do not establish equivariance; direct measurements on held-out rotations and transformation composition would provide stronger support.

We agree with the reviewer on both points.

We would first like to highlight that (in response to Q3) the regression and projections of Figure 2 are fitted and evaluated on separate data.

Action: We measure equivariance directly, following Eq. 2. For each frozen encoder ff we fit a latent action ρR\rho_R that depends only on the rotation and is linear in the latent, and test how well ρRf(x)\rho_R f(x) predicts f(DRx)f(D_R x). A linear action exists only if rotation acts on the latent in an organized way. It is fit on 2,000 clips and evaluated on 1,000 different clips whose sound assets and RIRs are disjoint from pretraining, each under 16 random rotations. Pairs with a relative rotation of 75–105° are excluded from the fit and used only for testing ("held-out"). Composition is never enforced in the fit. No labels are used.

EncoderDisp.R2R^2R2R^2 held-outH@1 (%)Two-step R2R^2
seq-JEPA0.420.840.8159.40.80
SIE (All)0.270.970.9786.30.97
SimCLR-EquiMod (All)0.070.910.9078.30.91
GRAM-T0.040.780.8044.60.75
AVSA0.070.180.170.20.11
EINV2 (supervised)0.730.910.9051.30.90
SELDnet (supervised)0.050.020.010.1−0.07

Disp.: distance between rotated views of a clip relative to the distance between clips (0 = invariant). R2R^2: fraction of the rotation-induced change explained by ρR\rho_R. H@1: ρRf(x)\rho_R f(x) retrieves the true rotated view among the clip's views (chance 6.3%). Two-step: ρR2ρR1f(x)\rho_{R_2}\rho_{R_1} f(x) against f(DR2DR1x)f(D_{R_2} D_{R_1} x).

Findings: (i) All three of our encoders admit a linear rotation action that generalizes to held-out rotations (R2R^2 within 0.03 of the overall value) and composes (two-step R2R^2 within 0.04 of one-step). (ii) SIE is the most exactly equivariant (R2R^2 0.97, two-step R2R^2 0.97). seq-JEPA exhibits robust equivariance (R2R^2 0.84, two-step R2R^2 0.80), where its latent responds most strongly to rotation among our methods (displacement 0.42). (iii) AVSA and SELDnet have no usable action. GRAM-T's response is structured (R2R^2 0.78) but small: its displacement is 0.04, ten times lower than seq-JEPA's, in line with its weaker localization. EINV2, trained with DoA labels, is also organized (R2R^2 0.91) but retrieves the correct view less often than any of our encoders (H@1 51.3%). Organized rotation structure is thus not exclusive to our objectives; what they contribute is obtaining it without labels.

We will replace Figure 2 in our manuscript with tables and plots from this direct equivariance analysis.

We are grateful for these requests, which led to the controls, the GRAM-T comparisons, and the measurements above. The reviewer noted that such results would bear directly on their assessment; we hope they resolve the concerns raised, and we remain open to clarifying any remaining items.

Comments on the paper

2 comments
AuthorsOpenReview ↗

A Summary of Our Rebuttal and Reviewer Comments [2/2]

Reviewer E9nu

W1. Concern: Semantic accuracy is not strong enough to justify this class of losses; if the idea is a joint spatial-semantic loss, the semantic performance should be state-of-the-art.

Action: We clarify that the accuracy the reviewer cites is from single-source synthetic linear probing (Table 1), not the downstream tasks using real-world data. The semantic performance on real-world tasks is comparable to the strongest baselines, and is already in Tables 2 (SELD) and 5 (captioning).

Finding: On real recordings, frozen seq-JEPA matches or beats the supervised EINV2 encoder, with a better SELD score (Table 2, 0.570 vs 0.578; lower is better) and better captioning (Table 5, SPIDEr 0.492 vs 0.463; higher is better). So it keeps the semantics that matter while adding the spatial structure these encoders lack; GRAM-T, the strongest semantic encoder, localizes worse than ours, which is the gap we aim to close using the joint objective in our work.

W2. Concern: It is not clear whether one of the three losses is consistently better; do the authors see a clear winner?

Action: We compare the three losses on SELD (Table 2) and cross-dataset (Table 3).

Finding: seq-JEPA is the clear winner (best SELD under both protocols, best cross-dataset localization error), and we adopt it for the language tasks.

W3. Concern: It is unclear how to compare the Table 1 baselines: are all methods trained on the same data, and what are the model sizes?

Action: We clarify the protocols (our three encoders share backbone, latent, data, and compute; SSL encoders are frozen-probed, supervised baselines trained end-to-end on labeled DCASE-Synth, all on the same held-out probe). We will add a parameter count per encoder in the final manuscript.

Finding: Differences among our methods come from the objective, not capacity or data.


Reviewer 7h83

W1. Concern: The demonstrated scope is limited to FOA; it is unclear whether the principle transfers to higher-order ambisonics or microphone-array recordings.

Action: We clarify that the rotation we use is the first-order case of the Wigner-DD action. Higher-order ambisonics can be handled by simply appending the corresponding higher-order blocks, leaving the encoder, objective, and training of our framework unchanged. Furthermore, microphone-array recordings can leverage the same pipeline through ambisonic encoding. We will make the FOA scope explicit in the abstract and introduction.

Finding: The formulation is format-general, and FOA is also the current industry standard and practically relevant setting. Furthermore, most available real-world ambisonic data is FOA, and even that is far from clean (e.g., YT-360 and related datasets). We aim to demonstrate the principle here, and leave extending to higher orders to future work.

AuthorsOpenReview ↗

A Summary of Our Rebuttal and Reviewer Comments [1/2]

We sincerely thank the three reviewers for their time and for feedback that was specific and constructive throughout. We are encouraged that the reviewers found the rotation-based training signal principled and well motivated (ZowE, 7h83), the exploration of invariance–equivariance losses for spatial audio valuable (E9nu), and the transfer to real recordings and audio–language tasks meaningful (ZowE, 7h83).

For each reviewer concern, we have summarized the action taken and the finding below.

Detailed responses follow in each reviewer thread. The revised manuscript, with all changes, will be uploaded at the end of the discussion period.


Reviewer ZowE

W1. Concern: The experiments do not sufficiently isolate the contribution of the central equivariance learning mechanism.

Action: We add ablation studies to the paper that hold data and compute fixed and isolate the effects of the invariance and equivariance objectives, rotation conditioning, and the number of views L=1L = 1 – 55.

Finding: Our experiments show that the rotation equivariance mechanism is the primary driver of good spatial performance, and that the combined invariance-equivariance objective best balances the semantic-spatial trade-off.

Removing rotation conditioning is the only change that breaks the spatial metrics: classification is preserved (63.6 → 66.2), but rotation error triples (7.4° → 23.9°) and azimuth error nearly doubles (28.4° → 48.1°). In addition, removing rotations altogether trades spatial structure for semantics: the objective then only asks views to agree on content, which benefits classification (63.6 → 68.6), while rotation error doubles (7.4° → 14.8°) and direction-of-arrival estimation degrades in both azimuth (28.4° → 29.2°) and elevation (13.4° → 17.6°). The full objective is the only variant that is best on all tasks at once. This isolates rotation-conditioned prediction, not the data, views, or architecture, as the source of the spatial structure, trading off a few points of classification accuracy.

W2. Concern: GRAM-T, the most relevant self-supervised baseline, is absent from the main SELD and language comparisons.

Action: We add GRAM-T to every downstream comparison (Tables 2–5 in the paper and excerpted in the comments below).

Finding: In our experiments, GRAM-T recovers semantics well but localizes worse than our encoders, and seq-JEPA performs the best overall on SELD tasks. This supports our claim that learning rotation equivariance prioritizes spatial precision, whereas reconstruction-based objectives prioritize semantics.

W3. Concern: The central representation is inconsistently defined.

Action: For seq-JEPA, we correct the notation (downstream uses the encoder output fθ(x)f_\theta(x); the aggregator and predictor are used only in pretraining). We will specify the full inference procedure for one recording in the final manuscript.

Finding: The inconsistency was a notational error. It was not a difference in our experiments.

W4. Concern: The smooth closed trajectories in Figure 2 do not establish equivariance; direct measurements on held-out rotations and transformation composition would provide stronger support.

Action: We conduct experiments to demonstrate equivariance more directly: we fit a linear map on training clips and test it on 1,000 held-out clips, with a held-out rotation band and a composition test.

Finding: For all three of our encoders (seq-JEPA, SIE, SimCLR-EquiMod), the action explains most of the rotation-induced change in the latent (coefficient of determination of the fit R2R^2 0.84, 0.97, 0.91 respectively). In the comments below, we detail how this transfers to rotations it was not trained on and how it composes without being trained to do so. We show that SIE is the most exactly equivariant; seq-JEPA is the least exactly linear of the three, yet its latent responds most strongly to direction of arrival.


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.