acceptodds
Under review as a conference paper at ICLR 2027

Robust Multimodal Representation Learning through Topology-Preserving Prototype Maps

Abstract

Learning representations that preserve relations between samples is a fundamental challenge in machine learning. For multimodal data, the representations should retain the information of each modality while remaining aligned across modalities. A common approach is to map all inputs into a shared latent space. However, this requires them to share a common geometry and may lead to the loss of modality-specific information. In this work, we propose Pesto, a representation learning method that uses a shared topology for alignment while learning a separate embedding space for each modality. We represent the topology as a grid of prototypes, where similar samples are mapped to neighbouring locations. For multimodal data, each modality has its own encoder and prototype matrix, and modalities are aligned through their distributions over the shared map. Thus, different modalities of the same instance are aligned to similar locations on the map, while similar samples are organised into neighbouring regions. We learn the representations and topology jointly by backpropagating the neighbourhood loss through the encoders and using it to update the prototypes. Since alignment takes place on the shared map rather than embedding spaces, each modality can be used separately at inference time without imputation or retraining. Experiments on real-world datasets show that our method retains the characteristics of individual modalities, produces consistent topological maps across modalities, and achieves competitive performance compared with existing methods.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multimodal representation learning, self-organising maps, missing modalities

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XPLROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a method for learning representation of paired data from several modalities/channels/sensors without labels. Each modality or data channel has its own encoder and set of prototype vectors. The prototypes are placed on a shared two-dimensional grid. The method encourages paired observations from different channels to select the same grid locations. It also encourages nearby locations to represent similar samples. After training, a classifier uses the grid to make predictions from any available subset of channels. Empirical tests on sleep stage identification and activity recognition show competitive results, including when some modalities/channels are missing.

Critical strengths
  • The main contribution is a shared grid that aligns modalities while keeping separate encoders and prototype vectors.
  • The method is simple and supports different modalities and embedding sizes.
  • It can use any available subset of modalities without retraining.
  • Empirical results on three datasets show competitive results, including when modalities are missing.
Critical weaknesses
  • The technical novelty seems limited. Equations (2) and (3) combine a standard contrastive loss with a SOM neighbourhood loss. Section 2 acknowledges prior work that also updates the encoder through the neighbourhood loss.
  • The experiments do not isolate the benefit of the grid. Section 4.4 tests whether the neighbourhood loss updates the encoder, but does not compare with the same prototype model without grid structure.
  • The introduction mentions preserving modality-specific information. However, Section 3 includes no explicit objective for preserving it, and Section 4 does not test this claim.
  • Section 3.1 introduces the grid and prototype tables before clearly stating the learning goal. It does not specify which information or relationships the representations should preserve.
  • The presentation of the results, especially in the main text, is hard to follow. It switches between datasets, protocols, and baseline types.
  • Section 4.2 states that PESTO “wins on every benchmark” and attributes the largest gains to the map. These claims need more support. For example, Table 6 shows similar full-input accuracy to NFC and lower IMU-only accuracy, and Table 4 shows lower macro-F1 than XSleepNet. Also, the comparisons do not separate the benefit of the grid from that of prototype-based alignment.
Questions for authors
  • What is new beyond combining contrastive learning with a SOM loss?
  • How does the method perform with the same prototypes but no grid structure?
  • What evidence shows that the method preserves modality-specific information?
  • Which baselines were run under the same conditions as PESTO?
6 comments
AuthorsOpenReview ↗

Answer to Reviewer XPLR (1/3)

We thank the reviewer for the detailed review and for the clear summary of PESTO. The reviewer's questions help us clarify the novelty, the role of the grid, and the evidence for our claims. We address the four questions and related weaknesses first, followed by the presentation (W4-W5) and the claims in Section 4.2 (W6). In each new experiment, we kept all settings other than the tested settings as in the original paper.

Q1

What is new beyond combining contrastive learning with a SOM loss? We thank the reviewer for this important question. PESTO aligns modalities by their positions on a shared topological map while retaining a separate embedding space and prototype vectors for each modality. This allows encoders with different output dimensions to be aligned without a common projection; on MM-Fi, for example, the keypoint encoder outputs 16 dimensions, while the depth and radar encoders output 64. The shared map also provides a readout for any available subset of sensors and a spatial organisation that can be inspected directly, as illustrated by the sleep-stage maps.

Following Section 2 and 3 in the original paper, each component of PESTO differs from existing work as follows:

ComponentExisting work (Section 2)PESTO
MapDeep SOMs (SOM-VAE, DESOM, SOM-CPC) learn one map over one representation space; ReSOM trains one SOM per modality on fixed features and connects the maps once both are fixed.All modalities share one map, each keeping its own encoder and prototypes, and the encoders are learned together with the map.
PrototypesVQ-VAE uses an unordered codebook; FSQ a fixed lattice, ordered by construction but not by the data; SwAV and SeLa unordered prototypes. In all of them, a prototype has an identity but no position.Prototypes carry grid coordinates and are arranged by the data through the neighbourhood term.
Cross-modal alignmentCMC and CLIP compare modalities in one shared embedding space.Each modality keeps its own embedding space, of any dimension, and the modalities are aligned on positions of the shared map.
Missing modalitiesGenerative methods reconstruct the absent modality; HeMIS averages embeddings in a common latent space; robust training drops modalities during training.Distributions over the shared map are averaged, so one classifier reads any subset without imputation, retraining or modality dropout.

Two controlled experiments in this response isolate these parts. Removing the neighbourhood term lowers accuracy on every dataset and removes the organisation of the map (Q2 below). Reading the same frozen encoders through the map instead of through concatenated embeddings keeps F1 at 0.638 rather than 0.393 when one sensor is left on Sleep-EDF-20 (please find our detailed response to Reviewer quAL, Q2).

W1

We see how presenting Equations (2) and (3) side by side could suggest a straightforward combination of existing losses. The two terms are, however, well motivated by what the shared map requires. First, the modalities of one sample should select the same map position while different samples remain apart, which calls for an alignment with a repulsive component; aligning the modalities' distributions directly, without it, collapses (see our response to Reviewer quAL, W2). Second, nearby positions should represent similar samples, which is what the neighbourhood term enforces; removing it removes the organisation of the map (Q2). What is new is therefore not a loss term, but the structure in which these terms operate and what it enables, as described in Q1.

Regarding the prior work in Section 2 that updates the encoder through the neighbourhood loss, these methods learn one map for one encoder and one modality; in PESTO, one map is shared by all modality encoders, and the neighbourhood gradient reaches each of them. We plan to revise the list of contributions accordingly, stating the third contribution in its multimodal form, and this will be reflected in the revised version.

AuthorsOpenReview ↗

Answer to Reviewer XPLR (2/3)

Q2

How does the method perform without grid structure? Reviewer Ptap asked for the same control, and we report it in full in our response to Reviewer Ptap (Q1). We retrained all four settings with the same prototypes, encoders, alignment term, readout, hyperparameters and seeds, and removed only the neighbourhood term:

SettingTest inputfrozen: PESTO → λ=0proto_head: PESTO → λ=0
UTD-MHAD (acc. %)IMU + skeleton96.6 → 90.598.1 → 96.4
IMU only89.9 → 81.692.7 → 89.0
skeleton only94.9 → 91.895.8 → 95.7
MM-Fi (acc. %)depth + radar81.0 → 78.082.1 → 79.9
depth only75.2 → 71.476.5 → 67.8
radar only58.7 → 54.656.1 → 47.7
Sleep-EDF-20 (F1)all sensors0.785 → 0.1650.791 → 0.635
missing at test0.729 → 0.1660.728 → 0.431
missing in training and test0.735 → 0.1460.735 → 0.514
Sleep-EDF-78 (acc. %)EEG + EOG83.9 → 72.184.3 → 75.2
EEG only82.7 → 59.483.2 → 65.8

Here λ is the weight of the neighbourhood term, and the PESTO rows report the originally submitted models, re-evaluated using the same pipeline as the ablations. Removing the neighbourhood term lowers accuracy on every dataset under both protocols, with the largest decreases in single-sensor settings such as IMU-only on UTD-MHAD (8.3 points under frozen) and EEG-only on Sleep-EDF-78 (23.3 points). The organisation of the map disappears with it: on UTD-MHAD the layout score falls from 71.9% of its attainable maximum to 1.3% (0 for random), and on Sleep-EDF the map stays concentrated on a single region. This control also separates the benefit of the grid from that of prototype-based alignment, which W6 asks for.

W2

Please see our response to Q2 above.

Q3

What evidence shows that the method preserves modality-specific information? By modality-specific we mean that each modality keeps its own encoder, its own embedding dimension and output scale, and its own prototype table; nothing projects the modalities into a common embedding space (Section 3.2). The testable consequence is that each modality, read alone through the map, remains informative, even though the classifier was trained on all modalities together. The paper reports these results across all three datasets:

Dataset (acc. %)Read from one modalityPESTOReference
MM-Fidepth only75.260.1, contrastive pre-training (Table 7)
radar only58.728.3, contrastive pre-training (Table 7)
UTD-MHADIMU only92.983.4, model trained on the IMU alone (Table 5)
Sleep-EDF-78, four sensorsEEG only83.579.6, SleepSMC (Table 8)
EOG only79.973.1, SleepSMC (Table 8)
EMG only56.152.9, SleepSMC (Table 8)

On MM-Fi, aligning the modalities in one shared embedding space leaves radar alone at 28.3%, whereas keeping each modality in its own space and aligning on the map gives 58.7%; the information carried by the weaker sensor is retained rather than absorbed into the stronger one. On UTD-MHAD, the IMU read alone from the two-modality model reaches 92.9%, above the 83.4% of a model trained on the IMU alone, so aligning with the skeleton does not overwrite what the IMU encodes. On Sleep-EDF-78, every single sensor read alone from PESTO is more accurate than SleepSMC, a supervised method designed specifically for missing sensors, including the chin EMG, the weakest of the four.

W3

We thank the reviewer for pointing out that this claim needs further clarification. We will clarify that PESTO maintains separate encoders and prototype tables for each modality, and connect this design to the single-modality results in Tables 5, 7 and 8, as discussed in Q3. We will also replace “retains the information of each modality” with the more precise statement that each modality can still support accurate predictions when read alone through the map. These changes will be reflected in the revised version.

AuthorsOpenReview ↗

Answer to Reviewer XPLR (3/3)

Q4

Which baselines were run under the same conditions as PESTO? In terms of data, splits and evaluation, every comparison in the paper follows the protocol of the corresponding work, because we evaluate PESTO under the protocol of each comparator; how each protocol was reproduced is detailed below.

In addition, two baselines were rerun by us with the same encoders as PESTO. On MM-Fi, contrastive pre-training with the multimodal objective of MMBind is trained with the same encoders, inputs, subject batches, optimiser and number of epochs, and read by the same fusion head (Table 7 and Appendix B). On UTD-MHAD, a model trained on the IMU alone with two augmented views uses the same encoder and classifier (Table 5).

For the other comparators (CroSSL, COCOA, XSleepNet, SeqSleepNet, MultiConsSleepNet, SleepSMC, NFC, HNS, CMC, Barlow Twins and the MMBind upper bound), we quote the published numbers and reproduce their protocols, preprocessing, data split, model-selection rule and metric, as closely as the published material allows:

  • Code released. For XSleepNet, CroSSL and MMBind, we reconstructed the protocols from the authors' released code. Where the code and the paper differ, we followed the code and noted the difference in Appendix B; for example, the missing-sensor evaluation of CroSSL does not run as released, so we implemented it as its paper describes. For MMBind, the script that produces its preprocessed data is not released; we asked the authors through the code repository and, without a reply, reconstructed this step from the released data loader and the paper.
  • Code and splits not released. For SleepSMC and MultiConsSleepNet, we followed the description in each paper.
  • Subject split. NFC, HNS, CMC and Barlow Twins are compared on the odd/even subject split shared by these works.

Appendix B documents each reconstruction. The quoted numbers are marked as such in the table captions, and we plan to state this distinction for every comparison in Appendix B.

W4 and W5

We agree that the better presentation can make the motivation and results easier to follow. We will open Section 3.1 with three design requirements: paired modalities should select the same map location, similar samples should select nearby locations, and each modality should retain its own encoder and embedding dimension. We will then introduce the grid and prototype tables in terms of these motivations. In Section 4.2, we will organise the comparisons according to the groups in Table 2 and integrate the new experiments reported in responses, so readers can follow the evidence for each claim.

W6

We agree that this is a wording issue. The sentence in Section 4.2 refers to the main comparison of each protocol, made with all sensors. Against self-supervised methods, PESTO is ahead of or on par with every comparator in every protocol: the margins are clear on Sleep-EDF-20 and MM-Fi (0.785 against 0.722, and 81.0% against 74.3%), and PESTO matches NFC and MultiConsSleepNet (96.2% against 96.06%, and 85.9% against 85.8%). The two examples the reviewer mentions fall outside this comparison. The IMU-only column of Table 6 is a single-sensor condition, where NFC is ahead (75.58% against 71.1%) while PESTO is ahead with the skeleton alone (93.6% against 86.05%); we will also correct the bold face in this column. XSleepNet in Table 4 is trained end-to-end with labels and belongs to the second part of the sentence: there, PESTO obtains a higher accuracy and κ (84.3 against 84.0, and 0.781 against 0.778) and a macro-F1 within 0.7 points (78.0 against 78.7), which is what we mean by matching. We plan to revise the sentence so that it states this scope explicitly.

Regarding the source of the gains, the ablation in Q2 now separates the grid from prototype-based alignment: removing the neighbourhood term lowers accuracy on every dataset, with the largest decreases in single-sensor settings, and removes the organisation of the map. The revised sentence will cite this control; these changes will be reflected in the revised version.

In summary, the new ablation isolates the contribution of the map, the component-wise comparison in Q1 sets out what PESTO adds to existing methods, and the single-modality results in Q3 support the claim on modality-specific information; the remaining points concern presentation, which we will address in the revision. We hope that, with these additions, the contribution of the paper comes across more clearly, and that the reviewer will consider them in the final evaluation. Any further questions are very welcome during the discussion period.

Reviewer_PtapOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Pesto learns modality-specific encoders and prototype tables indexed by a shared two-dimensional grid. It aligns paired modalities through their prototype-affinity distributions and uses a neighbourhood loss to organise the grid. A classifier reads the available modalities through the same map. Experiments cover sleep staging and activity recognition, including deployment with missing sensors.

Critical strengths
  1. The shared map provides a simple interface for different encoder dimensions and sensor subsets. The missing-sensor results give this design practical support: in the matched MM-Fi comparison, frozen Pesto improves accuracy from 74.3% to 81.0%, with larger gains for depth-only and radar-only deployment (Table 7).

  2. The evaluation includes subject-wise protocols and useful diagnostic analyses. The paired multimodal-versus-unimodal experiment improves IMU-only accuracy by 9.5 points (Table 5), while the sleep-stage maps and permutation references provide interpretable evidence of organisation beyond downstream classification.

Critical weaknesses
  1. The contribution of topology to representation quality is not isolated. The matched contrastive comparison changes both the prototype-based representation and the neighbourhood objective. Table 10 tests gradient routing, but both variants retain the grid loss; its reported benefit is in layout, while direct-readout accuracy decreases by 2.8 points at K=400. Without a matched version retaining the same prototype alignment and readout but removing the neighbourhood term, it remains unclear how much topology contributes to classification and missing-sensor robustness.

  2. The sleep results need more precise controls and qualification. Section 4.1 claims to match comparator encoders, but Table 3 quotes CroSSL's published results with its three-layer 1D CNN, whereas Table 9 specifies DeepSleepNet for Pesto. The 0.722-to-0.785 comparison therefore does not isolate the learning method. Appendix B also states that class-based oversampling applies to pretraining in this setting (lines 770–776), which conflicts with the claim of no labels before classifier training. Following the baseline's sampling protocol may be fair, but it does not establish label-free pretraining.

Questions for authors
  1. How do downstream accuracy, missing-sensor performance and map organisation change when the neighbourhood weight is zero, with the remaining architecture and training settings fixed?
  2. Does the Sleep-EDF-20 gain over CroSSL persist with a common encoder and pretraining on the original, unresampled training data?
5 comments
AuthorsOpenReview ↗

Answer to Reviewer Ptap (1/3)

We thank the reviewer for their careful assessment and for recognising the value of the shared map and missing-sensor evaluation. We hope the experiments and explanations below address both concerns: an ablation that isolates the neighbourhood term, and a sleep comparison using a matched encoder and unresampled pretraining data. Within each comparison, we kept all other settings the same as in the original paper and have revised the manuscript to reflect the results.

Q1

What changes when the neighbourhood weight is zero? We retrained all four settings with the neighbourhood weight set to zero and the encoders, prototype tables, alignment term, readout, hyperparameters and seeds unchanged. Accuracy decreases on every dataset under both protocols, and the decrease is generally larger when sensors are missing:

SettingTest inputfrozen: PESTO → λ=0proto_head: PESTO → λ=0
UTD-MHAD (acc. %)IMU + skeleton96.6 → 90.598.1 → 96.4
IMU only89.9 → 81.692.7 → 89.0
skeleton only94.9 → 91.895.8 → 95.7
MM-Fi (acc. %)depth + radar81.0 → 78.082.1 → 79.9
depth only75.2 → 71.476.5 → 67.8
radar only58.7 → 54.656.1 → 47.7
Sleep-EDF-20 (F1)all sensors0.785 → 0.1650.791 → 0.635
missing at test0.729 → 0.1660.728 → 0.431
missing in training and test0.735 → 0.1460.735 → 0.514
Sleep-EDF-78 (acc. %)EEG + EOG83.9 → 72.184.3 → 75.2
EEG only82.7 → 59.483.2 → 65.8

Here, λ denotes the weight of the neighbourhood term in Section 3.3. The PESTO rows report the originally submitted models, re-evaluated using the same pipeline as the ablations. Under the frozen protocol, only the classifier is trained, so these results reflect the quality of the pretrained representation. Removing the neighbourhood term reduces all-sensor accuracy by 3-12 percentage points across UTD-MHAD, MM-Fi, and Sleep-EDF-78, and single-sensor accuracy by up to 23 points.

And the organisation of the map? Without the neighbourhood term, the map loses its organisation on every dataset. On UTD-MHAD the layout score falls from 71.9% of its attainable maximum to 1.3% (0 for a random arrangement), and the topographic error rises from 0.80 to 0.99, the value of a random arrangement (0.98). On Sleep-EDF, 83.2% of adjacent cells share their sleep stage in PESTO (79.3% on the 20-subject subset), against 32.5% and 24.3% for random labels; without the neighbourhood term the map stays concentrated on a single region in both settings, and reading the grid directly returns the majority stage. On MM-Fi, a small dataset of 560 sequences, the share of adjacent cells holding the same action is 6.0%, above the 3.2% of random placement, and it falls to 0 for every seed without the neighbourhood term.

AuthorsOpenReview ↗

Answer to Reviewer Ptap (2/3)

W1

The W1 tied to the neighbourhood term is the ablation in Q1. We thank the reviewer for identifying precisely which comparison would isolate the contribution of the topology. The ablation in Q1 is this matched version: it keeps the same prototype alignment and readout and removes only the neighbourhood term. The two existing comparisons the reviewer mentions answer different questions, which we clarify below.

Table 10 isolates where the neighbourhood gradient goes, not the neighbourhood term itself. Both of its variants keep the neighbourhood term, and they differ only in whether its gradient reaches the encoder, with all other settings identical. Letting the gradient reach the encoder improves the layout at every grid size, by 22.1 points of the attainable ceiling at K=400K=400 and on all five splits. This comes with a small change in direct readout, which labels each cell by majority vote without a classifier: within 0.3 points at K=100K=100 and K=225K=225, and −2.8-2.8 and −2.9-2.9 points at K=400K=400 and K=900K=900. Table 10 therefore describes a trade-off within the topology design, instead of the contribution of the topology as a whole.

The MM-Fi contrastive comparison tests the shared-map objective as a whole. As the reviewer notes, it changes both the prototype-based representation and the neighbourhood objective, and it shows that learning on the shared map improves over contrastive pre-training in a common embedding space (74.3% to 81.0%). Separating the two components is what the new ablation adds.

The new ablation isolates the neighbourhood term. With the prototype alignment and readout unchanged, removing the neighbourhood term lowers accuracy on every dataset under both protocols, with the largest decreases in single-sensor settings such as IMU-only on UTD-MHAD, radar-only on MM-Fi and EEG-only on Sleep-EDF-78, and the map loses its organisation (Q1). We plan to place this ablation next to Table 10, so that Section 4.4 covers the full sequence: no neighbourhood term, a term that stops at the prototypes, and a term that reaches the encoders, this will be reflected in the revised version.

AuthorsOpenReview ↗

Answer to Reviewer Ptap (3/3)

Q2

We thank the reviewer for the precise reading of the sleep comparison. Table 3 quotes CroSSL as published, with its three-layer 1D CNN, while PESTO uses DeepSleepNet; the encoder statement in Section 4.1 is therefore too general for this comparison. As noted in Appendix B, we also followed CroSSL's released code in applying class-based oversampling during pretraining. Because this uses class labels, we have made the label-free claim in Section 4.1 precise for this protocol; as shown below, pretraining without the oversampling gives the same or higher accuracy. To answer Q2, we ran a comparison using CroSSL's encoder or DeepSleepNet, with or without oversampling in pretraining.

CroSSL's encoder is our PyTorch implementation of the encoder in CroSSL's released code. Without oversampling, pretraining uses each training sample once. The classifier and the test set follow the CroSSL protocol as in Table 3.

Sleep-EDF-20, PESTO (F1)all sensorsmissing at testmissing in training and test
PESTO, DeepSleepNet, oversampling (Table 3)0.7850.7290.735
PESTO, DeepSleepNet, no oversampling0.7890.7270.737
PESTO, CroSSL encoder, oversampling0.7820.7180.725
PESTO, CroSSL encoder, no oversampling0.7610.7110.713
CroSSL (quoted)0.7220.6940.667

Does the gain over CroSSL persist? Yes. With CroSSL's own encoder and pretraining on the original, unresampled data, the frozen representation remains ahead of CroSSL in all three conditions: 0.761 against 0.722 with all sensors, 0.711 against 0.694 when sensors are missing at test, and 0.713 against 0.667 when they are missing in training as well. The gain therefore holds when the encoder and the pretraining data are matched, so it does not come from the encoder or from the oversampling.

Is pretraining possible without labels? Yes, without loss of accuracy. Without oversampling, PESTO with its own encoder reaches 0.789 with frozen encoders (originally 0.785 in Table 3). The pretraining stage therefore does not need the label-based resampling.

W2

W2 is tied to the concerns raised in Q2, so please see our response to Q2 above.

In the revision, we will add the matched-encoder and unresampled results to Table 3, clarify which comparisons use the same encoder in Section 4.1, and correct the label-free claim for the oversampled pretraining protocol. We thank the reviewer for prompting these controls. We hope the neighbourhood-term ablation across all four settings and the sleep comparison controlled for encoder and pretraining data address the reviewer’s two concerns, and would appreciate their consideration in the final assessment. We would be happy to clarify any remaining details during the discussion period.

Reviewer_quALOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes PESTO, which replaces the traditional shared embedding space with a shared two-dimensional topology map (SOM), allowing each modality to retain its own encoder and prototype table. Through distributed alignment, it achieves multi-modal representation learning. The introduction of the topological structure into multi-modal alignment and the use of grid affinity distribution to naturally handle missing modalities have a certain degree of novelty and intuitive elegance. Some experimental results on Sleep-EDF, UTD-MHAD, and MM-Fi are also competitive.

Critical strengths
  1. This is the most fundamental and elegant contribution of the paper. Standard multimodal contrastive learning strictly requires projecting modalities into a common vector space of equal dimensions, which is often artificial and computationally rigid. By keeping modality-specific prototypes but sharing a 2D grid of coordinates, the authors propose a highly original alternative that decouples alignment from dimensionality.
  2. In representation learning, it is rare to see an unsupervised method produce a latent space that is both highly effective for downstream tasks and globally interpretable. The visualization and quantitative analysis demonstrate that the shared topology organizes sleep stages in a physiologically meaningful way. This proves that the model captures genuine physical structures rather than merely fitting a classification boundary.
Critical weaknesses
  1. The paper’s primary practical claim is that PESTO handles missing modalities out-of-the-box without imputation or retraining, simply by averaging the affinities of the available sensors. However, the authors completely omit how the downstream classification head is trained. If the classifier is trained solely on the full set of modalities, standard neural network behavior dictates it will suffer from severe distribution shift at test time when modalities are missing. If the classifier is trained using random modality dropout, this is a standard, well-known robust-training technique, not an inherent property of the proposed topological map.
  2. The paper advocates for aligning modalities via a shared topological map, yet the alignment loss (Eq. 2) relies on cross-entropy (InfoNCE) over in-batch negatives. Because each modality produces a probability distribution over the same K grid cells, aligning them should naturally be formulated as a direct distribution matching problem (e.g., minimizing KL divergence or Wasserstein distance). Using an InfoNCE loss means the alignment is still fundamentally dependent on batch size and contrastive sampling logic, rather than the geometry of the shared map.
Questions for authors
  1. Exactly how is the downstream classifier head trained? Is it trained exclusively on the full set of modalities and then applied to subsets at test time? Or is it trained using random modality dropout (training on subsets)?
  2. If the model is trained on all modalities and tested on subsets without any modality dropout during training, can you provide evidence to explain why the classifier does not break down due to distribution shift?
6 comments
AuthorsOpenReview ↗

Answer to Reviewer quAL (1/3)

We sincerely thank the reviewer for the careful reading of our manuscript and for the positive and constructive comments, and we fully acknowledge the suggestions raised during the review process. We are grateful that the reviewer captured the idea that motivated PESTO from the outset: to perform well on the task while preserving meaningful interpretability. All results below were obtained with every other component, hyperparameter and seed kept the same as in the original paper.

Q1

How is the head trained? For all main results on the three datasets in the paper, the head is trained with every modality present, then applied unchanged to each sensor subset at test time. It never sees a missing-modality example during training. Section 3.4 states this briefly (“the head is trained with all modalities and deployed on any observed subset SS”), and we will expand to make it clearer and discuss the relevant results in more detail.

Or is it trained using random modality dropout (training on subsets)? Does the robustness come from modality dropout? No. We understand that this concern may stem from the comparisons in Table 3. CroSSL reports results on Sleep-EDF-20 under three conditions: all sensors available, sensors missing at test time, and sensors missing during both training and testing. We evaluated PESTO under all three settings to allow a direct and fair comparison with CroSSL. The row names in original Table 3, describe when sensors are missing, but we would agree that they do not make clear how the head is trained. Specifically, in the “missing at test” row, the head is trained on all four sensors and applied unchanged to sensor subsets at test time. In the “missing in training and test” row, a random subset of sensors is removed from each training batch of the head (as described in Appendix B), corresponding to random modality dropout. We included this latter condition solely to match the CroSSL comparison protocol. The PESTO standard procedure, used for our main results, is to train the head with all sensors present and apply it directly to any observed subset at test time. With the head-training procedures made explicit, the results in Table 3 read:

Row in Table 3Head trained onSensors at testCroSSLPESTO
allall sensorsall0.7220.785 ± .004
missing at testall sensorsmissing0.6940.729 ± .005
missing in training and testmodality dropoutmissing0.6670.735 ± .004

We have revised the manuscript to clarify this distinction. With frozen encoders, the PESTO head trained only on complete inputs (0.729) performs on par with the head trained with modality dropout (0.735), while for CroSSL the head trained with modality dropout is lower (0.667 against 0.694). PESTO therefore does not rely on modality dropout: its robustness to missing sensors comes from reading through the shared map.

AuthorsOpenReview ↗

Answer to Reviewer quAL (2/3)

Q2

Why the classifier does not break down due to distribution shift? The head can be applied to unseen sensor subsets because its input remains in the same space. Whatever the subset SS, the head receives a distribution over the same KK cells, obtained by averaging the affinities of the sensors present (Equation 4). The head is trained only on training samples with all sensors present; at test time, the same operation is applied to the available sensors of a new sample. The resulting distribution is therefore defined over the same cells, even though neither the sample nor its sensor subset was seen during head training.

The alignment term encourages different modalities of a training sample to map to similar cells. When this learned alignment carries over to new samples, a subset produces a distribution similar to the one obtained from all sensors. The head can thus read an input with both a familiar form and similar content. With concatenated embeddings, by contrast, a missing sensor creates a zero-filled block that the head did not encounter during training.

Evidence 1 To clarify this point and provide empirical support, we first refer to the missing-sensor results reported in Section 4.3 and Figure 3. On Sleep-EDF-20, a head trained with all four sensors continues to perform well when fewer are available at test time: F1 reduces a little from 0.785 to 0.750 with two sensors for the frozen-encoder model, and from 0.791 to 0.746 for proto_head.

Sleep-EDF-20, head trained on all four sensors (F1)four sensorsthreetwo
frozen0.7850.7730.750
proto_head0.7910.7760.746

Evidence 2 We then ran an additional experiment to examine whether this behavior is associated with reading from the shared map. Using the same PESTO encoders and head-training protocol, we compared two inputs to the classifier: the distribution over the shared map and concatenated modality embeddings, with missing modalities set to zero as in CroSSL. Each head was trained with all four sensors present and then applied to each sensor subset at test time.

Sleep-EDF-20, frozen (F1)all sensorsmissing at testone sensor left
Map (PESTO readout)0.7850.7290.638
Concatenated embeddings0.7960.6330.393

With all sensors available, the two readouts achieve similar F1 scores. The difference emerges when sensors are removed: with only one sensor left, F1 is 0.638 for the map readout but falls to 0.393 for concatenated embeddings. Since the encoders and training protocol are held fixed, this comparison supports our view that the shared map helps the head retain useful performance under the distribution shift caused by missing sensors. We will add this comparison and discussion to the manuscript to support Section 3.4.

AuthorsOpenReview ↗

Answer to Reviewer quAL (3/3)

W2

We thank the reviewer for the careful reading and thoughtful suggestion. The contrastive alignment loss is a deliberate part of PESTO's design. Unlike embedding-level alignment, it operates on activation maps, encouraging the modalities to agree on positions in the shared map while separating different samples. The neighbourhood term further preserves the map’s local geometry. We agree that the paper should explain this motivation more clearly and how it differs from prior approaches more clearly, and we do so below.

Direct distribution matching has a trivial solution. Direct distribution matching, such as minimising a KL or Wasserstein divergence between the two maps of a sample, pulls the paired maps together, but no term in it distinguishes one sample from another. It therefore admits a trivial minimiser in which every sample, in every modality, is placed on the same few cells at zero loss. Section 3.3 notes that the alignment term “is the only term that separates samples, and it is what prevents collapse”, and Appendix G shows the same mechanism for the purely attractive neighbourhood term: trained alone, it collapses all prototypes (mean pairwise cosine 1.00001.0000, loss exactly zero). Some repulsion between samples is therefore needed, and the contrastive form of Equation (2) provides it.

A Wasserstein alignment collapses in practice. In fact, we considered aligning the two modalities' distributions over the cells directly at an early stage of this work. The two modalities then agreed well but accuracy fell to chance, because matching the paired distributions alone is minimised once all samples share the same cells. To better support our claims, we also tested the reviewer's suggestion directly, replacing InfoNCE on all four settings with a Wasserstein distance between the maps of the two modalities, with the grid distance between cells as ground cost (entropic Sinkhorn divergence). This is direct distribution matching that uses the geometry of the map and requires no negatives.

SettingProtocolInfoNCE (PESTO)WassersteinChance
UTD-MHAD (acc. %)frozen96.63.73.7
proto_head98.13.7
MM-Fi (acc. %)frozen81.07.17.1
proto_head82.17.1
Sleep-EDF-20 (acc. % / κ)frozen78.8 / 0.71440.2 / 0.00040.2 (majority)
proto_head79.1 / 0.71940.2 / 0.000
Sleep-EDF-78 (acc. % / κ)frozen83.9 / 0.77639.1 / 0.062
proto_head84.3 / 0.78140.7 / 0.089

In every setting the Wasserstein loss falls to almost zero while the map collapses onto a few cells (1 to 4 of 100 on MM-Fi, 7 to 33 of 400 on Sleep-EDF), leaving the classifier at chance or at the majority class. This is the trivial solution described above, which the contrastive term of Equation (2) rules out.

Accuracy is stable across batch sizes. We varied the batch size of learning with everything else fixed. Over an eight-fold range of batch sizes, the frozen accuracy on UTD-MHAD stays between 96.1% and 96.6% and the proto_head accuracy between 97.4% and 98.1%; on Sleep-EDF-20, a batch size of 64 already gives a similar F1 as 512.

UTD-MHAD, batch size3264 (paper)128256
frozen (acc. %)96.196.696.496.1
proto_head (acc. %)97.798.197.498.1
Sleep-EDF-20, batch size64128256 (paper)512
frozen (F1)0.7860.7810.7850.787
proto_head0.7880.7860.7910.791

The neighbourhood term gives the map its geometry. The alignment term encourages the modalities to agree on map positions, while the neighbourhood term encourages nearby cells to represent similar samples. These terms play complementary roles: the contrastive term explicitly separates different samples, whereas the neighbourhood term promotes local organisation on the map. Removing the neighbourhood term reduces accuracy on all evaluated datasets and disrupts the map's organisation (please see our response to Reviewer Ptap).

We will incorporate these clarifications and the new results into the revised version, in Sections 3.3, 3.4 and 4.3 and in the appendix. We thank the reviewer again for these questions, which have helped us clarify the mechanism behind PESTO. We hope the controlled comparisons and explanation above address the reviewer’s concerns about the training head and alignment loss, and we would be grateful if the reviewer could take them into account in the final assessment. We would be happy to clarify any remaining points during the discussion period.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response to Reviewers

Dear Reviewers,

We sincerely thank you for your detailed reviews and constructive feedback. We will respond to each reviewer in several comments. Once all responses have been posted, we will upload a revised version of the manuscript as soon as we can.

Best regards,

The Authors

AuthorsOpenReview ↗

Revised Manuscript

Dear Reviewers and Area Chairs,

We have uploaded the revised manuscript, with the changes highlighted. It includes the revisions described in our replies to each reviewer.



We have clarified how the classifier is trained and why it remains effective when sensors are missing. We revised the wording in several places and added experiments and comparisons to support both points.


To isolate the contribution of the map's topology, we added an ablation that removes only the neighbourhood term. We also explain why directly matching activation maps does not separate samples and report a comparison with Wasserstein alignment. Additional sleep experiments clarify the comparison with CroSSL across different encoder and pre-training settings.



Finally, we have clarified the learning goal and how it motivates the grid and prototype tables. We have also clarified the scope of our claims and reorganised the results section to make the findings and comparisons easier to follow.



We appreciate the time you have put into reviewing the paper. We hope the revised manuscript addresses your concerns and clarifies our contribution, and would be grateful if you could take these changes into account in your final assessment. We’re happy to clarify any remaining points during the discussion period.

Best regards,

The Authors

AuthorsOpenReview ↗

Follow-up before the end of the discussion period

Dear Reviewers and Area Chair,

We thank the reviewers for taking the time to read our responses and revised manuscript. We are glad the revisions addressed Reviewer Ptap’s concerns, and we appreciate their updated assessment.

With about 22 hours left in the discussion period, we wanted to make sure that the revised manuscript and our earlier replies reached you. We would be happy to respond to any further comments before the deadline.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.