- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a method for learning representation of paired data from several modalities/channels/sensors without labels. Each modality or data channel has its own encoder and set of prototype vectors. The prototypes are placed on a shared two-dimensional grid. The method encourages paired observations from different channels to select the same grid locations. It also encourages nearby locations to represent similar samples. After training, a classifier uses the grid to make predictions from any available subset of channels. Empirical tests on sleep stage identification and activity recognition show competitive results, including when some modalities/channels are missing.
Critical strengths
- The main contribution is a shared grid that aligns modalities while keeping separate encoders and prototype vectors.
- The method is simple and supports different modalities and embedding sizes.
- It can use any available subset of modalities without retraining.
- Empirical results on three datasets show competitive results, including when modalities are missing.
Critical weaknesses
- The technical novelty seems limited. Equations (2) and (3) combine a standard contrastive loss with a SOM neighbourhood loss. Section 2 acknowledges prior work that also updates the encoder through the neighbourhood loss.
- The experiments do not isolate the benefit of the grid. Section 4.4 tests whether the neighbourhood loss updates the encoder, but does not compare with the same prototype model without grid structure.
- The introduction mentions preserving modality-specific information. However, Section 3 includes no explicit objective for preserving it, and Section 4 does not test this claim.
- Section 3.1 introduces the grid and prototype tables before clearly stating the learning goal. It does not specify which information or relationships the representations should preserve.
- The presentation of the results, especially in the main text, is hard to follow. It switches between datasets, protocols, and baseline types.
- Section 4.2 states that PESTO “wins on every benchmark” and attributes the largest gains to the map. These claims need more support. For example, Table 6 shows similar full-input accuracy to NFC and lower IMU-only accuracy, and Table 4 shows lower macro-F1 than XSleepNet. Also, the comparisons do not separate the benefit of the grid from that of prototype-based alignment.
Questions for authors
- What is new beyond combining contrastive learning with a SOM loss?
- How does the method perform with the same prototypes but no grid structure?
- What evidence shows that the method preserves modality-specific information?
- Which baselines were run under the same conditions as PESTO?
6 comments
Answer to Reviewer XPLR (2/3)
Q2
How does the method perform without grid structure? Reviewer Ptap asked for the same control, and we report it in full in our response to Reviewer Ptap (Q1). We retrained all four settings with the same prototypes, encoders, alignment term, readout, hyperparameters and seeds, and removed only the neighbourhood term:
| Setting | Test input | frozen: PESTO → λ=0 | proto_head: PESTO → λ=0 |
|---|---|---|---|
| UTD-MHAD (acc. %) | IMU + skeleton | 96.6 → 90.5 | 98.1 → 96.4 |
| IMU only | 89.9 → 81.6 | 92.7 → 89.0 | |
| skeleton only | 94.9 → 91.8 | 95.8 → 95.7 | |
| MM-Fi (acc. %) | depth + radar | 81.0 → 78.0 | 82.1 → 79.9 |
| depth only | 75.2 → 71.4 | 76.5 → 67.8 | |
| radar only | 58.7 → 54.6 | 56.1 → 47.7 | |
| Sleep-EDF-20 (F1) | all sensors | 0.785 → 0.165 | 0.791 → 0.635 |
| missing at test | 0.729 → 0.166 | 0.728 → 0.431 | |
| missing in training and test | 0.735 → 0.146 | 0.735 → 0.514 | |
| Sleep-EDF-78 (acc. %) | EEG + EOG | 83.9 → 72.1 | 84.3 → 75.2 |
| EEG only | 82.7 → 59.4 | 83.2 → 65.8 |
Here λ is the weight of the neighbourhood term, and the PESTO rows report the originally submitted models, re-evaluated using the same pipeline as the ablations. Removing the neighbourhood term lowers accuracy on every dataset under both protocols, with the largest decreases in single-sensor settings such as IMU-only on UTD-MHAD (8.3 points under frozen) and EEG-only on Sleep-EDF-78 (23.3 points). The organisation of the map disappears with it: on UTD-MHAD the layout score falls from 71.9% of its attainable maximum to 1.3% (0 for random), and on Sleep-EDF the map stays concentrated on a single region. This control also separates the benefit of the grid from that of prototype-based alignment, which W6 asks for.
W2
Please see our response to Q2 above.
Q3
What evidence shows that the method preserves modality-specific information? By modality-specific we mean that each modality keeps its own encoder, its own embedding dimension and output scale, and its own prototype table; nothing projects the modalities into a common embedding space (Section 3.2). The testable consequence is that each modality, read alone through the map, remains informative, even though the classifier was trained on all modalities together. The paper reports these results across all three datasets:
| Dataset (acc. %) | Read from one modality | PESTO | Reference |
|---|---|---|---|
| MM-Fi | depth only | 75.2 | 60.1, contrastive pre-training (Table 7) |
| radar only | 58.7 | 28.3, contrastive pre-training (Table 7) | |
| UTD-MHAD | IMU only | 92.9 | 83.4, model trained on the IMU alone (Table 5) |
| Sleep-EDF-78, four sensors | EEG only | 83.5 | 79.6, SleepSMC (Table 8) |
| EOG only | 79.9 | 73.1, SleepSMC (Table 8) | |
| EMG only | 56.1 | 52.9, SleepSMC (Table 8) |
On MM-Fi, aligning the modalities in one shared embedding space leaves radar alone at 28.3%, whereas keeping each modality in its own space and aligning on the map gives 58.7%; the information carried by the weaker sensor is retained rather than absorbed into the stronger one. On UTD-MHAD, the IMU read alone from the two-modality model reaches 92.9%, above the 83.4% of a model trained on the IMU alone, so aligning with the skeleton does not overwrite what the IMU encodes. On Sleep-EDF-78, every single sensor read alone from PESTO is more accurate than SleepSMC, a supervised method designed specifically for missing sensors, including the chin EMG, the weakest of the four.
W3
We thank the reviewer for pointing out that this claim needs further clarification. We will clarify that PESTO maintains separate encoders and prototype tables for each modality, and connect this design to the single-modality results in Tables 5, 7 and 8, as discussed in Q3. We will also replace “retains the information of each modality” with the more precise statement that each modality can still support accurate predictions when read alone through the map. These changes will be reflected in the revised version.
Answer to Reviewer XPLR (3/3)
Q4
Which baselines were run under the same conditions as PESTO? In terms of data, splits and evaluation, every comparison in the paper follows the protocol of the corresponding work, because we evaluate PESTO under the protocol of each comparator; how each protocol was reproduced is detailed below.
In addition, two baselines were rerun by us with the same encoders as PESTO. On MM-Fi, contrastive pre-training with the multimodal objective of MMBind is trained with the same encoders, inputs, subject batches, optimiser and number of epochs, and read by the same fusion head (Table 7 and Appendix B). On UTD-MHAD, a model trained on the IMU alone with two augmented views uses the same encoder and classifier (Table 5).
For the other comparators (CroSSL, COCOA, XSleepNet, SeqSleepNet, MultiConsSleepNet, SleepSMC, NFC, HNS, CMC, Barlow Twins and the MMBind upper bound), we quote the published numbers and reproduce their protocols, preprocessing, data split, model-selection rule and metric, as closely as the published material allows:
- Code released. For XSleepNet, CroSSL and MMBind, we reconstructed the protocols from the authors' released code. Where the code and the paper differ, we followed the code and noted the difference in Appendix B; for example, the missing-sensor evaluation of CroSSL does not run as released, so we implemented it as its paper describes. For MMBind, the script that produces its preprocessed data is not released; we asked the authors through the code repository and, without a reply, reconstructed this step from the released data loader and the paper.
- Code and splits not released. For SleepSMC and MultiConsSleepNet, we followed the description in each paper.
- Subject split. NFC, HNS, CMC and Barlow Twins are compared on the odd/even subject split shared by these works.
Appendix B documents each reconstruction. The quoted numbers are marked as such in the table captions, and we plan to state this distinction for every comparison in Appendix B.
W4 and W5
We agree that the better presentation can make the motivation and results easier to follow. We will open Section 3.1 with three design requirements: paired modalities should select the same map location, similar samples should select nearby locations, and each modality should retain its own encoder and embedding dimension. We will then introduce the grid and prototype tables in terms of these motivations. In Section 4.2, we will organise the comparisons according to the groups in Table 2 and integrate the new experiments reported in responses, so readers can follow the evidence for each claim.
W6
We agree that this is a wording issue. The sentence in Section 4.2 refers to the main comparison of each protocol, made with all sensors. Against self-supervised methods, PESTO is ahead of or on par with every comparator in every protocol: the margins are clear on Sleep-EDF-20 and MM-Fi (0.785 against 0.722, and 81.0% against 74.3%), and PESTO matches NFC and MultiConsSleepNet (96.2% against 96.06%, and 85.9% against 85.8%). The two examples the reviewer mentions fall outside this comparison. The IMU-only column of Table 6 is a single-sensor condition, where NFC is ahead (75.58% against 71.1%) while PESTO is ahead with the skeleton alone (93.6% against 86.05%); we will also correct the bold face in this column. XSleepNet in Table 4 is trained end-to-end with labels and belongs to the second part of the sentence: there, PESTO obtains a higher accuracy and κ (84.3 against 84.0, and 0.781 against 0.778) and a macro-F1 within 0.7 points (78.0 against 78.7), which is what we mean by matching. We plan to revise the sentence so that it states this scope explicitly.
Regarding the source of the gains, the ablation in Q2 now separates the grid from prototype-based alignment: removing the neighbourhood term lowers accuracy on every dataset, with the largest decreases in single-sensor settings, and removes the organisation of the map. The revised sentence will cite this control; these changes will be reflected in the revised version.
In summary, the new ablation isolates the contribution of the map, the component-wise comparison in Q1 sets out what PESTO adds to existing methods, and the single-modality results in Q3 support the claim on modality-specific information; the remaining points concern presentation, which we will address in the revision. We hope that, with these additions, the contribution of the paper comes across more clearly, and that the reviewer will consider them in the final evaluation. Any further questions are very welcome during the discussion period.
Answer to Reviewer XPLR (1/3)
We thank the reviewer for the detailed review and for the clear summary of PESTO. The reviewer's questions help us clarify the novelty, the role of the grid, and the evidence for our claims. We address the four questions and related weaknesses first, followed by the presentation (W4-W5) and the claims in Section 4.2 (W6). In each new experiment, we kept all settings other than the tested settings as in the original paper.
Q1
What is new beyond combining contrastive learning with a SOM loss? We thank the reviewer for this important question. PESTO aligns modalities by their positions on a shared topological map while retaining a separate embedding space and prototype vectors for each modality. This allows encoders with different output dimensions to be aligned without a common projection; on MM-Fi, for example, the keypoint encoder outputs 16 dimensions, while the depth and radar encoders output 64. The shared map also provides a readout for any available subset of sensors and a spatial organisation that can be inspected directly, as illustrated by the sleep-stage maps.
Following Section 2 and 3 in the original paper, each component of PESTO differs from existing work as follows:
Two controlled experiments in this response isolate these parts. Removing the neighbourhood term lowers accuracy on every dataset and removes the organisation of the map (Q2 below). Reading the same frozen encoders through the map instead of through concatenated embeddings keeps F1 at 0.638 rather than 0.393 when one sensor is left on Sleep-EDF-20 (please find our detailed response to Reviewer quAL, Q2).
W1
We see how presenting Equations (2) and (3) side by side could suggest a straightforward combination of existing losses. The two terms are, however, well motivated by what the shared map requires. First, the modalities of one sample should select the same map position while different samples remain apart, which calls for an alignment with a repulsive component; aligning the modalities' distributions directly, without it, collapses (see our response to Reviewer quAL, W2). Second, nearby positions should represent similar samples, which is what the neighbourhood term enforces; removing it removes the organisation of the map (Q2). What is new is therefore not a loss term, but the structure in which these terms operate and what it enables, as described in Q1.
Regarding the prior work in Section 2 that updates the encoder through the neighbourhood loss, these methods learn one map for one encoder and one modality; in PESTO, one map is shared by all modality encoders, and the neighbourhood gradient reaches each of them. We plan to revise the list of contributions accordingly, stating the third contribution in its multimodal form, and this will be reflected in the revised version.