acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Multi-Objective Alignment in Pretraining Time-Series Foundation Models for EEG tasks

Abstract

Time-series foundation models have become a powerful approach for analyzing EEG data, showing strong performance across healthcare and neuroscience. However, most models rely on a single self-supervised pretraining objective, such as masked reconstruction, which may limit their ability to capture diverse features and hinder generalization across downstream tasks. Incorporating multiple learning objectives during pretraining is a promising direction, though disparities in gradient magnitudes can cause certain objectives to dominate the optimization, which reduces the benefits of joint training. In this work, we systematically study this phenomenon and explore the benefits of multi-objective pretraining of time-series foundation models. We introduce MAPE (Multi-objective Alignment in Pretraining time-series foundation models for EEG tasks), a theoretically grounded framework that efficiently aligns multiple objectives within a single training pipeline by analyzing their pretraining losses. Our straightforward, low-cost approach is applicable to any model. Our empirical analysis across three different encoders demonstrates that MAPE leads to improved performance compared to unaligned multi-objective pretraining and single self-supervised pretraining settings, highlighting its effectiveness in learning robust and generalizable EEG representations.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Multi-objective pretraining, Multi-objective alignment, Time-series foundation models, EEG foundation models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_T4MPOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a new method for using multi-objectives in Foundation model training. They tackle different limitations of existing multi-objectives method in the literature such as the impossibility of modality competition. Their method is motivated by theoretical insights and experiments over 3 backbones and different EEG datasets is done.

Critical strengths

First the paper reads well, the motivation of the work is well written and it is easy to follow where the authors wants to go. The method is also motivated by theoretical insights.

The paper tries to tackle an open question that is if multi-objectives can help for FM for EEGs.

Critical weaknesses

If the motivation are clear I am skeptical about the fact that this study and the results gives a clear conclusion of what it is interesting for EEG FM training:

  • First, the superiority of MAPE is not clear in all the study. The scores are often slightly above and the increase almost everytime inside the std range. Having a statistical test to show that the increase is significant could help to prove the utility of MAPE.
  • Second, an interesting part of the study is to understand if the loss used by the original authors of the model is really useful or not. For example in this workshop paper [1], the authors shows that a non-specific model Mantis is working well on EEG task. They claim that contrastive learning is helping for classification task, I'm wondering if losses of pre-training is task dependant. Adding other type of task could help to understand this behavior.
  • The settings of pre-training, except for Mantis with cauker, is not the one used orignal paper. For FM pre-training, you often need a lot of data, I'm wondering if the results are robust to the number of data in the pre-training. The conclusion will be strengthened by at least a study of the size of the pre-training datasets. I understand that computationally speaking training a FM is long, but for this particular study I think it matters.
  • Last, the authors claim in the abstract "lost-cost approach" and then in the limitations they say that "A limitation of our work is the inherently higher cost of pretraining". It should be clarify in the paper what the authors mean by "lost cost".

Small weaknesses:

  • The colors of Figure 1 are misleading. Each Losses have a color which is also in the latent space z. In the first place we want to link the colors between them.
  • OOD setting of Table 1 is never explained in the main paper nor criticised. It adds information but without any further information. Especially that in OOD settings MAPE seems to give less good results which is a setup that practitioners can deal with a lot.
  • You could only have one big Table with the two and three objectives, you will avoid duplicated lines and it will avoid back and forth between the tables for the reader.

Overall this study is interesting but it lack of further experiments and metrics to emphasise the need of their method.

[1] Gnassounou et. al., Leveraging Generic Time Series Foundation Models for EEG Classification, 2025

Questions for authors

See weaknesses.

Reviewer_TELiOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to combine multiple self-supervised objectives when pretraining time-series models for EEG tasks. It proposes MAPE, which adjusts objective weights using changes in training and validation losses. The aim is to prevent one objective from dominating training while retaining useful information from different objectives. The authors evaluate reconstruction, contrastive learning, and time-frequency reconstruction across MANTIS, CBraMod, and EEGPT. Downstream evaluation uses frozen representations with trained classifiers or adapters. Results show improvements over uniform weighting in several settings, although gains over the best single-objective baseline are generally modest and vary across tasks.

Critical strengths
  1. MAPE calculates weights from loss statistics without requiring separate gradient computations for each objective. This makes it potentially useful when the cost of balancing objectives matters.
  2. The evaluation covers different architectures and EEG tasks. Testing both a general time-series model and two EEG-specific models helps assess whether the approach depends on a particular architecture. The results provide useful evidence that combining objectives can help, but requires care.
Critical weaknesses
  1. MAPE uses normalized losses, but it is unclear whether the uniform baseline receives the same normalization. A baseline using normalized losses with fixed weights is essential: without it, the improvements could largely come from correcting loss scales rather than the proposed weighting rule.
  2. Although the paper defines objective weights as nonnegative, its unconstrained solution can produce negative values. A negative weight would encourage increasing the corresponding loss during training. The appendix explains how negative weights are handled for OGR, but does not provide an equivalent explanation for MAPE.
  3. The level of innovation and the completeness of the experiments are limited.
Questions for authors
  1. How was the 20% TUEG subset selected, and were pretraining training/validation splits separated by subject? Were all objective variants trained from scratch using the same initialization?
  2. DARE-EEG already combines masked reconstruction with contrastive mask alignment. What does MAPE add beyond this existing combination, and can you demonstrate its benefit against such a method under matched training and evaluation settings?
  3. The closed-form solution can produce negative weights, although the paper defines weights as nonnegative. How are these cases handled in MAPE, and does that handling preserve the claimed optimality?
  4. Several gains over the best single-objective baseline are small or inconsistent across tasks. Do these gains remain reliable across independent pretraining seeds and paired statistical comparisons?
Reviewer_CsJdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces MAPE (Multi-objective Alignment in Pretrainingtime-series foundation models for EEG tasks), a theoretically grounded frame-work that efficiently aligns multiple objectives within a single training pipeline by analyzing their pretraining losses. Our straightforward, low-cost approach is applicable to any model. The empirical analysis across three different encoders demonstrates that MAPE leads to improved performance compared to unaligned multi-objective pretraining and single self-supervised pretraining settings, high-lighting its effectiveness in learning robust and generalizable EEG representations.

Critical strengths
  1. Practical and theoretically motivated adaptive multi-objective alignment. The paper addresses an important limitation of multi-objective self-supervised pretraining: imbalance among objectives can prevent the model from benefiting from complementary learning signals. MAPE proposes a lightweight loss-based weighting mechanism with a closed-form solution, avoiding the computational overhead of explicitly manipulating gradients. Its model-agnostic design makes the approach potentially applicable to different foundation model architectures.
  2. Evaluation across multiple architectures and pretraining objectives. The paper evaluates MAPE on three distinct foundation model architectures and several downstream EEG tasks. The experiments include comparisons with single-objective pretraining, uniform multi-objective weighting, and OGR, as well as analyses of gradient magnitudes and objective weights. The extension to three objectives further supports the flexibility of the proposed framework.
Critical weaknesses
  1. Insufficient evidence distinguishing objective competition from cooperative optimization. The paper primarily identifies objective competition through gradient magnitude imbalance and unstable loss trajectories. However, a dominant gradient does not necessarily indicate harmful interference, as optimizing one objective may simultaneously benefit another. Without directly analyzing gradient directions or cross-objective interactions, it remains unclear whether the observed imbalance represents genuine objective competition or merely unequal contributions from cooperative objectives.
  2. Limited justification for gradient magnitude balancing. Although MAPE produces more balanced gradient magnitudes, the paper does not sufficiently justify why such balancing is necessarily beneficial for multi-objective pretraining. Different objectives may contribute unequally to representation learning, and balanced gradient magnitudes do not necessarily imply optimal joint optimization. Furthermore, the lack of comparisons with simpler normalization or gradient-balancing strategies makes it difficult to establish the necessity and distinctive benefits of the proposed alignment mechanism.
  3. Inconsistent and modest downstream performance improvements. Although MAPE achieves competitive results, its improvements over strong single-objective and multi-objective baselines are often modest and inconsistent across downstream datasets. In several cases, MAPE underperforms the strongest baselines, while adding a third pretraining objective does not consistently improve performance. Moreover, some improvements are small relative to the reported standard deviations, raising concerns about their statistical significance and the robustness of the claimed generalization benefits.
Questions for authors
  1. Distinguishing objective competition from cooperative optimization. The paper identifies objective competition primarily through gradient magnitude imbalance and unstable loss trajectories. However, a dominant gradient does not necessarily imply harmful interference, as optimizing one objective may simultaneously benefit another. How do the authors distinguish genuine objective conflicts from cooperative optimization with unequal gradient contributions? Could the authors provide direct evidence, such as gradient cosine similarities or cross-objective loss changes, to support the claimed competition?
  2. Justification and necessity of gradient magnitude balancing. MAPE produces more balanced gradient magnitudes, but why should such balancing necessarily lead to better multi-objective optimization? Different objectives may contribute unequally to downstream representation learning, and balanced gradients do not necessarily indicate an optimal trade-off. Could the authors clarify the theoretical or empirical justification for this assumption and compare MAPE with simpler loss-normalization or gradient-balancing strategies to demonstrate the necessity of the proposed mechanism?
  3. Consistency and statistical significance of performance improvements. Tables 1–3 show that MAPE does not consistently outperform the strongest baselines, and several improvements are relatively small compared with the reported standard deviations. Under what conditions does MAPE provide reliable benefits, and are its improvements statistically significant? Additionally, why does introducing a third pretraining objective sometimes fail to improve, or even degrade, downstream performance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.