acceptodds
Under review as a conference paper at ICLR 2027

FAME: Factorized Astronomical Multimodal Embeddings

Abstract

Stellar astrophysics observes the same stars through different instruments, across surveys with vastly different noise, resolution, and coverage, and some key properties, like stellar age, are not measured by any single one of them. Building a foundation model that combines these modalities therefore remains an open challenge. We present FAME, a foundation model for stellar astrophysics that combines seven modalities from six large-scale surveys via pretrained modality-specific encoders and a small alignment module. FAME departs from classical multimodal alignment in two ways. Its latent space is factorized into a shared and a residual subspace, so that aligning the modalities preserves what each one holds alone. And the shared subspace is trained with a joint-embedding objective extended by a graph-Laplacian penalty over catalogued physical coordinates. Under frozen linear probing, FAME outperforms unaligned features, a single-latent alignment baseline, and three existing astrophysics foundation models. Against fully supervised models it is more label-efficient on the labels that enter no pretraining objective and is beaten only on the one label read directly from a single observational capability; when the surveys carrying such a label are withheld, FAME still recovers it ( and ), where a supervised model trained under the same schedule retains little ( and ). Rare stellar populations are linearly decodable from the residual subspace, and the representation supports cross-modal spectral generation. Neither ingredient is specific to astronomy: the factorization applies wherever modalities carry private information, and the smoothness penalty wherever samples come with catalogued continuous coordinates. To our knowledge, FAME is the first foundation model for stellar astrophysics to combine time series, spectra and astrometry at scale.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
multimodal representation learning, self-supervised learning, joint-embedding methods, scientific foundation models, astrophysics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_Vi3VOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors use stellar astronomy data to train FAME, a foundation model for stellar astrophysics. The authors combine seven modalities from six surveys, crossmatched via Gaia DR3 source IDs.

They pretrain six encoders (one for each survey). Each encoder is trained both via a self-supervised objective alongside a supervised head. The encoders are then aligned to create the final model.

The paper represents substantial work, however I think it has some concerning flaws with evaluation and comparison with other astronomical foundation models. I go into these in my "weakness" section.

Critical strengths

This is a substantial piece of work. Training a ~1B parameter model is on the larger side of astronomy, and compiling the dataset to do this is a significant undertaking.

Critical weaknesses

I am concerned that astronomical labels are used at pre-training time. Particularly as the encoders are pre-trained with splits independent of the evaluation split. This seems to me to be a major oversight as it affects every downstream claimed result. The authors need to address this robustly.

No parameter scaling tests are carried out. A central expectation of a foundation model is that more data and more compute lead to better performance. This was shown for AION-1 with released variants from 300M to 3B params. FAME does not show this. Furthermore, your Figure 4 halts at 20k labels for downstream probe scaling, this hides the regime where purely supervised scaling would win out.

Linear probing is used for headline and external comparisons, whereas in AION-1 they use learned cross-attention as their probing method. An experiment using both non-linear probes and attentive methods is needed for a fair comparison.

AstroM3 and SpecCLIP are not compared (and SpecCLIP is not even cited!). Why not? They are highly relevant. The claim to be the first model combining time series, spectra and astrometry should be narrowed accordingly.

Questions for authors

Please see weaknesses.

Reviewer_yXLTOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper propose a foudnation model for astrophysics. Their model takes multimodal takes values from 7 surveys. They use dedicated encoders for each modality then a shared perciever-like module for multimodal tokens. They evaluate against competing models on probing tasks and show superior perforance across all data regimes.

Critical strengths
  • very interesting and under-explored topic

  • strong performance compared to the evaluated baselines

Critical weaknesses
  • the reviewer regret that there is not more exposition to the instrument and tasks. The community (or at least the reviewer) is not familiar at all with these and this would have helped making the paper more pleasant to read. The exact nature of the input / output is still quite vague after reading the entire paper.

  • The paper is overall quite hard to read and unclear. This comes from different reasons:

    • the authors do not introduce all their variables and losses, and some of the figure appear contradictory
    • the author use vague and imprecise statement relate to ML, likely heavily LMM- generated ("while the latent space geometry is illuminating, the ultimate test for Lsmooth is on downstream tasks")
    • many jargon heavy verbiage that were almost entirely indescribable by the reviewer
  • there is no ablation study to validate the different part of the model as necessary

  • some over-claiming, in particular that their approach is not "specific to astronomy" and "applies wherever modalities carry private information" yet this is not proven.

-the paper has some heavy structure issue: the reader is constantly referred to content from the appendix, and section 3 is just one subsection which is just one paragraph.

Due to the difficulty of reading the paper and the reviewer unfamiliarity with the topic and lack of exposition the reviewer is unable to assess whether the experimental protocol is rigorous or not.

Overall: The reviewer understands that the authors come from another community, which holds fascinating problems which could interest the ML community and benefit from its insight. However, ICLR remains a ML conference, and thus the authors should make an effort to make the paper readable by non-expert in their fields. As of now, it is very hard to get something out of this paper as a non expert as the reviewer only understand the (basic, but probably sound) model architecture, but neither the experiments settings not their analysis. The authors should either do a considerable effort in terms of pedagogy and exposition, or submit to a specialized workshop that would appreciate their contribution as experts.

Questions for authors

Details and questions:

"However, many of these works rely primarily on contrastive objectives, autoregressive models, or limited modalities." -> why are contrastive or autogressive models a bad thing?

"Because its continuum limit selects Laplacian eigenfunctions, the geometry it induces is predictable from the loss rather than empirical." The reveiwer could not follow

"This separation [of encoders] allows optimizing the architecture, maximizing training samples for each modality, and easily fine-tuning the model and adapting it to new tasks. " unclear why having seperate encoders does any of those things?

Figure 1 left: what are the the edges in the graph? Figure 1 right: a bit misealding, the encoders are not shared Figure1 right: does not represent the training objective (supervised and not) which is confusing

Figure2 : confusing, it looks like the modality are projected through a simple projector (linear?) when the text says an entire ransformer. Which produce one token per modality?? Unclear if the permodality encoder are transformers (fom the text), linear projectors, or MLPs

LSVICReg, Sres and Lrecon are not defined. Lsmooth does not appear in the figure 2

" [Lsmooth supplies] edges from catalogued coordinates, penalizing the squared difference of shared tokens that are close in the HR diagram:" The reveiwer could not follow

"ca is the min–max-normalized catalogue (Teff, log g) of star a" The reviewer could not follow

"and the per-node division by qaqb is the α = 1 member of the Coifman–Lafon family (Coifman and Lafon, 2006), which makes the continuum limit the unweighted Dirichlet energy R∥∇cs∥2 dc rather than one weighted by the sampling density."

éThe geometry is approximately translation-equivariant. Theeigenmodes of the bin-by-bin Gram matrix of the shared token are low-order sinusoids in Teff [...], the signature of a stationary kernel, and successive principal-component pairs trace smooth Lissajous curves along which Teff runs monotonically (upper middle and right), the geometry derive for a translation-symmetric representation; the same holdsfor log g (Figure 8). This follows from the loss: in its continuum limit Lsmooth is a Dirichlet energy whose operator is the Laplacian of parameter space, and its low eigenfunctions are Fourier modes The reviewer could not follow

Figure 3: The reviewer could not follow

Fine-tuned FAME reaches R2 = 0.736 (activity) and 0.921 (v sin i) with all modalities The reviewer could not follow

Reviewer_dm49OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper combines stellar light curves, spectra, and astrometry from multiple surveys. It uses pretrained encoders and an alignment module that learns shared information while retaining details from each observation type. A smoothness objective encourages stars with similar catalogue properties to have similar shared representations. The authors evaluate the model on stellar-property prediction, prediction with missing observations, identification of rare stellar populations, and generation of spectra from other observations.

Critical strengths
  1. The paper provide many experiments and analysis. Several results are promising. The Kepler–LAMOST comparison improves age prediction over DESA. The missing-modality experiments show large gains over the supervised baseline.
Critical weaknesses
  1. The training setup makes the physic parameter results difficult to assess. If I understand correctly, the model may have seen the test stars during encoder pretraining. For several parameters, it also saw most test stars’ labels (sec 5.2). Only age and activity were excluded from both the encoder and alignment training objectives. About 13% of the age-labelled probe stars have ages derived from rotation periods used in earlier training. The authors clearly disclose these limits in the main text and appendix, which I appreciate. However, this training setup makes it difficult to judge the performance gains over the baselines. As a result, the claims in the abstract and introduction (or other similar claim) should therefore be qualified:

(Abstract) Under frozen linear probing, FAME outperforms unaligned features, a single-latent alignment baseline, and three existing astrophysics foundation models.

(Introduction) Evidence that FAME outperforms three existing astrophysics foundation models under frozen linear probing, ...

Given the overlap between training and testing, I am not confident that these comparisons support such a strong conclusion without further qualification.

  1. The presentation makes the method and experimental results difficult to follow. Specifically,

    • Experimental structure: Section 5 contains five subsections but provides no opening overview of their purpose or how they connect.

    • Figures are cited out of numerical order in the main text. For example, sec 5.1 moves through figures 7 → 3 → 8 → 3 → 9 → 4. This makes the discussion difficult to follow. For example, the discussion around line 277 would be clearer if it identified figure 3 at the start, so readers know which figure to follow without switching back and forth to fig 7 and 8 in the appendix.

    • Some results that appear central to the paper are shown only in the appendix, even though the main text spends substantial space discussing them. Referring to appendix results is reasonable. However, for example 5.4 devotes an entire subsection to results (Table 8) appears only in the appendix. This separation makes the argument harder to follow.

    • Technical terms/details: Terms such as ACF and CQR pinball loss need clearer explanations where they first appear, with equations or clear references where appropriate. The light-curve encoder description in lines 202–209 is also difficult to follow. It lists AstroConformer, CNN, and DualFormer, but does not clearly explain how these components fit together or which losses train them. When the authors say “the two architectures differ,” where does that difference occur?

Together, these issues make it difficult to follow the method and assess the evidence supporting the paper’s claims.

Questions for authors
  1. In line 55 "Motivated by failure modes of classical multimodal objectives, ...", can author be more specific on which papers they refering to?

  2. Does the comparison with AION-1 change when it uses its published learned pooling procedure?

  3. Could you discuss how FAME’s method differs from FactorCL (Liang et al., NeurIPS 2023)? Both methods aim to preserve shared information and information unique to each modality.

Reviewer_K1d5OpenReview ↗
Rating
3(−1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

FAME is a foundation model for astronomical observations of stars drawn from 6 large-scale surveys. The model is trained in two phases: first, separate encoders are trained to model data from each survey using a combination of self-supervised and supervised objectives, then the encoders are connected to an alignment head to form an intermediate fusion model which is trained to produce embeddings which are factorised into a shared stream and a per-survey residual stream. The paper introduces SVICReg to train the shared stream. SVICReg extends VICReg with an additional objective which aligns embeddings of different stars with similar physical properties, leading to a latent space which is continuous in those properties, which is shown to improve performance on downstream tasks. The residual space is trained using reconstruction and supervised regression objectives to retain information which is not captured by the shared space.

This approach produces an embedding which retains more useful information than only training the shared stream with an alignment objective, which is demonstrated via an ablation. Extensive ablations and comparisons to strong baselines and other astronomy foundation models are carried out, and FAME consistently exhibits competitive or superior performance. Other experiments apply domain expertise to characterise the model's properties and prove salient advantages of the approach, such as improved performance when modelling rare subpopulations of stars.

Critical strengths

Point 1: successfully addresses an active open problem in machine learning for astronomy with an approach which will be interesting to the wider ICLR audience and applicable to other domains

Learning factorised multimodal embeddings from astronomical survey data is a fairly new and active problem in machine learning for astronomy, and there are multiple groups working on and interested in this problem. The authors do a good job of motivating the problem (I especially like figure 6) and explaining the value in tackling it. The problem is also being actively studied in the general CS / ML literature and other scientific domains, therefore it will be of interest to a wide audience. The authors have developed a novel solution to this problem and clearly demonstrated its effectiveness whilst also applying it to an unprecedented number and diversity of modalities. The comparsion to VICReg shows that this approach addresses an observed failure mode, and the ability of the residual space to improve performance on rare subpopulations of objects addresses a real deficiency of simple cross-modal alignment. This result will be very interesting to astronomers as modelling rare subpopulations effectively is especially valuable. Looking closely at the details of the model's design, all of the design decisions seem very sensible and respect the nature of the problem and data.

Point 2: well-considered, extensive, and careful evaluations are interpreted with domain-specific expertise to effectively support the paper's claims

I was very impressed by the extensive, appropriate, informative, and careful evaluations / experiments. The standard of the evaluations is exemplary and not typical of most papers in this area. There is not a canonical suite of benchmarks for foundation models in astronomy, so it requires significant effort to set up suitable evaluations, and I appreciate the effort which has gone into this. The results are interpreted well, often using relevant domain expertise to draw further insight, for example when discussing figure 5 in section 5.5. The experiments are very successful in supporting the claims in the text, for example figures 3 and 4 clearly demonstrate the value of L_smooth which is one of the key aspects of the paper's approach.

Note: in the "critical weaknesses" section, I discuss some concerns about the evaluations, however this does not contradict that the evaluations are of a high quality overall.

Critical weaknesses

Point 1: concerns about the evaluation protocol when comparing to other astronomy foundation models

My concerns centre around two themes:

  1. inputs provided to the models
  2. probing protocol may favour FAME

Note: despite these concerns, the performance of the model is strong and there are plenty of other supporting results (e.g. internal comparisons to other model architectures and objectives such as standard pre-training with VICReg), so the paper's contributions remain solid.

Details of concern 1

  • AION is provided with data from the Legacy Surveys including photometry scalars and photometric imaging, however these are OOD for AION (its Multimodal Universe pre-training dataset imposes selection cuts which would have filtered such objects). This data is also probably mostly redundant with the Gaia data provided alongside it, and the photometric imaging probably adds little information beyond that provided by the photometry scalars (since imaging a star effectively just smears the photometry scalars under the PSF).
  • When FAME is provided with similar inputs to AION, AION demonstrates superior performance, but when each model is given an expanded set of available modalities (note that these are different between FAME and AION), FAME demonstrates superior performance. I expect this is significantly a consequence of FAME being provided with modalities which are much more informative for predicting the probed quantities, rather than only a consequence of the differences in model architecture and objective. FAME receives spectra and timeseries (these contain a lot of information about the probed quantities), whereas AION Legacy Survey photometry and imaging (which adds little information beyond what is already present in the Gaia XP spectra). AION's performance is actually found to degrade when introducing the additional inputs. This is surprising, as I would expect the change to either be inert or improve performance (assuming that the information from the additional modalities is combined in such a way that the information from Gaia is not clobbered). This could suggest an issue in how the information from the modalities was combined, which relates to point 2 below.
  • When FAME is compared to Leung and Bovy's model, FAME is presented with all of the information that it can natively ingest (Gaia XP + astrometry), whereas Leung and Bovy's model is not afforded the same treatment as it can ingest Gaia BP−RP and 2MASS colours, a parallax-based pseudo-luminosity, and E(B−V).
  • Leung and Bovy's model is pre-trained to use only a random subset of up to 64 inputs but there are 110 Gaia XP coefficients. The authors do not state whether they (A) provided only a subset of the coefficients, which would limit the performance compared to FAME (which uses all 110 coefficients), or (B) provide all 110 coefficients which would be OOD for Leung and Bovy's model but might still improve performance.

Details of concern 2

  • Although linear probing is a standard way to assess downstream performance, it probably favours FAME over AION and Leung and Bovy's model. FAME is directly pre-trained to produce a compressed 'summary vector' with nice geometric and statistical properties which will make it particularly amenable to linear probing. AION and Leung and Bovy's model produce a high-dimensional transformer embedding sequence which is used to condition a generative decoder during pre-training. It is therefore necessary to use non-learned mean pooling to aggregate the embedding sequence into a vector for linear probing, and those models' pre-training objectives do not directly encourage this pooled vector to be amenable to linear probing. It is reasonable to expect that users of these models would choose a probe which is well-suited to the model they are using (whilst still being lightweight, cheap to train, and having limited capacity). AION uses a learned cross-attention pooling layer to extract information from its embedding sequence to form a compressed vector. I expect that this learned pooling would be particularly important when combining information from multiple modalities (which might address the degradation to AION's performance when adding additional Legacy Survey data).

Point 2: some related work on factorised representation learning is missing, despite the paper emphasising applicability outside of the astronomy setting

Section 2 (related work) has a paragraph on "shared and private representations", and elsewhere in the manuscript the authors mention that FAME's approach could be applied beyond an astronomy setting, however some relevant modern work on shared and private / factorised representations is omitted from this paragraph. Contextualising FAME in relation to this work would strengthen the manuscript and help to frame its contributions within the existing work on the topic. I suggest an incomplete list of relevant publications below:

There is also related modern work on using a label-informed contrastive objective to impose structure within the latent space which is relevant to the "Geometry-imposing regularization" paragraph:

Questions for authors
  1. By default, AION truncates the input sequence to 256 input tokens (but it can utilise >256 input tokens e.g. all 576 tokens from a Legacy Survey image). The evaluation protocol includes inputs with >256 input tokens. Did the authors ensure that the input sequence was not truncated to 256 tokens? If truncation occurred, this could potentially explain why adding more data seems to hurt AION's performance rather than being inert. Relatedly, when encoding multiple modalities with AION, were they passed through the encoder together or separately before concatenating the pooled embeddings from each modality? Running the modalities through AION separately (so that the Gaia block is unaltered by the presence of the Legacy Survey block) would assist in investigating why adding the Legacy Survey data seems to hurt performance.

  2. What were the dimensionalities of the embeddings derived from each model and provided to the probes for evaluation? This is not stated clearly in the text. Compact embeddings may be more amenable to the ridge probe.

Comments on the paper

1 comment

Hi reviewers,

Thank you for your reviewing service. The authors have provided rebuttals with new results. Please confirm that you have read them, state which of your concerns are or are not addressed, and whether that changes your recommendations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.