- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes PFS for multimodal contrastive learning. Instead of using a hand-designed similarity function, the authors pretrain a Transformer scorer on synthetic multimodal data generated from SCMs. The scorer is then frozen and used to train modality encoders on real datasets. The authors show that even a randomly initialized frozen Transformer works surprisingly well, and PFS further improves the results on four biomedical datasets.
Critical strengths
-
The motivation is quite clear. The paper is well written and easy to follow.
-
The idea of pretraining a scorer on synthetic data and reusing it across different datasets is reasonable and interesting.
-
The experiments on four real-world datasets provide some evidence that the proposed approach can work. The result that a randomly initialized frozen Transformer can already achieve competitive performance is also interesting.
Critical weaknesses
-
The main idea is quite straightforward. It mainly borrows the idea of pretraining on synthetic datasets, similar to TabPFN, but applies it to learning a scoring function. Learned similarity functions and neural critics have also been studied before. I think there is some novelty in this combination, but it is not particularly exciting.
-
The main experiments mostly compare PFS with hand-designed scoring functions, such as CLIP, Symile, Triangle, and Gram. However, there are already many works on learned multimodal scoring and matching. I would like to see stronger comparisons with these methods. Also, the trainable MLP scorer in Table 4 actually outperforms PFS on UKB, but this result is not included in the main comparison.
-
All four datasets are from the biomedical domain and have only three modalities. Although UKB has a relatively large cohort, the paper does not evaluate on larger and more diverse multimodal benchmarks. It is unclear how well this approach can scale to more modalities and larger datasets. Since the main idea is relatively simple, I would expect more solid experiments to support its general usefulness.
-
It is unclear how much of the improvement actually comes from the SCM structure. For example, Table 5 shows that using only linear SCMs gives almost the same performance as the full PFS. Removing DAG edges also preserves much of the improvement. I would like to see a comparison with simpler synthetic data generators, such as Gaussian latent-factor models.
-
Efficiency is not fully evaluated. Although the scorer is frozen, it still requires a Transformer forward pass for each candidate tuple. This could be expensive for large-scale retrieval. The paper reports training time, but I would also like to see inference cost and the total cost of synthetic pretraining.
Questions for authors
-
Why are stronger learned-scorer methods and recent multimodal contrastive learning methods, such as ConFu (cvpr2026), not included in the main comparison? Could the authors also explain why the trainable MLP performs better than PFS on UKB?
-
How important is the causal structure in the synthetic prior or it's just a direct adaptation fromTabPFN? Would a simpler synthetic data generator with shared latent factors achieve similar results?
-
Can the authors provide experiments on larger and non-biomedical datasets, or with more than three modalities? In particular, how would PFS handle additional modalities with new input projections?
-
What is the actual inference cost of PFS for large-scale retrieval? How large is the candidate set used to compute top-1 accuracy in Table 2?
-
The paper mentions that increasing the number of SCMs beyond 10k does not further improve performance. What do the authors think limits the scaling of prior-fitting?
5 comments
(Part 2)
[…] only three modalities.
Extending multimodal contrastive learning beyond three modalities is an important and relatively under-explored direction. Motivated by recent work on the UKB [1], we extend our evaluation to four modalities by adding touchscreen data.
The quadrimodal results further strengthen our main findings (Tab. R2): PFS achieves the highest retrieval performance, outperforming the strongest applicable handcrafted higher-order baseline. PFS also substantially improves upon the frozen random Transformer.
These results are based on a rigorous large-scale evaluation using UKB (~500k participants), including hyperparameter optimization and cross-validation. Triangle is not included because its scoring function is defined only for three modalities.
[1] T. Rheude, R. Eils, and B. Wild, “Fusion or Confusion? Multimodal Complexity Is Not All You Need,” in Advances in Neural Information Processing Systems (NeurIPS), 2026.
Tab R2: Higher-order baselines vs. PFS on the UKB with four modalities.
| Method | UKB M=4 (mean SE) |
|---|---|
| Higher-Order Baselines | |
| GRAM | |
| Symile | |
| Triangle | not defined - NaN |
| Proposed | |
| Transformer (frozen) | |
| PFS (frozen) |
W4 - Gain from SCM structure and comparisons with simpler synthetic data generators
We really thank the reviewer for this insightful suggestion since we also researched about this heavily in the last months. In fact, our only linear SCMs ablation (Tab. 5) evaluates a closely related generator. With Gaussian exogenous variables and linear structural equations and readouts, the resulting modalities follow a multivariate Gaussian distribution before normalization. The linear SCM can therefore be expressed as a Gaussian latent-factor model. We highly appreciate this alternative perspective on our existing baseline and will explicitly discuss its connection to Gaussian latent-factor models in the revised paper.
Our results suggest that the improvements arise mainly from prior-fitting on diverse latent cross-modal dependencies rather than from modeling nonlinear causal mechanisms. We see great potential in developing synthetic priors that better exploit causal structure and consider this an exciting direction for future work.
W5 - Efficiency (inference cost and pre-training cost)
We thank the reviewer for raising this point. We are happy to present additional experiments and studied whether a smaller scorer can retain the performance of the original PFS. This was especially helpful for faster development in the last months.
Interestingly, reducing the scorer parameters ( smaller) preserves or even improves downstream performance. As expected, retrieval scoring remains more expensive, reflecting the additional Transformer forward passes. However, total validation retrieval time is much closer (9.79 s vs. 8.93-9.39 s), indicating that scorer computation is not the dominant cost in this evaluation. Finally, synthetic pre-training of the smaller scorer requires only 1.45h. This is a one-time cost, as the resulting scorer can be reused across downstream datasets without further optimization.
Table R3: Timing analyses for all scoring functions on an A100-80GB.
| Scoring Function | Train time to lowest val loss (s) | Total train time (s) | Train time per step (ms) | Val retrieval time (s) | Scoring time per query (ms) |
|---|---|---|---|---|---|
| PFS-small | |||||
| Symile | |||||
| CLIP | |||||
| Triangle | |||||
| GRAM |
Table R4: Pre-training timing for PFS on an A100-80GB.
| Wall time total (h) | Wall time per step (s) | |
|---|---|---|
| PFS pre-training |
Table R5: New, smaller Transformer scorer.
| Setting | New Value | Old Value |
|---|---|---|
| Dimension | ||
| Layers | ||
| Heads | ||
| Head Dimension | ||
| FFN Dimension | ||
| Total Parameters |
(Part 3)
Q1.1 - Why are stronger learned-scorer methods and recent multimodal contrastive learning methods (ConFu), not included in the main comparison?
ConFu is related but addresses a complementary challenge: it introduces additional contrastive losses between individual and fused modalities, while we study the scoring function itself with PFS. In fact, ConFu still uses dot-product similarities. A direct comparison would therefore involve changing the training objective and introducing additional fusion networks, rather than isolating the effect of the scorer. We will clarify this and discuss ConFu in the main paper.
Regarding learned scorers, our submitted paper already includes trainable MLP and Transformer baselines (Tabs. 3-4). Since submission, we have substantially improved their optimization and extended the comparison to all four datasets (W2).
Q1.2 - Could the authors also explain why the trainable MLP performs better than PFS on UKB?
We find this very interesting! One possible explanation is that a task-specific MLP can adapt to UKB's structure, while PFS are a fixed inductive bias learned from synthetic data. We cannot yet establish this as the underlying cause, but we agree that this result highlights an important trade-off between task-specific adaptation and transferable scoring.
Q2 - How important is the causal structure in the synthetic prior or it's just a direct adaptation from TabPFN? Would a simpler synthetic data generator with shared latent factors achieve similar results?
Our existing only linear SCMs ablation (Tab. 5) is already equivalent to a Gaussian latent-factor model at the observational level (see W4). The similar performance of the linear-only and full priors suggests that nonlinear causal mechanisms are not the core. Instead, the benefits mainly come from learning diverse latent cross-modal dependencies.
Q4 - Can the authors provide experiments on larger and non-biomedical datasets, or with more than three modalities? In particular, how would PFS handle additional modalities with new input projections?
See W3.
Extending PFS beyond three modalities is straightforward. For example, for four modalities, one would have to change the assignment matrix and add another modality-specific input projection. The contrastive objective and the training would stay the same without much higher computational costs since its simply another token for the Transformer scorer in the end since, e.g., Triangle is not even defined for more than three modalities.
Q5.1 - What is the actual inference cost of PFS for large-scale retrieval?
We have added a more extensive efficiency analysis (see W5).
Q5.2 - How large is the candidate set used to compute top-1 accuracy in Table 2?
Thanks for pointing this out. The candidate set size is 128 (1 positive and 127 negatives). We will add this information in the main paper.
Q6 - What do the authors think limits the scaling of prior-fitting?
This is the main question about which we do research since submission for follow-up work! We currently observe only limited benefits from modeling more complex causal relationships (W4). We believe that larger and more diverse SCMs could improve performance, but it is not yet clear how to best exploit this.
(Part 1)
We thank the reviewer for the extensive feedback. We are happy to provide answers and new experiments, and hope that this will clarify the concerns since this represents a lot of research in the last months since submission.
W1 - Novelty w.r.t. learned similarity functions and neural critics
We agree that synthetic pre-training and learned similarity functions have been studied (Sec. 2). Our contribution goes beyond combining these ideas: A scoring function learned from synthetic data generalizes to real-world datasets without any downstream adaptation. In contrast to learned critics, which are typically trained jointly with the encoders, PFS provide a reusable, frozen scoring function. We show that this can outperform both handcrafted scoring functions and trained neural critics. We believe this is a promising new direction for multimodal contrastive learning and is an exciting possibility for future work.
W2 - Comparisons with other learned multimodal scoring functions
We agree that learned similarity functions and neural critics have been studied previously (Sec. 2) and our submitted paper already partly includes trainable MLP and Transformer scorers (Tabs. 3-4).
However, we agree that these comparisons deserve greater prominence, especially given the MLP result on the UKB.
Since submission, we have done a lot of research for answering why learned scorers sometimes but not always lead to improvements. We found that this is primarily related to optimization landscapes and extended their evaluation to all four datasets.
We identified two important factors:
These results confirm that learned scoring functions are strong alternatives when carefully optimized. At the same time, PFS offer a different approach: learning a transferable scoring function entirely from synthetic data and keeping it frozen during downstream training. In summary, the finetuned PFS still outperforms or is at least competitive to the trainable scorers.
Tab R1: Trainable scorers and finetuned PFS after fixing optimization landscape challenges.
W3 - More modalities, larger datasets, and other domains
We agree that non-biomedical datasets would be interesting and this is explained as limitation in the main paper (Sec. 5). However, multimodal datasets are actually the biggest bottleneck. We believe it is important to set existing datasets in context. So for example, MOSI/MOSEI primarily target sentiment classification rather than retrieval, while Symile-M3 already achieves around 90% retrieval accuracy in the original Symile paper, leaving limited room for improvement. We focus on healthcare because it provides different modalities and opportunities to extend contrastive learning beyond three modalities in future work. Overall, our evaluation covers four datasets with extensive tuning and cross-validations, providing an extensive assessment of generalizability especially compared to related work.
Larger datasets are valuable. However, we implemented four real-world datasets, with extensive hyperparameter tuning across all methods to ensure fair comparisons. Further, we also include the large-scale UKB with ~500k samples. Extending this evaluation to substantially larger datasets would require much more additional computational resources, especially under the same tuning protocol. We would welcome suggestions for multimodal retrieval benchmarks that the reviewer considers relevant.