acceptodds
Under review as a conference paper at ICLR 2027

Prior-Fitted and Frozen Scorers for Multimodal Contrastive Learning

Abstract

Multimodal contrastive learning typically relies on manually specified scoring functions, such as cosine similarity, to quantify the similarity between representations from different modalities. Several such functions have been recently proposed to extend contrastive learning beyond two modalities. In principle, however, any function that assigns high compatibility to matched samples and low compatibility to mismatched ones could serve as a scorer. Motivated by this broader view, we instantiate the scorer with a Transformer and find that even a randomly initialized model, when kept frozen, can outperform strong handcrafted multimodal contrastive baselines, whereas optimizing the scorer jointly with the encoders is less effective. Building on this observation, we introduce the Prior-Fitted Scorer (PFS): a Transformer scorer pre-trained on synthetic multimodal data generated from structural causal model (SCM) priors before transferring it to real-world data. The SCM exposes the scorer to diverse dependency structures, including shared, pairwise, and private information, providing a structured prior over higher-order cross-modal similarity. Across four real-world datasets, frozen Transformer scorers are competitive to contrastive baselines, and prior-fitting further improves the random initialization. More broadly, our results suggest that multimodal contrastive learning need not rely on manual similarity functions: Transformers can provide strong inductive biases, while causal pretraining can further shape these biases.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
multimodal contrastive learning, structural causal models, prior-fitted networks, synthetic pretraining, transformer scorer, multimodal retrieval, representation learning, causal priors

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_R8TTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PFS for multimodal contrastive learning. Instead of using a hand-designed similarity function, the authors pretrain a Transformer scorer on synthetic multimodal data generated from SCMs. The scorer is then frozen and used to train modality encoders on real datasets. The authors show that even a randomly initialized frozen Transformer works surprisingly well, and PFS further improves the results on four biomedical datasets.

Critical strengths
  • The motivation is quite clear. The paper is well written and easy to follow.

  • The idea of pretraining a scorer on synthetic data and reusing it across different datasets is reasonable and interesting.

  • The experiments on four real-world datasets provide some evidence that the proposed approach can work. The result that a randomly initialized frozen Transformer can already achieve competitive performance is also interesting.

Critical weaknesses
  • The main idea is quite straightforward. It mainly borrows the idea of pretraining on synthetic datasets, similar to TabPFN, but applies it to learning a scoring function. Learned similarity functions and neural critics have also been studied before. I think there is some novelty in this combination, but it is not particularly exciting.

  • The main experiments mostly compare PFS with hand-designed scoring functions, such as CLIP, Symile, Triangle, and Gram. However, there are already many works on learned multimodal scoring and matching. I would like to see stronger comparisons with these methods. Also, the trainable MLP scorer in Table 4 actually outperforms PFS on UKB, but this result is not included in the main comparison.

  • All four datasets are from the biomedical domain and have only three modalities. Although UKB has a relatively large cohort, the paper does not evaluate on larger and more diverse multimodal benchmarks. It is unclear how well this approach can scale to more modalities and larger datasets. Since the main idea is relatively simple, I would expect more solid experiments to support its general usefulness.

  • It is unclear how much of the improvement actually comes from the SCM structure. For example, Table 5 shows that using only linear SCMs gives almost the same performance as the full PFS. Removing DAG edges also preserves much of the improvement. I would like to see a comparison with simpler synthetic data generators, such as Gaussian latent-factor models.

  • Efficiency is not fully evaluated. Although the scorer is frozen, it still requires a Transformer forward pass for each candidate tuple. This could be expensive for large-scale retrieval. The paper reports training time, but I would also like to see inference cost and the total cost of synthetic pretraining.

Questions for authors
  • Why are stronger learned-scorer methods and recent multimodal contrastive learning methods, such as ConFu (cvpr2026), not included in the main comparison? Could the authors also explain why the trainable MLP performs better than PFS on UKB?

  • How important is the causal structure in the synthetic prior or it's just a direct adaptation fromTabPFN? Would a simpler synthetic data generator with shared latent factors achieve similar results?

  • Can the authors provide experiments on larger and non-biomedical datasets, or with more than three modalities? In particular, how would PFS handle additional modalities with new input projections?

  • What is the actual inference cost of PFS for large-scale retrieval? How large is the candidate set used to compute top-1 accuracy in Table 2?

  • The paper mentions that increasing the number of SCMs beyond 10k does not further improve performance. What do the authors think limits the scaling of prior-fitting?

5 comments

(Part 1)

We thank the reviewer for the extensive feedback. We are happy to provide answers and new experiments, and hope that this will clarify the concerns since this represents a lot of research in the last months since submission.


W1 - Novelty w.r.t. learned similarity functions and neural critics

We agree that synthetic pre-training and learned similarity functions have been studied (Sec. 2). Our contribution goes beyond combining these ideas: A scoring function learned from synthetic data generalizes to real-world datasets without any downstream adaptation. In contrast to learned critics, which are typically trained jointly with the encoders, PFS provide a reusable, frozen scoring function. We show that this can outperform both handcrafted scoring functions and trained neural critics. We believe this is a promising new direction for multimodal contrastive learning and is an exciting possibility for future work.

W2 - Comparisons with other learned multimodal scoring functions

We agree that learned similarity functions and neural critics have been studied previously (Sec. 2) and our submitted paper already partly includes trainable MLP and Transformer scorers (Tabs. 3-4).

However, we agree that these comparisons deserve greater prominence, especially given the MLP result on the UKB.

Since submission, we have done a lot of research for answering why learned scorers sometimes but not always lead to improvements. We found that this is primarily related to optimization landscapes and extended their evaluation to all four datasets.

We identified two important factors:

  1. Scaling the unit-normalized embeddings by (128)\sqrt(128) for the MLP, preventing its GELU activations from acting mainly in the near-linear regime, and
  2. Using a separate, approximately 5-10× smaller learning rate for the scorer.

These results confirm that learned scoring functions are strong alternatives when carefully optimized. At the same time, PFS offer a different approach: learning a transferable scoring function entirely from synthetic data and keeping it frozen during downstream training. In summary, the finetuned PFS still outperforms or is at least competitive to the trainable scorers.

Tab R1: Trainable scorers and finetuned PFS after fixing optimization landscape challenges.

ScorerINSPECT ↑\uparrowUKB ↑\uparrowMIMIC-Symile ↑\uparrowMIMIC-IV ↑\uparrow
MLP (trainable)0.7461±0.0010.7461 \pm 0.0010.6105±0.0070.6105 \pm 0.0070.4950±0.0160.4950 \pm 0.0160.6701±0.0060.6701 \pm 0.006
Transformer (trainable)0.7592±0.0040.7592 \pm 0.0040.5950±0.0180.5950 \pm 0.0180.4940±0.0120.4940 \pm 0.0120.6380±0.0030.6380 \pm 0.003
PFS (finetuned)0.7792±0.002\mathbf{0.7792 \pm 0.002}0.5970±0.0080.5970 \pm 0.0080.5100±0.1100.5100 \pm 0.1100.6787±0.002\mathbf{0.6787 \pm 0.002}

W3 - More modalities, larger datasets, and other domains

“All four datasets are from the biomedical domain”

We agree that non-biomedical datasets would be interesting and this is explained as limitation in the main paper (Sec. 5). However, multimodal datasets are actually the biggest bottleneck. We believe it is important to set existing datasets in context. So for example, MOSI/MOSEI primarily target sentiment classification rather than retrieval, while Symile-M3 already achieves around 90% retrieval accuracy in the original Symile paper, leaving limited room for improvement. We focus on healthcare because it provides different modalities and opportunities to extend contrastive learning beyond three modalities in future work. Overall, our evaluation covers four datasets with extensive tuning and cross-validations, providing an extensive assessment of generalizability especially compared to related work.

[…] The paper does not evaluate on larger and more diverse multimodal benchmarks.

Larger datasets are valuable. However, we implemented four real-world datasets, with extensive hyperparameter tuning across all methods to ensure fair comparisons. Further, we also include the large-scale UKB with ~500k samples. Extending this evaluation to substantially larger datasets would require much more additional computational resources, especially under the same tuning protocol. We would welcome suggestions for multimodal retrieval benchmarks that the reviewer considers relevant.

(Part 2)

[…] only three modalities.

Extending multimodal contrastive learning beyond three modalities is an important and relatively under-explored direction. Motivated by recent work on the UKB [1], we extend our evaluation to four modalities by adding touchscreen data.

The quadrimodal results further strengthen our main findings (Tab. R2): PFS achieves the highest retrieval performance, outperforming the strongest applicable handcrafted higher-order baseline. PFS also substantially improves upon the frozen random Transformer.

These results are based on a rigorous large-scale evaluation using UKB (~500k participants), including hyperparameter optimization and cross-validation. Triangle is not included because its scoring function is defined only for three modalities.

[1] T. Rheude, R. Eils, and B. Wild, “Fusion or Confusion? Multimodal Complexity Is Not All You Need,” in Advances in Neural Information Processing Systems (NeurIPS), 2026.

Tab R2: Higher-order baselines vs. PFS on the UKB with four modalities.

MethodUKB M=4 (mean ±\pm SE) ↑\uparrow
Higher-Order Baselines
GRAM0.4453±0.0080.4453 \pm 0.008
Symile0.4012±0.0100.4012 \pm 0.010
Trianglenot defined - NaN
Proposed
Transformer (frozen)0.3452±0.0140.3452 \pm 0.014
PFS (frozen)0.5215±0.017\mathbf{0.5215 \pm 0.017}

W4 - Gain from SCM structure and comparisons with simpler synthetic data generators

We really thank the reviewer for this insightful suggestion since we also researched about this heavily in the last months. In fact, our only linear SCMs ablation (Tab. 5) evaluates a closely related generator. With Gaussian exogenous variables and linear structural equations and readouts, the resulting modalities follow a multivariate Gaussian distribution before normalization. The linear SCM can therefore be expressed as a Gaussian latent-factor model. We highly appreciate this alternative perspective on our existing baseline and will explicitly discuss its connection to Gaussian latent-factor models in the revised paper.

Our results suggest that the improvements arise mainly from prior-fitting on diverse latent cross-modal dependencies rather than from modeling nonlinear causal mechanisms. We see great potential in developing synthetic priors that better exploit causal structure and consider this an exciting direction for future work.

W5 - Efficiency (inference cost and pre-training cost)

We thank the reviewer for raising this point. We are happy to present additional experiments and studied whether a smaller scorer can retain the performance of the original PFS. This was especially helpful for faster development in the last months.

Interestingly, reducing the scorer parameters (30×30\times smaller) preserves or even improves downstream performance. As expected, retrieval scoring remains more expensive, reflecting the additional Transformer forward passes. However, total validation retrieval time is much closer (9.79 s vs. 8.93-9.39 s), indicating that scorer computation is not the dominant cost in this evaluation. Finally, synthetic pre-training of the smaller scorer requires only 1.45h. This is a one-time cost, as the resulting scorer can be reused across downstream datasets without further optimization.

Table R3: Timing analyses for all scoring functions on an A100-80GB.

Scoring FunctionTrain time to lowest val loss (s) ↓\downarrowTotal train time (s) ↓\downarrowTrain time per step (ms) ↓\downarrowVal retrieval time (s) ↓\downarrowScoring time per query (ms) ↓\downarrow
PFS-small361361575575309.8309.89.799.791.1441.144
Symile294294485485272.3272.38.968.960.1410.141
CLIP362362550550261.3261.38.938.930.2340.234
Triangle486486676676263.7263.79.129.120.3420.342
GRAM410410603603264.5264.59.399.390.5870.587

Table R4: Pre-training timing for PFS on an A100-80GB.

Wall time total (h) ↓\downarrowWall time per step (s) ↓\downarrow
PFS pre-training1.451.450.0670.067

Table R5: New, smaller Transformer scorer.

SettingNew Value ↓\downarrowOld Value ↓\downarrow
Dimension128128512512
Layers2222
Heads2222
Head Dimension6464256256
FFN Dimension128128512512
Total Parameters298K298K8.9M8.9M

(Part 3)

Q1.1 - Why are stronger learned-scorer methods and recent multimodal contrastive learning methods (ConFu), not included in the main comparison?

ConFu is related but addresses a complementary challenge: it introduces additional contrastive losses between individual and fused modalities, while we study the scoring function itself with PFS. In fact, ConFu still uses dot-product similarities. A direct comparison would therefore involve changing the training objective and introducing additional fusion networks, rather than isolating the effect of the scorer. We will clarify this and discuss ConFu in the main paper.

Regarding learned scorers, our submitted paper already includes trainable MLP and Transformer baselines (Tabs. 3-4). Since submission, we have substantially improved their optimization and extended the comparison to all four datasets (W2).

Q1.2 - Could the authors also explain why the trainable MLP performs better than PFS on UKB?

We find this very interesting! One possible explanation is that a task-specific MLP can adapt to UKB's structure, while PFS are a fixed inductive bias learned from synthetic data. We cannot yet establish this as the underlying cause, but we agree that this result highlights an important trade-off between task-specific adaptation and transferable scoring.

Q2 - How important is the causal structure in the synthetic prior or it's just a direct adaptation from TabPFN? Would a simpler synthetic data generator with shared latent factors achieve similar results?

Our existing only linear SCMs ablation (Tab. 5) is already equivalent to a Gaussian latent-factor model at the observational level (see W4). The similar performance of the linear-only and full priors suggests that nonlinear causal mechanisms are not the core. Instead, the benefits mainly come from learning diverse latent cross-modal dependencies.

Q4 - Can the authors provide experiments on larger and non-biomedical datasets, or with more than three modalities? In particular, how would PFS handle additional modalities with new input projections?

See W3.

Extending PFS beyond three modalities is straightforward. For example, for four modalities, one would have to change the assignment matrix and add another modality-specific input projection. The contrastive objective and the training would stay the same without much higher computational costs since its simply another token for the Transformer scorer in the end since, e.g., Triangle is not even defined for more than three modalities.

Q5.1 - What is the actual inference cost of PFS for large-scale retrieval?

We have added a more extensive efficiency analysis (see W5).

Q5.2 - How large is the candidate set used to compute top-1 accuracy in Table 2?

Thanks for pointing this out. The candidate set size is 128 (1 positive and 127 negatives). We will add this information in the main paper.

Q6 - What do the authors think limits the scaling of prior-fitting?

This is the main question about which we do research since submission for follow-up work! We currently observe only limited benefits from modeling more complex causal relationships (W4). We believe that larger and more diverse SCMs could improve performance, but it is not yet clear how to best exploit this.

Reviewer_GjYXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether a learned scorer can replace a hand-designed similarity function in multimodal contrastive learning. The authors train a Transformer scorer on synthetic data from structural causal models, freeze it, and then train modality encoders on real data. The method improves top-1 retrieval over the listed fixed-form baselines on four trimodal biomedical datasets.

Critical strengths
  1. The paper asks a clear question about how multimodal similarity should be defined. Prior-fitting a scorer on synthetic data is a direct way to test this idea.

  2. The method and transfer procedure are easy to follow. The frozen random Transformer and trainable-scorer controls help separate the effects of architecture, prior-fitting, and downstream training.

  3. The paper reports five-fold results on four datasets and ablates the synthetic prior. The same frozen scorer is used across datasets without dataset-specific prior-fitting.

Critical weaknesses
  1. The strongest learned-scorer result is missing from the main comparison. In Table 4, a trainable MLP reaches 0.6115 top-1 accuracy on UKB, above the frozen PFS at 0.5745. Its results on all four datasets are needed to qualify the claim that PFS is the strongest scorer.

  2. The need for the nonlinear part of the SCM prior is not established. In Table 5, a linear-only prior reaches 0.5733 on UKB, close to 0.5745 for the full prior. The paper should distinguish the benefit of cross-modal structure from that of nonlinear mechanisms.

  3. The accuracy gain comes with substantial cost. Table 6 reports 0.244 seconds per training step for PFS versus 0.038 for Symile, before counting prior-fitting. Total training cost should be reported alongside retrieval gains.

  4. All four datasets are trimodal and biomedical, and the main task is retrieval. The paper does not test a new modality count or a different domain. The general claim about reusable multimodal scoring should be limited to the evaluated settings.

Questions for authors

Please address the above weaknesses and answer the following questions.

  1. Can you evaluate the trainable MLP scorer on all four datasets under the Table 2 setting? Where does PFS outperform it after comparable tuning?

  2. Can you compare the full SCM prior with a simpler linear prior on a second dataset? Which part of the prior explains the improvement over a frozen random Transformer?

  3. What are the total GPU-hours for prior-fitting and downstream training for each scorer? How does PFS compare at a matched compute budget?

  4. Does the same frozen scorer work when the number or type of modalities changes? If this is not tested, please narrow the transfer claim.

3 comments

(Part 1)

We thank the reviewer for this extensive feedback. We are happy to provide answers and new experiments and hope that this will clarify the concerns since this represents a lot of research in the last months since submission.


W1 - Learned scorers in main comparison

We agree that these comparisons deserve greater prominence, especially given the MLP result on the UKB. Since submission, we have done a lot of research for answering why learned scorers sometimes but not always lead to improvements. We found that this is primarily related to optimization landscapes and extended their evaluation to all four datasets.

We identified two important factors:

  1. Scaling the unit-normalized embeddings by (128)\sqrt(128) for the MLP, preventing its GELU activations from acting mainly in the near-linear regime, and
  2. Using a separate, approximately 5-10× smaller learning rate for the scorer.

These results confirm that learned scoring functions are strong alternatives when carefully optimized. At the same time, PFS offers a different approach: learning a transferable scoring function entirely from synthetic data and keeping it frozen during downstream training. In summary, the finetuned PFS still outperforms or is at least competitive to the trainable scorers.

Tab R1: Trainable scorers and finetuned PFS after fixing optimization landscape challenges.

ScorerINSPECT ↑\uparrowUKB ↑\uparrowMIMIC-Symile ↑\uparrowMIMIC-IV ↑\uparrow
MLP (trainable)0.7461±0.0010.7461 \pm 0.0010.6105±0.0070.6105 \pm 0.0070.4950±0.0160.4950 \pm 0.0160.6701±0.0060.6701 \pm 0.006
Transformer (trainable)0.7592±0.0040.7592 \pm 0.0040.5950±0.0180.5950 \pm 0.0180.4940±0.0120.4940 \pm 0.0120.6380±0.0030.6380 \pm 0.003
PFS (finetuned)0.7792±0.002\mathbf{0.7792 \pm 0.002}0.5970±0.0080.5970 \pm 0.0080.5100±0.1100.5100 \pm 0.1100.6787±0.002\mathbf{0.6787 \pm 0.002}

W2 - Need for nonlinearity and its distinction to cross-modal structure

We really thank the reviewer for this insightful suggestion since we also researched about this heavily in the last months. Our results suggest that the improvements arise mainly from prior-fitting on diverse latent cross-modal dependencies rather than from modeling nonlinear causal mechanisms. We see great potential in developing synthetic priors that better exploit causal structure and consider this an exciting direction for future work.

W3 - Efficiency and total training time

We thank the reviewer for raising this point. We are happy to present additional experiments and studied whether a smaller scorer can retain the performance of the original PFS. This was especially helpful for faster development in the last months since submission.

Interestingly, reducing the scorer parameters (30×30\times smaller) preserves or even improves downstream performance. As expected, retrieval scoring remains more expensive, reflecting the additional Transformer forward passes. However, total validation retrieval time is much closer (9.79 s vs. 8.93-9.39 s), indicating that scorer computation is not the dominant cost in this evaluation. Finally, synthetic pre-training of the smaller scorer requires only 1.45h. This is a one-time cost, as the resulting scorer can be reused across downstream datasets without further optimization.

Table R2: Timing analyses for all scoring functions on an A100-80GB.

Scoring FunctionTrain time to lowest val loss (s) ↓\downarrowTotal train time (s) ↓\downarrowTrain time per step (ms) ↓\downarrowVal retrieval time (s) ↓\downarrowScoring time per query (ms) ↓\downarrow
PFS-small361361575575309.8309.89.799.791.1441.144
Symile294294485485272.3272.38.968.960.1410.141
CLIP362362550550261.3261.38.938.930.2340.234
Triangle486486676676263.7263.79.129.120.3420.342
GRAM410410603603264.5264.59.399.390.5870.587

Table R3: Pre-training timing for PFS on an A100-80GB.

Wall time total (h) ↓\downarrowWall time per step (s) ↓\downarrow
PFS pre-training1.451.450.0670.067

Table R4: New, smaller Transformer scorer.

SettingNew Value ↓\downarrowOld Value ↓\downarrow
Dimension128128512512
Layers2222
Heads2222
Head Dimension6464256256
FFN Dimension128128512512
Total Parameters298K298K8.9M8.9M

(Part 2)

W4 - Limitation to trimodal, biomedical datasets

“All four datasets are […] biomedical, and the main task is retrieval.”

We agree that non-biomedical datasets would be interesting and this is explained as limitation in the main paper (Sec. 5). We believe it is important to set existing datasets in context. So for example, MOSI/MOSEI primarily target sentiment classification rather than retrieval, while Symile-M3 already achieves around 90% retrieval accuracy in the original Symile paper, leaving limited room for improvement. We focus on healthcare because it provides different modalities and opportunities to extend contrastive learning beyond three modalities in future work. Further, Related ML work in multimodal contrastive learning mainly covers retrieval tasks. Overall, our evaluation covers four datasets with extensive tuning and cross-validations, providing an extensive assessment of generalizability especially compared to related work.

“The paper does not test a new modality count.”

Extending multimodal contrastive learning beyond three modalities is an important and relatively under-explored direction. Motivated by recent work on the UKB [1], we extend our evaluation to four modalities by incorporating touchscreen data.

The quadrimodal results further strengthen our main findings (Tab. R): PFS achieves the highest retrieval performance, outperforming the strongest applicable handcrafted higher-order baseline. PFS also substantially improves upon the frozen random Transformer.

These results are based on a rigorous large-scale evaluation using UKB (~500k participants), including hyperparameter optimization and cross-validation. Triangle is not included because its scoring function is defined specifically for three modalities.

[1] T. Rheude, R. Eils, and B. Wild, “Fusion or Confusion? Multimodal Complexity Is Not All You Need,” in Advances in Neural Information Processing Systems (NeurIPS), 2026.

Tab R5: Higher-order baselines vs. PFS on the UKB with four modalities.

MethodUKB M=4 (mean ±\pm SE) ↑\uparrow
Higher-Order Baselines
GRAM0.4453±0.0080.4453 \pm 0.008
Symile0.4012±0.0100.4012 \pm 0.010
Trianglenot defined - NaN
Proposed
Transformer (frozen)0.3452±0.0140.3452 \pm 0.014
PFS (frozen)0.5215±0.017\mathbf{0.5215 \pm 0.017}

Q1 - Can you evaluate the trainable MLP scorer on all four datasets under the Table 2 setting? Where does PFS outperform it after comparable tuning?

See W1.

Q2 - Can you compare the full SCM prior with a simpler linear prior on a second dataset? Which part of the prior explains the improvement over a frozen random Transformer?

We have extended the linear-only ablation to MIMIC-Symile. Consistent with Tab. 5, the linear-only prior achieves similar performance. This further supports that the improvements mainly come from pre-training on cross-modal structures rather than nonlinear causal mechanisms (see W2).

Table R6: Linear-only ablation on a second dataset (Mimic-Symile)

Top-1 Acc ↑\uparrow
Transformer (frozen)0.5021±0.0190.5021 \pm 0.019
Only linear SCMs0.5130±0.0080.5130 \pm 0.008
PFS (frozen)0.5164±0.0070.5164 \pm 0.007

Q3 - What are the total GPU-hours for prior-fitting and downstream training for each scorer? How does PFS compare at a matched compute budget?

We have added an extended efficiency analysis covering prior-fitting, downstream training, and inference costs (W3). While these experiments provide a detailed comparison, they do not show a strictly matched-compute evaluation. We will clarify this in the main paper.

Q4 - Does the same frozen scorer work when the number or type of modalities changes? If this is not tested, please narrow the transfer claim.

See W4.

Thanks for the additional experiments, which directly answer my questions. Please incorporate these results into the manuscript. However, the trainable MLP still performs better than the reported PFS, and the efficiency analysis does not include a matched-compute comparison. These careful evaluations are important to demonstrate the effects, so I would like to keep my score.

Reviewer_5gcWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates replacing handcrafted similarity functions with Transformer-based scorers for multimodal contrastive learning. The authors first observe that a randomly initialized and frozen Transformer can guide modality-specific encoders to learn effective representations. They then propose the Prior-Fitted Scorer (PFS), which is pretrained on synthetic multimodal vectors generated from structural causal models and subsequently frozen while the modality encoders are trained on real-world data. Experiments cover four trimodal biomedical retrieval datasets, with ablations on scorer architectures, initialization schemes, and pretraining configurations. PFS outperforms the handcrafted scoring baselines in the main results table, although the training objectives, baseline implementations, and practical computational costs require further clarification.

Critical strengths
  1. A randomly initialized and frozen Transformer achieves competitive performance across multiple datasets, demonstrating that a fixed neural scoring function can effectively guide modality encoders. This observation broadens the design space of scoring functions for multimodal contrastive learning.

  2. The authors report reusing a single pretrained scorer across four biomedical datasets with different modality combinations, consistently improving upon the randomly initialized frozen Transformer. Ablations on scorer architecture, initialization, and synthetic priors provide initial evidence for understanding these improvements.

Critical weaknesses
  1. The paper does not explicitly provide the complete optimization objective, including aggregation across retrieval directions, the number of negatives, and the sampling procedure. The synthetic pretraining and real-data training also lack a consolidated description. Furthermore, while CLIP, Triangle, Gram, and Symile are briefly introduced, their training and retrieval implementations in this trimodal setting are not sufficiently detailed, nor is the precise consistency of candidate construction, sampling, and tuning protocols across methods. These omissions make it difficult to reconstruct the experiments and determine whether the performance differences can be attributed primarily to scorer design.

  2. Table 6 shows that PFS has substantially higher training time than the handcrafted scoring baselines, while also requiring a separate synthetic pretraining stage. Since Figure 3 uses epochs as the horizontal axis, it demonstrates convergence in terms of training epochs but does not establish shorter training time. The paper does not adequately report the total pretraining cost, the training time or inference costs in different tasks. These measurements are important for assessing the practicality and scalability of the method.

Questions for authors
  1. Could the authors provide the complete optimization objective, specifying how losses are aggregated across modality retrieval directions, the number of negatives, and the negative sampling procedure? Please also clarify whether synthetic pretraining and real-data training differ in their objectives or sample construction.

  2. Could the authors provide the scoring functions used for CLIP, Triangle, Gram, and Symile, and explain how each method constructs its training loss and performs retrieval in the trimodal setting?

  3. Could the authors provide unified pseudocode or an algorithm table specifying the inputs, trainable parameters, and frozen parameters in synthetic pretraining and real-data training, as well as the relationship between the scorer and modality encoders?

  4. Could the authors report the total cost of PFS pretraining and the wall-clock time required by each method to reach a common retrieval accuracy? How does inference time scale with the number of retrieval candidates?

3 comments

With the rebuttal phase coming to an end, we thank the reviewer again for their time and feedback. Overall, the new results support our central findings.

We would be grateful if the reviewer could let us know whether any concerns remain before the rebuttal period ends.

Thanks for the response, which directly addressed my question. It remains unclear whether the experiments in Table 2 of the manuscript and Table R2 of the response used the same PFS model. If different models were used, it would be difficult to determine whether scorer computation is indeed not the dominant computational cost. I believe this analysis is important and I decide to keep my score.

We are grateful that our response addressed all questions. We would like to clarify a misunderstanding for the model choice. We addressed in our initial rebuttal (W2) that a larger/smaller PFS lead to similar results: “Interestingly, reducing the scorer parameters (30× smaller) preserves or even improves downstream performance.”. We agree that this analysis is important and included it therefore. However, as shown in the rebuttal, scorer computation is not the dominant computational cost.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.