- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper introduces a preference-conditioned multimodal learning framework that parameterizes a family of representations within a single model.
Critical strengths
The paper idea is simple and it seems effective.
Critical weaknesses
Overall, I find the paper hard to read and it should be considerably revised, providing limited novelty, and with some concerns on the experiments.
Some examples on he writing style are:
The introduction should severely revised. The logical flow is unclear, the motivation is alternated with proposals, the equation is not needed and confuses the reader.
contributions 3 and 4 are not actually contributions, they are just experimental verification of the method.
The problem setup paragraph is actually the method itself.
Notation should be improved: matrices and vectors must be bold, otherwise they are scalars.
On the experimental part, instead, if I am not wrong, the authors in the appendix say they follow CoMM for architectures design on the Multibench benchmark. However, the CoMM baseline in the original paper achieves much higher results than the one shown in this paper. Can the authors discuss on this or rerun the experiments to provide a clearer or fair comparison?
Main serious concern:
The paper severely lacks a discussion on the majority of works that are effectively related to the method, it just cites eight papers (among which only CoMM and FactorCL are tested in the experiments). Without a proper related work section, it is hard to understand whether the paper is truly novel. Indeed, the novelty is unclear and very limited: the formulation resembles the LoRA one and it seems just adapted in the context of multimodal encoders and everything relies in a set of three fixed parameters. I do not think this would be relevant for the scientific community.
Questions for authors
see weaknesses
5 comments
2.Novelty and Relationship to Prior Work
Reviewer: The paper severely lacks a discussion of the majority of works that are effectively related to the method. Without a proper related work section, it is hard to understand whether the paper is truly novel. The novelty is unclear and very limited: the formulation resembles the LoRA one and seems just adapted in the context of multimodal encoders, with everything relying on a set of three fixed parameters.
Related work. We expanded the Related Work section to include additional contrastive learning approaches, notably SimCLR and Cross+Self, and clarified their relationship to more recent multimodal representation learning methods, including CLIP, GMC, FactorCL, and CoMM.
We also clarified STEER's relationship to PaLoRA, acknowledging that preference-conditioned adaptation and multi-objective optimization build on existing mechanisms.
Novelty and main contribution. While STEER builds on these techniques, we believe its main contribution lies in the learning problem it formulates and addresses.
To the best of our knowledge, prior work has not explicitly addressed the problem of learning a preference-conditioned family of shared and modality-specific representations during multimodal self-supervised pretraining, for downstream tasks that are unknown at training time.
FactorCL focuses on learning shared and modality-specific information within a fixed representation, while PaLoRA provides a mechanism for adapting a model across different objective preferences. STEER connects these directions by making the shared–unique information balance a controllable property of a single pretrained multimodal model, allowing downstream tasks to select their preferred information profiles without retraining the encoders.
We would also welcome suggestions of specific closely related work that we may have overlooked, particularly studies addressing the same setting, as this would help us further improve the completeness of our discussion.
3.CoMM Baseline Comparison
Reviewer: In the appendix, the authors say they follow CoMM for architecture design on the MultiBench benchmark. However, the CoMM baseline in the original paper achieves much higher results than the ones shown in this paper. Can the authors discuss this or rerun the experiments to provide a clearer or fair comparison?
When stating that we follow CoMM, we refer primarily to the benchmark settings and encoder architectures, rather than reproducing its complete training and downstream evaluation pipeline.
In particular, our original evaluation fitted logistic regression probes on the validation split, whereas CoMM fits probes on the training split, selects hyperparameters using validation data, and refits on the combined training and validation data.
This difference is particularly relevant for UR-FUNNY, where the validation split contains only 1,034 samples, compared with 8,074 training samples. Consequently, our original probes were fitted on substantially fewer examples.
To investigate the discrepancy directly, we re-evaluated CoMM, STEER, and the other baselines on UR-FUNNY using CoMM's published linear-probing implementation, applied to the same frozen representations across five seeds.
The results are summarized below (accuracy in %, mean ± standard deviation):
| Method | Original protocol | CoMM probing protocol |
|---|---|---|
| STEER | 58.03 ± 1.06 | 60.96 ± 0.72 |
| CoMM | 54.56 ± 1.62 | 58.47 ± 0.75 |
| GMC | 55.18 ± 0.51 | 59.02 ± 1.02 |
| CLIP | 54.63 ± 0.94 | 58.53 ± 0.62 |
| FactorCL | 56.86 ± 0.81 | 58.19 ± 0.87 |
The results show that the downstream evaluation protocol substantially influences CoMM's reported performance. Under our original protocol, CoMM achieves 54.56 ± 1.62%, whereas applying its published probing procedure to the same frozen representations increases its accuracy to 58.47 ± 0.75%.
Importantly, STEER's advantage over CoMM is preserved under the same probing protocol, achieving 60.96 ± 0.72% versus 58.47 ± 0.75%. This ordering is also consistent across all five paired seeds.
We observe similar improvements among the other evaluated baselines, suggesting that the lower absolute accuracies in our original comparison are not specific to CoMM but are substantially influenced by the downstream probing procedure.
We emphasize that this experiment changes the downstream probing protocol without retraining the representations. It therefore helps explain the discrepancy, although it does not constitute a complete reproduction of CoMM's original pipeline.
Finally, we clarified these experimental choices in Appendix B.2, including the architectures, training configurations, and downstream evaluation protocols. We also provide our code in an anonymized repository to facilitate reproducibility.
We hope these additional results and clarifications provide a more transparent basis for interpreting our comparison with CoMM.
1.Presentation and organization
Thank you for your detailed feedback and constructive recommendations. We have carefully considered your comments and revised the manuscript to improve its clarity and organization.
We substantially reorganized the Introduction to establish a clearer progression from the motivation and problem formulation to the proposed approach. We also revised the contributions to distinguish the methodological contributions from their experimental validation.
In the Method section, we separated the general problem setup from the preference-conditioned formulation, making the distinction between the learning problem and our proposed solution more explicit.
We improved the notation by consistently using bold symbols for vectors and matrices, numbering and referencing the relevant equations, and adding two algorithms describing the pretraining and downstream evaluation procedures.
With these revisions, we believe the paper is now better structured, clearer, and easier to follow.