acceptodds
Under review as a conference paper at ICLR 2027

Beyond a Single Representation: Steering Shared and Modality-Specific Information in Multimodal SSL

Abstract

Multimodal self-supervised learning aims to learn general representations by ex- ploiting interactions between multiple sources of information. However, most existing approaches learn a single representation with a fixed balance between information shared across modalities and information specific to each modality. We argue that such a fixed operating point might be a limitation for multimodal foundation models, where future downstream tasks are unknown during pretraining and may require different information profiles. Hence, we introduce STEER, a preference-conditioned multimodal learning framework that parameterizes a family of representations within a single model. STEER controls the balance between shared and modality-specific information through a preference vector defined on a simplex, and uses low-rank preference-conditioned adaptations to continuously steer the representations toward different information regimes. This allows the de- sired operating point to be automatically selected after pretraining according to the downstream task, without retraining the representation model. We evaluate STEER in both controlled and real-world multimodal settings. In the former, we show that the learned preference space recovers the expected shared and unique information structure and expands the attainable Pareto region beyond fixed-representation baselines. In the latter, STEER outperforms fixed-representation baselines on sev- eral tasks while selecting distinct operating points from the same pretrained model. Importantly, the selected preferences also provide an interpretable description of the balance between modalities favored by each task, which is particularly valuable in scientific applications such as neuroimaging.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
multimodal self-supervised learning, multimodal representation learning, information decomposition, neuroimaging

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1AKtOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces a preference-conditioned multimodal learning framework that parameterizes a family of representations within a single model.

Critical strengths

The paper idea is simple and it seems effective.

Critical weaknesses

Overall, I find the paper hard to read and it should be considerably revised, providing limited novelty, and with some concerns on the experiments.

Some examples on he writing style are:

The introduction should severely revised. The logical flow is unclear, the motivation is alternated with proposals, the equation is not needed and confuses the reader.

contributions 3 and 4 are not actually contributions, they are just experimental verification of the method.

The problem setup paragraph is actually the method itself.

Notation should be improved: matrices and vectors must be bold, otherwise they are scalars.

On the experimental part, instead, if I am not wrong, the authors in the appendix say they follow CoMM for architectures design on the Multibench benchmark. However, the CoMM baseline in the original paper achieves much higher results than the one shown in this paper. Can the authors discuss on this or rerun the experiments to provide a clearer or fair comparison?

Main serious concern:

The paper severely lacks a discussion on the majority of works that are effectively related to the method, it just cites eight papers (among which only CoMM and FactorCL are tested in the experiments). Without a proper related work section, it is hard to understand whether the paper is truly novel. Indeed, the novelty is unclear and very limited: the formulation resembles the LoRA one and it seems just adapted in the context of multimodal encoders and everything relies in a set of three fixed parameters. I do not think this would be relevant for the scientific community.

Questions for authors

see weaknesses

5 comments
AuthorsOpenReview ↗

1.Presentation and organization

Thank you for your detailed feedback and constructive recommendations. We have carefully considered your comments and revised the manuscript to improve its clarity and organization.

Reviewer: The introduction should be severely revised. The logical flow is unclear, the motivation is alternated with proposals, and the equation is not needed and confuses the reader. Contributions 3 and 4 are not actually contributions; they are just experimental verification of the method. The problem setup paragraph is actually the method itself. Notation should be improved: matrices and vectors must be bold.

We substantially reorganized the Introduction to establish a clearer progression from the motivation and problem formulation to the proposed approach. We also revised the contributions to distinguish the methodological contributions from their experimental validation.

In the Method section, we separated the general problem setup from the preference-conditioned formulation, making the distinction between the learning problem and our proposed solution more explicit.

We improved the notation by consistently using bold symbols for vectors and matrices, numbering and referencing the relevant equations, and adding two algorithms describing the pretraining and downstream evaluation procedures.

With these revisions, we believe the paper is now better structured, clearer, and easier to follow.

AuthorsOpenReview ↗

2.Novelty and Relationship to Prior Work

Reviewer: The paper severely lacks a discussion of the majority of works that are effectively related to the method. Without a proper related work section, it is hard to understand whether the paper is truly novel. The novelty is unclear and very limited: the formulation resembles the LoRA one and seems just adapted in the context of multimodal encoders, with everything relying on a set of three fixed parameters.

Related work. We expanded the Related Work section to include additional contrastive learning approaches, notably SimCLR and Cross+Self, and clarified their relationship to more recent multimodal representation learning methods, including CLIP, GMC, FactorCL, and CoMM.

We also clarified STEER's relationship to PaLoRA, acknowledging that preference-conditioned adaptation and multi-objective optimization build on existing mechanisms.

Novelty and main contribution. While STEER builds on these techniques, we believe its main contribution lies in the learning problem it formulates and addresses.

To the best of our knowledge, prior work has not explicitly addressed the problem of learning a preference-conditioned family of shared and modality-specific representations during multimodal self-supervised pretraining, for downstream tasks that are unknown at training time.

FactorCL focuses on learning shared and modality-specific information within a fixed representation, while PaLoRA provides a mechanism for adapting a model across different objective preferences. STEER connects these directions by making the shared–unique information balance a controllable property of a single pretrained multimodal model, allowing downstream tasks to select their preferred information profiles without retraining the encoders.

We would also welcome suggestions of specific closely related work that we may have overlooked, particularly studies addressing the same setting, as this would help us further improve the completeness of our discussion.

AuthorsOpenReview ↗

3.CoMM Baseline Comparison

Reviewer: In the appendix, the authors say they follow CoMM for architecture design on the MultiBench benchmark. However, the CoMM baseline in the original paper achieves much higher results than the ones shown in this paper. Can the authors discuss this or rerun the experiments to provide a clearer or fair comparison?

When stating that we follow CoMM, we refer primarily to the benchmark settings and encoder architectures, rather than reproducing its complete training and downstream evaluation pipeline.

In particular, our original evaluation fitted logistic regression probes on the validation split, whereas CoMM fits probes on the training split, selects hyperparameters using validation data, and refits on the combined training and validation data.

This difference is particularly relevant for UR-FUNNY, where the validation split contains only 1,034 samples, compared with 8,074 training samples. Consequently, our original probes were fitted on substantially fewer examples.

To investigate the discrepancy directly, we re-evaluated CoMM, STEER, and the other baselines on UR-FUNNY using CoMM's published linear-probing implementation, applied to the same frozen representations across five seeds.

The results are summarized below (accuracy in %, mean ± standard deviation):

MethodOriginal protocolCoMM probing protocol
STEER58.03 ± 1.0660.96 ± 0.72
CoMM54.56 ± 1.6258.47 ± 0.75
GMC55.18 ± 0.5159.02 ± 1.02
CLIP54.63 ± 0.9458.53 ± 0.62
FactorCL56.86 ± 0.8158.19 ± 0.87

The results show that the downstream evaluation protocol substantially influences CoMM's reported performance. Under our original protocol, CoMM achieves 54.56 ± 1.62%, whereas applying its published probing procedure to the same frozen representations increases its accuracy to 58.47 ± 0.75%.

Importantly, STEER's advantage over CoMM is preserved under the same probing protocol, achieving 60.96 ± 0.72% versus 58.47 ± 0.75%. This ordering is also consistent across all five paired seeds.

We observe similar improvements among the other evaluated baselines, suggesting that the lower absolute accuracies in our original comparison are not specific to CoMM but are substantially influenced by the downstream probing procedure.

We emphasize that this experiment changes the downstream probing protocol without retraining the representations. It therefore helps explain the discrepancy, although it does not constitute a complete reproduction of CoMM's original pipeline.

Finally, we clarified these experimental choices in Appendix B.2, including the architectures, training configurations, and downstream evaluation protocols. We also provide our code in an anonymized repository to facilitate reproducibility.

We hope these additional results and clarifications provide a more transparent basis for interpreting our comparison with CoMM.

Reviewer_scXjOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Different downstream tasks have different requirements for shared multimodal information and modality-specific information. Therefore, an ideal multimodal representation should be able to adjust its information composition based on specific task needs. Existing multimodal self-supervised learning methods, such as FactorCL and CoMM, typically learn a fixed representation. Even when these methods can encode both R and U components, they lack the flexibility to adjust their relative proportions or the allocation of representation capacity after pre-training. The authors propose STEER, which uses preference vectors defined on a simplex. By combining cross-modal contrastive learning, intra-modal contrastive learning, and information disentanglement constraints, STEER allows a single model to learn a family of representations, each with a different preference for R versus U information.

Critical strengths

S1. The observation that different downstream tasks may require different amounts of redundant and modality-specific information raises an important problem.

S2. The use of preference-conditioned LoRA provides a parameter-efficient way to adjust representations for different R/U preferences within a single model.

Critical weaknesses

W1. The method mainly combines existing techniques, with limited innovation in how R and U information is learned.

The approach to learning R/U information is largely based on the disentanglement ideas introduced in FactorCL, while preference-conditioned LoRA, Pareto optimization, and preference annealing are largely drawn from PaLoRA. Thus, the method is better viewed as an application of existing multi-objective optimization methods to multimodal SSL, rather than a new learning objective or theoretical mechanism.

Furthermore, existing methods are not necessarily unable to adjust how information is used for downstream tasks. The authors need to explain more clearly why the allocation of representation capacity must be adjusted during representation learning, rather than achieving a similar result through a fixed, information-rich representation combined with task-specific feature selection.

W2. The experiments are not sufficient to fully support the claims.

First, significant performance gains are observed on only two of the three datasets. In addition, the baseline performance on the UR-FUNNY dataset is notably low. For example, FactorCL reported an accuracy of over 60%. The number of comparison methods is also limited. Therefore, the experiments do not provide strong enough evidence to support the authors' conclusions.

Questions for authors

Q1. STEER relies on a predefined preference simplex and selects the optimal λ\lambda through downstream validation. Could the authors explain why the preference vector needs to be manually specified and searched, rather than learned automatically from the downstream task? In particular, is it possible to learn a task-conditioned preference predictor that estimates the required balance between redundant and modality-specific information?

Q2. The preference vector λ\lambda controls both the encoder parameters through LoRA and the allocation of R/U features. How much does each mechanism contribute to the final performance gains? Could the authors provide ablation studies to separate their contributions?

6 comments
AuthorsOpenReview ↗

1.Novelty and Contribution Beyond Existing Methods

Thank you for your insightful feedback. We sincerely appreciate your thoughtful comments and suggestions.

Reviewer: The method mainly combines existing techniques, with limited innovation in how R and U information is learned. The approach to learning R/U information is largely based on the disentanglement ideas introduced in FactorCL, while preference-conditioned LoRA, Pareto optimization, and preference annealing are largely drawn from PaLoRA. Thus, the method is better viewed as an application of existing multi-objective optimization methods to multimodal SSL, rather than a new learning objective or theoretical mechanism.

We agree that the individual components used in STEER are not new in isolation. The shared and modality-specific objectives build on ideas introduced by FactorCL, while preference-conditioned adaptation and multi-objective optimization are closely related to PaLoRA. We therefore do not claim a new R/U estimator, a new contrastive objective, or a new LoRA mechanism.

Our main contribution lies in the learning formulation. As the reviewer highlighted among the strengths, different downstream tasks may require different balances of shared and modality-specific information. STEER builds directly on this observation: rather than committing to a single fixed multimodal representation, a single pretrained model learns a family of shared–unique information profiles, allowing downstream tasks to select their preferred operating point without retraining the encoders.

To the best of our knowledge, this specific problem of learning a preference-conditioned family of shared and modality-specific representations for previously unknown downstream tasks has not been explicitly addressed in multimodal self-supervised learning. While FactorCL focuses on learning shared and modality-specific information within a fixed representation, and PaLoRA provides mechanisms for preference-conditioned multi-objective optimization, STEER makes the information balance itself a controllable property of a task-agnostic pretrained model.

This distinction also has practical consequences. In our additional readout-only ablation, a preference-independent encoder trained at a fixed compromise does not match Full STEER, even when equipped with the same adaptive readout. This supports the value of learning preference-dependent representations during pretraining, rather than relying solely on downstream feature allocation.

STEER also provides an interpretable view of multimodal representation learning. In applications such as medical imaging, it can be useful not only to obtain good predictive performance but also to understand whether a downstream task primarily benefits from information shared across modalities or information specific to one modality. The selected preference provides an indication of this balance.

We therefore believe that STEER's contribution is not the novelty of its individual optimization components, but the formulation and empirical investigation of information-level preference conditioning for task-agnostic multimodal self-supervised learning.

AuthorsOpenReview ↗

2.Experimental Evidence and FactorCL Baseline Comparison

Reviewer: The experiments are not sufficient to fully support the claims. First, significant performance gains are observed on only two of the three datasets. In addition, the baseline performance on the UR-FUNNY dataset is notably low. For example, FactorCL reported an accuracy of over 60%. The number of comparison methods is also limited. Therefore, the experiments do not provide strong enough evidence to support the authors' conclusions.

We agree that the discrepancies with the originally published baseline results deserve further clarification. We therefore investigated the evaluation procedure on UR-FUNNY and found that differences in the downstream probing protocol contribute substantially to the lower accuracies reported in our paper.

In our original experiments, we fitted the linear probe on the validation split, whereas FactorCL uses training data to fit its downstream classifier. To assess the effect of this difference, we repeated the evaluation using a train-fitted probing protocol while keeping the pretrained representations unchanged.

Under this protocol, FactorCL achieves 58.58 ± 0.90%, while STEER reaches 61.51 ± 0.92%, averaged over five seeds. This brings our FactorCL result considerably closer to the 60.5 ± 0.8% reported in the original FactorCL paper (Appendix D, Table 6), which was also obtained over five independent runs.

We also noticed that FactorCL uses dedicated augmentation strategies for UR-FUNNY and MOSEI, including noise injection restricted to silent frames (Table 5). These differences, together with other implementation choices, may contribute to the remaining discrepancy between our reproduced FactorCL result and its published performance.

Importantly, STEER maintains its advantage over FactorCL when both methods are evaluated under the same train-fitted probing protocol. STEER's accuracy is also slightly higher than FactorCL's published self-supervised result, although we acknowledge that the two studies do not use identical experimental settings.

Regarding the observation that performance gains are more pronounced on two of the three datasets, we agree that the improvements are not uniform across all benchmarks. STEER is designed to accommodate different downstream information requirements, and its benefits may therefore vary depending on the dataset and task. We view the results as evidence of the framework's potential rather than a claim of universal superiority.

Finally, regarding the choice of baselines, we focused on CLIP, GMC, CoMM, and FactorCL because they cover complementary approaches to multimodal self-supervised learning, ranging from cross-modal alignment to explicit shared and modality-specific information learning. We considered FactorCL and CoMM particularly relevant comparisons because of their focus on multimodal information structure.

We hope these additional experiments and clarifications provide a more transparent interpretation of our results and strengthen the experimental support for STEER.

AuthorsOpenReview ↗

3.Downstream Preference Selection and Task-Conditioned Learning

Thank you for this valuable suggestion.

Reviewer: STEER relies on a predefined preference simplex and selects the optimal through downstream validation. Could the authors explain why the preference vector needs to be manually specified and searched, rather than learned automatically from the downstream task? In particular, is it possible to learn a task-conditioned preference predictor that estimates the required balance between redundant and modality-specific information?

We chose a predefined preference simplex with downstream validation-based selection for two main reasons.

1. Task-agnostic pretraining. During self-supervised pretraining, future downstream tasks are unknown, so there is no task-specific supervision from which to infer their preferred information profiles. STEER therefore learns a reusable family of representations spanning different balances of shared and modality-specific information. Each downstream task can subsequently select its preferred operating point without retraining the representation model.

2. Simple and flexible downstream selection. Once the pretrained model is frozen, selecting the operating point becomes a lightweight model-selection problem over the preference grid. We evaluate candidate preferences using downstream probes and select the preferred operating point based on validation performance. The grid resolution can be adjusted to explore the preference space more or less finely, while our noise-aware consensus procedure (Appendix A.4) helps stabilize the selection across seeds.

We agree that learning a task-conditioned preference predictor is a promising extension, particularly in large-scale multi-task settings where such a predictor could be trained across multiple tasks. It could also be valuable in few-shot and zero-shot scenarios, where validation-based selection becomes less reliable or is unavailable. Such a predictor could potentially infer the appropriate information balance from a task description or limited task context.

We therefore consider validation-based selection a simple and transparent choice for the setting studied here. Improving downstream operating-point selection was already identified as a future direction in the Discussion and Future Directions section of our original submission.

Reviewer_2AJ7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes STEER, a preference-conditioned framework for multimodal self-supervised learning. It uses a preference vector to balance objectives associated with shared and modality-specific information, while injecting the same preference into low-rank adapters within the encoders.

Critical strengths

The motivation is interesting, particularly the emphasis on modeling and preserving information unique to each modality.

Critical weaknesses
  1. It remains unclear why STEER can control modality-specific information. The paper explains how the preference vector changes the loss weights and effective encoder parameters. However, changing model parameters as a function of a preference does not, by itself, establish that the intended information component is being controlled. The connection between these operations and the claimed information-level behavior requires further explanation. The paper needs a clearer operational definition of modality-specific information, an explanation of the assumptions underlying this interpretation, and a stronger account of why increasing a corresponding preference coordinate should enhance that information.

  2. The experiments do not sufficiently separate encoder conditioning from preference-dependent readout. Also, the existing placement ablation compares conditioning the encoder, the projection heads, or both. However, it does not directly answer the critical question: Could similar behavior be obtained with a preference-independent encoder and only a preference-dependent readout?

  3. The method section needs more context, design rationale, and explanation of the complete pipeline.

  4. The coverage and discussion of related work are insufficient to clearly establish the contribution.

Questions for authors

see weakness

6 comments
AuthorsOpenReview ↗

1. Preference semantics and PID interpretation

Thank you for your constructive feedback.

Reviewer: It remains unclear why STEER can control modality-specific information. The paper needs a clearer operational definition of modality-specific information, an explanation of the assumptions underlying this interpretation, and a stronger account of why increasing a corresponding preference coordinate should enhance that information.

We agree that changing the loss weights or conditioning the encoder is not, by itself, sufficient to establish that the corresponding shared or modality-specific information is actually being controlled.

We would first like to clarify that STEER does not interpret its training objectives as exact PID estimators. Our formulation is motivated by the redundancy–uniqueness view of PID (Liang et al., 2023a), but uses tractable surrogate objectives to encourage different shared and modality-specific information profiles.

We consider modality-specific information to be information carried by one modality that is not shared with the other. Our approach assumes that augmentations preserve meaningful shared and modality-specific factors while perturbing nuisance variations. Under these assumptions, cross-modal InfoNCE encourages information common to both modalities, while within-modality NT-Xent preserves stable features that may include both shared and modality-specific information. The CLUB separation term further discourages the modality-specific branch from duplicating information captured by the shared branch.

Increasing a modality-specific preference coordinate gives greater weight to its corresponding preservation and separation objective, while preference-conditioned LoRA allows the encoder to adapt accordingly. Together, these mechanisms encourage the intended information balance, although exact separation and monotonicity with respect to the preference coordinates are not theoretically guaranteed.

This choice of surrogate objectives is also practical. Direct PID estimation remains challenging for high-dimensional continuous representations. Classical formulations can involve optimization over probability distributions (Bertschinger et al., 2014), while high-dimensional estimation can suffer from finite-sample bias (Venkatesh et al., 2023). Continuous variables introduce additional difficulties (Ehrlich et al., 2024). For this reason, we optimize scalable surrogate objectives during training and empirically examine whether the resulting representations exhibit the intended redundancy–uniqueness structure.

Importantly, this analysis was already included in our original submission through the controlled Trifeature experiment. Trifeature provides a setting where the underlying factors are known: Shape is shared across modalities, while Deformation and Texture are specific to modalities 1 and 2, respectively.

We complement downstream probing with a BROJA-style bivariate PID decomposition (Bertschinger et al., 2014; Makkeh et al., 2018). In this controlled setting, the synthetic factors allow us to analyze discrete information sources and prediction targets rather than estimating PID directly from high-dimensional continuous neural representations. A shuffled-target estimate is subtracted to reduce the finite-sample estimation floor.

Across the complete 15-point preference simplex, the estimated information components exhibit strong correlations with their corresponding preference coordinates:

  • Redundancy: ρ(λ_R, R) = 0.968
  • Modality-1 uniqueness: ρ(λ_U1, U1) = 0.968
  • Modality-2 uniqueness: ρ(λ_U2, U2) = 0.952

The vertex-level results in Table 14 of Appendix D further illustrate this behavior. At the shared-information vertex (1, 0, 0), redundant shape information reaches 3.886 ± 0.012 bits. At the modality-1-specific vertex (0, 1, 0), unique deformation information reaches 1.111 ± 0.067 bits, while the modality-2-specific vertex (0, 0, 1) yields 1.317 ± 0.078 bits of texture-specific information.

These findings show that the model's predictions exhibit the intended changes in information profiles as the preference varies. They support the interpretation of the preference vector without implying that the surrogate objectives recover exact PID quantities or that the learned representation blocks are perfectly disentangled.

We agree that this connection, although already investigated in Appendix D of the original submission, was not sufficiently visible in the main text. We have therefore clarified the distinction between surrogate training objectives and PID quantities in the Method section, and explicitly directed readers to the existing empirical analysis in Appendix D.

The references supporting this response are provided in a separate follow-up comment below.

AuthorsOpenReview ↗

Refrences follow up

References

  1. Liang et al. (2023). Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework. NeurIPS 36, 27351–27393.
  2. Bertschinger et al. (2014). Quantifying Unique Information. Entropy, 16(4), 2161–2183. DOI: 10.3390/e16042161.
  3. Venkatesh et al. (2023). Gaussian Partial Information Decomposition: Bias Correction and Application to High-Dimensional Data. NeurIPS 36.
  4. Ehrlich et al. (2024). Partial Information Decomposition for Continuous Variables Based on Shared Exclusions: Analytical Formulation and Estimation. Physical Review E, 110(1), 014115. DOI: 10.1103/PhysRevE.110.014115.
  5. Makkeh et al. (2018). BROJA-2PID: A Robust Estimator for Bivariate Partial Information Decomposition. Entropy, 20(4). DOI: 10.3390/e20040271.
AuthorsOpenReview ↗

2. Preference-conditioned encoder versus preference-dependent readout

We thank the reviewer for this important question.

We agree that it is necessary to distinguish the contribution of encoder conditioning from that of the preference-dependent downstream readout.

To directly address the question, we conducted an additional experiment using a preference-independent encoder trained with a fixed compromise between the shared and modality-specific objectives. We then applied the same preference-dependent readout used by STEER to the resulting representations. In this configuration, the encoder does not change with the preference, and only the downstream allocation of shared and modality-specific information depends on the selected operating point.

We compare this readout-only baseline against Full STEER, which conditions both the encoder and downstream readout on the preference vector.

The results are summarized below (accuracy in %, mean ± standard deviation):

MethodCMU-MOSEIShapeDeformationTextureTrifeature Avg.
Readout-only69.27 ± 0.8386.23 ± 1.4422.28 ± 1.0820.14 ± 2.1242.88
Full STEER73.14 ± 0.9499.98 ± 0.0529.16 ± 2.6637.46 ± 3.9655.53

The results show that a preference-independent encoder with an adaptive readout can capture useful task-specific information, but does not achieve the same performance as Full STEER in the settings tested.

On CMU-MOSEI, Full STEER achieves 73.14 ± 0.94%, compared with 69.27 ± 0.83% for the readout-only baseline.

On Trifeature, Full STEER performs better across all three tasks. The difference is especially notable for Texture (37.46% versus 20.14%) and Shape (99.98% versus 86.23%). Deformation also improves from 22.28% to 29.16%.

Overall, the average Trifeature accuracy increases from 42.88% to 55.53%.

These findings suggest that preference-dependent downstream allocation alone does not fully explain STEER's performance. Conditioning the encoder allows the feature extraction process itself to adapt to the selected information profile, rather than relying exclusively on different allocations of a fixed representation.

We further investigate this distinction in Appendix C.3, where we decouple the preference supplied to the encoder from the preference used by the readout. By holding the readout preference fixed and varying only the encoder preference, we observe variations in downstream accuracy of 2.23 points on CMU-MOSEI and 4.28 points on Trifeature. These values provide complementary evidence that the encoder contributes preference-dependent behavior independently of the readout.

We also evaluated the complementary configuration in which we retain the preference-conditioned encoder but remove the adaptive downstream allocation. In this additional experiment, STEER was trained from scratch in a separate run and the complete 192-dimensional R/U representation was used without preference-dependent dimensional allocation. On CMU-MOSEI, this configuration achieves 73.45 ± 0.38%, compared with 73.94 ± 0.74% for Full STEER in the same ablation run.

These results indicate that substantial task-relevant information is already present in the conditioned representation.

Taken together, these experiments clarify the complementary roles of the two mechanisms. The preference-conditioned encoder adapts the learned features toward different shared and modality-specific information profiles, while the preference-dependent readout allocates the resulting information under a fixed downstream dimensional budget.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers

We sincerely thank Reviewers 2AJ7, scXj, and 1AKt for their careful evaluations and constructive feedback. We appreciate the time and effort they devoted to reviewing our work.

We are encouraged that the reviewers recognized several positive aspects of STEER:

  1. Motivation and relevance: Reviewer 2AJ7 highlighted the interesting motivation behind STEER, particularly the importance of preserving modality-specific information in multimodal representation learning. Reviewer scXj also recognized the relevance of learning task-dependent balances between shared and modality-specific information, supporting the motivation for considering different information requirements across downstream tasks.

  2. Methodological simplicity and efficiency: Reviewer 1AKt acknowledged the simplicity and apparent effectiveness of the proposed approach. Reviewer scXj further highlighted the parameter efficiency of preference-conditioned LoRA, which allows different operating points to be represented within a single model rather than requiring independently trained encoders.

  3. Flexibility of multimodal representations: The observations from Reviewers 2AJ7 and scXj underscore the relevance of moving beyond representations that emphasize a single balance of shared and modality-specific information. This is the central question STEER aims to address: learning a family of information profiles during task-agnostic pretraining so that previously unknown downstream tasks can select the representation most suited to their requirements.

Building on these observations, we have carefully addressed the reviewers' concerns through methodological clarifications, revisions to the manuscript, and additional experiments, which we summarize below:

  1. Presentation and methodological clarity (1AKt, 2AJ7, scXj). We reorganized the Introduction to better separate motivation, problem formulation, and contributions. We also improved the notation, clarified the Method section, expanded the experimental setup, and added algorithms describing the pretraining and downstream evaluation procedures.

  2. Novelty and relationship to prior work (1AKt, 2AJ7, scXj). We expanded the Related Work section and clarified STEER's relationship to FactorCL, CoMM, and PaLoRA. While STEER builds on existing contrastive objectives and preference-conditioned optimization mechanisms, to the best of our knowledge, prior work has not explicitly addressed the problem of learning a preference-conditioned family of shared and modality-specific representations during multimodal self-supervised pretraining, for downstream tasks that are unknown at training time.

  3. Preference semantics and PID interpretation (2AJ7, scXj). We clarified that STEER uses PID-inspired surrogate objectives rather than direct PID estimators, and explained more explicitly how the preference vector encourages shared and modality-specific information profiles. We also strengthened the connection to the existing empirical PID analysis in Appendix D, which was already included in our original submission.

  4. Encoder conditioning versus downstream readout (2AJ7, scXj). We introduced a direct readout-only ablation showing that Full STEER outperforms a preference-independent encoder with the same adaptive readout on CMU-MOSEI and Trifeature. Together with the additional analysis in Appendix C.3, these results support the contribution of encoder conditioning beyond downstream feature allocation.

  5. Experimental setup and baseline comparisons (1AKt, scXj). We clarified the experimental protocols in Appendix B.2 and conducted additional evaluations explaining the discrepancies with published CoMM and FactorCL results. STEER maintains its advantage under consistent probing protocols, strengthening the reliability of our comparisons.

We believe these revisions and additional analyses substantially improve the clarity, experimental support, and positioning of STEER. We thank the reviewers again for their valuable feedback and provide further details in our individual responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.