acceptodds
Under review as a conference paper at ICLR 2027

Acquisition Policies Change the Bayes Rule: Policy-Conditioned Posterior Correction for Multimodal Learning

Abstract

Multimodal models usually predict from available modalities as if an observed subset had fixed predictive meaning. We show that this can fail after an acquisition-policy shift. The same observed content and mask can imply a different class posterior. We formulate policy-conditioned multimodal learning and define the observed acquisition mass, the class-conditional probability that a policy produces an observed record after marginalizing unobserved content. Its ratio transforms a source posterior into a policy-specific posterior, while a posterior-weighted Jensen-Shannon divergence quantifies irreducible log-loss without policy information. Our Policy-Conditioned Observed-Mass Model (PCOM) combines a frozen source predictor, source-trained mass estimation, and policy inference from unlabeled context-mask logs. Record-level mixtures are corrected in mass space before normalization. Across CMU-MOSEI, MELD, MM-IMDb, and UPMC Food-101, PCOM improves endpoint log loss and retains positive mean gains at held-out mixture proportions. Ablations verify the roles of candidate-class conditioning and policy state, while stress tests show that state fidelity alone does not guarantee utility. Thus, policy variation should not be removed through invariance when it changes an observation's predictive meaning.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
multimodal learning, bayes rule

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nbgaOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies multimodal prediction under shifting acquisition policies, where the same observed modalities may correspond to different class posteriors depending on how they were selected. The authors propose PCOM, which uses class-conditional acquisition mass to correct predictions from a frozen source model.

Critical strengths
  • The paper studies an interesting problem.
  • The authors provide a detailed theoretical formulation and extensive experiments across four datasets.
Critical weaknesses
  • The experimental setting does not fully support the paper's motivation of generalizing to unseen acquisition policies. Most of the positive results come from policy endpoints that are already represented during source training, or mixtures between known endpoints. For the mixture experiments, the endpoint identity is explicitly provided, and only the mixture proportion needs to be inferred. This seems substantially easier than adapting to a genuinely unseen acquisition policy. More importantly, the continuous-path experiments show negative average gains on CMU-MOSEI and MELD, while the hidden-content experiments also fail to improve over the source anchor. I understand that the authors acknowledge these limitations, but they significantly weaken the practical motivation of the method.

  • I am not convinced that the performance improvements can be attributed to the proposed posterior correction. PCOM receives context-mask logs containing modality statistics and source-supervised triage features, while the main baselines do not receive the same information. There is also a mismatch between the theoretical formulation and its implementation. Equation (2) uses a source posterior conditioned on the retained context, but the frozen source predictor explicitly excludes this context. This raises questions about whether the improvement comes from correcting the acquisition shift or simply introducing additional predictive information through the correction module.

  • The necessity of the proposed learned mass estimator remains unclear. In the main experiments, the acquisition policy is generated as a function of context, label, and policy state, which makes the corresponding observed mass analytically available. This is particularly concerning because Appendix Table 18 shows that analytic mass performs substantially better than the learned estimator on CMU-MOSEI.

  • Some experimental comparisons and ablations are not sufficiently convincing. For example, removing class conditioning or policy-state information in Table 18 produces zero correction by construction. These results mostly confirm the algebraic behavior of the proposed formulation rather than demonstrate the benefit of the learned architecture. The mixture experiments also lack comparisons against the adaptation baselines used in the endpoint experiments. In addition, the posterior-averaging control achieves better NLL on MELD and MM-IMDb, although it uses the true mixture proportion.

  • The presentation is unnecessarily complicated and could be substantially improved. The paper reads more like a technical handbook than a focused research paper in several places. For example, Figure 2 introduces a large number of components and intermediate operations, while the surrounding text spends considerable space explaining the implementation workflow. Similar descriptions appear throughout the appendices, with many tables and diagnostic quantities that do not always help establish the main contribution. I found it difficult to identify which components are essential and which are implementation choices.

Questions for authors
  • Could the authors provide results on genuinely unseen acquisition policies, where neither the endpoint identity nor the policy family is available during source training?
  • Could the authors include simpler correction baselines and more controlled ablations? In particular, it would be useful to compare against a direct context-conditioned correction model and evaluate whether the individual PCOM components improve performance beyond these simpler alternatives.
4 comments

We are grateful for your detailed comments. We respond to each question below.

Q1. Unseen acquisition policies

A1. Thank you for highlighting this distinction. The submitted mixtures test new proportions along specified directions. Following your suggestion, we evaluated mechanisms excluded from fitting and selection with PCOM frozen. Prediction used no endpoint identity, acquisition probabilities, or new branch.

Table R2 in our response to aMNR reports average gains of 0.01230.0123 on CMU-MOSEI and 0.01520.0152 on MELD across the four mechanisms, with equal weights. Hidden content gains remain −0.0040-0.0040 and −0.0080-0.0080. Both intervals against the direct predictor with the same information include zero (Table R3). These results support transfer to some tested mechanisms, without establishing a general advantage on excluded policies.

We also separated the effects of source conditioning, mass estimation, and state inference on continuous paths (Table R14). Providing the state changes the original CMU-MOSEI gain from −0.0391 to −0.0388, while MELD remains at −0.0469. Analytic mass improves the comparison with the anchor and inferred state fixed. This identifies a limitation of the learned mass estimate.

Table R14. NLL gains on continuous paths over each configuration's corresponding source anchor.

AnchorMassStateCMU-MOSEIMELD
OriginalLearnedInferred−0.0391−0.0469
OriginalLearnedSupplied−0.0388−0.0469
OriginalAnalyticInferred0.01100.01100.01600.0160
OriginalAnalyticSupplied0.01140.01140.01620.0162
With contextLearnedInferred−0.0020-0.0020−0.0060-0.0060
With contextLearnedSupplied−0.0018-0.0018−0.0058-0.0058
With contextAnalyticInferred0.02000.02000.02500.0250
With contextAnalyticSupplied0.02030.02030.02520.0252

Table R15. Learned NLL minus analytic NLL with the anchor including context and inferred state fixed. Paired 95% CI.

CMU-MOSEIMELD
0.0220  [0.0100,  0.0340]0.0220\;[0.0100,\;0.0340]0.0310  [0.0180,  0.0440]0.0310\;[0.0180,\;0.0440]

To examine correction when acquisition reads hidden content, we added a finite-state diagnostic with exact references (Table R16). Here YY is balanced, X,HX,H are independent conditional on YY, and P(X=y∣Y=y)=0.7P(X=y\mid Y=y)=0.7, P(H=y∣Y=y)=0.8P(H=y\mid Y=y)=0.8. We always observe XX and set C=XC=X. The source hides HH with probability 0.5. The target hides it with probability 0.9 when H=1H=1 and 0.1 when H=0H=0. The rule uses content that can become hidden without reading the label.

Table R16. Target NLL within the diagnostic's specified policy family.

Source posteriorMassState informationTarget NLL
ExactNo correctionNot used0.52903345
ExactExactSupplied0.47826179
ExactLearnedSupplied0.48900.4890
FittedExactSupplied0.48200.4820
FittedLearnedSupplied0.49300.4930
FittedLearnedInferred0.49700.4970
FittedNo correctionNot used0.53500.5350

The population gain is 0.05077166 nats. Fully estimated correction gains 0.0380  [0.0280,  0.0480]0.0380\;[0.0280,\;0.0480] over the fitted anchor, with paired 95% CI (Table R16). This diagnostic measures estimation error within a specified family. We will keep it distinct from transfer to excluded mechanisms and the unresolved multimodal failures.

Q2. Correction and ablations

A2. Thank you for this suggestion. Equation 2 conditions the source posterior on retained context, while the submitted anchor omits it. We tested PCOM-C with this conditioning and report it separately. With V=(Xr,R)V=(X_r,R), starting from ηs(y∣V)\eta_s(y\mid V) requires the additional factor ηs(y∣V,C)/ηs(y∣V)\eta_s(y\mid V,C)/\eta_s(y\mid V). The current context construction does not justify omitting this factor.

Adding context reduces anchor NLL by 0.27700.2770 on CMU-MOSEI and 0.22880.2288 on MELD (Table R4 in our response to aMNR). With the anchor, logs, environments, supervision, masks, and tuning budget matched, PCOM-C improves over the direct residual by 0.0140  [0.0050,  0.0230]0.0140\;[0.0050,\;0.0230] and 0.0300  [0.0150,  0.0450]0.0300\;[0.0150,\;0.0450]. These paired intervals support an endpoint gain with the same information (Table R4). The direct residual learns an update for each class.

Q2. Correction and ablations, continued

A2. Analytic mass implements the correction exactly for the main generator with a single label. Its submitted CMU-MOSEI gain is 0.5711, versus 0.3503 with learned PCOM (submission Table 18). Table R17 holds the anchor with context, records, and inferred state weights fixed.

Table R17. Endpoint NLL with learned and analytic mass.

DatasetLearned NLLAnalytic NLLLearned minus analytic, paired 95% CI
CMU-MOSEI0.21800.12000.0980  [0.0720,  0.1240]0.0980\;[0.0720,\;0.1240]
MELD1.03000.90000.1300  [0.0950,  0.1650]0.1300\;[0.0950,\;0.1650]

The gap in Table R17 measures error in estimating the correction target. We will state when analytic evaluation is available. For MM-IMDb, correcting one bit can require averaging over other unknown labels, so its reference depends on that approximation. Exact target policy evaluation was not used for excluded mechanisms.

The zero-gain rows in submission Table 18 check the update's algebra. We will label this role explicitly. The direct residual gains 0.15300.1530 and 0.30500.3050 over the anchor with context (Table R4). Hard selection and marginalization give 0.3506 and 0.3503 on CMU-MOSEI, and 0.0082 and 0.0102 on MM-IMDb (submission Table 18). Removing overlap regularization gives 0.4294 on CMU-MOSEI and also changes policy generation. We will separate measured benefits from structural checks.

For mixtures, we compared mass mixing and posterior averaging with the same true or inferred coefficients, anchor, mass models, and records (Table R18). All intervals include zero. The difference remains inconclusive and does not establish equivalence.

Table R18. Posterior averaging NLL minus mass mixing NLL with matched coefficients. Positive values favor mass mixing.

DatasetTrue coefficientInferred coefficient
CMU-MOSEI0.0047  [−0.0008,  0.0102]0.0047\;[-0.0008,\;0.0102]0.0041  [−0.0007,  0.0089]0.0041\;[-0.0007,\;0.0089]
MELD−0.0006  [−0.0018,  0.0006]-0.0006\;[-0.0018,\;0.0006]−0.0005  [−0.0019,  0.0009]-0.0005\;[-0.0019,\;0.0009]
MM-IMDb−0.0001  [−0.0005,  0.0003]-0.0001\;[-0.0005,\;0.0003]−0.0001  [−0.0005,  0.0003]-0.0001\;[-0.0005,\;0.0003]
UPMC0.0011  [−0.0002,  0.0024]0.0011\;[-0.0002,\;0.0024]0.0009  [−0.0004,  0.0022]0.0009\;[-0.0004,\;0.0022]

Masses combine before normalization for mixtures across records. Observing a record changes the relative weights of the component mechanisms, so averaging endpoint posteriors with the original coefficient generally gives a different posterior.

We also added ERM and direct residual comparisons beyond endpoints with the same anchor and information (Tables R19 and R20). PCOM-C improves over the direct residual on several mixture benchmarks. On MELD paths, the interval favors the direct residual. The other adapted methods were tested only at endpoints.

Table R19. Direct residual NLL minus PCOM-C NLL with paired 95% CI. Positive values favor PCOM-C.

DatasetMixturesContinuous paths
CMU-MOSEI0.0090  [0.0030,  0.0150]0.0090\;[0.0030,\;0.0150]−0.0050  [−0.0110,  0.0010]-0.0050\;[-0.0110,\;0.0010]
MELD0.0100  [0.0030,  0.0170]0.0100\;[0.0030,\;0.0170]−0.0110  [−0.0190,  −0.0030]-0.0110\;[-0.0190,\;-0.0030]
MM-IMDb0.0005  [−0.0002,  0.0012]0.0005\;[-0.0002,\;0.0012]0.0002  [−0.0004,  0.0008]0.0002\;[-0.0004,\;0.0008]
UPMC0.0080  [0.0030,  0.0130]0.0080\;[0.0030,\;0.0130]0.0100  [0.0040,  0.0160]0.0100\;[0.0040,\;0.0160]

Table R20. ERM NLL minus PCOM-C NLL. Positive values favor PCOM-C.

DatasetERM minus PCOM-C on mixturesERM minus PCOM-C on paths
CMU-MOSEI0.02200.0220−0.0010-0.0010
MELD0.03500.0350−0.0050-0.0050

Q3. Presentation

A3. Thank you for this helpful feedback. We will organize the text and Figure 2 around the source posterior, observed mass ratio, and policy information from logs. We will explain how each quantity is estimated and state its assumptions. We will also distinguish uncertainty about the endpoint from mixtures across records. Implementation details and supporting diagnostics will move out of the main argument.

We sincerely appreciate the time and care you have put into reviewing our paper. We would welcome your feedback on our response, especially on whether the explanations and additional evidence address your concerns. If any points remain unclear, we would be happy to discuss them further. Thank you again for helping us strengthen our work.

I appreciate the authors' additional experiments, but I do not think they fully address my main concerns. The learned mass estimator still underperforms the analytic alternative, and the new experiments provide limited evidence of generalization to unseen acquisition policies. I also believe the presentation issues require substantial revision and have not yet been addressed in the manuscript, so I will maintain my original score.

Reviewer_4wH4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper says: if the acquisition policy (which modalities get recorded) depends on the class, the same observed record has a different posterior under a new policy. The fix is to multiply the source posterior by the ratio of "observed acquisition masses" and renormalize (Prop. 1). They also show a single policy-agnostic predictor pays a Jensen-Shannon regret (Eq. 3). PCOM is a frozen source predictor plus three learned parts: a policy generator, a state encoder that infers the policy from unlabeled context-mask logs, and a mass estimator. All are trained on source data only. Tested on MOSEI, MELD, MM-IMDb and UPMC with fixed features, 5 seeds. Big NLL gains at "learned endpoint" policies (MOSEI 0.64 to 0.23 vs DAMS). Gains also at held-out mixture proportions. Gains are negative on continuous paths and hidden-content policies, and the authors say so.

Critical strengths
  1. The main point the paper is trying to make is right and validated. Class-dependent acquisition changes the Bayes rule. The toy study (Table 1) shows it clearly.
  2. The paper has a good audit of its own failures. It reports all 1,840 cells (160 endpoint, 800 continuous, 800 mixture, 80 hidden-content) with positive-cell counts per regime (Table 15). 438 are negative. Fig. 4 shows inferred state can match the known state even where both are worse than the anchor (L429-432). So "state fidelity is not utility" is shown, not just stated. This would be useful for anyone who builds on this.
  3. DAMS is included at endpoints, and there are paired CIs over seeds.
Critical weaknesses
  1. "Unseen" policies are not unseen. The paper says at L290-291 that neither study holds out endpoint identities or policy families. The target policies are states of the authors' own generator, and PCOM's state encoder, mass models and scalar estimators are trained on the same generator. Endpoint state inference is a nine-way classification of generator codes, and the gap to the state-known predictor is under 0.001 NLL. That shows the encoder recovers codes it was trained on. It says nothing about a policy the method has not met.
  2. The main results are synthetic. Masks are generated from the label and context by a network trained to hurt the frozen anchor (L204, L283). MOSEI AUROC rises from 0.70 to 0.96 and NLL from 0.64 to 0.23. I read that as decoding a label-to-mask dependence the authors built, not as reading multimodal content. There is no real acquisition shift. The source requires complete labeled records and a uniform source policy over all nonempty masks (L198). The features are cheap fixed vectors with small MLPs (0.2M to 0.6M parameters), and no modern vision-language model is tested. It is unknown whether the result survives another model family or scale.
  3. The novel part is not exercised. Proposition 1 is Bayes' rule with a likelihood ratio, which the paper itself cites Rockenschaub for (L040). Eq. 3 is the standard KL decomposition. The one new object is the observed mass, which averages the policy over unobserved content (Eq. 1). But the generator depends only on the label and the observed context (Eq. 5), so the mass is available in closed form. Where hidden content actually matters, PCOM is worse than no correction. What remains is: know the family of label-dependent mask policies, guess which one is active, apply the ratio.
  4. Minor. The correction costs 0.70 to 24.1 ms per record against 0.014 to 0.12 ms for the anchor, 50 to 200 times more, and the paper does not discuss whether the gain justifies it.
Questions for authors
  1. What happens when the source masks are not uniform, the source records are incomplete, or the content distribution also shifts?
  2. Can you hold out whole policy families? Train on some generator states and test on a different generator architecture, hand-written rules such as "drop audio when sentiment is strong," and policies that do not depend on the label. What happens when the true policy is outside the family?
  3. Can you add ERM given the same context vector C, baselines trained on the same environments, label-shift EM, a few target labels, and Stokes / Rockenschaub / MIRRAMS? How does DAMS do when mask marginals change?
4 comments

We appreciate your valuable feedback. Our responses follow below.

Q1. Source assumptions

A1. Thank you for this question. Nonuniform masks require source support wherever the target joint distribution is positive and the correct source mass. Each anchor was trained under its source policy. Table R5 tests the denominator.

Table R5. CMU-MOSEI NLL gain over each source anchor with 95% CI.

Source policy and massGain
Nonuniform, γ=0.20\gamma=0.200.1260  [0.1010,  0.1510]0.1260\;[0.1010,\;0.1510]
Uses class, correct mass0.1180  [0.0900,  0.1460]0.1180\;[0.0900,\;0.1460]
Uses class, estimated mass0.1010  [0.0750,  0.1270]0.1010\;[0.0750,\;0.1270]
Uses class, mass incorrectly ignores class0.0410  [0.0140,  0.0680]0.0410\;[0.0140,\;0.0680]

The first row uses πs(r)=(1−γ)1[r=r0]+γ/A\pi_s(r)=(1-\gamma)\mathbf1[r=r_0]+\gamma/A, where r0r_0 is a fixed mask and AA counts available masks. All actions have positive probability.

For incomplete views, complete audit records supply donors (Table R6). Ordinary training views remain incomplete. Donors must be representative or have known positive inclusion probabilities. Cross-fitting controls record reuse. Unknown MNAR missingness still needs additional assumptions.

Table R6. CMU-MOSEI NLL gain over the audit study's anchor. Intervals are 95% CI where available.

Audit settingGain
100%100\% complete audit0.14200.1420
50%50\% complete audit0.12800.1280
25%25\% complete audit0.1100  [0.0820,  0.1380]0.1100\;[0.0820,\;0.1380]
Selected audit, inverse probability weights0.0980  [0.0700,  0.1260]0.0980\;[0.0700,\;0.1260]
Selected audit, unweighted donors0.0410  [0.0130,  0.0690]0.0410\;[0.0130,\;0.0690]

We multiplied prior odds by 2.02.0 with P(X,C∣Y)P(X,C\mid Y) fixed (Table R7). The known prior ratio provides extra information. Broader content shifts remain open.

Table R7. CMU-MOSEI NLL gain over the same anchor with context under each target distribution.

Acquisition shiftPrior shiftCorrectionGain [95% CI]
AbsentAbsentPCOM-C0.0000  [−0.0004,  0.0004]0.0000\;[-0.0004,\;0.0004]
PresentAbsentPCOM-C0.1670  [0.1450,  0.1890]0.1670\;[0.1450,\;0.1890]
AbsentPresentPCOM-C−0.0002  [−0.0006,  0.0002]-0.0002\;[-0.0006,\;0.0002]
PresentPresentPCOM-C0.0940  [0.0650,  0.1230]0.0940\;[0.0650,\;0.1230]
PresentPresentPCOM-C with known prior ratio0.1260  [0.1000,  0.1520]0.1260\;[0.1000,\;0.1520]

Q2. Unseen policy families

A2. Following your suggestion, we froze PCOM and excluded target mechanisms from fitting and selection, without target identity, probabilities, or a new branch. Table R2 in our response to aMNR reports positive gains for the alternative generator and handwritten rule using labels. Hidden content remains a failure case.

A rule without labels can still matter when it reads content that is later hidden. If it reads only retained context, its factor cancels when the source posterior conditions on that context and source mass is the same across classes.

Q3. Baselines

A3. Thank you for suggesting these controls. Table R4 in our response to aMNR compares ERM and direct correction with the same information. Table R8 adds EM and the requested adaptations. With 500500 labels from a separate target partition, the direct residual has lower NLL. PCOM-C uses no target labels.

Table R8. Additional endpoint comparisons. Lower NLL is better.

MethodCMU-MOSEIMELD
Label-shift EM0.38700.38701.36801.3680
NeuMISE, Rockenschaub et al., adapted0.24700.24701.09401.0940
Stokes et al., adapted0.25100.25101.10601.1060
MIRRAMS, adapted0.26000.26001.11801.1180
PCOM-C, no target labels0.21800.21801.03001.0300
Direct residual with 100100 target labels0.22900.22901.05501.0550
Direct residual with 500500 target labels0.20500.20501.01501.0150

EM is a diagnostic because the benchmark changes modalities but retains all records and the label distribution. The DAMS-Mask-IW adaptation benefits more when mask marginals shift, while PCOM-C retains the larger gain (Table R9).

Table R9. CMU-MOSEI NLL gains over the anchor with context, with 95% CI. Both methods use context.

Target mask marginalsDAMS-Mask-IWPCOM-C
Approximately preserved0.0080  [0.0020,  0.0140]0.0080\;[0.0020,\;0.0140]0.1670  [0.1450,  0.1890]0.1670\;[0.1450,\;0.1890]
Shifted0.0410  [0.0260,  0.0560]0.0410\;[0.0260,\;0.0560]0.1320  [0.1080,  0.1560]0.1320\;[0.1080,\;0.1560]

Q4. Synthetic acquisition

A4. Thank you for this remark. Controlled acquisition keeps the population fixed so we can isolate how selection changes the posterior. Tables R5 and R6 in Part 1 address nonuniform source masks and incomplete views. We also tested frozen SigLIP 2 Base features on UPMC while keeping policies and masks fixed. PCOM-C improves over the direct residual in this feature setting (Table R10). Naturally occurring shifts remain untested.

Table R10. UPMC NLL with frozen SigLIP 2 Base features.

MethodNLL
Source anchor with context1.91001.9100
Direct residual1.84201.8420
PCOM-C1.81501.8150
Direct minus PCOM-C, paired 95% CI0.0270  [0.0150,  0.0390]0.0270\;[0.0150,\;0.0390]

Q5. Novelty

A5. Thank you for this remark. Our contribution connects observed mass correction with policy inference from unlabeled logs. When a policy reads hidden content, its observed mass averages over that content conditional on what is retained. For the main generator with a single label, this mass is available exactly. Analytic and learned masses therefore implement the same correction target.

The submitted CMU-MOSEI gains are 0.5711 with analytic mass and 0.3503 with learned PCOM (submission Table 18). Table R17 in our response to nbga compares them with the same anchor and state weights.

Table R16 in our response to nbga measures estimation error with exact references in a finite-state family. It supports correction within that family. The multimodal failures in Table R2 remain unresolved.

The information diagnostic in Table R11 is also consistent with the population log-loss identity. It does not establish Bayes optimality of the fitted predictors.

Table R11. Information diagnostic in nats.

QuantityValue
Estimated conditional information0.052825
Learned NLL reduction from shared to supplied state0.052780
Paired difference and interval−0.000044  [−0.001870,  0.001781]−0.000044\;[−0.001870,\;0.001781]

Q6. Runtime

A6. Thank you for raising this cost. We reused deterministic evaluations of repeated states and batched class and branch computations. This preserves state marginalization and the mass ensemble. Table R12 reports the resulting correction times under the same batching protocol. Mean NLL changes by 0.00000.0000 at the reported precision.

Table R12. Warm, amortized correction time under the same batching protocol.

DatasetSubmitted, ms/recordOptimized, ms/recordSpeed ratio
CMU-MOSEI0.70350.43000.43001.64×1.64\times
MELD1.87021.12001.12001.67×1.67\times
MM-IMDb12.01456.80006.80001.77×1.77\times
UPMC24.115013.900013.90001.73×1.73\times

Table R13 gives warm latency for a single record over 100100 synchronized repetitions, excluding feature extraction, loading, and calibration. It also reports the separate ensemble comparison.

Table R13. UPMC latency and ensemble comparison.

QuantityValue
Warm single record latency, median186.0186.0 ms
Warm single record latency, 95th percentile207.0207.0 ms
One-fold variant, amortized time3.60003.6000 ms/record
NLL before reducing the ensemble2.2661
NLL with one fold2.27502.2750

The one-fold variant changes the estimator and increases NLL (Table R13). We will report both the remaining overhead and this tradeoff. These timings do not establish cost effectiveness against the direct residual.

Thank you again for your careful review and helpful suggestions. We hope our response and the revised manuscript have clarified the points you raised. We would be grateful to hear your thoughts and learn whether there are any remaining questions we could address. Your further feedback would help us improve the clarity and presentation of our work.

I thank the authors for their detailed response to my questions.

  • The rebuttal partly answers my unseen-family question. it adds baselines. label-free PCOM-C beats them on both datasets. the baseline adaptations are not described.
  • A residual with 500 labels still wins. it wins on both datasets. with 100 labels, PCOM-C wins on both datasets. intervals are missing, so the gaps are unclear.
  • The core concerns remain. masks and policies are still synthetic. PCOM-C does not correct label-prior shift. hidden content still fails.

I intend to maintain my original score.

Reviewer_aMNROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies multimodal prediction under shifts in acquisition policies, showing that the same observed modalities and missingness pattern can lead to different class posteriors. The authors derive a posterior correction based on class-conditional observed acquisition mass and propose PCOM, which combines a frozen source predictor, mass estimation, and policy inference from unlabeled context-mask logs. Experiments on four multimodal datasets show improvements in log loss under learned policy endpoints and held-out mixture proportions. Ablation studies and stress tests further examine the contributions of different components and the limitations of the method under more challenging policy shifts.

Critical strengths
  1. The paper raises an important point that is often overlooked in missing-modality learning: the same observation can have different predictive meanings under different acquisition policies. The observed-mass formulation provides a clear theoretical basis for correcting the posterior.

  2. The experiments cover four multimodal datasets and several types of policy shifts. PCOM shows consistent gains on learned policy endpoints and held-out mixture proportions. The ablations and stress tests also provide useful evidence about when the method works and when it fails.

Critical weaknesses
  1. The paper's main claim of generalization to unseen acquisition policies is not convincingly supported. Most positive results are obtained within a learned policy family, with known endpoints or directions. Performance also degrades under some continuous and hidden-content shifts. This raises doubts about whether PCOM can handle genuinely unseen acquisition mechanisms, which is central to the paper's motivation.

  2. The comparisons do not adequately control for differences in information access. PCOM uses context-mask logs and a learned policy family, while the baselines receive substantially less information. Without information-matched baselines, it is difficult to determine whether the large performance gains come from the proposed posterior correction or simply from the additional information. This weakens the empirical evidence for the method's main contribution.

Questions for authors
  1. How does PCOM perform when the target policy comes from an unseen family and its endpoint is unknown? The current interpolation experiments do not seem to cover this case.

  2. Could the authors include baselines that use the same context-mask logs as PCOM? Without such comparisons, it is hard to separate the benefit of the mass correction from that of the extra information.

2 comments

Thank you for recognizing the problem and suggesting further tests of transfer and access to information. We address each point below.

Q1. Unseen policy families

A1. Thank you for highlighting this distinction. The submitted mixtures test new proportions along specified acquisition directions. Their mean gains appear in Table R1.

Table R1. Submitted NLL gains on mixtures.

CMU-MOSEIMELDMM-IMDbUPMC
0.14370.19020.00270.0705

Following your suggestion, we froze all PCOM components and excluded the target mechanisms from policy fitting and model selection. Prediction used the observed record, permitted logs of context and masks, and existing branches. It used no target endpoint identity, policy code, acquisition probabilities, or new branch.

Table R2. NLL gain over the frozen source anchor with paired 95% confidence intervals.

Excluded acquisition mechanismCMU-MOSEIMELD
Different generator architecture0.0320  [0.0200,  0.0440]0.0320\;[0.0200,\;0.0440]0.0410  [0.0250,  0.0570]0.0410\;[0.0250,\;0.0570]
Handwritten rule using observed context0.0002  [−0.0010,  0.0014]0.0002\;[-0.0010,\;0.0014]−0.0003  [−0.0019,  0.0013]-0.0003\;[-0.0019,\;0.0013]
Rule using hidden content without labels−0.0040  [−0.0065,  −0.0015]-0.0040\;[-0.0065,\;-0.0015]−0.0080  [−0.0120,  −0.0040]-0.0080\;[-0.0120,\;-0.0040]
Handwritten rule using labels0.0210  [0.0100,  0.0320]0.0210\;[0.0100,\;0.0320]0.0280  [0.0140,  0.0420]0.0280\;[0.0140,\;0.0420]

The average gains across the four mechanisms are 0.01230.0123 and 0.01520.0152, with equal weights (Table R2). The alternative generator and the handwritten rule using labels give positive gains. Selection based on hidden content remains a failure case. These results support transfer to the tested mechanisms. A different generator architecture alone does not establish that every resulting policy lies outside the learned family.

We also tested a direct predictor with the same context, logs, and source environments. Both intervals in Table R3 include zero, so this comparison does not establish an advantage for PCOM on excluded mechanisms.

Table R3. Direct predictor NLL minus PCOM NLL on excluded mechanisms with paired 95% CI.

CMU-MOSEIMELD
0.0020  [−0.0020,  0.0060]0.0020\;[-0.0020,\;0.0060]0.0010  [−0.0040,  0.0060]0.0010\;[-0.0040,\;0.0060]

Q2. Baselines with the same information

A2. Thank you for this suggestion. We added a source anchor with context, ERM on the same environments, and a direct residual using the same context and logs as PCOM-C (Table R4).

PCOM-C and the direct residual use the same anchor qsC(y∣xr,r,c)q_s^C(y\mid x_r,r,c), log representation, source environments, amount of source supervision, target masks, and tuning budget. The direct model learns a residual for each class. For multilabel prediction, it updates Bernoulli logits. For a single label, its update is

qdir(y∣o,W)=Normalizey[qsC(y∣o)expd(o,h(W))].q_{\text{dir}}(y\mid o,W)=\text{Normalize}_y[q_s^C(y\mid o) \text{exp} d(o, h(W)) ].

Table R4. Endpoint NLL with the same information. Lower NLL is better.

ModelCMU-MOSEIMELDMM-IMDbUPMC
Submitted source anchor0.66201.59380.27162.4748
Source anchor with context0.38500.38501.36501.36500.26700.26702.41002.4100
ERM on the same environments0.25500.25501.11001.11000.26000.26002.29002.2900
Direct residual0.23200.23201.06001.06000.25500.25502.24502.2450
PCOM-C0.21800.21801.03001.03000.25200.25202.21502.2150
Direct minus PCOM-C, paired 95% CI0.0140  [0.0050,  0.0230]0.0140\;[0.0050,\;0.0230]0.0300  [0.0150,  0.0450]0.0300\;[0.0150,\;0.0450]0.0030  [0.0008,  0.0052]0.0030\;[0.0008,\;0.0052]0.0300  [0.0120,  0.0480]0.0300\;[0.0120,\;0.0480]

Adding context reduces source anchor NLL by 0.27700.2770 on CMU-MOSEI and 0.22880.2288 on MELD (Table R4). We then keep this anchor fixed to assess the correction. These reductions do not form an additive breakdown of the original PCOM gain because the anchor changes.

With this anchor fixed, PCOM-C improves over the direct residual by 0.01400.0140 and 0.03000.0300, respectively (Table R4). The paired intervals are positive on all four datasets. The endpoint advantage therefore persists when the direct predictor receives the same information.

We report PCOM-C separately because the submitted anchor omits context. We keep the acquisition generator and masks fixed in these comparisons and will clarify each implementation's conditioning. The endpoint result applies to these fitted methods. Transfer to excluded mechanisms is assessed separately in Tables R2 and R3.

We sincerely thank you for the time and effort you have devoted to reviewing our work. We would greatly appreciate your thoughts on our response and the additional results, particularly whether any concerns still need clarification. We would be happy to discuss any remaining questions and would value your further feedback as we refine the manuscript.

Comments on the paper

1 comment

We thank reviewers aMNR, 4wH4, and nbga for their careful reading and helpful suggestions. We appreciate your interest in the acquisition problem and the evidence needed to understand when correction is useful.

Following your suggestions, we tested mechanisms excluded from fitting and model selection with PCOM frozen. Table R2 in our response to aMNR reports gains beyond new mixture proportions, alongside failures with hidden content. The comparison with a direct predictor remains inconclusive on these mechanisms (Table R3). We also added baselines with the same context, logs, and training environments. PCOM-C retains an endpoint advantage over the direct residual on all four datasets (Table R4). Its source anchor includes retained context, consistent with Equation 2, and is reported separately from the submitted implementation.

Analytic mass and the finite-state diagnostic help distinguish the correction target from estimation error (Tables R17 and R16 in our response to nbga). We will clarify this distinction and the conditions under which the results apply. The individual responses include the full tables, source assumptions, and remaining limitations. We will also simplify the text and Figure 2 to make the main argument easier to follow. We would be happy to clarify any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.