- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies multimodal prediction under shifting acquisition policies, where the same observed modalities may correspond to different class posteriors depending on how they were selected. The authors propose PCOM, which uses class-conditional acquisition mass to correct predictions from a frozen source model.
Critical strengths
- The paper studies an interesting problem.
- The authors provide a detailed theoretical formulation and extensive experiments across four datasets.
Critical weaknesses
-
The experimental setting does not fully support the paper's motivation of generalizing to unseen acquisition policies. Most of the positive results come from policy endpoints that are already represented during source training, or mixtures between known endpoints. For the mixture experiments, the endpoint identity is explicitly provided, and only the mixture proportion needs to be inferred. This seems substantially easier than adapting to a genuinely unseen acquisition policy. More importantly, the continuous-path experiments show negative average gains on CMU-MOSEI and MELD, while the hidden-content experiments also fail to improve over the source anchor. I understand that the authors acknowledge these limitations, but they significantly weaken the practical motivation of the method.
-
I am not convinced that the performance improvements can be attributed to the proposed posterior correction. PCOM receives context-mask logs containing modality statistics and source-supervised triage features, while the main baselines do not receive the same information. There is also a mismatch between the theoretical formulation and its implementation. Equation (2) uses a source posterior conditioned on the retained context, but the frozen source predictor explicitly excludes this context. This raises questions about whether the improvement comes from correcting the acquisition shift or simply introducing additional predictive information through the correction module.
-
The necessity of the proposed learned mass estimator remains unclear. In the main experiments, the acquisition policy is generated as a function of context, label, and policy state, which makes the corresponding observed mass analytically available. This is particularly concerning because Appendix Table 18 shows that analytic mass performs substantially better than the learned estimator on CMU-MOSEI.
-
Some experimental comparisons and ablations are not sufficiently convincing. For example, removing class conditioning or policy-state information in Table 18 produces zero correction by construction. These results mostly confirm the algebraic behavior of the proposed formulation rather than demonstrate the benefit of the learned architecture. The mixture experiments also lack comparisons against the adaptation baselines used in the endpoint experiments. In addition, the posterior-averaging control achieves better NLL on MELD and MM-IMDb, although it uses the true mixture proportion.
-
The presentation is unnecessarily complicated and could be substantially improved. The paper reads more like a technical handbook than a focused research paper in several places. For example, Figure 2 introduces a large number of components and intermediate operations, while the surrounding text spends considerable space explaining the implementation workflow. Similar descriptions appear throughout the appendices, with many tables and diagnostic quantities that do not always help establish the main contribution. I found it difficult to identify which components are essential and which are implementation choices.
Questions for authors
- Could the authors provide results on genuinely unseen acquisition policies, where neither the endpoint identity nor the policy family is available during source training?
- Could the authors include simpler correction baselines and more controlled ablations? In particular, it would be useful to compare against a direct context-conditioned correction model and evaluate whether the individual PCOM components improve performance beyond these simpler alternatives.
4 comments
Q2. Correction and ablations, continued
A2. Analytic mass implements the correction exactly for the main generator with a single label. Its submitted CMU-MOSEI gain is 0.5711, versus 0.3503 with learned PCOM (submission Table 18). Table R17 holds the anchor with context, records, and inferred state weights fixed.
Table R17. Endpoint NLL with learned and analytic mass.
| Dataset | Learned NLL | Analytic NLL | Learned minus analytic, paired 95% CI |
|---|---|---|---|
| CMU-MOSEI | 0.2180 | 0.1200 | |
| MELD | 1.0300 | 0.9000 |
The gap in Table R17 measures error in estimating the correction target. We will state when analytic evaluation is available. For MM-IMDb, correcting one bit can require averaging over other unknown labels, so its reference depends on that approximation. Exact target policy evaluation was not used for excluded mechanisms.
The zero-gain rows in submission Table 18 check the update's algebra. We will label this role explicitly. The direct residual gains and over the anchor with context (Table R4). Hard selection and marginalization give 0.3506 and 0.3503 on CMU-MOSEI, and 0.0082 and 0.0102 on MM-IMDb (submission Table 18). Removing overlap regularization gives 0.4294 on CMU-MOSEI and also changes policy generation. We will separate measured benefits from structural checks.
For mixtures, we compared mass mixing and posterior averaging with the same true or inferred coefficients, anchor, mass models, and records (Table R18). All intervals include zero. The difference remains inconclusive and does not establish equivalence.
Table R18. Posterior averaging NLL minus mass mixing NLL with matched coefficients. Positive values favor mass mixing.
| Dataset | True coefficient | Inferred coefficient |
|---|---|---|
| CMU-MOSEI | ||
| MELD | ||
| MM-IMDb | ||
| UPMC |
Masses combine before normalization for mixtures across records. Observing a record changes the relative weights of the component mechanisms, so averaging endpoint posteriors with the original coefficient generally gives a different posterior.
We also added ERM and direct residual comparisons beyond endpoints with the same anchor and information (Tables R19 and R20). PCOM-C improves over the direct residual on several mixture benchmarks. On MELD paths, the interval favors the direct residual. The other adapted methods were tested only at endpoints.
Table R19. Direct residual NLL minus PCOM-C NLL with paired 95% CI. Positive values favor PCOM-C.
| Dataset | Mixtures | Continuous paths |
|---|---|---|
| CMU-MOSEI | ||
| MELD | ||
| MM-IMDb | ||
| UPMC |
Table R20. ERM NLL minus PCOM-C NLL. Positive values favor PCOM-C.
| Dataset | ERM minus PCOM-C on mixtures | ERM minus PCOM-C on paths |
|---|---|---|
| CMU-MOSEI | ||
| MELD |
Q3. Presentation
A3. Thank you for this helpful feedback. We will organize the text and Figure 2 around the source posterior, observed mass ratio, and policy information from logs. We will explain how each quantity is estimated and state its assumptions. We will also distinguish uncertainty about the endpoint from mixtures across records. Implementation details and supporting diagnostics will move out of the main argument.
We sincerely appreciate the time and care you have put into reviewing our paper. We would welcome your feedback on our response, especially on whether the explanations and additional evidence address your concerns. If any points remain unclear, we would be happy to discuss them further. Thank you again for helping us strengthen our work.
I appreciate the authors' additional experiments, but I do not think they fully address my main concerns. The learned mass estimator still underperforms the analytic alternative, and the new experiments provide limited evidence of generalization to unseen acquisition policies. I also believe the presentation issues require substantial revision and have not yet been addressed in the manuscript, so I will maintain my original score.
We are grateful for your detailed comments. We respond to each question below.
Q1. Unseen acquisition policies
A1. Thank you for highlighting this distinction. The submitted mixtures test new proportions along specified directions. Following your suggestion, we evaluated mechanisms excluded from fitting and selection with PCOM frozen. Prediction used no endpoint identity, acquisition probabilities, or new branch.
Table R2 in our response to aMNR reports average gains of 0.0123 on CMU-MOSEI and 0.0152 on MELD across the four mechanisms, with equal weights. Hidden content gains remain −0.0040 and −0.0080. Both intervals against the direct predictor with the same information include zero (Table R3). These results support transfer to some tested mechanisms, without establishing a general advantage on excluded policies.
We also separated the effects of source conditioning, mass estimation, and state inference on continuous paths (Table R14). Providing the state changes the original CMU-MOSEI gain from −0.0391 to −0.0388, while MELD remains at −0.0469. Analytic mass improves the comparison with the anchor and inferred state fixed. This identifies a limitation of the learned mass estimate.
Table R14. NLL gains on continuous paths over each configuration's corresponding source anchor.
Table R15. Learned NLL minus analytic NLL with the anchor including context and inferred state fixed. Paired 95% CI.
To examine correction when acquisition reads hidden content, we added a finite-state diagnostic with exact references (Table R16). Here Y is balanced, X,H are independent conditional on Y, and P(X=y∣Y=y)=0.7, P(H=y∣Y=y)=0.8. We always observe X and set C=X. The source hides H with probability 0.5. The target hides it with probability 0.9 when H=1 and 0.1 when H=0. The rule uses content that can become hidden without reading the label.
Table R16. Target NLL within the diagnostic's specified policy family.
The population gain is 0.05077166 nats. Fully estimated correction gains 0.0380[0.0280,0.0480] over the fitted anchor, with paired 95% CI (Table R16). This diagnostic measures estimation error within a specified family. We will keep it distinct from transfer to excluded mechanisms and the unresolved multimodal failures.
Q2. Correction and ablations
A2. Thank you for this suggestion. Equation 2 conditions the source posterior on retained context, while the submitted anchor omits it. We tested PCOM-C with this conditioning and report it separately. With V=(Xr,R), starting from ηs(y∣V) requires the additional factor ηs(y∣V,C)/ηs(y∣V). The current context construction does not justify omitting this factor.
Adding context reduces anchor NLL by 0.2770 on CMU-MOSEI and 0.2288 on MELD (Table R4 in our response to aMNR). With the anchor, logs, environments, supervision, masks, and tuning budget matched, PCOM-C improves over the direct residual by 0.0140[0.0050,0.0230] and 0.0300[0.0150,0.0450]. These paired intervals support an endpoint gain with the same information (Table R4). The direct residual learns an update for each class.