acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Modality Fusion at Test Time: Label-Free Selection for Robust Multimodal Sensing

Abstract

Multimodal sensing benefits from complementary observations, but distribution shift can change the utility of individual modalities, making indiscriminate fusion detrimental. Selecting useful modalities without target labels is challenging: incorrect predictions can remain confident, and cross-modal agreement can be inflated by shared predicted-class preferences. We propose Test-Time Modality Selection (TMS), a label-free framework that selects a shared modality subset for an unlabeled target batch. Using cached predictions from classifiers frozen after source training or few-shot adaptation, TMS combines prediction coverage screening, cross-modal support estimation through empirical-marginal-corrected agreement, and reference-based marginal contribution assessment. Auxiliary calibration supports coalition comparisons, while final predictions use unweighted averaging of the selected modalities' original logits. Selection requires no source calibration data, target labels, parameter updates, or additional classifier forward passes. Experiments on MM-Fi and XRF55 evaluate cross-environment transfer under zero-, one-, and two-shot adaptation and controlled modality corruption. TMS improves mean zero-shot accuracy on MM-Fi from 32.16% to 58.79% and one-shot accuracy under severe mmWave corruption on XRF55 from 20.60% to 35.12%. These results demonstrate the value of reassessing modality participation at test time using unlabeled target predictions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Modality Selection, Multimodal Sensing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FgHGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Authors propose TMS, a label-free method to select a subset of modalities at test time for multimodal classification under distribution shift. Each modality has its own classifier, which is frozen after source training or few-shot adaptation. Using only the cached logits of an unlabeled target batch, the method screens modalities by the coverage of their predicted classes, then by chance-corrected Cohen’s κ agreement with other modalities, and finally by a marginal contribution test against held-out reference modalities, removing at most one modality. The selected modalities are fused by averaging their raw logits. Experiments on MM-Fi and XRF55 under 0/1/2-shot adaptation and synthetic corruption show that the method improves zero-shot accuracy on MM-Fi from 32.16% to 58.79%, close to the label-based oracle (59.14%), while the improvements after adaptation are small.

Critical strengths

The paper chooses a practical problem and sets a strict protocol for it: frozen classifiers, no target labels, no source data, and an oracle under the same fusion rule as reference. Under this protocol, the zero-shot improvement on MM-Fi is large, and the method is never worse than full fusion in all reported tables, which is important for deployment. In addition, the results are reported per transfer and per budget with the selected subsets and full algorithm details, so the claims can be checked easily.

Critical weaknesses

This paper proposes a relatively complete work, but I have the following concerns:

  1. The contribution of the selector is insufficient. Each stage is close to existing ideas that are not discussed in the paper. The coverage screening is very similar to the prediction dispersity in “Confidence and Dispersity Speak: Characterizing Prediction Matrix for Unsupervised Accuracy Estimation”, where dispersity was already used as a label-free signal of accuracy under distribution shift, since poorly generalizing models tend to assign most samples to a few classes. Using pairwise κ to select classifiers can be traced back to Kappa pruning in “Pruning Adaptive Boosting” (ICML 1997), although κ there is used for diversity rather than agreement. The rank-one agreement model in Eq. (9) is the one-coin Dawid–Skene model, and the weak supervision line such as FlyingSquid (ICML 2020) already estimates the accuracies of noisy sources without labels. What I doubt is whether combining these components by several hand-set rules (strongest partner, Stage 3 only when |S1| ≥ 3, removing at most one modality, and the fallbacks) is sufficient as a contribution, because there is no unified objective or identifiability analysis behind it.
  2. The applicable scope of the method is not discussed. When there are only two modalities, s_1 = s_2 and Stage 3 is not triggered, so the method can only rely on coverage screening and cannot decide which of the two disagreeing modalities should be removed. Two-modality settings are common, such as the video and audio benchmarks used in READ. Moreover, both the strongest partner rule and the rank-one model assume that the errors of different modalities are roughly independent. If two modalities fail in the same way, they will support each other. Appendix A.4 mentions that correlated errors do not cancel, but no experiment analyzes this case.
  3. The experiments cannot effectively support that the gain comes from test-time selection. According to Appendix C.1, in the zero-shot setting TMS mainly keeps mmWave on MM-Fi and selects radar only in all XRF55 scenes, and after one-shot adaptation it almost always keeps RGB, mmWave and LiDAR. However, the accuracy of each single modality and fixed-policy baselines (e.g., always using mmWave) are not reported. I wonder how much of the 26-point improvement remains when compared with such a fixed policy. In addition, the baselines are weak: Ent. is identical to full fusion in all settings, Conf. never exceeds full fusion, and ATC drops to 3.72–3.95% on four MM-Fi zero-shot transfers, which is close to chance for 27 classes. The most direct baseline, selecting or weighting modalities by the p̂_m already computed in Eq. (10), is missing. I believe that comparisons with label-model-based selection, dispersity-based selection and per-sample weighting such as QMF (ICML 2023) are needed in order to be convincing.
  4. The improvements after adaptation are only 0.88 and 0.35 points on MM-Fi, and no variance over seeds or support-set draws is reported. It is hard to judge whether these improvements are significant.
Questions for authors
  1. Please report the single-modality accuracy for every transfer and budget, together with the mmWave-only and leave-one-transfer-out fixed-subset results. This is the most important point for my rating.
  2. How does TMS compare with selection or weighting by p̂_m or Dawid–Skene EM, dispersity-based selection, and ATC combined with coverage screening? Are the subsets selected by ATC on E02→E01, E02→E04, E04→E02 and E04→E03 collapsed to a few classes?
  3. What are the results on all two-modality pairs of MM-Fi and XRF55, and in a controlled case where two modalities share the same failure (e.g., a duplicated or jointly corrupted modality)?
  4. Is N the whole target query set? Is the selection stable with small batches or streaming data?
  5. Stage 3 scores coalitions with the weights w_m/T_m, but the final prediction uses equal weights on raw logits. Why is the deployed rule not used for scoring, and would it change the selected subsets?
8 comments
AuthorsOpenReview ↗

Response to Reviewer FgHG

We thank the reviewer for the detailed and insightful review, and for recognizing the strict protocol, the large zero shot improvement, the fact that TMS is never worse than full fusion in all reported settings, and the transparency of our reporting. We first answer the question that the reviewer identified as most important.

Q1 and W3. Single modality accuracy and fixed policies

Tables R1 and R2 report the accuracy of every single modality together with full fusion, the leave one transfer out fixed subset, TMS, and the oracle. The leave one transfer out fixed subset is the subset with the best mean accuracy on the other transfers of the same dataset, selected with their target labels and then applied to the held out transfer. On MM-Fi, the other transfers include transfers into the same target environment, so this fixed subset is selected with labels from the target environment itself. It is therefore an optimistic reference for fixed policies. Results for every individual transfer will be included in the appendix.

Table R1. MM-Fi accuracy in %, averaged over 12 transfers.

ShotsRGBDepthmmWaveLiDARWiFiFull fusionLeave one transfer out fixed subsetTMSOracle
05.3813.6657.6625.333.8632.1657.6658.7959.14
159.3152.8369.6271.2615.8786.0086.8886.8886.88
266.1760.3070.0673.5320.8887.3487.6587.6987.91

Table R2. XRF55 accuracy in %, averaged over three target scenes.

ShotsmmWaveWiFiRFIDFull fusionLeave one transfer out fixed subsetTMSOracle
017.052.091.845.3217.0517.0517.05
138.1939.418.9454.8556.7157.4857.48
251.7252.4313.3167.2369.8870.1070.10

Without adaptation, mmWave is the dominant reliable modality on clean data. TMS identifies it without any labels and adds depth or LiDAR in the transfers where they help, reaching 58.79% compared with 57.66% for mmWave alone. After adaptation, the best subset changes to RGB, mmWave, and LiDAR, and every single modality falls far below TMS. TMS matches or exceeds the leave one transfer out fixed subset in every setting without using any labels. The main value of test time selection appears when the sensor condition changes after a policy has been fixed, as Table R3 shows.

Table R3. Accuracy in % when sensors fail after the policy has been fixed.

SettingFull fusionmmWave onlyLeave one transfer out fixed subsetTMS
XRF55, one shot, severe mmWave corruption20.601.8518.0735.12
MM-Fi, zero shot, severe mmWave corruption at input level14.214.764.7623.18
MM-Fi, zero shot, moderate corruption of both mmWave and WiFi at input level14.634.654.6527.12

The leave one transfer out fixed subset is competitive on clean data, but it is chosen before deployment and cannot react when a sensor fails. It degrades together with the failed sensor, while TMS detects the failure on the target batch and selects the remaining modalities. No single fixed policy is suitable across adaptation budgets and sensor conditions, and this is the gain that test time selection provides.

AuthorsOpenReview ↗

Response to Reviewer FgHG

Q2 and W3. Stronger baselines and the ATC collapse

Table R4 adds selection by the estimated accuracy p̂ from Eq. 10, Dawid Skene selection, dispersity selection, agreement based weighting with the spectral meta learner, ATC with coverage screening, and QMF. QMF here weights each modality per sample by its confidence on the frozen classifiers.

Table R4. MM-Fi accuracy in %, averaged over 12 transfers.

MethodZero shotOne shotTwo shots
Full fusion32.1686.0087.34
Majority voting33.5683.8585.34
Spectral meta learner32.9786.0886.68
Selection by p̂33.5985.9686.75
Dawid Skene selection54.9185.9686.75
Dispersity selection55.1683.5283.29
QMF33.4986.2387.41
ATC21.8883.0784.26
ATC with coverage screening56.4483.0784.26
TMS58.7986.8887.69
Oracle59.1486.8887.91

TMS is the best method without labels in all three settings. Selection by p̂ and QMF stay close to full fusion at zero shot, because a collapsed modality still produces confident predictions that these scores cannot distinguish from reliable ones. On the four transfers where ATC drops to between 3.72% and 3.95%, its accuracy equals the chance level of 1/27. On E02→E01, E02→E04, E04→E02, and E04→E03, ATC selects depth alone or RGB alone. The effective number of predicted classes of these subsets is between 1.00 and 1.08, and at least 98.6% of their predictions fall into a single class. ATC rewards these subsets because their predictions remain highly confident. This is exactly the failure that coverage screening detects, and adding coverage screening to ATC raises its zero shot accuracy from 21.88% to 56.44%.

W1. Contribution and identifiability

TMS is related to the works the reviewer mentions, and we will discuss all of them in the revised version. The relation, however, lies in the statistics used, while the selection target and the way the statistics are combined are different. Dispersity estimates the accuracy of a single model. TMS uses coverage only as a relative screen among modalities on the same batch and never as an accuracy estimate, and dispersity used alone for selection reaches 55.16% at zero shot, below 58.79% for TMS. Kappa pruning keeps learners with low κ to increase diversity in boosting. TMS uses κ corrected by the empirical prediction marginals in the opposite direction, as evidence of cross modal support, and this correction alone contributes 4.69 points at zero shot. Eq. 9 is the one coin Dawid Skene model, and FlyingSquid estimates source accuracies with triplet methods. These label models answer how accurate each source is. TMS asks whether adding a modality improves the fused decision. For this reason, the rank one model serves only as auxiliary calibration, and Stage 3 compares coalitions with and without a modality against held out references. Dawid Skene selection reaches 54.91% at zero shot, below TMS.

The rules also follow from an identifiability analysis. Under the one coin model with conditional independence and uniform errors, κ between modalities i and j equals q_i q_j, where q denotes chance corrected competence. The strongest partner score of modality m is therefore q_m times the largest competence among the other modalities, and Stage 2 keeps exactly the modalities whose competence is at least τ times that of the second strongest modality. With only two modalities, agreement identifies only the product of their competences. This matches the triplet requirement in FlyingSquid and explains why Stage 3 requires at least three modalities. Removing at most one modality keeps the decision conservative when all evidence comes from a single unlabeled batch. All thresholds are fixed to 0.5 across datasets, adaptation budgets, and corruption conditions without tuning. Each rule is supported by the ablation: removing coverage screening, replacing the empirical marginal correction with a uniform baseline, or removing the marginal assessment lowers zero shot accuracy to 52.68%, 54.10%, and 51.92%, respectively.

AuthorsOpenReview ↗

Response to Reviewer FgHG

W2 and Q3. Two modalities and shared failures

With two modalities, the identifiability analysis above shows that pairwise agreement determines only the product of the two competences. No method based on agreement can decide which of two disagreeing modalities is reliable, so TMS then relies on coverage screening. We evaluated all two modality pairs, 10 on MM-Fi and 3 on XRF55. At zero shot, TMS still removes the collapsed modality and improves the average accuracy from 24.61% to 30.51%. After adaptation, TMS selects the same subset as full fusion in every pair, so it is never worse than full fusion. TMS is designed for sensing systems with three or more modalities, which is the common setting in multimodal sensing, with five modalities in MM-Fi and three in XRF55. We will state this scope explicitly.

For shared failures, we test both cases the reviewer suggests. In the joint corruption case, two modalities are degraded by the same type of corruption.

Table R5. Accuracy in % when mmWave and WiFi are both moderately corrupted.

SettingFull fusionTMS
XRF55, one shot10.3712.84
XRF55, two shots13.3019.12
MM-Fi, zero shot, input level corruption14.6327.12

TMS improves over full fusion in all three cases. In the duplication case, we copy the mmWave classifier as an additional modality, which is the worst case of perfectly correlated errors. The original rule keeps both copies in all 45 settings across datasets, transfers, and shot numbers, and on XRF55 with one shot this removes WiFi and lowers accuracy from 57.48% to 38.19%. Merging modality pairs whose κ exceeds 0.90 before computing the support scores restores the selection obtained without the duplicate in all 45 settings. It does not change any selection on clean data, where the largest pairwise κ is 0.65. We will include this step in the revised method.

W4. Significance of the gains after adaptation

The gains after adaptation are bounded by the oracle, which uses target labels under the same fusion rule. Table R6 reports a sign test over the 12 MM-Fi transfers, with ties excluded.

Table R6. MM-Fi accuracy in % and sign test of TMS against full fusion over 12 transfers.

ShotsFull fusionTMSOracleWins / ties / lossesOne sided p
032.1658.7959.1412 / 0 / 00.00024
186.0086.8886.8811 / 1 / 00.00049
287.3487.6987.9110 / 2 / 00.00098

With one shot, the oracle also reaches 86.88%, so TMS captures all of the available headroom. With two shots, it captures 0.35 of the 0.57 points available. On XRF55, TMS improves full fusion by 2.62 and 2.87 points with one and two shots and equals the oracle in both cases. TMS never loses to full fusion on any transfer. To measure the variance over support sets, we keep the source encoders fixed and draw the support set again with two independent random seeds on all 12 MM-Fi transfers.

Table R7. MM-Fi accuracy in % for different support draws, averaged over 12 transfers.

Support drawOne shot full fusionOne shot TMSGainTwo shot full fusionTwo shot TMSGain
Original split86.0086.880.8887.3487.690.35
Random draw, seed 10189.0589.870.8393.1893.530.36
Random draw, seed 30388.7589.650.9092.9093.270.37

The original split uses a fixed deterministic choice of support samples, and random draws give higher absolute accuracy. The gain of TMS over full fusion, however, is stable across all support sets, at 0.83 to 0.90 points with one shot and 0.35 to 0.37 points with two shots. The two random draws also select exactly the same subset on all 12 transfers. In addition, when the query set is resampled with bootstrap, the selected subset remains unchanged in 34 of 36 MM-Fi settings.

Reviewer_JYJKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Test-Time Modality Selection (TMS), a label-free framework for selecting reliable modalities under distribution shifts. The method combines prediction-coverage screening, chance-corrected cross-modal agreement, and reference-based marginal contribution assessment to select a shared modality subset without updating model parameters. Experiments on MM-Fi and XRF55 demonstrate substantial improvements over full fusion and confidence-based selection, particularly under zero-shot transfer and modality corruption.

Critical strengths

The paper addresses an important challenge in multimodal learning by dynamically selecting reliable modalities without target labels or model updates. The proposed framework is lightweight and compatible with pretrained modality-specific classifiers.

Critical weaknesses

1.The proposed approach relies heavily on cross-modal agreement as a proxy for reliability. However, modalities may exhibit correlated errors under shared distribution shifts, causing mutually incorrect predictions to reinforce each other. The auxiliary rank-one model also assumes conditional independence and simplified error distributions. The consequences of violating these assumptions are not adequately investigated.

2.Prediction-coverage screening assumes that reliable modalities should maintain relatively diverse predicted-class distributions. However, legitimate target batches may contain highly imbalanced class frequencies or only a small subset of activities. Under such conditions, a reliable modality could be incorrectly excluded. The paper lacks systematic evaluations of batch size, class imbalance, and changing target distributions.

3.The experiments mainly compare TMS with full fusion, confidence-based selection, entropy-based selection, and ATC. More direct comparisons with existing unsupervised ensemble ranking, agreement-based weighting, and strong single-modality selection strategies would better establish its advantages. This is particularly important because the reported zero-shot improvements frequently arise from retaining only a small number of modalities.

Questions for authors

1.What are the fundamental methodological contributions of TMS beyond integrating existing chance-corrected agreement and unsupervised ensemble selection techniques?

  1. How sensitive is the proposed method to target-batch size and class distribution? Could the authors evaluate class-imbalanced batches and smaller online inference windows?
4 comments
AuthorsOpenReview ↗

Response to Reviewer JYJK

We thank the reviewer for the careful reading and the constructive comments, and for recognizing that TMS addresses an important challenge in multimodal learning while remaining lightweight and compatible with pretrained modality specific classifiers. We address each concern below.

W1. Correlated errors and the rank one assumptions

TMS is designed to limit the influence of correlated errors at several levels. The rank one model is used only to calibrate the auxiliary coalition predictions in Stage 3. It provides neither the support scores in Stage 2 nor the weights of the final prediction, which averages the original logits with equal weights, so a violation of its assumptions affects only the relative scale inside the coalition comparisons. The most common form of correlated error under shared shift is that several modalities collapse toward the same few classes. The empirical marginal correction in Stage 2 removes this shared preference, and replacing it with a uniform chance baseline lowers zero shot accuracy on MM-Fi from 58.79% to 54.10%. In Stage 3, every coalition is compared against a held out reference modality, so a modality is never used to validate itself. The paper also evaluates a shared corruption setting in which WiFi and mmWave are corrupted together on XRF55.

Table R1. XRF55 accuracy in % when WiFi and mmWave are both moderately corrupted.

ShotsFull fusionTMSOracle
One10.3712.8413.64
Two13.3019.1219.39

Even when two modalities are degraded by a common corruption, TMS improves over full fusion and stays close to the oracle, which shows that the selection remains effective when the errors of different modalities are correlated.

W2 and Q2. Batch size, class imbalance, and changing target distributions

Coverage screening uses a relative threshold. A modality is excluded only when its effective number of predicted classes falls below half of the widest coverage among the modalities on the same batch. When the true class distribution is imbalanced, reliable modalities lose coverage together and their ratios are largely preserved. Excluded modalities also remain available for the later agreement statistics. To measure sensitivity, we select the subset on random target batches of different sizes and on batches that contain only three classes, and then evaluate the selected subset on the full MM-Fi target set with zero shots.

Table R2. MM-Fi zero shot accuracy in % of the subset selected on each batch, averaged over 12 transfers. Values after ± are standard deviations over repeated random draws.

Batch used for selectionAccuracy
Full target set58.79
Random batch of 256 samples58.28 ± 1.0
Random batch of 16 samples50.16 ± 10.3
Batch of 256 samples with only three classes51.0
Full fusion without selection32.16

Selection on 256 samples is almost identical to selection on the full target set, and even online windows of 16 samples remain far above full fusion. Batches with only three classes are the most extreme case of class imbalance. Accuracy decreases to 51.0% but still remains far above full fusion. We further examined the mechanism described by the reviewer, in which a near random modality reaches very high coverage, raises the reference, and causes a reliable modality to be excluded. When we add a random modality whose effective number of classes equals the number of classes, TMS never selects it, but in 2 of 12 transfers it raises the reference and removes LiDAR. On E03→E04, for example, accuracy drops from 60.19% to 54.46%. Computing the reference coverage only over modalities that have a positive κ partner restores the original subset in both transfers. We will add this correction and discuss the behavior on small batches in the revised version.

AuthorsOpenReview ↗

Response to Reviewer JYJK

W3. Unsupervised ensembles, agreement based weighting, and single modality selection

Table R3 compares TMS with unsupervised ensemble ranking and agreement based weighting under the same frozen classifiers.

Table R3. MM-Fi accuracy in %, averaged over 12 transfers.

MethodZero shotOne shotTwo shots
Full fusion32.1686.0087.34
Majority voting33.5683.8585.34
Spectral meta learner32.9786.0886.68
Selection by rank one accuracy estimate33.5985.9686.75
Dawid Skene selection54.9185.9686.75
Dispersity selection55.1683.5283.29
ATC21.8883.0784.26
ATC with coverage screening56.4483.0784.26
TMS58.7986.8887.69
Oracle59.1486.8887.91

TMS is the best method without labels in all three settings. Majority voting, the spectral meta learner, and selection by estimated accuracy stay close to full fusion at zero shot, because they estimate how accurate each modality is but are not designed to detect modalities that collapse to a few classes. To isolate the effect of test time selection, we also compare TMS with fixed single modality and fixed subset policies.

Table R4. Accuracy in % of fixed policies on clean data.

DatasetShotsmmWave onlyBest single modalityFixed subset from other transfersTMS
MM-Fi057.6657.6657.6658.79
MM-Fi169.6271.2686.8886.88
MM-Fi270.0673.5387.6587.69
XRF55017.0517.0517.0517.05
XRF55138.1939.4156.7157.48
XRF55251.7252.4369.8870.10

The best single modality is chosen with target labels, and the fixed subset is chosen with target labels from the other transfers, so both use more supervision than TMS. TMS matches or exceeds all fixed policies in every setting without using any labels. After adaptation, every single modality is far below TMS, and TMS keeps three of the five modalities on MM-Fi, so it does not favor small subsets. On E02→E01 with one shot, TMS keeps three modalities while the oracle keeps all five, and both reach 89.41%. A fixed policy also cannot follow changes in sensor condition, as Table R5 shows.

Table R5. Accuracy in % under severe mmWave corruption.

SettingFull fusionmmWave onlyTMS
XRF55, one shot20.601.8535.12
MM-Fi, zero shot, input level corruption14.214.7623.18

When mmWave fails, a policy fixed on mmWave collapses to 1.85% and 4.76%, while TMS selects the remaining modalities and reaches 35.12% and 23.18%.

Q1. Methodological contribution

TMS builds on established statistics, but its contribution lies in the problem formulation and in a selection principle that existing ensemble methods do not provide. We formulate deployment time selection of a modality subset for fusion with frozen classifiers and no source data, and we show that the best subset changes with both the target environment and the adaptation budget. TMS also separates reliability from contribution. Unsupervised ensemble methods such as the Dawid Skene model and the spectral meta learner estimate how accurate each source is, whereas subset selection must determine whether adding a modality improves the fused decision. Our marginal assessment compares coalitions with and without a modality against held out reference modalities, so a modality never validates itself. TMS further organizes its components by the strength of their assumptions. Selection decisions rely on coverage, chance corrected agreement, and reference agreement, which require only weak assumptions, while the rank one model is used only for auxiliary calibration. Under the one coin model with conditional independence and uniform errors, Stage 2 keeps exactly the modalities whose chance corrected competence is at least τ times that of the second strongest modality, which gives the rule a clear interpretation. The results confirm that this design goes beyond combining existing tools. Dawid Skene selection and dispersity selection reach 54.91% and 55.16% at zero shot, below 58.79% for TMS, and removing coverage screening, the empirical marginal correction, or the marginal assessment lowers zero shot accuracy to 52.68%, 54.10%, and 51.92%, respectively.

We hope that the new results and clarifications address the reviewer's main concerns. If so, we would be grateful if the reviewer could kindly take them into account when reconsidering the overall assessment.

AuthorsOpenReview ↗

Friendly follow-up

Thank you again for recognizing that TMS addresses an important challenge in multimodal learning while remaining lightweight and compatible with pretrained modality specific classifiers. We hope our analysis of correlated errors with the joint corruption results, the sensitivity study on batch size and class imbalance, the comparison with unsupervised ensemble and agreement based baselines, the fixed policy comparisons, and the clarification of our methodological contribution address your concerns. If any issue remains unclear, please let us know. We would be glad to discuss it and provide further clarification promptly.

Thank you again for your careful reading and for recognizing that TMS addresses an important challenge in multimodal learning while remaining lightweight and compatible with pretrained modality-specific classifiers. We hope our analyses of correlated errors, batch-size and class-imbalance sensitivity, unsupervised-ensemble and fixed-policy comparisons, and the clarification of the methodological contribution have addressed your concerns. If no issues remain, we would be grateful if you could consider updating your assessment. If any point is still unclear, we would be happy to clarify it promptly.

Reviewer_eXiZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a test-time fusion framework for multimodal inference. The goal is to perform modality selection to filter corrupt logit through multimodal classification logit agreement. Evaluation on two sensing datasets under synthetic noise, dropping and environment transfer shows improved performance over test-time selection baselines.

Critical strengths
  • The motivation of the problem is relevant and clear.
  • I appreciate the goal of not adding additional modules for adaptation while improving the performance.
Critical weaknesses
  • The evaluation compares the proposed methods with older baselines, overlooking more the broader multimodal fusion frameworks in the literature. Its unclear how the work compares with existing multimodal fusion frameworks which handle robustness at training time. The authors also mention other test-time fusion, such as attention-based fusion, but does not compare them in the evaluation. Data selection or modality selection under noisy inputs could also be compared.
  • The authors evaluate them on supervised classifiers. Recent advances in self-supervised learning provides more robust approach for classification, where the representation shows great generalizability. Moreover, the intermediate representations (for both self-supervised methods and supervised classifiers) also provide additional information about the data corruption. It is unclear how the methods compare with using those rich intermediate representations rather than class logits. Evaluation across encoders and training approaches would be beneficial.
Questions for authors
  1. What is the intuition behind using class logits for selection? Why not using intermediate representations (embedding before the class mapping)?
  2. Can you provide additional evaluation with SOTA robust multimodal fusion baselines (including attention-based, training-based)?
  3. Beyond cross-environment transfer, is there real-world noise measurements over synthetic corruptions to evaluate the robustness of the proposed methods in real sensing applications?
6 comments
AuthorsOpenReview ↗

Response to Reviewer eXiZ

We thank the reviewer for the thoughtful review and the constructive suggestions, and for recognizing that the motivation of our work is relevant and clear and that TMS improves performance without adding any adaptation modules. We address each concern below.

W1 and Q2. Comparison with robust fusion and selection under noisy inputs

Training time robust fusion and TMS address different stages of the same goal. Methods such as X-Fi learn to fuse any available combination of modalities, but they assume that the modalities provided are informative. TMS decides at deployment which available modalities should not participate. The two are therefore complementary. We compare TMS with both kinds of methods and also apply TMS on top of robust fusion models.

Robust fusion. We train an X-Fi style attention fusion model and a late fusion model on the same source encoders. Both are trained with random modality dropout.

Table R1. MM-Fi zero shot accuracy in %, averaged over 12 transfers.

MethodAll modalitiesWith TMS subset
Logit averaging32.1658.79
Attention fusion with modality dropout19.1453.23
Late fusion with modality dropout33.0257.50

Training time robustness does not prevent collapse under environment shift. With all modalities as input, the attention model reaches 19.14% and the late fusion model 33.02%. Applying TMS to the same models raises them to 53.23% and 57.50%. On XRF55, the late fusion model improves from 5.93% to 15.82% with TMS.

Selection under noisy inputs. We compare TMS with QMF, ATC, and a fixed modality policy. Our QMF weights each modality per sample by its confidence on the frozen classifiers.

Table R2. XRF55 one shot accuracy in % under severe corruption, averaged over three target scenes.

MethodmmWave severeWiFi severe
Full fusion20.6031.33
ATC19.0526.78
QMF19.1231.12
Fixed mmWave only1.8538.19
TMS35.1238.20

Table R3. MM-Fi zero shot accuracy in % under input level corruption, averaged over 12 transfers.

MethodmmWave severeBoth moderate
Full fusion14.2114.63
Fixed subset selected on clean data14.7214.30
Fixed mmWave only4.764.65
TMS23.1827.12

QMF stays close to full fusion because a failed sensor still makes confident predictions on individual samples. A fixed policy degrades together with the failed sensor. TMS detects the failure from batch level statistics and selects the remaining reliable modalities.

W2 and Q1. Why class logits, and evaluation across encoders

We use class logits for several reasons. The modalities in our setting use very different encoders, and their embeddings differ in dimension and geometry. Comparing such embeddings requires a learned alignment, which needs extra training and paired data, whereas the logits of all modalities share the same label space, so agreement between modalities is directly defined on every sample. The failure we target also appears in the predicted labels. Under environment shift, a failed modality assigns most samples to a few classes with high confidence. Coverage and chance corrected agreement measure this directly, while a shift in the embedding shows how far a modality has moved but not whether its predictions are still useful. In addition, cached logits are sufficient, so TMS works with frozen or black box classifiers and needs no extra forward pass. We agree that intermediate representations carry useful information, and we tested a variant that replaces Cohen's κ with linear CKA between the penultimate features of each pair of modalities.

Table R4. MM-Fi zero shot accuracy in %, averaged over 12 transfers.

Agreement measureAccuracy
Cohen's κ on predicted labels58.79
Linear CKA on penultimate features46.49

As shown in Table R4, CKA reaches 46.49%, well below 58.79% for κ, which confirms that label agreement is the more reliable signal for this purpose. TMS only needs the prediction of each modality and makes no assumption about how the encoders are trained. Besides supervised unimodal classifiers, Table R1 applies TMS to attention fusion and late fusion trained with modality dropout, and TMS improves both. We did not train self supervised encoders within the rebuttal period and will list this as a limitation. Stronger representations improve accuracy on clean data, but they cannot restore information that a failed sensor no longer captures, so selection remains necessary under sensor failure.

AuthorsOpenReview ↗

Response to Reviewer eXiZ

Q3. Real world noise

Our main results already involve real deployment shift. The four MM-Fi environments differ in room layout, multipath propagation, and subjects, and the XRF55 scenes differ in physical environment. No synthetic noise is added in this setting, and TMS still improves substantially over full fusion, reaching or approaching the label based oracle.

Table R5. Zero shot accuracy in % under real environment shift.

DatasetFull fusionTMSOracle
MM-Fi32.1658.7959.14
XRF555.3217.0517.05

The controlled corruptions complement this setting by varying the severity of sensor failure. They are applied to the raw inputs, and the frozen encoders are run again on the corrupted data. Dropping spatial bins or points in mmWave corresponds to occlusion and sparse radar returns, and Gaussian noise on WiFi CSI corresponds to interference and a low signal to noise ratio. The paper reports these corruptions on XRF55, and we now apply the same protocol to MM-Fi. Tables R2 and R3 show that TMS improves over full fusion in every case, for example from 14.21% to 23.18% under severe mmWave corruption on MM-Fi.

We hope that the new comparisons and clarifications address the reviewer's main concerns. If so, we would be grateful if the reviewer could kindly take them into account when reconsidering the overall assessment.

AuthorsOpenReview ↗

Friendly follow-up

Thank you again for recognizing that the motivation of our work is relevant and clear, and that TMS improves performance without adding any adaptation modules. We hope our comparisons with robust fusion models trained with modality dropout, the evaluation against QMF, ATC, and fixed policies under noisy inputs, the new input level corruption results on MM-Fi, and the comparison between label agreement and feature level CKA address your concerns. If any issue remains unclear, please let us know. We would be glad to discuss it and provide further clarification promptly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.