- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Authors propose TMS, a label-free method to select a subset of modalities at test time for multimodal classification under distribution shift. Each modality has its own classifier, which is frozen after source training or few-shot adaptation. Using only the cached logits of an unlabeled target batch, the method screens modalities by the coverage of their predicted classes, then by chance-corrected Cohen’s κ agreement with other modalities, and finally by a marginal contribution test against held-out reference modalities, removing at most one modality. The selected modalities are fused by averaging their raw logits. Experiments on MM-Fi and XRF55 under 0/1/2-shot adaptation and synthetic corruption show that the method improves zero-shot accuracy on MM-Fi from 32.16% to 58.79%, close to the label-based oracle (59.14%), while the improvements after adaptation are small.
Critical strengths
The paper chooses a practical problem and sets a strict protocol for it: frozen classifiers, no target labels, no source data, and an oracle under the same fusion rule as reference. Under this protocol, the zero-shot improvement on MM-Fi is large, and the method is never worse than full fusion in all reported tables, which is important for deployment. In addition, the results are reported per transfer and per budget with the selected subsets and full algorithm details, so the claims can be checked easily.
Critical weaknesses
This paper proposes a relatively complete work, but I have the following concerns:
- The contribution of the selector is insufficient. Each stage is close to existing ideas that are not discussed in the paper. The coverage screening is very similar to the prediction dispersity in “Confidence and Dispersity Speak: Characterizing Prediction Matrix for Unsupervised Accuracy Estimation”, where dispersity was already used as a label-free signal of accuracy under distribution shift, since poorly generalizing models tend to assign most samples to a few classes. Using pairwise κ to select classifiers can be traced back to Kappa pruning in “Pruning Adaptive Boosting” (ICML 1997), although κ there is used for diversity rather than agreement. The rank-one agreement model in Eq. (9) is the one-coin Dawid–Skene model, and the weak supervision line such as FlyingSquid (ICML 2020) already estimates the accuracies of noisy sources without labels. What I doubt is whether combining these components by several hand-set rules (strongest partner, Stage 3 only when |S1| ≥ 3, removing at most one modality, and the fallbacks) is sufficient as a contribution, because there is no unified objective or identifiability analysis behind it.
- The applicable scope of the method is not discussed. When there are only two modalities, s_1 = s_2 and Stage 3 is not triggered, so the method can only rely on coverage screening and cannot decide which of the two disagreeing modalities should be removed. Two-modality settings are common, such as the video and audio benchmarks used in READ. Moreover, both the strongest partner rule and the rank-one model assume that the errors of different modalities are roughly independent. If two modalities fail in the same way, they will support each other. Appendix A.4 mentions that correlated errors do not cancel, but no experiment analyzes this case.
- The experiments cannot effectively support that the gain comes from test-time selection. According to Appendix C.1, in the zero-shot setting TMS mainly keeps mmWave on MM-Fi and selects radar only in all XRF55 scenes, and after one-shot adaptation it almost always keeps RGB, mmWave and LiDAR. However, the accuracy of each single modality and fixed-policy baselines (e.g., always using mmWave) are not reported. I wonder how much of the 26-point improvement remains when compared with such a fixed policy. In addition, the baselines are weak: Ent. is identical to full fusion in all settings, Conf. never exceeds full fusion, and ATC drops to 3.72–3.95% on four MM-Fi zero-shot transfers, which is close to chance for 27 classes. The most direct baseline, selecting or weighting modalities by the p̂_m already computed in Eq. (10), is missing. I believe that comparisons with label-model-based selection, dispersity-based selection and per-sample weighting such as QMF (ICML 2023) are needed in order to be convincing.
- The improvements after adaptation are only 0.88 and 0.35 points on MM-Fi, and no variance over seeds or support-set draws is reported. It is hard to judge whether these improvements are significant.
Questions for authors
- Please report the single-modality accuracy for every transfer and budget, together with the mmWave-only and leave-one-transfer-out fixed-subset results. This is the most important point for my rating.
- How does TMS compare with selection or weighting by p̂_m or Dawid–Skene EM, dispersity-based selection, and ATC combined with coverage screening? Are the subsets selected by ATC on E02→E01, E02→E04, E04→E02 and E04→E03 collapsed to a few classes?
- What are the results on all two-modality pairs of MM-Fi and XRF55, and in a controlled case where two modalities share the same failure (e.g., a duplicated or jointly corrupted modality)?
- Is N the whole target query set? Is the selection stable with small batches or streaming data?
- Stage 3 scores coalitions with the weights w_m/T_m, but the final prediction uses equal weights on raw logits. Why is the deployed rule not used for scoring, and would it change the selected subsets?
8 comments
Response to Reviewer FgHG
Q2 and W3. Stronger baselines and the ATC collapse
Table R4 adds selection by the estimated accuracy p̂ from Eq. 10, Dawid Skene selection, dispersity selection, agreement based weighting with the spectral meta learner, ATC with coverage screening, and QMF. QMF here weights each modality per sample by its confidence on the frozen classifiers.
Table R4. MM-Fi accuracy in %, averaged over 12 transfers.
| Method | Zero shot | One shot | Two shots |
|---|---|---|---|
| Full fusion | 32.16 | 86.00 | 87.34 |
| Majority voting | 33.56 | 83.85 | 85.34 |
| Spectral meta learner | 32.97 | 86.08 | 86.68 |
| Selection by p̂ | 33.59 | 85.96 | 86.75 |
| Dawid Skene selection | 54.91 | 85.96 | 86.75 |
| Dispersity selection | 55.16 | 83.52 | 83.29 |
| QMF | 33.49 | 86.23 | 87.41 |
| ATC | 21.88 | 83.07 | 84.26 |
| ATC with coverage screening | 56.44 | 83.07 | 84.26 |
| TMS | 58.79 | 86.88 | 87.69 |
| Oracle | 59.14 | 86.88 | 87.91 |
TMS is the best method without labels in all three settings. Selection by p̂ and QMF stay close to full fusion at zero shot, because a collapsed modality still produces confident predictions that these scores cannot distinguish from reliable ones. On the four transfers where ATC drops to between 3.72% and 3.95%, its accuracy equals the chance level of 1/27. On E02→E01, E02→E04, E04→E02, and E04→E03, ATC selects depth alone or RGB alone. The effective number of predicted classes of these subsets is between 1.00 and 1.08, and at least 98.6% of their predictions fall into a single class. ATC rewards these subsets because their predictions remain highly confident. This is exactly the failure that coverage screening detects, and adding coverage screening to ATC raises its zero shot accuracy from 21.88% to 56.44%.
W1. Contribution and identifiability
TMS is related to the works the reviewer mentions, and we will discuss all of them in the revised version. The relation, however, lies in the statistics used, while the selection target and the way the statistics are combined are different. Dispersity estimates the accuracy of a single model. TMS uses coverage only as a relative screen among modalities on the same batch and never as an accuracy estimate, and dispersity used alone for selection reaches 55.16% at zero shot, below 58.79% for TMS. Kappa pruning keeps learners with low κ to increase diversity in boosting. TMS uses κ corrected by the empirical prediction marginals in the opposite direction, as evidence of cross modal support, and this correction alone contributes 4.69 points at zero shot. Eq. 9 is the one coin Dawid Skene model, and FlyingSquid estimates source accuracies with triplet methods. These label models answer how accurate each source is. TMS asks whether adding a modality improves the fused decision. For this reason, the rank one model serves only as auxiliary calibration, and Stage 3 compares coalitions with and without a modality against held out references. Dawid Skene selection reaches 54.91% at zero shot, below TMS.
The rules also follow from an identifiability analysis. Under the one coin model with conditional independence and uniform errors, κ between modalities i and j equals q_i q_j, where q denotes chance corrected competence. The strongest partner score of modality m is therefore q_m times the largest competence among the other modalities, and Stage 2 keeps exactly the modalities whose competence is at least τ times that of the second strongest modality. With only two modalities, agreement identifies only the product of their competences. This matches the triplet requirement in FlyingSquid and explains why Stage 3 requires at least three modalities. Removing at most one modality keeps the decision conservative when all evidence comes from a single unlabeled batch. All thresholds are fixed to 0.5 across datasets, adaptation budgets, and corruption conditions without tuning. Each rule is supported by the ablation: removing coverage screening, replacing the empirical marginal correction with a uniform baseline, or removing the marginal assessment lowers zero shot accuracy to 52.68%, 54.10%, and 51.92%, respectively.
Response to Reviewer FgHG
W2 and Q3. Two modalities and shared failures
With two modalities, the identifiability analysis above shows that pairwise agreement determines only the product of the two competences. No method based on agreement can decide which of two disagreeing modalities is reliable, so TMS then relies on coverage screening. We evaluated all two modality pairs, 10 on MM-Fi and 3 on XRF55. At zero shot, TMS still removes the collapsed modality and improves the average accuracy from 24.61% to 30.51%. After adaptation, TMS selects the same subset as full fusion in every pair, so it is never worse than full fusion. TMS is designed for sensing systems with three or more modalities, which is the common setting in multimodal sensing, with five modalities in MM-Fi and three in XRF55. We will state this scope explicitly.
For shared failures, we test both cases the reviewer suggests. In the joint corruption case, two modalities are degraded by the same type of corruption.
Table R5. Accuracy in % when mmWave and WiFi are both moderately corrupted.
| Setting | Full fusion | TMS |
|---|---|---|
| XRF55, one shot | 10.37 | 12.84 |
| XRF55, two shots | 13.30 | 19.12 |
| MM-Fi, zero shot, input level corruption | 14.63 | 27.12 |
TMS improves over full fusion in all three cases. In the duplication case, we copy the mmWave classifier as an additional modality, which is the worst case of perfectly correlated errors. The original rule keeps both copies in all 45 settings across datasets, transfers, and shot numbers, and on XRF55 with one shot this removes WiFi and lowers accuracy from 57.48% to 38.19%. Merging modality pairs whose κ exceeds 0.90 before computing the support scores restores the selection obtained without the duplicate in all 45 settings. It does not change any selection on clean data, where the largest pairwise κ is 0.65. We will include this step in the revised method.
W4. Significance of the gains after adaptation
The gains after adaptation are bounded by the oracle, which uses target labels under the same fusion rule. Table R6 reports a sign test over the 12 MM-Fi transfers, with ties excluded.
Table R6. MM-Fi accuracy in % and sign test of TMS against full fusion over 12 transfers.
| Shots | Full fusion | TMS | Oracle | Wins / ties / losses | One sided p |
|---|---|---|---|---|---|
| 0 | 32.16 | 58.79 | 59.14 | 12 / 0 / 0 | 0.00024 |
| 1 | 86.00 | 86.88 | 86.88 | 11 / 1 / 0 | 0.00049 |
| 2 | 87.34 | 87.69 | 87.91 | 10 / 2 / 0 | 0.00098 |
With one shot, the oracle also reaches 86.88%, so TMS captures all of the available headroom. With two shots, it captures 0.35 of the 0.57 points available. On XRF55, TMS improves full fusion by 2.62 and 2.87 points with one and two shots and equals the oracle in both cases. TMS never loses to full fusion on any transfer. To measure the variance over support sets, we keep the source encoders fixed and draw the support set again with two independent random seeds on all 12 MM-Fi transfers.
Table R7. MM-Fi accuracy in % for different support draws, averaged over 12 transfers.
| Support draw | One shot full fusion | One shot TMS | Gain | Two shot full fusion | Two shot TMS | Gain |
|---|---|---|---|---|---|---|
| Original split | 86.00 | 86.88 | 0.88 | 87.34 | 87.69 | 0.35 |
| Random draw, seed 101 | 89.05 | 89.87 | 0.83 | 93.18 | 93.53 | 0.36 |
| Random draw, seed 303 | 88.75 | 89.65 | 0.90 | 92.90 | 93.27 | 0.37 |
The original split uses a fixed deterministic choice of support samples, and random draws give higher absolute accuracy. The gain of TMS over full fusion, however, is stable across all support sets, at 0.83 to 0.90 points with one shot and 0.35 to 0.37 points with two shots. The two random draws also select exactly the same subset on all 12 transfers. In addition, when the query set is resampled with bootstrap, the selected subset remains unchanged in 34 of 36 MM-Fi settings.
Response to Reviewer FgHG
We thank the reviewer for the detailed and insightful review, and for recognizing the strict protocol, the large zero shot improvement, the fact that TMS is never worse than full fusion in all reported settings, and the transparency of our reporting. We first answer the question that the reviewer identified as most important.
Q1 and W3. Single modality accuracy and fixed policies
Tables R1 and R2 report the accuracy of every single modality together with full fusion, the leave one transfer out fixed subset, TMS, and the oracle. The leave one transfer out fixed subset is the subset with the best mean accuracy on the other transfers of the same dataset, selected with their target labels and then applied to the held out transfer. On MM-Fi, the other transfers include transfers into the same target environment, so this fixed subset is selected with labels from the target environment itself. It is therefore an optimistic reference for fixed policies. Results for every individual transfer will be included in the appendix.
Table R1. MM-Fi accuracy in %, averaged over 12 transfers.
Table R2. XRF55 accuracy in %, averaged over three target scenes.
Without adaptation, mmWave is the dominant reliable modality on clean data. TMS identifies it without any labels and adds depth or LiDAR in the transfers where they help, reaching 58.79% compared with 57.66% for mmWave alone. After adaptation, the best subset changes to RGB, mmWave, and LiDAR, and every single modality falls far below TMS. TMS matches or exceeds the leave one transfer out fixed subset in every setting without using any labels. The main value of test time selection appears when the sensor condition changes after a policy has been fixed, as Table R3 shows.
Table R3. Accuracy in % when sensors fail after the policy has been fixed.
The leave one transfer out fixed subset is competitive on clean data, but it is chosen before deployment and cannot react when a sensor fails. It degrades together with the failed sensor, while TMS detects the failure on the target batch and selects the remaining modalities. No single fixed policy is suitable across adaptation budgets and sensor conditions, and this is the gain that test time selection provides.