acceptodds
Withdrawn from ICLR 2027

Beyond Fusion Confidence: Adaptive Three-Way Calibration for Semi-Supervised Multimodal Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Semi-supervised multimodal learning often uses fusion confidence to select or weight pseudo-labels. Yet predictions with similar confidence can have different correctness depending on whether modality-specific heads support the predicted class. We propose Adaptive Three-Way Calibration (ATC), a teacher–student framework that distinguishes full, partial, and absent support as Accept, Boundary, and Reject regions. These regions condition pseudo-label admission, weighting, and target construction. Reliability estimated by replaying labeled training examples further controls target concentration and confidence alignment, separating the teacher's class proposal from its training-time reliability. Experiments on five heterogeneous benchmarks compare ATC with nine semi-supervised and multimodal baselines under a common feature and optimization protocol. ATC ranks first or second in 19 of 20 primary dataset–metric comparisons. Confidence-bin diagnostics reveal both regional correctness stratification and distinct sample-count trends. CUB ablations show that regional supervision accounts for most of the classification gain, while retaining Boundary yields lower mean ECE than either binary partition. Together, these results support using modality-head evidence to condition pseudo-label reliability rather than relying on fusion confidence alone.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Semi-supervised Learning, Multimodal Learning, Model Calibration, Three-way Decisions

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KSA8OpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies pseudo-label reliability in semi-supervised multimodal learning. The authors observe that samples with similar fusion confidence can have substantially different correctness depending on whether modality-specific heads support the fused prediction. Based on this observation, they propose Adaptive Three-Way Calibration (ATC), which partitions unlabeled samples into Accept, Boundary, and Reject regions according to modality-head support. These regions control pseudo-label admission, weighting, and target construction. The method further estimates region-level reliability by replaying labeled training examples and uses the resulting targets for pseudo-label shaping and confidence alignment. Experiments are conducted on five multimodal benchmarks with several SSL and multimodal baselines, together with calibration analyses and ablation studies.

Critical strengths
  1. The paper identifies a meaningful limitation of fusion confidence in multimodal SSL: even at similar fusion confidence, modality-head support provides additional information about pseudo-label correctness.

  2. The evaluation goes beyond final classification accuracy and includes calibration metrics, confidence-conditioned reliability analysis, regional diagnostics, and ablation studies.

Critical weaknesses
  1. The necessity of the full three-way formulation is not convincingly established. In the CUB ablation, binary support partitions already recover nearly all of the classification gain, while Full ATC provides only a very small additional improvement in ACC/F1. Moreover, several gains over the strongest baseline are small, and the paper reports only three seeds without significance tests or confidence intervals.

  2. I have concerns about the validity and reproducibility of the baseline comparisons. In the released supplementary code, the MixMatch, CoMatch, and SimMatch branches appear to omit important components of the original methods, and some of these branches use averaged modality-head predictions rather than the EMA fusion-head prediction described in the paper. If the main results in Table 1 were produced by these implementations, this would materially affect the interpretation of the empirical comparison.

Questions for authors
  1. Were the MixMatch, CoMatch, and SimMatch results in Table 1 generated using the corresponding branches in the released supplementary code? If yes, please explain how these implementations preserve the key mechanisms of the original methods. If not, please provide the exact implementations and configurations used for the reported results.

  2. Can the authors provide results comparing the three-way partition against simpler alternatives, such as binary support partitioning or a continuous support-ratio formulation, across multiple datasets? This would help establish whether the Boundary region is genuinely necessary.

  3. Can the authors evaluate regional reliability using cross-fitting or a held-out calibration subset rather than replaying labeled examples that were already used for model training? I am concerned that the current replay procedure may yield optimistic in-sample reliability estimates.

  4. How were the dataset-specific values of τg, wg, bg, λu, and λcal selected? Please report the search space, validation criterion, tuning budget, and whether comparable tuning resources were provided to the baselines.

Reviewer_pJyMOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Adaptive Three-Way Calibration (ATC) for semi-supervised multimodal learning. An EMA teacher generates fusion pseudo-labels, while modality-specific heads identify full, partial, or absent support for the proposed class. These patterns define Accept, Boundary, and Reject regions that regulate pseudo-label admission, weighting, and target construction. Regional reliability is estimated by replaying labeled training examples after warm-up and is used to shape supervision and align student confidence. Experiments on five benchmarks show competitive classification and calibration performance. However, the justification of the confidence-alignment objective and the evidence supporting the additional three-way and adaptive components remain insufficient.

Critical strengths
  1. The empirical motivation is useful. Predictions within the same fusion-confidence bin exhibit different correctness rates across modality-support regions, suggesting that modality-head evidence provides information beyond fusion confidence.
  2. The evaluation covers five heterogeneous benchmarks under a shared feature and training protocol and includes both classification and calibration metrics, providing useful evidence for support-conditioned pseudo-label supervision.
Critical weaknesses
  1. The confidence-alignment objective is insufficiently justified. Equation (8) estimates teacher pseudo-label correctness, whereas Equation (12) supervises the student’s maximum prediction probability. When teacher and student predict different classes, these quantities concern different correctness events. Appendix B.4 does not resolve this mismatch, and replay on training examples may further produce optimistic reliability targets.
  2. The methodological novelty appears incremental, combining a conventional teacher–student framework with support-based grouping, regional weighting, target shaping, and confidence regularization. The adaptive component is also limited: reliability targets are estimated once after warm-up and then frozen, while thresholds, weights, and blending coefficients remain fixed.
  3. Several improvements over the strongest competing baselines are modest. Table 1 reports ACC gains of only 0.01 percentage points on Food101-MM, 0.36 on DVM, 0.41 on PAD-UFES20, and 0.39 on MOSI. Although CUB shows a clearer improvement, the blanket claim of remarkable outperformance is stronger than the overall evidence supports.
  4. The statistical evidence does not adequately establish robustness under limited supervision. All three random seeds use the same labeled/unlabeled split, so the reported standard deviations capture training randomness but not uncertainty from selecting different small labeled subsets. Paired confidence intervals or significance analyses are also absent.
  5. The ablations do not convincingly establish the necessity and independent contributions of the additional components. They are restricted to CUB and omit uncertainty estimates. Full ATC improves ACC over the stronger binary partition by only 0.09 percentage points, while replay reduces ECE from 2.00 to 1.96. Thresholding, weighting, blending, shrinkage, and ordering are not sufficiently isolated.
  6. Presentation and reproducibility need improvement. Figures 1–2 contain notation inconsistencies, and the method, checkpoint, and data split underlying Figure 1 are unclear. Important controller settings and configuration-selection details remain unspecified. ECE lacks a formal definition, and parts of the experimental discussion and conclusion repeat earlier material.
Questions for authors
  1. Under what assumptions can teacher pseudo-label correctness serve as a target for the student’s maximum prediction probability? Please report teacher–student disagreement rates and calibration results for agreement and disagreement cases, and quantify potential bias from estimating reliability on training examples.
  2. Can the binary-versus-three-way and fixed-versus-replay ablations be repeated on additional datasets with uncertainty estimates? How does the three-way partition compare with directly using the continuous proportion of supporting modality heads?
  3. Can the main comparisons be evaluated over independently sampled labeled subsets, with paired confidence intervals against the strongest baselines? This would help establish whether the smaller gains persist under low-label sampling uncertainty.
  4. Please specify the warm-up duration, regional priors, minimum replay counts, regional bounds, calibration scope, and exact validation rule for selecting target placement. Please also report the selected configuration, class count, and actual labeled sample count for each dataset.
  5. Which method, checkpoint, and data split produced Figure 1? Please reconcile the figure notation with the equations, distinguish diagnostic confidence bins from pooled training targets, and clarify the definition and averaging procedure of ECE.
Reviewer_n5gkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a limitation of semi-supervised multimodal learning methods that rely only on fusion-head confidence to select pseudo-labels: predictions with similar confidence may have substantially different correctness depending on how strongly they are supported by the modality-specific heads. The authors propose Adaptive Three-way Calibration (ATC), which partitions unlabeled examples into Accept, Boundary, and Reject regions according to whether all, some, or none of the modality heads support the fusion prediction. These regions control pseudo-label admission, weighting, and target construction. The method also replays a small number of labeled training examples to estimate empirical regional reliability, while shrinkage and monotonic projection are used to stabilize sparse estimates. Experiments on five multimodal datasets show that ATC generally outperforms a range of SSL and multimodal baselines in classification, F1, ECE, and Brier score. The ablation suggests that region-aware supervision produces most of the classification improvement.

Critical strengths

The motivation is clear and directly supported by empirical evidence. Rather than merely asserting that fusion confidence is unreliable, the paper demonstrates substantial correctness differences among support regions within the same confidence bins. Treating partial modality agreement as a separate Boundary condition is more reasonable than collapsing the evidence into a binary agreement/disagreement partition and provides a meaningful degree of novelty. The evaluation covers five types of multimodal tasks, several label ratios, and a broad set of classification and calibration metrics. Results are reported as means and standard deviations over three runs. The fact that only the fusion head is retained at inference time, with no additional inference overhead, is also practically attractive.

Critical weaknesses

The evaluation protocol does not yet fully establish that the improvements arise from the proposed method itself. The feature extractors are frozen, so it remains unclear whether ATC is effective under end-to-end training or with stronger pretrained backbones. The central ablations are conducted mainly on CUB, without demonstrating that the contribution of each component is consistent across datasets. Some improvements are comparable to or smaller than the reported standard deviations, and the PAD-UFES20 F1 variance is particularly large, yet no statistical significance tests are provided. Reliability targets are estimated by replaying examples from the training set, which may introduce optimistic bias. Correlated modality errors may also cause examples in the Accept region to be confidently wrong. Finally, the region-specific thresholds, weights, and shrinkage strengths introduce additional tuning flexibility, and the paper does not fully establish that all baselines received an equivalent tuning budget.

Questions for authors
  1. Could the authors report the binary-partition, three-way-partition, no-replay, and no-calibration ablations on all five datasets rather than only on CUB?
  2. How were the regional thresholds, weights, and shrinkage strengths selected? Were any test results used, and were these quantities tuned separately for each dataset? Did the baselines receive the same hyperparameter-search budget?
  3. Could the authors provide paired significance tests or confidence intervals, especially for settings where the gains are small or the variance is large?
  4. Do the conclusions remain unchanged if regional reliability is estimated from an independent validation set rather than by replaying labeled training examples?
  5. How does ATC behave under end-to-end fine-tuning, missing modalities, modality corruption, or strongly correlated errors across modalities?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.