acceptodds
Under review as a conference paper at ICLR 2027

CHEC: Counterfactual Low-Rank Adaptation with Historical Evidence Correction for Continual Missing Modality Learning

Abstract

Continual Missing Modality Learning (CMML) is important for multimodal systems that must continually acquire new knowledge while remaining reliable under intermittent modality absence. However, existing methods still face two limitations that require further improvement in performance: (1) they mainly compensate for missing-side information, without explicitly adapting the surviving modality after its cross-modal context changes; and (2) preserving historical knowledge does not ensure that its contribution to the current prediction is supported by the current input. To address these problems, this paper proposes CHEC, a novel CMML framework combining Counterfactual Low-Rank Adaptation with Historical Evidence Correction. During training, CHEC adds a controller conditioned on the current sample and modality availability to LoRA, allowing the transformation applied to the surviving modality to adapt to each input. It further introduces Utility-Gated Counterfactual Functional Correction (UG-CFC), which projects the functional difference between paired Complete and Missing views into a counterfactual correction attainable in the Missing-view rank space, and retains it only when the correction is predicted to benefit classification. This resolves the mismatch between the desired functional change and what the Missing representation can realize, while filtering harmful corrections without increasing LoRA rank. During inference, Historical Evidence Correction (HEC) replaces task-level hard routing with class-wise evidence correction. It retains all learned classes and calibrates historical evidence according to support from the current input, avoiding both premature class exclusion and unsupported contributions from historical knowledge. Across two existing CMML benchmarks, CHEC achieves the state-of-the-art performance among six baselines in all 30 settings. Further analyses show relative reductions of 14.0–40.4% in the Complete–Missing functional gap and 27.4–32.2% in the final historical-class error rate.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Missing Modality Learning, Low-Rank Adaptation, Counterfactual Functional Supervision, Historical Evidence Correction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qa9xOpenReview ↗
Rating
1
Confidence
3(−1)
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CHEC, a framework for Continual Missing Modality Learning (CMML), to address two challenges: adapting the surviving modality when other modalities become unavailable and mitigating unreliable contributions from historical knowledge during task-agnostic inference. Specifically, the authors introduce Conditional LoRA and Utility-Gated Counterfactual Functional Correction (UG-CFC) during training, leveraging the functional differences between Complete and Missing views to guide adaptation. During inference, Historical Evidence Correction (HEC) adjusts historical-class prediction scores based on the support provided by the current input features.

Critical strengths

UG-CFC integrates paired Complete/Missing views, rank-space projection, and utility gating to guide conditional adaptation. This design is technically well-motivated and provides a meaningful approach to improving feature adaptation under missing-modality conditions.

Critical weaknesses

1. Insufficient validation of the reliability and statistical assumptions underlying HEC. HEC applies non-positive log-support corrections exclusively to historical classes, while leaving current-task prediction scores unchanged. Consequently, even when an input is consistent with the feature distribution of a historical class, its correct prediction score may still be suppressed. Although the paper demonstrates a reduction in historical-class prediction errors, it does not sufficiently investigate whether these improvements come at the cost of increased false negatives or reduced recall for historical classes.

Furthermore, HEC uses shared class statistics and calibration distance distributions across different modality-availability states. However, different missing-modality conditions may induce distinct feature distributions, potentially resulting in inaccurate support estimates for certain samples under shared calibration. Although UG-CFC reduces the functional gap between Complete and Missing views, this does not directly establish that the final classification feature distributions are sufficiently aligned. These concerns leave the claimed reliability of HEC insufficiently validated.

2. Insufficient evidence for the independent contribution of key components and the performance advantages over recent methods. The current ablation studies primarily focus on UG-CFC and HEC, while all configurations retain Conditional LoRA. Without a direct comparison between standard LoRA and Conditional LoRA, it is difficult to determine the independent contribution of the dynamic controller. Additionally, the authors employ Stop-Gradient to separate the gradient update paths of the original LoRA and the conditional controller, but the necessity of this design has not been sufficiently demonstrated.

Meanwhile, the experimental comparisons do not adequately cover recent relevant studies. Considering that CHEC achieves average AP improvements of only 1.73 and 0.63 percentage points over the strongest baseline, DeLo, the lack of more comprehensive comparisons with recent methods weakens the evidence for its performance advantages while most of baselines are works in 2024, 2025. Together, these limitations make it difficult to fully assess the novelty and practical contributions of the proposed framework.

Questions for authors

Q1. HEC applies one-sided negative corrections exclusively to historical classes. Could the authors provide historical-class Precision, Recall, and False Negative Rate before and after applying HEC to clarify whether the reduction in historical-class errors comes at the cost of suppressing correct historical predictions? Additionally, does the class-wise correction in HEC provide clear advantages over a simple uniform penalty applied to historical-class logits?

Q2. HEC uses shared class statistics and calibration distance distributions across different modality-availability states. Could the authors compare shared calibration with modality-specific calibration to validate the appropriateness of this design?

Q3. Could the authors provide a direct ablation comparing standard LoRA with Conditional LoRA and further clarify the advantages of the Stop-Gradient design over joint optimization without gradient separation?

Q4. Given the relatively modest performance improvements over the strongest baseline, could the authors include comparisons with more recent representative methods under consistent experimental settings to better establish the competitiveness of CHEC?

Q5. CHEC introduces additional counterfactual correction and gradient computations during training and evaluates all learned task experts during inference. Could the authors provide comparisons of training time, inference latency, and memory consumption to better assess the trade-off between computational cost and performance gains?

5 comments

W1 — Statistical assumptions and historical errors

Thank you for asking how historical recall and pooled calibration behave. At 70% Mixed, the first table reports historical diagnostics. The second table compares distance references by availability state on the same final expert features.

Metric at 70% Mixed (%)Native all-headUniform penaltyHEC
Food101 historical macro-recall66.9566.4967.36
MM-IMDb Task 5 historical micro-recall29.8027.9328.64
MM-IMDb Task 5 historical micro-FNR70.2072.0771.36
MM-IMDb Task 5 historical micro-FPR5.203.943.48

On Food101, HEC improves historical recall. On MM-IMDb, it raises FNR relative to native inference, but retains more recall at lower FPR than the uniform penalty.

CalibrationFood101 AP ↑Food101 FG ↓MM-IMDb AP ↑MM-IMDb FG ↓
Fully shared69.8812.6729.8429.93
Shared statistics, state-specific distance references69.7412.5329.9429.88
Fully state-specific69.6012.6129.8729.84

State-specific fitting changes AP only modestly in this fixed-expert comparison. Text-missing positives more often have low support, yet pooled HEC improves AP in every availability state on both datasets. The calibration evidence comes directly from the final expert features.

We have clarified empirical support in Section 4.3, Equation (8), and added the historical error and calibration comparisons in Appendix I, Tables 20 and 28.

W2 — Independent contributions and recent comparisons

Thank you for asking us to isolate Conditional LoRA and Stop-Gradient and extend our baseline comparison. At 70% Mixed, the controller ablation fixes rank and inference. The gradient ablation fixes objective and parameter count and reports separate minus joint updates to A/B over three paired seeds (mean ± sample SD).

ControllerFood101 AP ↑MM-IMDb AP ↑
Static LoRA59.8725.32
State only (capacity matched)61.1425.88
Sample + state61.7326.05

Conditional LoRA improves AP over static LoRA on both datasets. The capacity-matched comparison also supports conditioning on individual inputs.

DatasetΔAP (pp)ΔFG (pp)
Food101+1.27 ± 0.07−1.16 ± 0.07
MM-IMDb+0.24 ± 0.06+0.40 ± 0.09

Gradient separation improves paired mean AP on both datasets. The FG changes show that the benefit is strongest for AP across the two datasets.

DatasetSettings with higher CHEC AP than adapted MoRA (seed 42)
Food10115 / 15
MM-IMDb15 / 15

We adapted MoRA to the same continual learning protocol, sharing the frozen ViLT backbone, splits, masks and all-head HEC inference while keeping each method’s own adapter configuration. CHEC’s AP exceeds that of adapted MoRA in every reported seed-42 setting on both datasets.

We have added the controller ablation in Section 5.3, Table 3, and the gradient controls in Appendix G, Tables 11 and 13. We have added the MoRA comparison in Section 5.2, Tables 1 and 2, and described the adaptation and shared inference protocol in Appendix B.

Q1 — Historical Precision, Recall and FNR versus a uniform penalty

Thank you for requesting this breakdown. Food101 uses macro metrics over historical classes. MM-IMDb uses micro metrics over Task 5’s 16 historical labels. The uniform penalty is tuned on validation data.

DatasetInference rulePrecision (%)Recall (%)FNR (%)
Food101Native all-head79.7566.9533.05
Food101Uniform penalty81.2066.4933.51
Food101HEC84.3567.3632.64
MM-IMDbNative all-head30.9029.8070.20
MM-IMDbUniform penalty35.6227.9372.07
MM-IMDbHEC39.1128.6471.36

Relative to native inference, HEC improves both precision and recall on Food101. On MM-IMDb it gains 8.21 precision points while giving up 1.16 recall points. Against the uniform penalty, it improves both metrics on both datasets, supporting more selective attenuation.

We have added the historical precision, recall, and FNR results for native inference, HEC, and the uniform penalty in Appendix I, Table 20.

Q2 — Shared versus state-specific calibration

Thank you for suggesting this control. All three calibration variants use the same trained experts.

CalibrationFood101 AP ↑Food101 FG ↓MM-IMDb AP ↑MM-IMDb FG ↓
Fully shared69.8812.6729.8429.93
Shared statistics, state-specific distance references69.7412.5329.9429.88
Fully state-specific69.6012.6129.8729.84

Fully shared calibration gives the highest Food101 AP, while state-specific distance references improve MM-IMDb AP by 0.10 points. Sharing retains more validation positives per class reference and gives comparable AP in this fixed-expert comparison. The state-specific variants also give slightly lower FG than pooling on both datasets.

We have added the comparisons of shared and state-specific calibration in Appendix I, Table 28.

Q3 — Standard LoRA and Stop-Gradient versus joint optimization

Thank you for suggesting direct controls for both design choices. The controller test fixes rank and inference. The paired gradient test fixes forward architecture and loss. In its joint control, the residual classification and UG-CFC gradients both reach A/B. The gradient table reports separated minus joint updates (mean ± sample SD).

ControllerFood101 AP ↑MM-IMDb AP ↑
Static LoRA59.8725.32
State only (capacity matched)61.1425.88
Sample + state61.7326.05

At fixed rank and inference, Conditional LoRA outperforms static LoRA in AP on both datasets.

DatasetΔAP (pp)ΔFG (pp)
Food101+1.27 ± 0.07−1.16 ± 0.07
MM-IMDb+0.24 ± 0.06+0.40 ± 0.09

Separating the two paths improves paired mean AP on both datasets over the joint control while leaving the auxiliary correction to the controller. FG also falls on Food101 and rises by 0.40 points on MM-IMDb.

We have clarified the gradient paths in Section 4.2, Equation (7), and added the controller and gradient controls in Section 5.3, Table 3, and Appendix G, Tables 11–13.

Q4 — Recent representative baselines under consistent settings

Thank you for suggesting a comparison with a more recent method. We adapted MoRA to the continual learning protocol. At 70% Mixed, both methods share the frozen ViLT backbone, task splits, missing-modality masks and all-head HEC inference. Each uses its own adapter rank and layer coverage.

DatasetMethodAP ↑FG ↓
Food101Adapted MoRA + HEC66.5611.27
Food101CHEC69.8812.67
MM-IMDbAdapted MoRA + HEC28.3232.26
MM-IMDbCHEC29.8429.93

Under the shared inference rule, CHEC improves AP by 3.32 and 1.52 points on Food101 and MM-IMDb, respectively. It also lowers MM-IMDb FG by 2.33 points, while Food101 FG rises by 1.40 points.

We have added the MoRA comparison across all settings in Section 5.2, Tables 1 and 2, and described the adaptation and shared inference protocol in Appendix B.

Q5 — Training time, inference latency and memory

Thank you for asking for measured costs. Both methods use matched optimizer steps at 70% Mixed, and the inference measurements use each method’s full final-stage pipeline.

MetricFood101 DeLoFood101 CHECMM-IMDb DeLoMM-IMDb CHEC
End-to-end training (h)12.9416.215.087.16
Single-sample latency (ms)21.47174.8326.38110.92
Peak training memory (GiB)10.7213.9413.4116.68
Peak inference memory (GiB)3.163.484.024.31

The absolute measurements show modestly longer training and a larger latency increase from the complete inference path.

Expert set, native all-head → HECFood101 ΔAP (pp)MM-IMDb ΔAP (pp)
CHEC+1.38+1.97

Training adds paired-view and utility-probe computation. The larger deployment cost includes evaluating every expert, which keeps all learned classes eligible. On those same experts, HEC improves AP by 1.38 points on Food101 and 1.97 on MM-IMDb.

We have added the training time, single-sample latency, and peak memory measurements in Appendix J, Table 31. Section 5.4, Table 5, reports AP on the same CHEC experts.

Dear Reviewer, thank you for your time and valuable feedback. We are writing to gently follow up on our submitted rebuttal. We wonder if our response has addressed your concerns and clarified the issues raised.

If you have any remaining questions, please feel free to let us know—we would be more than happy to provide further clarification. If our response has resolved your concerns, we would be very grateful if you could kindly consider updating your score.

Thank you again for your constructive review.

Reviewer_Px6qOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies continual learning for image-text classification when the image or the text can be missing during training and testing, and when the task identity is unknown at test time. It builds on a frozen ViLT model with separate LoRA adapters for each task. The method, CHEC, has three parts. A small controller network makes the LoRA adapter depend on each sample and on which modality is missing. A training loss, UG-CFC, pushes the adapter outputs of a missing-modality version of a sample toward those of the complete version, and keeps this loss only when it also helps classification. At test time, a rule called HEC runs all task models and lowers the scores of old models when the input does not look like their classes. On UPMC-Food101-CMML and MM-IMDb-CMML, CHEC reports the best accuracy among six baselines.

Critical strengths

The paper addresses a relevant and challenging problem in continual multimodal learning. The proposed framework is technically reasonable and combines training-time adaptation with inference-time correction in a coherent way. The results show consistent improvements across different missing-modality settings. The authors also provide useful ablations showing that the proposed components contribute to performance within their framework.

Critical weaknesses

W1 My main concern is the degree of novelty relative to recent work, particularly DeLo. Input-dependent low-rank adaptation and consistency learning under missing modalities have already been explored. The proposed counterfactual correction introduces a different formulation, but its conceptual distinction from existing alignment and consistency objectives is not sufficiently established. Similarly, the historical evidence correction relies on established distance-based calibration ideas. Overall, the method appears to combine and refine existing techniques rather than introduce a substantially new approach. A clearer discussion of the differences and advantages over closely related methods is needed.

W2 Although CHEC consistently outperforms the reported baselines, the margins over DeLo are relatively small, particularly on MM-IMDb. More importantly, the experiments do not convincingly demonstrate that these gains arise from the specific mechanisms proposed in the paper. The training and inference changes are not fully separated in the main comparison, and simpler alternatives are missing from the ablations. For example, it remains unclear whether the counterfactual correction offers a meaningful advantage over standard paired-view consistency learning or additional training on constructed missing-modality samples. The improvements could therefore be explained partly by additional supervision or a different inference strategy rather than the proposed correction mechanism itself.

W3 The paper claims that HEC suppresses unreliable historical evidence while preserving useful knowledge. However, since HEC systematically reduces scores for historical classes without similarly correcting the newest classes, its improvements may partly result from favoring recent tasks rather than identifying unreliable evidence. The proposed historical-class error metric does not adequately distinguish these explanations, particularly because reducing historical scores directly reduces certain types of false positives. Furthermore, HEC increases the reported forgetting on Food101. Stronger analysis of old-task performance and comparisons with simple score-calibration baselines are needed to establish that HEC works for the claimed reason.

W4 Most experiments and ablations are reported for a single random seed, with limited analysis of robustness across different settings. Given the small improvements over the strongest baseline, stronger statistical evidence is important. The paper also introduces additional training computation and evaluates multiple task-specific models at inference, unlike DeLo's task-selection approach. Without reporting computational and memory costs, it is difficult to assess whether the modest performance improvements justify the additional complexity.

Questions for authors

Q1) Can the authors provide stronger evidence that the proposed counterfactual correction outperforms simpler consistency-learning objectives, and that the improvement is not primarily due to additional training supervision?

Q2) How much of the improvement over DeLo comes from the training mechanism versus the inference strategy? Would applying HEC to DeLo produce similar gains?

Q3) Can the authors demonstrate that HEC genuinely improves the reliability of historical predictions rather than simply favoring recent classes, for example through old-class recall and comparisons with simple score penalties?

Q4) Can the authors provide more robust multi-seed results and a comparison of training and inference costs against DeLo?

6 comments

W1 — Novelty relative to DeLo and calibration

Thank you for asking us to clarify the contribution relative to existing alignment and calibration. DeLo aligns routing queries and outputs. RebQ reconstructs missing queries for prompt retrieval. Guo et al. and MoRA share cross-modal information through prompts and low-rank parameters, respectively. UG-CFC specifies an attainable correction to the adapter’s action on surviving tokens, using paired views of the same instance and a classification-gradient test. HEC addresses which historical scores to attenuate when all classes compete, using a separate validation-positive reference for each class.

Controlled comparison at 70% MixedFood101 ΔAP (pp)MM-IMDb ΔAP (pp)
UG-CFC vs. logit distillation, same views and inference+1.08+0.46
HEC vs. uniform penalty, fixed experts+0.59+0.82

The matched-view results support choosing a realizable correction to the surviving-token adapter action, with the utility gate testing whether that correction is predicted to help classification. The fixed-expert results support input- and class-specific attenuation when all learned classes compete. These comparisons separately support the training and inference decisions that distinguish CHEC.

We have clarified the training and inference mechanisms in Section 4.2, Equations (2)–(5), and Section 4.3, Equations (8) and (9). We have added their controlled comparisons in Sections 5.3 and 5.4, Tables 3 and 6.

W2 — Mechanism attribution and simpler alternatives

Thank you for asking us to separate training and inference effects. The training comparison uses the same paired views, controller and inference. The inference comparison holds each set of experts fixed.

ComparisonFood101 ΔAP (pp)MM-IMDb ΔAP (pp)
UG-CFC vs. missing-view classification+2.32+0.84
UG-CFC vs. logit distillation+1.08+0.46
Add HEC to CHEC experts+1.38+1.97
Add HEC to DeLo experts+2.19+1.99

The matched-view controls give UG-CFC an AP advantage even with the same supervision inputs. The inference gains are similar across the two expert sets on MM-IMDb.

MM-IMDb inference at 70% MixedAP ↑FG ↓
DeLo + HEC29.5530.30
CHEC29.8429.93

Under shared HEC, CHEC still gains 0.29 AP points and lowers FG by 0.37 points on MM-IMDb. The smaller same-view training gain over distillation and the similar inference gains on both expert sets help explain the modest AP margin.

We have added the same-view controls in Section 5.3, Table 3, and the common-HEC comparison in Section 5.4, Table 6. We have discussed the smaller MM-IMDb margin in Appendix G.

W3 — Historical reliability and recency bias

Thank you for raising the recency explanation. On Food101, the current task accounts for about 73% of the AP gain, and current-class intrusion increases from 19.30% to 21.43%.

Post-hoc Food101 diagnostic (matched test HCER 9.92%)Old-task accuracy (%)
Uniform penalty66.72
HEC67.71

At the same test HCER, HEC preserves 0.99 points more old-task accuracy than the uniform penalty in this post-hoc diagnostic, supporting input-dependent selection alongside the recency shift.

Correction rule, fixed experts at 70% MixedFood101 AP ↑Food101 FG ↓MM-IMDb AP ↑MM-IMDb FG ↓
Symmetric HEC69.6412.8529.6630.19
Original HEC69.8812.6729.8429.93

On Food101, FG rises from 8.02 to 12.67 as the mean old-task peak improves more than final old-task accuracy, which still rises from 67.30% to 67.71%. The original rule also gives higher AP and lower FG than the tested symmetric variant on both datasets. That variant changes the historical correction as well, so the comparison evaluates the complete rules.

We have revised Section 5.4 and added the AP breakdown and correction-rule comparisons in Appendix I, Tables 20–25 and 30.

W4 — Robustness and computational cost

Thank you for asking about stability and cost. The paired results use seeds 17, 42 and 73 (CHEC minus DeLo, mean ± sample SD). Costs compare matched training steps at 70% Mixed and each method’s full inference path.

DatasetMixed rateΔAP (pp)ΔFG (pp)
Food10170%+2.20 ± 0.23+3.19 ± 0.23
Food10190%+2.25 ± 0.26+3.12 ± 0.27
MM-IMDb70%+0.89 ± 0.15−1.88 ± 0.25
MM-IMDb90%+0.65 ± 0.11−1.58 ± 0.14

AP rises in every pair. MM-IMDb FG falls in every pair, whereas Food101 FG rises in every pair.

Metric at 70% MixedFood101MM-IMDb
CHEC / DeLo training time1.25×1.41×
CHEC / DeLo single-sample latency8.14×4.20×
Training peak memory increase (GiB)+3.22+3.27
Inference peak memory increase (GiB)+0.32+0.29
HEC vs. native all-head on CHEC experts (ΔAP, pp)+1.38+1.97

Training time increases by 25% on Food101 and 41% on MM-IMDb, with a larger latency increase for full all-expert inference. That scoring path keeps every learned class eligible. Within it, fixed-expert comparisons show that HEC adds 1.38 and 1.97 AP points, respectively.

We have added the paired results in Appendix E, Tables 7 and 8, and summarized the inference latency in Section 5.4, with measured costs in Appendix J, Table 31. Section 5.4, Table 5, reports the AP comparison on fixed CHEC experts.

Q1 — Evidence against augmented classification or consistency

Thank you for suggesting simpler objectives. All four variants use the same extra Missing views, controller and inference at 70% Mixed.

Paired-view supervisionFood101 AP ↑MM-IMDb AP ↑
Missing-view classification66.1827.03
Feature consistency66.7627.16
Logit distillation67.4227.41
UG-CFC68.5027.87

UG-CFC improves AP over all three objectives that already supervise the extra views, including gains of 1.08 and 0.46 points over logit distillation. With the view set held fixed, these gains support the choice of counterfactual adapter supervision.

We have added the comparisons with alternative objectives using the same constructed views in Section 5.3, Table 3.

Q2 — Training, inference and HEC on DeLo

Thank you for asking how the gains break down. At 70% Mixed, the training comparison holds native all-head inference fixed, while each inference comparison holds the trained experts fixed.

Controlled comparisonFood101 ΔAP (pp)MM-IMDb ΔAP (pp)
Add UG-CFC within CHEC+6.77+1.82
Add HEC to CHEC experts+1.38+1.97
Add HEC to original DeLo experts+2.19+1.99
CHEC vs. DeLo under shared HEC+0.78+0.29

Within CHEC, UG-CFC contributes the larger AP increment on Food101, and the training and inference increments are similar on MM-IMDb. HEC also improves fixed DeLo experts, with nearly the same MM-IMDb gain as on CHEC experts. After equalizing inference with HEC, CHEC retains AP gains of 0.78 and 0.29 points over DeLo.

We have reported the training and inference ablations in Appendix F, Table 9, and Section 5.4, Tables 5 and 6. We have added the DeLo crossover in Appendix G, Table 10.

Q3 — Reliability versus a uniform recency shift

Thank you for suggesting simple penalties. The experts are fixed at 70% Mixed, and penalty strengths are selected on validation data. MM-IMDb metrics cover the 16 historical labels on the Task 5 test set.

Inference ruleFood101 old-task accuracy (%)Food101 HCER (%)MM-IMDb micro-recall (%)MM-IMDb micro-FPR (%)
Native all-head67.3013.6729.805.20
Uniform penalty66.8410.6827.933.94
Task-wise penalty67.1610.1228.303.61
HEC67.719.9228.643.48

On Food101, HEC gives higher old-task accuracy and lower HCER than either fixed penalty. On MM-IMDb, HEC reduces recall relative to native inference. Compared with the two simple penalties, it retains more recall and yields fewer false positives. The fixed-expert results support choosing which historical scores to attenuate for each input.

We have added comparisons with simpler penalties in Section 5.4, Table 6, and historical error and operating-point results in Appendix I, Tables 20 and 24.

Q4 — Multi-seed results and training and inference costs

Thank you for asking for paired runs and runtime comparisons. We use paired seeds 17, 42 and 73 (CHEC minus DeLo, mean ± sample SD) and match training steps for the cost measurements at 70% Mixed.

DatasetMixed rateΔAP (pp)ΔFG (pp)
Food10170%+2.20 ± 0.23+3.19 ± 0.23
Food10190%+2.25 ± 0.26+3.12 ± 0.27
MM-IMDb70%+0.89 ± 0.15−1.88 ± 0.25
MM-IMDb90%+0.65 ± 0.11−1.58 ± 0.14

AP improves in every tested pair. FG falls on MM-IMDb and rises on Food101 across both rates.

MetricFood101 DeLoFood101 CHECMM-IMDb DeLoMM-IMDb CHEC
End-to-end training (h)12.9416.215.087.16
Single-sample latency (ms)21.47174.8326.38110.92
Peak training memory (GiB)10.7213.9413.4116.68
Peak inference memory (GiB)3.163.484.024.31

The additional single-sample latency accompanies the full all-expert inference path, which keeps all learned classes eligible.

Expert set, native all-head → HECFood101 ΔAP (pp)MM-IMDb ΔAP (pp)
CHEC+1.38+1.97

Within the all-expert path, HEC improves AP on the same trained CHEC experts. This isolates the score-correction gain while the reported latency compares the complete deployment paths.

We have added the paired runs in Section 5.2 and Appendix E, Tables 7 and 8, and the runtime comparison in Appendix J, Table 31. Section 5.4, Table 5, reports the fixed-expert ablation.

Reviewer_8vXBOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies Continual Missing Modality Learning (CMML): a vision-language classifier learns tasks in sequence, inputs may lack text or image at both training and test time, and task identity is unknown at test time. The authors identify two gaps. First, existing methods compensate for the missing modality but do not adapt how the surviving modality is processed once its cross-modal context changes. Second, preserving historical knowledge does not ensure that historical evidence is supported by the current input. CHEC addresses the first gap during training. Each task's per-modality LoRA is augmented with a sample- and availability-conditioned r×r controller, giving W h + B(I + Rₓ)Ah. The controller is supervised by Utility-Gated Counterfactual Functional Correction (UG-CFC). For each Complete training sample, the method constructs text-missing and image-missing views and computes the displacement between the Complete-view and Missing-view rank-space actions. It projects this displacement through a ridge regression onto directions the Missing-view coordinates can realize, and uses the result as a detached target for the controller, measured as a functional error through B. A first-order utility gate admits the correction only when its direction has a negative inner product with the classification gradient. Auxiliary gradients reach only the controller. At inference, Historical Evidence Correction (HEC) runs all task experts. For each historical expert, it computes a Mahalanobis distance to each class mean, converts it into an empirical p-value against validation distances, and adds log p to that class's evidence; current-task evidence is left uncorrected. On UPMC-Food101-CMML and MM-IMDb-CMML across 30 missingness settings, CHEC has the best AP in every setting, ahead of DeLo by 1.73 (Food101) and 0.63 (MM-IMDb) points on average. Diagnostic analyses show smaller Complete–Missing functional gaps and lower historical-class error rates.

Critical strengths

1. The paper frames the problem well. Separating "adapt the surviving modality" from "control unsupported historical evidence", and placing them at training and inference time respectively, is a useful decomposition for CMML.

2. UG-CFC is carefully constructed. The ridge projection gives a realizable target, the functional loss through sg(B) and sg(Z_M) ignores operator errors that do not change the output (Eq. 4 in the appendix makes this explicit), and the gradient separation keeps the auxiliary signal from altering the native LoRA or the classifier.

3. The pairing control is informative. Comparing same-label CFC with same-instance paired CFC (Table 3) cleanly separates instance correspondence from label correspondence and shows a clear gain on Food101.

4. HEC is a principled, training-free alternative to hard task routing. It keeps all candidate classes, uses a conformal-style empirical p-value, and corrects in one direction only. The paper also explains the counter-intuitive FG behavior through the decomposition into peak and final performance instead of hiding it.

Critical weaknesses

1. The base model behind CHEC is substantially weaker than DeLo, which raises the question of where the final margin comes from. In Table 6, the configuration with neither component (conditional adapter, native all-head inference) reaches 61.73 AP on Food101 at 70% Mixed, versus 67.42 for DeLo in Table 1. UG-CFC and HEC together add 8.15 points, yet CHEC ends only 2.46 points above DeLo. The paper needs to explain the 5.7-point deficit and should report DeLo + UG-CFC and DeLo + HEC. This would show whether the components are complementary to a strong CMML method or mainly recover what the native all-head setup loses.

2. The first claimed contribution, the conditional controller, is never ablated. Every row of Tables 3 and 6 keeps the conditional adapter. There is no comparison between static task LoRA and LoRA plus controller under the same training and inference. Whether making the LoRA action sample- and availability-dependent helps at all is therefore unverified.

3. A key control for UG-CFC is missing. The "compute-matched control" sets the CFC weight to zero, so the constructed Missing views receive no supervision. The obvious alternative uses the same constructed views as modality-dropout augmentation under Lcls, or as feature- or logit-level distillation from the Complete view. Because the utility gate already admits only corrections aligned with the classification gradient, UG-CFC may behave like filtered augmentation. Without these baselines, the specific value of the ridge-projected counterfactual target is not established. An ablation that removes the ridge projection (raw target D) is also needed to support the realizability argument.

4. HEC is asymmetric in a way that may hide errors. Only historical evidence is corrected (I[t < k]), so current-task classes keep their full evidence. In class-incremental settings this asymmetry usually increases task-recency bias. HCER counts only historical false predictions. It does not count old-task samples absorbed by current-task classes, which is the error HEC would increase. The Food101 FG rise from 8.02 to 12.67 is consistent with this. Please report the full accuracy matrix a_{i,j} and a symmetric current-class intrusion rate.

5. The calibration and the treatment of missing inputs in HEC are questionable. One covariance and one set of validation-distance distributions are shared across Complete, TM and IM inputs. Missing-modality features are shifted, so their Mahalanobis distances, and hence their p-values, will be systematically smaller. HEC may therefore suppress evidence for exactly the incomplete samples the paper targets. Please break down HEC's effect by availability state. Please also clarify whether baselines had access to the per-task validation data that HEC uses for calibration.

6. The statistical support is thin for small margins. All 30 main results use one seed (42). On MM-IMDb, 9 of 15 settings have AP margins over DeLo below 0.6 points, which is within the seed-to-seed spread one would expect from Table 5. Only one setting has three seeds. "State of the art in all 30 settings" should be supported by multi-seed means for at least a representative subset on both datasets. The paper should also state whether the DeLo numbers were reproduced under the same code and seeds.

7. The functional-gap diagnostic is circular. Figure 3 measures the normalized gap between the Complete-view and Missing-view LoRA outputs, which is close to the quantity UG-CFC directly minimizes. A reduction is therefore expected and is not independent evidence of better representations. A diagnostic linking the gap reduction to per-sample accuracy gains on Missing inputs would be more convincing.

8. Directly relevant baselines are missing. MoRA (missing-modality LoRA), PAL and Guo et al. (2025) are discussed as the closest related work but are not compared. For a LoRA-based CMML method, MoRA adapted to the continual protocol is the most direct comparison point.

Questions for authors

1. What fraction of (instance, missing-state) pairs does the utility gate admit, and how does this fraction change across epochs and tasks? If it is close to 0% or 100%, the gate's contribution in Table 3 needs another explanation.

2. Rₓ is shared by all tokens of a sample, site and modality, and the target ∆* is fitted over n aligned tokens. What is the typical rank of Z_M relative to r = 8? In other words, how often is the ridge problem effectively underdetermined?

3. In HEC, when expert t evaluates an input from a different task, its feature f_t(x) comes from the adapter of task t. Are the per-class distance distributions calibrated only on in-task validation data, so that out-of-task inputs are never part of the calibration? How does this affect the meaning of p as "support"?

4. Is the evidence scale e_{t,c} = z_{t,c} − log Σ_{c′≠c} exp z_{t,c′} comparable across experts trained in different stages, given that each expert has its own classifier head with different logit scales?

5 comments

W1 — DeLo baseline and component transfer

Thank you for suggesting these crossover tests. Both models use frozen ViLT. The table separates routing, training transfer and inference transfer at 70% Mixed.

Comparison (AP and ΔAP)Food101MM-IMDb
DeLo Task-Key → native all-head (AP)67.42 → 66.9128.97 → 27.56
DeLo paired-view control → DeLo + UG-CFC (native all-head inference in both)67.10 → 67.83 (+0.73)27.70 → 27.83 (+0.13)
Fixed original DeLo experts: native → HEC (AP, ΔAP)66.91 → 69.10 (+2.19)27.56 → 29.55 (+1.99)

Routing accounts for 0.51 of the 5.69-point Food101 AP gap between Task-Key DeLo and the conditional-only CHEC control (61.73 AP). The remaining 5.18 points compare the base models under the same native all-head inference. In DeLo, UG-CFC raises AP when the added controller, paired views and routing weights are fixed. HEC also raises AP on the original fixed experts. Both components therefore improve the DeLo configurations in which they were tested.

We have added the fixed-expert HEC comparison in Section 5.4, Table 6, and the matched transfer experiments in Appendix G, Table 10.

W2 — Static versus Conditional LoRA

Thank you for suggesting a direct controller ablation. Training, LoRA rank and native all-head inference are held fixed at 70% Mixed.

ControllerFood101 AP ↑MM-IMDb AP ↑
Static LoRA59.8725.32
State only (capacity matched)61.1425.88
Sample + state61.7326.05

Conditional LoRA improves AP over static LoRA by 1.86 points on Food101 and 0.73 on MM-IMDb. Adding sample information to the capacity-matched state-only controller gives a further 0.59 and 0.17 points, respectively. Shuffling conditions within each availability state also reduces paired mean AP by 0.55 and 0.14 points (Appendix G, Table 14), supporting the value of matching the condition to the input.

We have added controls for static LoRA, state-only conditioning, and sample-plus-state conditioning in Section 5.3, Table 3.

W3 — Simpler paired supervision and the raw target

Thank you for proposing these controls. At 70% Mixed we hold the controller, constructed views and native all-head inference fixed.

Paired-view supervisionFood101 AP ↑MM-IMDb AP ↑
Missing-view classification66.1827.03
Feature consistency66.7627.16
Logit distillation67.4227.41
Direct post-B matching + gate68.2227.75
UG-CFC68.5027.87

UG-CFC improves AP over the strongest conventional objective, logit distillation, by 1.08 points on Food101 and 0.46 on MM-IMDb under the same view construction. With the up-projection, features and detached target held fixed when computing the gradient, the unregularized pseudoinverse gives the same correction gradient as direct post-B matching. Holding the gate fixed, ridge fitting adds 0.28 and 0.12 AP points over that unregularized control, identifying the benefit of regularizing the target in this comparison.

We have added the same-view supervision and raw-target comparisons in Section 5.3, Table 3, and the ridge sensitivity analysis in Appendix G, Table 16.

W4 — Accuracy matrix and current-class intrusion

Thank you for asking for the complementary error direction. At the final Food101 stage in 70% Mixed, we compare the same trained experts under two inference rules. Current-class intrusion is the fraction of old-task test examples predicted as current-task classes, averaged equally over old tasks.

Metric (%)Native all-headHEC
Current-task accuracy79.2989.43
Old-task accuracy67.3067.71
Old-to-current intrusion19.3021.43
Old-to-other-historical error13.4010.86

FG rises from 8.02 to 12.67 because the mean old-task peak increases by 5.06 percentage points and the final old-task accuracy increases by 0.41. Current-class intrusion shows a recency shift, and Task 10 accounts for about 73% of the final AP gain. For old tasks, the 2.54-point decrease in errors into other old classes exceeds the 2.13-point increase in errors into current classes, leaving a net gain of 0.41 points in final old-task accuracy.

We have expanded Section 5.4. Appendix I, Table 21 reports the full Food101 task-by-stage accuracy matrix, and Table 23 reports current-class intrusion by stage.

W5 — Availability-specific calibration and validation access

Thank you for asking about the availability states and validation access. The methods in the controlled comparison had access to the same per-task validation splits. At 70% Mixed, we compare pooled HEC with native all-head inference on the same experts.

Availability stateFood101 ΔAP (pp)MM-IMDb ΔAP (pp)
Complete+1.42+1.57
Text missing+1.58+1.86
Image missing+1.16+1.74

Text-missing positives enter the low-support tail (p ≤ 0.05) more often than Complete positives. Even with that shift, pooled HEC improves AP relative to native inference in every availability state on both datasets, including both incomplete states.

We have described the shared splits and missingness assignments in Appendix B and added the calibration comparison and discussion by availability state in Appendix I, including Table 28.

W6 — Small margins and reproduction

Thank you for asking us to verify the smaller margins. The confirmed DeLo version, splits and masks reproduce the published seed-42 scores. The table shows CHEC minus DeLo (mean ± sample SD over paired seeds 17, 42 and 73).

DatasetMixed rateΔAP (pp)ΔFG (pp)
Food10170%+2.20 ± 0.23+3.19 ± 0.23
Food10190%+2.25 ± 0.26+3.12 ± 0.27
MM-IMDb70%+0.89 ± 0.15−1.88 ± 0.25
MM-IMDb90%+0.65 ± 0.11−1.58 ± 0.14

CHEC improves AP in all 12 paired comparisons across the four tested dataset–rate settings. FG falls in every MM-IMDb pair and rises in every Food101 pair. The smaller MM-IMDb gains thus recur across these seeds at both Mixed rates, while the full 30-setting comparison reports seed 42.

We have described the DeLo reproduction in Appendix B, summarized the paired MM-IMDb results in Section 5.2, and added the individual runs and paired summaries for both datasets in Appendix E, Tables 7 and 8.

W7 — Functional-gap diagnostic and classification gains

Thank you for requesting a diagnostic tied to classification. We compare paired validation predictions. Each missing state has 2,043 Food101 examples or 30,420 MM-IMDb seen-label decisions. The Spearman correlations relate gap reduction to margin improvement.

DatasetStateWrong → RightRight → WrongNet gain (pp)Spearman
Food101TM16784+4.060.19
Food101IM12971+2.840.16
MM-IMDbTM1,4751,150+1.070.11
MM-IMDbIM1,2841,117+0.550.09

Across both missing states, more predictions or seen-label decisions switch from wrong to right than from right to wrong. This provides classification-level evidence alongside Figure 3’s functional-gap reduction. The modest Spearman correlations indicate a limited association with per-input margin improvement.

We have added the paired prediction transitions and margin correlations in Appendix H, Table 19, alongside the functional-gap analysis in Section 5.3, Figure 3.

W8 — Recent missing-modality baselines

Thank you for pointing out the missing comparisons with recent work. PAL combines prompting with an analytic classifier, and Guo et al. use prompt-based adaptation. As you noted, MoRA is the most direct comparison for CHEC’s low-rank adaptation. We adapted MoRA to our continual learning protocol while preserving its modality-specific factors and cross-modal Gram interactions. At 70% Mixed, both methods share the frozen ViLT backbone, task splits, missing-modality masks and all-head HEC inference. Each uses its own adapter rank and layer coverage.

DatasetMethodAP ↑FG ↓
Food101Adapted MoRA + HEC66.5611.27
Food101CHEC69.8812.67
MM-IMDbAdapted MoRA + HEC28.3232.26
MM-IMDbCHEC29.8429.93

With the same backbone and all-head HEC inference, CHEC improves AP over the adapted MoRA by 3.32 points on Food101 and 1.52 on MM-IMDb at 70% Mixed. MM-IMDb also improves by 2.33 FG points, while Food101 FG increases by 1.40 points. The AP advantage covers both benchmarks under the tested adapter configurations.

We have added the MoRA results for all settings in Section 5.2, Tables 1 and 2, and described the adaptation and shared inference protocol in Appendix B.

Q1 — Fraction admitted by the utility gate

Thank you for asking how selective the gate is. We aggregate eligible instance–missing-state pairs at 70% Mixed.

DatasetAggregationTM accepted (%)IM accepted (%)
Food101Pooled46.8453.67
Food101First task40.9047.40
Food101Final task51.1060.60
MM-IMDbPooled47.0953.04
MM-IMDbFirst task41.2046.30
MM-IMDbFinal task49.1555.70

Pooled acceptance is close to half. Across tasks, it rises through epoch 7 and changes by at most 0.04 points thereafter. From epoch 1 to 7, the TM/IM rates move from 38.84%/46.22% to 49.85%/56.47% on Food101 and from 39.89%/46.49% to 49.79%/55.51% on MM-IMDb.

ComparisonFood101 ΔAP (pp)MM-IMDb ΔAP (pp)
Utility gate vs. acceptance-matched random gate+1.32+0.38

At the same acceptance rate, gradient-based selection yields 1.32 and 0.38 more AP points than random selection on Food101 and MM-IMDb. The gain therefore comes from which corrections the gate selects under this control.

We have added pooled gate acceptance rates and discussed their variation across tasks and epochs in Appendix H, including Table 17. We have added the acceptance-matched random-gating control in Appendix G, Table 15.

Q2 — Rank of the Missing coordinates

Thank you for asking when the local fit is underdetermined. The typical rank of Z_M is eight.

DatasetRank-deficient TM (%)Rank-deficient IM (%)
Food1010.180.90
MM-IMDb0.050.26

Rank deficiency occurs in at most 0.90% of the reported sample–site pairs. A positive ridge coefficient gives a unique fit in those cases. When n > 8, even full row rank restricts the fitted displacement to an eight-dimensional row space within the n-dimensional token space. The resulting target is therefore constrained to changes the Missing view can realize.

We have clarified the ridge fit and its row-space constraint in Section 4.2, Equations (2) and (3), and Appendix A, Equation (12), and added the observed ranks in Appendix H, Table 18.

Q3 — Out-of-task inputs and the meaning of support

Thank you for asking how the reference set is formed. Each expert uses its own in-task validation positives for the class reference and encodes every query, including out-of-task examples. The support p is the smoothed fraction of reference distances at least as large as the query distance.

DatasetCross-task AUROC across availability states
Food1010.67–0.75
MM-IMDb0.61–0.68

The cross-task AUROC ranges show that p carries compatibility information in the expert’s feature space, with weaker separation when text is missing. Its meaning is an empirical tail rank relative to in-task positive examples.

We have clarified the validation reference and empirical support in Section 4.3, Equation (8), and Appendix C, and added cross-task AUROC in Appendix I.

Q4 — Comparability of independently trained heads

Thank you for raising head comparability. On Food101, one-vs-rest log-odds remove a common additive shift within each head, while cross-head scale differences can remain. MM-IMDb uses native logits. We fit a temperature per head on task validation data and apply the same scaling to both inference variants.

Head-wise scaling and inferenceFood101 AP ↑MM-IMDb AP ↑
Temperature, native all-head69.0527.87
Temperature + HEC70.0330.01
Affine, native all-head69.2228.63
Affine + HEC70.0830.10

With both variants using the same temperature scaling, HEC still improves AP by 0.98 points on Food101 and 2.14 on MM-IMDb. The affine-scaling control shows the same direction on both datasets. Class-specific support therefore adds predictive value under both tested head-wise rescalings.

We have clarified native evidence in Section 4.3 and Appendix B, Equation (15), and added the head-wise scaling comparisons in Appendix I, Table 29.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.