- Rating
- 1
- Confidence
- 3(−1)
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes CHEC, a framework for Continual Missing Modality Learning (CMML), to address two challenges: adapting the surviving modality when other modalities become unavailable and mitigating unreliable contributions from historical knowledge during task-agnostic inference. Specifically, the authors introduce Conditional LoRA and Utility-Gated Counterfactual Functional Correction (UG-CFC) during training, leveraging the functional differences between Complete and Missing views to guide adaptation. During inference, Historical Evidence Correction (HEC) adjusts historical-class prediction scores based on the support provided by the current input features.
Critical strengths
UG-CFC integrates paired Complete/Missing views, rank-space projection, and utility gating to guide conditional adaptation. This design is technically well-motivated and provides a meaningful approach to improving feature adaptation under missing-modality conditions.
Critical weaknesses
1. Insufficient validation of the reliability and statistical assumptions underlying HEC. HEC applies non-positive log-support corrections exclusively to historical classes, while leaving current-task prediction scores unchanged. Consequently, even when an input is consistent with the feature distribution of a historical class, its correct prediction score may still be suppressed. Although the paper demonstrates a reduction in historical-class prediction errors, it does not sufficiently investigate whether these improvements come at the cost of increased false negatives or reduced recall for historical classes.
Furthermore, HEC uses shared class statistics and calibration distance distributions across different modality-availability states. However, different missing-modality conditions may induce distinct feature distributions, potentially resulting in inaccurate support estimates for certain samples under shared calibration. Although UG-CFC reduces the functional gap between Complete and Missing views, this does not directly establish that the final classification feature distributions are sufficiently aligned. These concerns leave the claimed reliability of HEC insufficiently validated.
2. Insufficient evidence for the independent contribution of key components and the performance advantages over recent methods. The current ablation studies primarily focus on UG-CFC and HEC, while all configurations retain Conditional LoRA. Without a direct comparison between standard LoRA and Conditional LoRA, it is difficult to determine the independent contribution of the dynamic controller. Additionally, the authors employ Stop-Gradient to separate the gradient update paths of the original LoRA and the conditional controller, but the necessity of this design has not been sufficiently demonstrated.
Meanwhile, the experimental comparisons do not adequately cover recent relevant studies. Considering that CHEC achieves average AP improvements of only 1.73 and 0.63 percentage points over the strongest baseline, DeLo, the lack of more comprehensive comparisons with recent methods weakens the evidence for its performance advantages while most of baselines are works in 2024, 2025. Together, these limitations make it difficult to fully assess the novelty and practical contributions of the proposed framework.
Questions for authors
Q1. HEC applies one-sided negative corrections exclusively to historical classes. Could the authors provide historical-class Precision, Recall, and False Negative Rate before and after applying HEC to clarify whether the reduction in historical-class errors comes at the cost of suppressing correct historical predictions? Additionally, does the class-wise correction in HEC provide clear advantages over a simple uniform penalty applied to historical-class logits?
Q2. HEC uses shared class statistics and calibration distance distributions across different modality-availability states. Could the authors compare shared calibration with modality-specific calibration to validate the appropriateness of this design?
Q3. Could the authors provide a direct ablation comparing standard LoRA with Conditional LoRA and further clarify the advantages of the Stop-Gradient design over joint optimization without gradient separation?
Q4. Given the relatively modest performance improvements over the strongest baseline, could the authors include comparisons with more recent representative methods under consistent experimental settings to better establish the competitiveness of CHEC?
Q5. CHEC introduces additional counterfactual correction and gradient computations during training and evaluates all learned task experts during inference. Could the authors provide comparisons of training time, inference latency, and memory consumption to better assess the trade-off between computational cost and performance gains?
5 comments
Q2 — Shared versus state-specific calibration
Thank you for suggesting this control. All three calibration variants use the same trained experts.
| Calibration | Food101 AP ↑ | Food101 FG ↓ | MM-IMDb AP ↑ | MM-IMDb FG ↓ |
|---|---|---|---|---|
| Fully shared | 69.88 | 12.67 | 29.84 | 29.93 |
| Shared statistics, state-specific distance references | 69.74 | 12.53 | 29.94 | 29.88 |
| Fully state-specific | 69.60 | 12.61 | 29.87 | 29.84 |
Fully shared calibration gives the highest Food101 AP, while state-specific distance references improve MM-IMDb AP by 0.10 points. Sharing retains more validation positives per class reference and gives comparable AP in this fixed-expert comparison. The state-specific variants also give slightly lower FG than pooling on both datasets.
We have added the comparisons of shared and state-specific calibration in Appendix I, Table 28.
Q3 — Standard LoRA and Stop-Gradient versus joint optimization
Thank you for suggesting direct controls for both design choices. The controller test fixes rank and inference. The paired gradient test fixes forward architecture and loss. In its joint control, the residual classification and UG-CFC gradients both reach A/B. The gradient table reports separated minus joint updates (mean ± sample SD).
| Controller | Food101 AP ↑ | MM-IMDb AP ↑ |
|---|---|---|
| Static LoRA | 59.87 | 25.32 |
| State only (capacity matched) | 61.14 | 25.88 |
| Sample + state | 61.73 | 26.05 |
At fixed rank and inference, Conditional LoRA outperforms static LoRA in AP on both datasets.
| Dataset | ΔAP (pp) | ΔFG (pp) |
|---|---|---|
| Food101 | +1.27 ± 0.07 | −1.16 ± 0.07 |
| MM-IMDb | +0.24 ± 0.06 | +0.40 ± 0.09 |
Separating the two paths improves paired mean AP on both datasets over the joint control while leaving the auxiliary correction to the controller. FG also falls on Food101 and rises by 0.40 points on MM-IMDb.
We have clarified the gradient paths in Section 4.2, Equation (7), and added the controller and gradient controls in Section 5.3, Table 3, and Appendix G, Tables 11–13.
Q4 — Recent representative baselines under consistent settings
Thank you for suggesting a comparison with a more recent method. We adapted MoRA to the continual learning protocol. At 70% Mixed, both methods share the frozen ViLT backbone, task splits, missing-modality masks and all-head HEC inference. Each uses its own adapter rank and layer coverage.
| Dataset | Method | AP ↑ | FG ↓ |
|---|---|---|---|
| Food101 | Adapted MoRA + HEC | 66.56 | 11.27 |
| Food101 | CHEC | 69.88 | 12.67 |
| MM-IMDb | Adapted MoRA + HEC | 28.32 | 32.26 |
| MM-IMDb | CHEC | 29.84 | 29.93 |
Under the shared inference rule, CHEC improves AP by 3.32 and 1.52 points on Food101 and MM-IMDb, respectively. It also lowers MM-IMDb FG by 2.33 points, while Food101 FG rises by 1.40 points.
We have added the MoRA comparison across all settings in Section 5.2, Tables 1 and 2, and described the adaptation and shared inference protocol in Appendix B.
Q5 — Training time, inference latency and memory
Thank you for asking for measured costs. Both methods use matched optimizer steps at 70% Mixed, and the inference measurements use each method’s full final-stage pipeline.
| Metric | Food101 DeLo | Food101 CHEC | MM-IMDb DeLo | MM-IMDb CHEC |
|---|---|---|---|---|
| End-to-end training (h) | 12.94 | 16.21 | 5.08 | 7.16 |
| Single-sample latency (ms) | 21.47 | 174.83 | 26.38 | 110.92 |
| Peak training memory (GiB) | 10.72 | 13.94 | 13.41 | 16.68 |
| Peak inference memory (GiB) | 3.16 | 3.48 | 4.02 | 4.31 |
The absolute measurements show modestly longer training and a larger latency increase from the complete inference path.
| Expert set, native all-head → HEC | Food101 ΔAP (pp) | MM-IMDb ΔAP (pp) |
|---|---|---|
| CHEC | +1.38 | +1.97 |
Training adds paired-view and utility-probe computation. The larger deployment cost includes evaluating every expert, which keeps all learned classes eligible. On those same experts, HEC improves AP by 1.38 points on Food101 and 1.97 on MM-IMDb.
We have added the training time, single-sample latency, and peak memory measurements in Appendix J, Table 31. Section 5.4, Table 5, reports AP on the same CHEC experts.
Dear Reviewer, thank you for your time and valuable feedback. We are writing to gently follow up on our submitted rebuttal. We wonder if our response has addressed your concerns and clarified the issues raised.
If you have any remaining questions, please feel free to let us know—we would be more than happy to provide further clarification. If our response has resolved your concerns, we would be very grateful if you could kindly consider updating your score.
Thank you again for your constructive review.
W1 — Statistical assumptions and historical errors
Thank you for asking how historical recall and pooled calibration behave. At 70% Mixed, the first table reports historical diagnostics. The second table compares distance references by availability state on the same final expert features.
On Food101, HEC improves historical recall. On MM-IMDb, it raises FNR relative to native inference, but retains more recall at lower FPR than the uniform penalty.
State-specific fitting changes AP only modestly in this fixed-expert comparison. Text-missing positives more often have low support, yet pooled HEC improves AP in every availability state on both datasets. The calibration evidence comes directly from the final expert features.
We have clarified empirical support in Section 4.3, Equation (8), and added the historical error and calibration comparisons in Appendix I, Tables 20 and 28.
W2 — Independent contributions and recent comparisons
Thank you for asking us to isolate Conditional LoRA and Stop-Gradient and extend our baseline comparison. At 70% Mixed, the controller ablation fixes rank and inference. The gradient ablation fixes objective and parameter count and reports separate minus joint updates to A/B over three paired seeds (mean ± sample SD).
Conditional LoRA improves AP over static LoRA on both datasets. The capacity-matched comparison also supports conditioning on individual inputs.
Gradient separation improves paired mean AP on both datasets. The FG changes show that the benefit is strongest for AP across the two datasets.
We adapted MoRA to the same continual learning protocol, sharing the frozen ViLT backbone, splits, masks and all-head HEC inference while keeping each method’s own adapter configuration. CHEC’s AP exceeds that of adapted MoRA in every reported seed-42 setting on both datasets.
We have added the controller ablation in Section 5.3, Table 3, and the gradient controls in Appendix G, Tables 11 and 13. We have added the MoRA comparison in Section 5.2, Tables 1 and 2, and described the adaptation and shared inference protocol in Appendix B.
Q1 — Historical Precision, Recall and FNR versus a uniform penalty
Thank you for requesting this breakdown. Food101 uses macro metrics over historical classes. MM-IMDb uses micro metrics over Task 5’s 16 historical labels. The uniform penalty is tuned on validation data.
Relative to native inference, HEC improves both precision and recall on Food101. On MM-IMDb it gains 8.21 precision points while giving up 1.16 recall points. Against the uniform penalty, it improves both metrics on both datasets, supporting more selective attenuation.
We have added the historical precision, recall, and FNR results for native inference, HEC, and the uniform penalty in Appendix I, Table 20.