- The reliability signal may largely measure tumour size, and the robustness claim that motivates it is never tested.
My most serious concern is what the "observable quality" score actually captures. For both CT and radiomics, quality is the fraction of the tumour delineation that fits inside a fixed 64 crop, lowered further if the tumour touches the crop border. LUNG1 is a cohort of locally advanced tumours, and many of them are larger than 64. In practice, a patient will usually get a low "quality" score not because the scan is poor, but because the tumour is big. Tumour size and volume are among the strongest known predictors of survival in this cohort, as shown in Aerts et al. (2014), which the authors themselves cite.
As a result, the channel described as outcome-independent reliability is very likely feeding prognostic information into the gate. That directly undercuts the paper's central claim that relevance and reliability are kept separate. The loss of "quality" is also created by the authors' own choice of crop size, not by acquisition artefacts or segmentation errors, which are what the introduction uses to motivate the method. And because CT and radiomics share the same quality score, the reliability term can never shift weight between those two modalities, only between imaging and clinical data.
There is also reason to think the mechanism does very little for most patients. The paper itself notes that the gate is unaffected by a shift applied equally to all modalities. So for a patient whose tumour fits fully in the crop and whose clinical record is complete, reliability can change the gate only through the small certainty term, which is worth at most about 0.05 in logit units. The paper never shows how quality, reliability or fusion weights are distributed across patients, nor how often a modality is actually missing in LUNG1. Without that, it is possible that the missing-modality handling is never exercised by the real data, and that any effect of reliability is concentrated in the large-tumour subgroup described above.
Finally, the robustness claim that motivates the paper is never tested directly. There is no controlled experiment that degrades the inputs (adding noise, eroding masks, corrupting radiomic features, or deleting clinical fields) and shows that CAER holds up better than plain late fusion. There is no evaluation with modalities removed at test time, and no external cohort. LUNG1 comes from a single institution (MAASTRO), so its acquisition variability is limited to begin with.
The falsifiability argument in Section 4.6 does not fill this gap. Holding the content scores fixed, the claimed directional behaviour holds by construction, so no data could contradict it. The section also cites Eq. 21 where Eq. 20 is meant. Likewise, the auditability contribution is asserted but never demonstrated: there are no case studies and no analysis of how the fusion weights behave.
- The reported gains are within the range of sampling and training noise, and the comparison set does not test the fusion claim.
All of the evidence rests on a single test split of 85 patients. There are no confidence intervals, no paired significance tests, and no variance across seeds, because predictions from the five seeds are averaged before evaluation. At this sample size and with LUNG1's event rate, the standard deviation of Harrell's C is also could not found. Moreover, 1-year AUC of 0.5768 is close to chance and rests on a few dozen events. The IBS differences in the ablation, around 0.002, are negligible. Reporting results to four decimal places suggests a precision the data cannot support.
The paper's own ablation table makes the noise problem concrete. The "Equal residual" variant adds the same amount to every available modality's gate score. Since the gate ignores such a shift, this variant computes the same function as plain late fusion. Yet the two are reported at 0.5846 and 0.5745 Harrell C. That gap of about 0.010 between models that should be equivalent is a direct measure of run-to-run variation. It is about half the 0.022 gain credited to adding the certainty term on top of quality alone. By the authors' own bounds, that term can change the balance between any two modalities by only a very small amount, so attributing a 0.022 gain in concordance to it is not convincing.
The baselines also do not test the claim the paper makes. The paper is about how to fuse modalities, yet none of the six comparison methods is a multimodal fusion model; each simply receives one concatenated feature vector. Missing comparisons include:
a. QMF adapted to survival (cited by the authors but not compared);
b. evidential or trusted fusion (e.g., Han et al., ICLR 2021);
c. mixture-of-experts gating;
d. simple averaging of the branch predictions;
e. a clinical-only Cox model on stage and age, which is the reference clinicians use.
Baseline tuning is not documented. DeepHit's IBS of 0.337 and the 3-year AUCs of about 0.51 for MTLR and DeepHit suggest these baselines were under-tuned rather than inherently weak. CAER is also effectively a five-model ensemble, while CoxNet is a single deterministic model. we observed that plain late fusion without any reliability term already beats every baseline. About 85% of the IBS improvement over the best baseline comes from the per-branch architecture rather than from reliability. This contradicts the discussion's suggestion that the gains come from how evidence is combined rather than from model capacity.