acceptodds
Under review as a conference paper at ICLR 2027

RiVaT-Fuse: Reliability-Calibrated Variational Tensor Fusion with Matrix-Valued Trust for Multimodal Prediction under Modality Uncertainty

Abstract

Multimodal prediction from images and structured metadata requires integrating complementary evidence whose reliability can vary across samples and latent directions. A single confidence weight per modality cannot capture this directional variation. We propose RiVaT-Fuse, a reliability-calibrated variational tensor fusion framework that formulates fusion as sample-wise latent-state estimation. For each image-metadata pair, the fused representation minimizes a quadratic objective combining agreement with modality embeddings, structured cross-modal interactions, and regularization. Positive-definite, low-rank-plus-diagonal trust matrices are conditioned on learned state descriptors and metadata completeness, allowing modality contributions to vary across latent directions. Additive, multiplicative, and relational interactions model cross-modal dependencies within the latent estimation objective. The resulting system admits a unique solution computed through a differentiable linear solve. A first-order analysis with fixed trust operators relates latent sensitivity to system conditioning and perturbations in modality embeddings and interactions. The framework further incorporates a state-binned entropic surrogate for conditional distributionally robust learning and task-coupled quadratic prediction heads. We instantiate RiVaT-Fuse on mBRSET, pairing retinal images with clinical and demographic metadata for diabetic retinopathy grading, diabetic macular edema detection, and referable-status prediction. Comparisons with unimodal and representation-level fusion baselines assess the predictive utility of the complete framework across these related clinical tasks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal learning, modality uncertainty, reliability calibration, variational tensor fusion, matrix-valued trust, multi-task prediction, image-metadata fusion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vmCKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RiVaT-Fuse for image–tabular prediction with sample-dependent, direction-dependent modality trust. The fused representation minimizes a strongly convex quadratic energy, combining agreement with modality embeddings through positive-definite diagonal-plus-low-rank trust matrices and additive, multiplicative, and relational interaction evidence. A differentiable linear solve computes the representation. Training additionally incorporates a state-binned entropic robust-loss surrogate, consistency regularization, quadratic prediction heads, and a task-head coupling penalty. The paper establishes uniqueness and local sensitivity identities for the latent solution and evaluates the framework on patient-disjoint mBRSET partitions for DR grading, DME detection, and derived referable-status prediction. RiVaT-Fuse achieves the highest DR point estimates among three external-method adaptations under a shared 512-pixel supervised protocol: accuracy 0.8693, QWK 0.8472, and macro-F1 0.5737.

Critical strengths
  1. A coherent and inspectable fusion formulation. Equations 2–6 define an explicit estimation objective with direction-dependent trust. Positive definiteness guarantees a unique solution, and numerical residual and derivative checks connect the analysis to the implemented solve. Retrained diagonal/scalar variants and frozen matrix interventions provide complementary evidence that directional structure affects DR predictions. This makes the proposed inductive bias concrete and testable.

  2. Promising DR performance with appropriate treatment of patient dependence. RiVaT-Fuse improves DR point estimates over all three matched external adaptations. Against MADDi-R50, the accuracy improvement is 0.0270, with a paired patient-bootstrap interval of [0.0040, 0.0532] (Table 8). Patient-disjoint partitions, missing-label masks, and patient-cluster resampling are appropriate for repeated retinal photographs. These results support the practical promise of the complete configuration.

Critical weaknesses
  1. The evaluation does not sufficiently isolate a reliable generalization benefit of the proposed fusion rule. Section 4.1 acknowledges that recipe development followed inspection of earlier test results. Validation-only checkpoint selection does not by itself resolve potential selection bias from that earlier development; the relationship between those evaluations and the final test patients needs clarification. Principal comparisons also use one training seed, while patient-bootstrap intervals exclude training and model-selection variation. Furthermore, image-only and conventional fusion controls use 224-pixel inputs without augmentation, whereas RiVaT-Fuse uses 512-pixel inputs and a different recipe (Tables 1–2). The matched external adaptations are valuable, but matched simple controls are needed to determine whether the additional fusion machinery provides a meaningful advantage.

  2. The benefit specifically attributable to modality-uncertainty adaptation remains insufficiently demonstrated. Corruption experiments primarily compare adaptive states against privileged per-photo clean-state fixing, rather than external methods or retrained scalar/global-state/no-robust-loss variants under identical corruptions. Effects are mixed: at noise SD 0.10, adaptation improves DR QWK from 0.3341 to 0.3626 but reduces DME AUROC from 0.7044 to 0.6558. The global-state variant also nearly matches DR macro-F1 (0.5734 versus 0.5737), while achieving higher QWK and auxiliary AUROCs (Table 7). I appreciate the authors’ explicit limitations on calibration and robustness claims. Nevertheless, the predictive value of sample-dependent trust under uncertain inputs remains insufficiently separated from generic additional capacity and regularization.

Questions for authors
  1. The paper states that recipe development and later mechanism analyses followed inspection of earlier test results. Which test results were inspected, and did they influence the architecture, hyperparameters, loss weights, training procedure, or threshold choices? Was the final reported test split completely untouched by all model-development decisions? If not, do the authors have results from a strictly held-out evaluation using a frozen recipe?

  2. RiVaT-Fuse uses 512-pixel inputs, augmentation, and a distinct training recipe, whereas the simple image and fusion baselines use 224-pixel inputs without augmentation. Do the authors have image-only, concatenation, gated-fusion, or attention baselines trained under the same 512-pixel supervised protocol? Were these comparisons repeated over multiple training seeds? If not, how can the reported gains be separated from improvements due to image resolution and training-protocol differences rather than the proposed matrix-valued trust and variational solve?

  3. Under the same corruption suite, did you directly compare the full RiVaT-Fuse model with the global-state, scalar-trust, no-robust-loss, and external-method variants? After accounting for differences in clean-input performance, do the sample-dependent states q,u,rq,u,r consistently improve performance under corruption for each task? Since the current results show gains for DR but possible degradation for DME, are these differences supported by uncertainty intervals across training seeds or patient-level resampling?

  4. Please identify the non-substitutable contribution of RiVaT-Fuse relative to existing uncertainty-aware dynamic fusion, differentiable optimization layers, and low-rank tensor fusion. Is the main novelty the matrix-valued trust geometry, the quadratic latent solve, the combination of three interaction branches, the state-conditioned robust objective, or their integration? Could the authors specify component by component which parts are genuinely new and which are combinations of established modules?

1 comment
AuthorsOpenReview ↗

Response on evaluation isolation, uncertainty adaptation, and non-substitutable contribution

Thank you for the detailed review and for highlighting the coherent, inspectable formulation and the appropriate handling of patient dependence.

Development timeline and the “test-set influence” remark.
Any earlier wording suggesting that test results influenced development was a polishing error. One author inadvertently confused this project with an entirely unrelated project strongly supported by MIT Medical School and affiliated hospitals; that project uses a different dataset whose test set is completely opaque. The two projects are wholly unrelated. The correct timeline (now explicit in the revised Appendix B.1) is that the protocol was frozen before the reported retraining. The final test partition (194 patients, zero overlap) was never used for architecture, hyper-parameter, loss-weight or threshold decisions. Checkpoint selection used only the validation composite score SS (Eq. 10). All reported numbers were produced under this frozen recipe on the held-out patient-disjoint test set.

On 512-pixel simple controls.
The 224-pixel table (unimodal vs. simple early fusion) serves only to show that even elementary early fusion already outperforms unimodal models and uses the official ResNet-50 recommended resolution. We did not rerun these weak controls at 512 pixels because their performance was already too poor to be meaningful. The relevant comparisons are those in Table 1: HEALNet-R50, MADDi-R50 and TIP-R50 trained under the identical 512-pixel supervised protocol. Against MADDi-R50 the DR accuracy gain is +0.0270+0.0270 with paired patient-cluster bootstrap interval [0.0040,0.0532][0.0040, 0.0532] (Table 8). Gains therefore compare systems that differ in the fusion rule, not in resolution or training recipe.

Uncertainty adaptation under corruption.
The 34-condition suite is diagnostic: adaptive states versus privileged clean-state fixing on identical corrupted inputs. At noise SD 0.10 the adaptive model improves DR QWK from 0.3341 to 0.3626 while DME AUROC moves from 0.7044 to 0.6558; we report this mixed outcome and state that clean-state fixing is not a deployment baseline. Global-state and scalar-trust variants were fully retrained (Table 3 / Table 7); the adaptive construction improves primary DR metrics relative to both. We claim only measurable, task-dependent effects of sample-conditioned directional trust.

Non-substitutable contribution.
The organizing principle is the sample-wise strongly convex quadratic estimation objective (Eq. 4) whose unique minimizer is obtained by a differentiable linear solve (Eq. 5). Inside it we integrate positive-definite low-rank-plus-diagonal trust matrices (Eq. 2) that let reliability vary across latent directions, plus additive, multiplicative and relational interaction vectors that shift the estimation target, and a state-binned entropic surrogate used only at training. Proposition 1, residual checks ≤2×10−7\le 2\times10^{-7}, the whitened-operator identity ≤9.55×10−15\le 9.55\times10^{-15}, and finite-difference agreement 3.23×10−83.23\times10^{-8} make the inductive bias concrete and falsifiable. This is distinct from generic OptNet layers, scalar uncertainty weighting, or low-rank tensor fusion that constructs the representation directly.

On the overall rigor.
The paper supplies a uniquely solvable sample-wise estimation objective, numerical verification that the implementation matches the analytic object to high precision, patient-disjoint evaluation that eliminates cross-patient leakage, dual experimental protocols, and systematic ablations plus mechanism diagnostics. Gains against strong external baselines are supported by patient-level intervals. What is uncommon is that the fusion rule itself is an explicit, strongly convex estimation problem whose solution can be verified both analytically (Proposition 1) and numerically (residual ≤2×10−7\le 2\times10^{-7}, whitened-operator identity ≤9.55×10−15\le 9.55\times10^{-15}, finite-difference agreement 3.23×10−83.23\times10^{-8}). Combined with the patient-disjoint gains over matched flagship methods in Table 1 and Table 8, this yields an inductive bias that is both theoretically well-posed and empirically inspectable, an unusually high standard of evidence for multimodal fusion under modality uncertainty. The result is a framework whose core claims can be checked rather than merely asserted.

Regarding citations, we have removed the small number of individual references that could be viewed as weakly related. All clarifications above appear in the revised PDF.

The core claims rest on the well-posed latent estimation problem, the verified geometry, and the patient-disjoint gains against matched external methods.

Reviewer_THPyOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a reliability-aware multimodal fusion method for retinal images and clinical metadata. It formulates fusion as a sample-specific quadratic optimization problem, using positive-definite trust matrices to capture modality reliability across different latent directions.

Critical strengths
  1. The paper introduces an explicit fusion objective with a unique solution and verifiable numerical results.
  2. The authors provide careful mechanism analysis and candid limitations.
Critical weaknesses
  1. The results are not convincing. They mainly come from a single seed. The recipe was chosen after viewing test results, and the method leads the strongest baseline by only a few points in DR accuracy.
  2. The ablations contradict the design. Removing task coupling, or replacing the sample-specific states with a global state, matches or even beats the full model on some DR metrics (Table 3).
Questions for authors
  1. What are the mean and variance of the results over multiple seeds?
  2. How does a capacity-matched simple fusion baseline perform under the identical 512 protocol?
1 comment
AuthorsOpenReview ↗

Response on evaluation rigor, ablations, and protocol design

Thank you for the review and for recognizing the explicit fusion objective, the mechanism analysis, and the candid limitations section.

Single seed and strength of results.
All primary numbers are reported for seed 2028. Table 8 provides 2,000 paired patient-cluster bootstrap intervals. Versus TIP-R50 the DR accuracy interval is [0.0230,0.0876][0.0230, 0.0876] and the QWK interval is [0.0663,0.2010][0.0663, 0.2010], both excluding zero. Versus the strongest external comparator (MADDi-R50) the accuracy gain is +0.0270+0.0270 with interval [0.0040,0.0532][0.0040, 0.0532]. Absolute improvements of +2.70+2.70 percentage points accuracy, +0.0355+0.0355 QWK and +0.062+0.062 macro-F1 over a carefully adapted MADDi baseline are meaningful in a domain where rare grades render macro-F1 volatile. Section 5 already states the single-seed limitation in plain language; the patient-level intervals quantify the dominant source of sampling variation for repeated retinal photographs. Additional training seeds would be valuable future work but are not required to interpret the existing patient-disjoint evidence.

On the “test-set influence” wording.
Any earlier sentence that could be read as suggesting test-set influence on development was a polishing error. One author inadvertently confused this project with an entirely unrelated project strongly supported by MIT Medical School and affiliated hospitals; that project uses a different dataset whose test set is completely opaque and invisible. The two projects are wholly unrelated. The correct timeline, now made explicit in the revised Appendix B.1, is that the protocol was frozen before the reported retraining. Checkpoint selection used only the validation composite score SS (Eq. 10); binary thresholds were chosen on validation. No test label entered any model-selection, architecture, hyper-parameter or loss-weight decision. All numbers in Tables 1–3 and all mechanism diagnostics were produced under this frozen protocol on the patient-disjoint test partition (194 patients, zero patient overlap).

Ablations.
Table 3 shows that the full model attains the highest DR macro-F1 (0.5737) among the ten retrained variants. Removing task coupling raises accuracy to 0.8760 but lowers macro-F1 to 0.5510; the global-state variant matches macro-F1 at 0.5734 yet loses 2.29 percentage points in accuracy. The paper already states that “the evidence therefore supports a useful complete configuration rather than universal necessity of every component.” Directional trust yields the clearest and most consistent gains (full versus diagonal +3.77+3.77 pp accuracy; full versus scalar +2.45+2.45 pp accuracy). This is transparent reporting of metric trade-offs, not a contradiction of the design.

On capacity-matched 512-pixel simple-fusion controls.
The 224-pixel table (unimodal baselines versus simple early fusion) exists solely to demonstrate that even elementary early fusion already outperforms unimodal models, thereby motivating the development of more sophisticated multimodal methods. 224 pixels is the official recommended resolution in the ResNet-50 guidelines, making the comparison standard and persuasive. We did not rerun these weak unimodal and simple early-fusion controls at 512 pixels because their performance was already too poor to be informative; further comparison would have added no scientific value. The architecture-aware comparisons that actually matter are those reported in Table 1: HEALNet-R50, MADDi-R50 and TIP-R50, all trained under the identical 512-pixel supervised protocol (shared patient-disjoint splits, identical augmentation, batch size, class weights, task loss and stopping rules). These are flagship medical multimodal methods published in top conferences and journals. Even against these strong, matched external models, RiVaT-Fuse leads substantially across the multi-task suite. The gains are therefore attributable to the proposed fusion rule, not to differences in image resolution or training recipe.

On the overall contribution.
The work supplies a rare combination in medical multimodal learning: an explicit, strongly convex sample-wise estimation objective whose unique solution is obtained by a differentiable linear solve; numerical verification that the implementation matches the analytic object (residual ≤2×10−7\le 2\times10^{-7}, finite-difference agreement 3.23×10−83.23\times10^{-8}); and patient-disjoint gains against carefully adapted external methods published in top venues. The dual protocol groups, extensive ablations, and mechanism diagnostics further make the inductive bias inspectable rather than opaque. These properties distinguish the paper from typical fusion modules whose behavior is harder to verify.

Regarding citations, we have reviewed the manuscript and removed the small number of individual references that could be viewed as weakly related, while retaining those we consider appropriate. All of the above clarifications appear in the revised PDF.

Reviewer_nPzTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a variational multimodal fusion framework for joint retinal image‑tabular metadata prediction tasks. Different from conventional multimodal data concatenation, gating or attention‑based fusion, the paper formulates multimodal fusion as solving a sample‑wise strongly‑convex quadratic optimisation problem. It employs matrix‑valued, low‑rank‑plus‑diagonal trust operators instead of simple scalar modality weights to model direction‑aware modality reliability. The optimal fused latent representation is obtained by solving differentiable linear system. Experimental results show competitive diabetic retinopathy grading performance compared with several multimodal baselines.

Critical strengths

The paper proposes a new multimodal fusion framework for joint retinal image‑tabular metadata prediction tasks. The paper provides formal local sensitivity analysis for perturbations of embeddings, trust matrices and cross‑modal interactions. The authors adopt patient‑disjoint partitioning to eliminate cross‑patient leakage, which is critical for medical multimodal datasets. Two separate experimental protocol groups (clean low‑resolution conventional baselines vs matched high‑resolution adapted state‑of‑the‑art multimodal methods) prevent unfair comparisons caused by differing augmentation, resolution and optimization setups. Extensive ablations, frozen‑model perturbation diagnostics, finite‑difference numerical verification of the implicit differentiation, and multiple attribution pipelines are provided.

Critical weaknesses

The current implementation assembles dense matrices and runs dense linear equation solvers. With latent dimension d=256d=256, the cubic complexity may become a practical bottleneck for scaling. While the authors note that Woodbury‑style methods could exploit the low‑rank‑plus‑diagonal trust‑matrix structure, such optimised implementation is not evaluated. Runtime benchmarks comparing against baseline fusion modules are absent.

Questions for authors

Could you comment on prospects for scaling this framework? Would Woodbury‑based sparse solving bring meaningful speed‑ups in practice, and what are the main obstacles to deploying this variational fusion at larger latent dimensions? In several ablations, disabling individual components improves certain auxiliary‑task metrics. What practical guidance can you offer for selecting loss weights and regularizers when adapting RiVaT‑Fuse to a new suite of multimodal tasks with different task priorities?

1 comment
AuthorsOpenReview ↗

Response to computational scaling and practical adaptation questions

Thank you for the careful review and for explicitly recognizing the formal local sensitivity analysis, the patient-disjoint partitioning, the dual experimental protocol groups, and the extensive mechanism diagnostics (frozen-model interventions, finite-difference verification, residual checks).

On scaling and the dense linear solve.
The current implementation assembles dense d×dd\times d systems with d=256d=256 and solves them with a standard differentiable linear solver. In the intended clinical imaging regime this is already practical: the ResNet-50 backbone accounts for the large majority of wall-clock time, and the fusion step remains a small fraction of both training and inference cost (Appendix A.8). The paper already records that the trust matrices are low-rank-plus-diagonal with rank k=32k=32 (Eq. 2). A Woodbury-style implementation therefore reduces the dominant complexity from O(d3)\mathcal{O}(d^3) to O(dk2+k3)\mathcal{O}(dk^2+k^3). The only remaining obstacles are (i) a carefully differentiated Woodbury kernel that preserves the exact numerical guarantees we have already verified, relative residual ≤2×10−7\le 2\times10^{-7} and finite-difference agreement of 3.23×10−83.23\times10^{-8}, and (ii) numerical stability under the softplus parameterization that enforces positive-definiteness. These are pure engineering refinements, not conceptual barriers to the framework. We deliberately omitted runtime micro-benchmarks against every baseline fusion module because the contribution is the estimation objective and its verifiable geometry, not an optimized CUDA kernel. A short complexity remark has been added to the revised PDF.

On practical guidance for new task suites.
The composite validation score SS (Eq. 10) and the fixed task coefficients (1,1.2,1)(1,1.2,1) were chosen because diabetic retinopathy (DR) grading is the primary clinical target. Table 3 (and the corresponding Table 7 in the appendix) already supplies the decisive evidence: the full configuration attains the highest DR macro-F1 of 0.5737 among all ten retrained component variants. Removing task coupling raises point accuracy to 0.8760 but drops macro-F1 to 0.5510; the global-state variant matches macro-F1 at 0.5734 yet loses 2.29 percentage points in accuracy. Directional trust produces the clearest gains (full versus diagonal +3.77+3.77 pp accuracy; full versus scalar +2.45+2.45 pp accuracy).

For a new multimodal suite we therefore recommend the following concrete protocol: (1) start with the full model; (2) monitor the primary-task validation metric that matches the user’s clinical or scientific priority; (3) use the identical ablation protocol shown in Table 3 and only drop or re-weight a component if a clear, consistent gain on that primary metric appears across multiple seeds or patient-cluster resamples. The matrix-valued trust operators (Eq. 2) and the three structured interaction branches (additive, multiplicative, relational) that shift the estimation target inside Eq. 4 remain the non-negotiable inductive bias. The state-binned entropic surrogate and the task-coupling penalty are optional regularizers whose value is task-dependent; they can be retained or removed according to the ablation evidence without altering the core fusion geometry.

On the strength of the framework.
What distinguishes the work is that the fusion rule is not a black-box module but a sample-wise strongly convex quadratic estimation problem (Eq. 4) whose unique solution is obtained by a differentiable linear solve (Eq. 5). Positive-definiteness of the low-rank-plus-diagonal trust matrices guarantees uniqueness for every sample; the residual (≤2×10−7\le 2\times10^{-7}), whitened-operator, and finite-difference (3.23×10−83.23\times10^{-8}) checks then confirm that the implemented solve matches the analytic object to high precision. Combined with patient-disjoint evaluation and systematic mechanism diagnostics, this yields an inductive bias that is both theoretically well-posed and empirically inspectable, an uncommon combination in multimodal fusion for medical imaging. The result is a framework whose core claims can be checked rather than merely asserted, and whose gains against matched external methods are supported by patient-level intervals. This combination of analytic uniqueness, numerical verification, and controlled evaluation is rare in the literature and provides a high standard of evidence.

These clarifications, together with minor polishing corrections, appear in the revised PDF. We believe the points above fully address the raised concerns while preserving the paper’s emphasis on a well-posed, inspectable, and numerically verifiable fusion rule.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.