acceptodds
Under review as a conference paper at ICLR 2027

CBF: Severity-Aware Multimodal Fatty Liver Disease Grading via Collaborative Branch Fine-Tuning

Abstract

Combining tongue images with routine physiological indicators offers a non-invasive means of grading fatty liver disease (FLD). Multimodal models such as HM-TDF fuse both inputs to improve overall accuracy. However, we find that this gain has a clinical cost: the fused model recognizes fewer moderate-to-severe cases than a model that uses the physiological indicators alone. Averaging the predictions of single-input branches at inference raises accuracy but does not recover these cases, because the branches are never trained for the shared decision. To make fusion severity-aware, we propose Collaborative Branch Fine-Tuning (CBF), which keeps tongue-only, physiology-only, and joint predictors and fine-tunes them together with a loss on their averaged prediction and on each branch. We prove that this objective reallocates each record's update toward the branches that carry the correct signal, while the per-branch term keeps every branch a competent predictor on its own. Extensive experiments on Tongue-FLD show that CBF achieves the best accuracy and macro-F1 among all compared methods and restores the moderate-to-severe recall lost by fusion. More importantly, the recovered predictions rely more on the patient's own tongue image.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal learning, collaborative fine-tuning, fatty liver disease grading, tongue image analysis, physiological indicators

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5s3hOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

I find this an interesting and practically relevant research direction. Prior work has already demonstrated benefits from combining unimodal information with cross-modal interactions, providing empirical support for the general approach. My main concern is therefore not whether the direction is worthwhile, but how much new understanding this paper adds. The predictor composition and training objective are closely related to existing methods, and the methodological novelty appears limited. The grade-specific observations are useful, but the current analysis does not yet establish a new mechanism or principle beyond what is already understood about ensemble training. At present, I view the paper more as a valuable application and validation of established ideas than as a sufficiently developed novel contribution.

Critical strengths

Table 2 and Figures 1, 3, and 4 show that a multimodal model with higher overall accuracy can recognize fewer moderate-to-severe cases than a physiology-only predictor. The grade-specific recall and paired prediction changes make this discrepancy concrete. This is a useful contribution because it shows why an apparent improvement in overall performance may not reflect improvement across disease grades. The analysis provides more insight than another aggregate accuracy comparison alone.

Critical weaknesses
  1. Figure 2 appear to have been prepared with insufficient care, and the overall workflow is not clearly illustrated. CBF’s three-predictor design and combined individual–ensemble supervision have close precedents. The paper currently reads more as an application of established ensemble ideas to fatty liver grading than as a new severity-modeling mechanism or a new empirical principle. My concern is not whether the direction is effective, but how far this study advances the existing understanding.
  2. The ablation confounds collaborative supervision with additional fine-tuning, and does not test whether class reweighting or simple output correction produces comparable gains. It therefore remains unclear whether the improvement in moderate-to-severe recall reflects better multimodal cooperation, additional optimization, or a shift in class-prediction preferences.
Questions for authors
  1. Please compare CBF with I2M2 using the same V/S/JV/S/J encoders, initialization checkpoints, and fine-tuning budget. An additional product-aggregation variant with the same individual-branch supervision as CBF would separate the effects of aggregation and auxiliary supervision. This would establish whether the benefit comes from CBF’s specific training configuration or from the existing three-predictor framework, rather than relying only on a descriptive distinction from prior work. The difference between I2M2’s product aggregation and CBF’s probability averaging should be preserved and explicitly documented.
  2. Starting from identical checkpoints, compare individual-branch cross-entropy only, averaged-prediction cross-entropy only, and full CBF. Match optimization steps and selection rules, and control the overall loss scale and tuning budget. Also report continued fine-tuning of the joint predictor JJ alone. The key question is whether CBF consistently outperforms these additionally trained controls—not merely a frozen average—and whether both loss terms are necessary.
Reviewer_wrihOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed a multi-modal learning method dubbed Collaborative Branch Fine-Tuning for fatty liver grading using tongue images and physiological indicators. The authors noticed a phenomenon that multimodal fusion improves overall classification accuracy but reduces recall for intermediate cases. To address this, the proposed framework jointly fine-tunes three predictors. Experiments demonstrated improvements in overall accuracy. Overall I think it is an interesting paper, but it was limited to a single dataset.

Critical strengths
  1. The authors addressed an important issue in multimodal medical learning, and the motivation is reasonable and clearly presented.

  2. The experimental analysis is quite thorough, and the paper is easy to follow.

  3. The branch-level analysis and tongue-replacement tests provide additional insights into collaborative fine-tuning.

Critical weaknesses
  1. The proposed objective is similar to the methods from Jeffares et al. (NeurIPS 2023). The paper would benefit from a clearer explanation of new theoretical insights.

  2. All experiments are conducted on a single dataset. While the availability of paired tongue-image and physiological data is understandably limited, the proposed method is presented as a general multimodal learning framework.

Questions for authors
  1. Can the framework be evaluated on additional multimodal datasets to show its general applicability?
Reviewer_m3BbOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Collaborative Branch Fine-Tuning (CBF) for three-grade fatty liver disease classification using tongue images and physiological indicators. The authors identify an important limitation of conventional multimodal fusion: although HM-TDF improves overall accuracy, it reduces recall for moderate-to-severe cases compared with a physiology-only predictor.

CBF maintains three predictors using tongue images, physiological indicators, and both modalities, respectively. Their probability outputs are averaged, and all three branches are jointly fine-tuned using cross-entropy on the averaged prediction and auxiliary cross-entropy losses on individual branches. The authors also derive the corresponding gradient weighting behavior.

Experiments on the Tongue-FLD dataset show improvements in accuracy and macro-F1 over the reproduced HM-TDF baseline. Notably, moderate-to-severe recall increases from 24.85% to 34.37%. Additional paired statistical tests, branch-level analyses, Grad-CAM visualizations, and tongue-replacement experiments investigate the effects of collaborative training.

Critical strengths
  1. The paper identifies a clinically relevant limitation of multimodal fusion that can be obscured by overall accuracy. The focus on moderate-to-severe recall is meaningful given the class imbalance and the potential consequences of missing more severe cases. The paired per-grade analysis provides convincing evidence of improved recognition for this subgroup.

  2. The proposed method is simple and well characterized. The gradient analysis explains how supervising the averaged prediction changes the contribution of individual branches, while the ablation experiments distinguish collaborative fine-tuning from inference-only averaging. The additional branch-level and tongue-replacement analyses provide useful evidence about changes in model behavior.

Critical weaknesses
  1. The methodological novelty is limited. The proposed objective largely combines joint supervision of an ensemble prediction with auxiliary cross-entropy losses on individual branches, closely related to existing joint ensemble training strategies. Despite the severity-aware framing, the objective does not explicitly incorporate disease severity or class-dependent misclassification costs. Comparisons with class-balanced losses, cost-sensitive learning, and established multimodal gradient-balancing methods are needed to determine whether the observed improvement requires the proposed approach.

  2. The empirical validation is limited to a single dataset and one underlying multimodal architecture. Moreover, the improvement in overall accuracy over direct averaging is not statistically significant (McNemar p = 0.107), while moderate-to-severe recall remains only 34.37%. Evaluation with additional multimodal backbones or datasets would be important to establish the generality of the approach and the practical significance of its severity-sensitive performance.

Questions for authors
  1. How does CBF compare with class-balanced cross-entropy, focal loss, logit adjustment, or cost-sensitive training applied to the original HM-TDF model? Could these simpler alternatives recover moderate-to-severe recall while maintaining comparable overall accuracy?

  2. What is the specific methodological distinction between CBF and existing jointly trained deep ensemble objectives with individual-member supervision, particularly the framework discussed by Jeffares et al. (2023)? Does the proposed method provide a new optimization mechanism beyond applying this objective to heterogeneous modality branches?

  3. Is the improvement consistent when applying CBF to other multimodal architectures or when varying the auxiliary-loss weight? Such evidence would help establish whether the benefits arise from a general collaborative training principle rather than the particular HM-TDF configuration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.