CBF: Severity-Aware Multimodal Fatty Liver Disease Grading via Collaborative Branch Fine-Tuning

Combining tongue images with routine physiological indicators offers a non-invasive means of grading fatty liver disease (FLD). Multimodal models such as HM-TDF fuse both inputs to improve overall accuracy. However, we find that this gain has a clinical cost: the fused model recognizes fewer moderate-to-severe cases than a model that uses the physiological indicators alone. Averaging the predictions of single-input branches at inference raises accuracy but does not recover these cases, because the branches are never trained for the shared decision. To make fusion severity-aware, we propose Collaborative Branch Fine-Tuning (CBF), which keeps tongue-only, physiology-only, and joint predictors and fine-tunes them together with a loss on their averaged prediction and on each branch. We prove that this objective reallocates each record's update toward the branches that carry the correct signal, while the per-branch term keeps every branch a competent predictor on its own. Extensive experiments on Tongue-FLD show that CBF achieves the best accuracy and macro-F1 among all compared methods and restores the moderate-to-severe recall lost by fusion. More importantly, the recovered predictions rely more on the patient's own tongue image.