- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes DRA (Decide, Read, Adapt), a framework for missing-modality fine-tuning that allocates additional evidence based on modality availability. Supported by Bayes-risk analysis, DRA selectively reads additional text when images are missing, employs shared low-rank adaptation, and optimizes classification thresholds. Experiments on three benchmarks, along with ablation studies and controlled comparisons, demonstrate the effectiveness and efficiency of the proposed framework.
Critical strengths
This paper provides a solid theoretical foundation by analyzing how missing modalities affect the value of additional evidence through Bayes-risk analysis, offering theoretical support for state-conditioned evidence allocation. The experimental evaluation is comprehensive, covering multiple benchmarks, ablation studies, controlled comparisons, and computational cost analysis, which collectively demonstrate the effectiveness and efficiency of the proposed design.
Critical weaknesses
W1. The Decide module essentially performs conventional validation-based threshold optimization and is applied only to MM-IMDb, limiting its novelty and general applicability.
W2. Although allocating additional evidence based on modality availability is a meaningful idea, the proposed method heavily relies on CLIP's limited text context length, restricting its theoretical and practical generalizability.
W3. Despite introducing multiple modules and additional computational overhead, DRA improves upon MoRA by less than one percentage point in most settings, raising concerns about whether the added complexity is justified.
W4. DRA modifies both the adapter architecture and training configurations of MoRA, making it difficult to isolate the sources of the overall performance improvements.
W5. Although improvements are reported on three datasets, the core State-conditioned Evidence Allocation mechanism is evaluated only on MM-IMDb, lacking sufficient cross-dataset validation.
Questions for authors
Q1. Are the training hyperparameters consistent between DRA and MoRA? If not, how can the authors rule out the possibility that the improvements primarily result from more favorable hyperparameter settings rather than the proposed method?
Q2. Does State-conditioned Evidence Allocation remain effective for vision-language models with longer context windows or tasks without text truncation?
Q3. Given the detailed analysis of threshold optimization in Section 5.3, why are similar threshold-related issues not observed or discussed on the other datasets?
Q4. Can the authors isolate the contributions of the modified adapter architecture, Read module, and State-conditioned Allocation under consistent training configurations? Why does directly adding Read to MoRA lead to a performance decrease in Table 3(a)?
Q5. Can the authors evaluate State-conditioned Evidence Allocation on additional datasets with longer textual inputs to demonstrate its generalizability?
Thank you for the detailed questions. We answer Q1–Q5 in order. Method, theory and reported DRA numbers are unchanged; the revision adds the controls you asked for and narrows the claims.
Q1 (W4). Same hyperparameters as MoRA? Could the gain come from the recipe?
They were not the same (Table 33), so the original margins over MoRA mix method and recipe. Matched controls now separate the two.
On N24News, the new long-text dataset, the gain is mostly the method. Official MoRA scores 70.76, our recipe without Read 71.53 and with Read 73.73 (Table 4a). Of the +2.97 gap to MoRA, +2.20 [1.75, 2.64] comes from adding Read under one recipe.
On MM-IMDb it is mostly the recipe. We retrained every arm with six paired seeds (both-missing 70%, validation-fitted macro-F1; Table 13):
Recipe and initialization move F1 from 55.20 to 57.59; swapping the adapter makes no measurable difference (57.59 vs 57.55); Read adds +0.50 [0.29, 0.72] under one recipe. "23 of 27" is removed from the abstract, Sec. 5.2 presents Table 1 as a recipe-level comparison, and Adapt is no longer presented as a contribution.
Q2 (W2). Longer context windows, or tasks without truncation?
Partly. With Long-CLIP (248 positions) on MM-IMDb, giving the long window to image-missing rows instead of random rows improves F1 by +0.58 [0.15, 1.02] (Table 11). On N24News, 86.5% of articles still have text after 248 positions and reading it adds +1.93 [1.08, 2.77], but there choosing by state is no better than choosing at random (+0.07 [−0.19, 0.33]). So reading stays useful with a longer window when text remains, and the state rule beat random selection in one of the two tests.
Without truncation the text reader has nothing to read (App. A.2). The idea is not tied to text, though: in a Food101 test where titles fit the window, the extra evidence is a second image view, and removing the text raises its benefit by +1.05 [0.63, 1.46] (App. A.9). The effect is small, and a new view was not clearly better than re-encoding the original.
Q3 (W1). Why are thresholds discussed only for MM-IMDb?
MM-IMDb is the only one of the three benchmarks whose metric needs a threshold: macro-F1 requires a yes/no decision per label, while Food101 uses argmax accuracy and Hateful Memes AUROC. We agree that Decide is standard validation threshold fitting; it is no longer presented as a contribution.
Q4 (W4). Isolating adapter, Read and allocation; why did MoRA + Read drop?
The table under Q1 is this isolation: rows change the adapter, columns the reading policy. Of Read's +0.50, reading random rows gives +0.27. Choosing image-missing rows adds +0.36 AP [0.22, 0.49] on top; its F1 increment, +0.23 [−0.04, 0.50], has an interval that includes zero.
The −0.34 for MoRA + Read in Table 3(a) was a three-seed average. With six seeds it is −0.03 [−0.78, 0.72] under the original checkpoint rule and +0.44 [−0.03, 0.92] under the selection rule of the table above (55.20 vs 55.65). Neither is a measurable drop. We also froze six official MoRA models and trained only the reader. F1 at MoRA's fixed threshold rises by +0.67 [0.57, 0.77], or +0.20 [0.07, 0.33] with refitted thresholds, while re-reading the prefix instead of the unread text gives +0.001 [−0.053, 0.055] (Table 30). So the reader helps on top of an unmodified MoRA model; whether it helps under MoRA's joint training, six seeds cannot tell.
Q5 (W5). A second dataset with longer text
On N24News, random reading at the same budget reaches 73.35 (Table 4a), so most of the +2.20 comes from reading more text. Choosing the articles by state is worth a further +0.44 accuracy [0.18, 0.71] over random (eight seeds; Table 7). With the model and the articles fixed, hiding the image raises the benefit of reading the tail by +4.26 points [2.60, 5.85]. The same test on MM-IMDb gives +0.00839 nats per label [0.00767, 0.00905] (Table 3b). This is the interaction that Prop. 1 describes, observed on two datasets: a missing image makes the unread text worth more.
W3. Is the added complexity justified?
The only added component is Read: one 512×512 matrix, no policy network, and extra text passes only for image-missing rows. Its GPU forward overhead is 8.0% on MM-IMDb, or 4.0% when empty windows are skipped (Table 17), and 17% on N24News against 32% for reading everything (Table 19). There the state rule reads 54% of the extra windows and keeps 62% of the gain from reading everything (random selection: 51%; App. A.1). On MM-IMDb the gain from Read is indeed under one point; on N24News it is +2.20.