acceptodds
Under review as a conference paper at ICLR 2027

DRA: State-Conditioned Evidence Allocation for Missing-Modality Fine-Tuning

Abstract

When a modality disappears, the value of evidence still available but unread can change. We study state-conditioned evidence allocation: using observed modality availability to direct a limited budget of additional text encoding. A Bayes-risk characterization relates this choice to information substitutes and complements:the preferred state can reverse with the loss, and at an interior budget, state-blind allocation is optimal only at equal value per cost. DRA (Decide, Read, Adapt)implements the allocation with a state-gated residual reader that reuses an adapted CLIP text tower; we retain shared adaptation and validation thresholds as interfaces. On MM-IMDb, same-movie image removal with symmetric reader training gives a positive log-loss interaction of 0.00839 nats per label (movie-cluster95% CI [0.00767, 0.00905]). With the reader and extra-window budget fixed, targeting image-missing rows improves macro-F1 by +0.18 and AP by +0.33 over state-blind allocation. In retrained controls, DRA comes within 0.19 F1 of all-row reading using 53% of the extra windows. Under published evaluation rules across MM-IMDb, UPMC-Food101 and Hateful Memes, DRA exceeds reported MoRA in 23 of 27 benchmark settings

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Parameter-Efficient Tuning, Multimodal Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CezmOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes DRA (Decide, Read, Adapt), a framework for missing-modality fine-tuning that allocates additional evidence based on modality availability. Supported by Bayes-risk analysis, DRA selectively reads additional text when images are missing, employs shared low-rank adaptation, and optimizes classification thresholds. Experiments on three benchmarks, along with ablation studies and controlled comparisons, demonstrate the effectiveness and efficiency of the proposed framework.

Critical strengths

This paper provides a solid theoretical foundation by analyzing how missing modalities affect the value of additional evidence through Bayes-risk analysis, offering theoretical support for state-conditioned evidence allocation. The experimental evaluation is comprehensive, covering multiple benchmarks, ablation studies, controlled comparisons, and computational cost analysis, which collectively demonstrate the effectiveness and efficiency of the proposed design.

Critical weaknesses

W1. The Decide module essentially performs conventional validation-based threshold optimization and is applied only to MM-IMDb, limiting its novelty and general applicability.

W2. Although allocating additional evidence based on modality availability is a meaningful idea, the proposed method heavily relies on CLIP's limited text context length, restricting its theoretical and practical generalizability.

W3. Despite introducing multiple modules and additional computational overhead, DRA improves upon MoRA by less than one percentage point in most settings, raising concerns about whether the added complexity is justified.

W4. DRA modifies both the adapter architecture and training configurations of MoRA, making it difficult to isolate the sources of the overall performance improvements.

W5. Although improvements are reported on three datasets, the core State-conditioned Evidence Allocation mechanism is evaluated only on MM-IMDb, lacking sufficient cross-dataset validation.

Questions for authors

Q1. Are the training hyperparameters consistent between DRA and MoRA? If not, how can the authors rule out the possibility that the improvements primarily result from more favorable hyperparameter settings rather than the proposed method?

Q2. Does State-conditioned Evidence Allocation remain effective for vision-language models with longer context windows or tasks without text truncation?

Q3. Given the detailed analysis of threshold optimization in Section 5.3, why are similar threshold-related issues not observed or discussed on the other datasets?

Q4. Can the authors isolate the contributions of the modified adapter architecture, Read module, and State-conditioned Allocation under consistent training configurations? Why does directly adding Read to MoRA lead to a performance decrease in Table 3(a)?

Q5. Can the authors evaluate State-conditioned Evidence Allocation on additional datasets with longer textual inputs to demonstrate its generalizability?

1 comment

Thank you for the detailed questions. We answer Q1–Q5 in order. Method, theory and reported DRA numbers are unchanged; the revision adds the controls you asked for and narrows the claims.

Q1 (W4). Same hyperparameters as MoRA? Could the gain come from the recipe?

They were not the same (Table 33), so the original margins over MoRA mix method and recipe. Matched controls now separate the two.

On N24News, the new long-text dataset, the gain is mostly the method. Official MoRA scores 70.76, our recipe without Read 71.53 and with Read 73.73 (Table 4a). Of the +2.97 gap to MoRA, +2.20 [1.75, 2.64] comes from adding Read under one recipe.

On MM-IMDb it is mostly the recipe. We retrained every arm with six paired seeds (both-missing 70%, validation-fitted macro-F1; Table 13):

Recipe, adapterNo ReadRandom ReadState Read
Official MoRA55.2055.4655.65
Our recipe, MoRA's normalized adapter57.5957.8458.08
Our recipe, our additive adapter57.5557.8458.06

Recipe and initialization move F1 from 55.20 to 57.59; swapping the adapter makes no measurable difference (57.59 vs 57.55); Read adds +0.50 [0.29, 0.72] under one recipe. "23 of 27" is removed from the abstract, Sec. 5.2 presents Table 1 as a recipe-level comparison, and Adapt is no longer presented as a contribution.

Q2 (W2). Longer context windows, or tasks without truncation?

Partly. With Long-CLIP (248 positions) on MM-IMDb, giving the long window to image-missing rows instead of random rows improves F1 by +0.58 [0.15, 1.02] (Table 11). On N24News, 86.5% of articles still have text after 248 positions and reading it adds +1.93 [1.08, 2.77], but there choosing by state is no better than choosing at random (+0.07 [−0.19, 0.33]). So reading stays useful with a longer window when text remains, and the state rule beat random selection in one of the two tests.

Without truncation the text reader has nothing to read (App. A.2). The idea is not tied to text, though: in a Food101 test where titles fit the window, the extra evidence is a second image view, and removing the text raises its benefit by +1.05 [0.63, 1.46] (App. A.9). The effect is small, and a new view was not clearly better than re-encoding the original.

Q3 (W1). Why are thresholds discussed only for MM-IMDb?

MM-IMDb is the only one of the three benchmarks whose metric needs a threshold: macro-F1 requires a yes/no decision per label, while Food101 uses argmax accuracy and Hateful Memes AUROC. We agree that Decide is standard validation threshold fitting; it is no longer presented as a contribution.

Q4 (W4). Isolating adapter, Read and allocation; why did MoRA + Read drop?

The table under Q1 is this isolation: rows change the adapter, columns the reading policy. Of Read's +0.50, reading random rows gives +0.27. Choosing image-missing rows adds +0.36 AP [0.22, 0.49] on top; its F1 increment, +0.23 [−0.04, 0.50], has an interval that includes zero.

The −0.34 for MoRA + Read in Table 3(a) was a three-seed average. With six seeds it is −0.03 [−0.78, 0.72] under the original checkpoint rule and +0.44 [−0.03, 0.92] under the selection rule of the table above (55.20 vs 55.65). Neither is a measurable drop. We also froze six official MoRA models and trained only the reader. F1 at MoRA's fixed threshold rises by +0.67 [0.57, 0.77], or +0.20 [0.07, 0.33] with refitted thresholds, while re-reading the prefix instead of the unread text gives +0.001 [−0.053, 0.055] (Table 30). So the reader helps on top of an unmodified MoRA model; whether it helps under MoRA's joint training, six seeds cannot tell.

Q5 (W5). A second dataset with longer text

On N24News, random reading at the same budget reaches 73.35 (Table 4a), so most of the +2.20 comes from reading more text. Choosing the articles by state is worth a further +0.44 accuracy [0.18, 0.71] over random (eight seeds; Table 7). With the model and the articles fixed, hiding the image raises the benefit of reading the tail by +4.26 points [2.60, 5.85]. The same test on MM-IMDb gives +0.00839 nats per label [0.00767, 0.00905] (Table 3b). This is the interaction that Prop. 1 describes, observed on two datasets: a missing image makes the unread text worth more.

W3. Is the added complexity justified?

The only added component is Read: one 512×512 matrix, no policy network, and extra text passes only for image-missing rows. Its GPU forward overhead is 8.0% on MM-IMDb, or 4.0% when empty windows are skipped (Table 17), and 17% on N24News against 32% for reading everything (Table 19). There the state rule reads 54% of the extra windows and keeps 62% of the gain from reading everything (random selection: 51%; App. A.1). On MM-IMDb the gain from Read is indeed under one point; on N24News it is +2.20.

Reviewer_cB7bOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies missing-modality fine-tuning of CLIP. It formalizes this as state-conditioned evidence allocation under a reading budget, with a Bayes-risk analysis relating the choice to information substitutes and complements. The proposed DRA combines a state-gated residual reader (Read), an additive MoRA-style adapter (Adapt), and validation-fitted thresholds (Decide), and it exceeds reported MoRA results in 23 of 27 settings across MM-IMDb, UPMC-Food101, and Hateful Memes.

Critical strengths
  1. The observation is concrete and practically relevant: 62% of MM-IMDb summaries exceed CLIP’s text window, so a model can lose its image while discarding text it still has.
  2. The controlled experiments are careful. They include same-movie image removal, symmetric reader training, fixed-reader allocation at matched budgets, and prefix re-reading as a compute-matched control.
  3. The method is lightweight and Adapt can be merged into the backbone at inference.
Critical weaknesses
  1. The core contribution applies to a single dataset. Read is used only on MM-IMDb, since the texts in Food101 and Hateful Memes fit within the prefix. Consequently, the allocation reduces to a hand-coded rule (read more text when the image is missing) evaluated on one dataset
  2. The novelty of Adapt is limited. It differs from MoRA only by removing magnitude normalization, and adding Read to MoRA lowers F1 (−0.34; Table 3a), suggesting that Read’s benefit depends on DRA’s training recipe rather than being a general principle.
  3. The state-of-the-art claim in Table 1 is selective. Table 1 adopts the official fixed-threshold rule (0.5) and compares only up to MoRA, yet under this same rule the reimplemented DPL outperforms DRA in all three scenarios. DPL, TLP, and MCULoRA are absent from the main table. DRA’s advantage lies in ranking (higher macro-AP) and in F1 under validated thresholds, which should be stated explicitly rather than implied by the “23 of 27” summary.
Questions for authors

Please refer to the weakness.

2 comments

Thank you for the careful reading. We agree with all three points, including that our Table 1 summary was selective, and we have changed the paper accordingly.

W1. One dataset, and a hand-coded rule

We added a second long-text dataset, N24News. Three results carry over (Table 4a, Table 7, App. A.1):

  • Read helps under one recipe: 71.53 without Read, 73.73 with it, a gain of +2.20 [1.75, 2.64]. Official MoRA scores 70.76.
  • A missing image makes the unread text more valuable. With the model and the articles fixed, hiding the image raises the benefit of reading the tail by +4.26 accuracy points [2.60, 5.85]. This is the effect the rule relies on.
  • Choosing by state beats choosing at random at the same budget: +0.66 [0.58, 0.74] with the reader held fixed, and +0.44 [0.18, 0.71] when each policy is trained separately (eight seeds each).

On the rule being hand-coded: it is, and the paper now presents it that way. Availability is known before any forward pass and needs no fitted selector, so it is the simplest prior, and it is the one our analysis is about. It is not the best selector. Ranking articles by first-pass entropy at the same budget beats the state rule on N24News by +0.98 [0.90, 1.05] (Table 8; App. A.1). Once entropy is used, adding availability to it gives +0.22 [0.19, 0.25] on readers trained to read every article (average over nine budgets; Table 10) and nothing measurable on the DRA model itself (+0.010 [−0.062, 0.082]; Table 9). On MM-IMDb the interval of the entropy−state F1 difference includes zero in all six settings we tested (Table 6). The abstract, Sec. 5.5 and the limitations (Sec. 5.8) now state that the rule is not the best selector.

W2. Novelty of Adapt, and MoRA + Read (−0.34)

We agree about Adapt and no longer present it as a contribution. In a six-seed control under one recipe, MoRA's normalized adapter and our additive adapter score 57.59 and 57.55 without Read (Table 13).

The −0.34 was a three-seed average. With six seeds it is −0.03 [−0.78, 0.72] under the original checkpoint rule and +0.44 [−0.03, 0.92] when checkpoints are selected with validation thresholds. We can no longer say that Read lowers MoRA's F1, and we cannot say that it raises it. Three checks address whether Read depends on our recipe:

  • Common recipe, either adapter. Read adds +0.50 [0.29, 0.72] averaged over both adapters and +0.49 [0.23, 0.76] with MoRA's normalized adapter alone. These use checkpoints selected with validation thresholds; with the original fixed-cutoff selection the normalized-adapter gain is +0.19 [−0.06, 0.43].
  • Unmodified MoRA. We froze six official MoRA models and trained only the reader. F1 at MoRA's fixed threshold rises by +0.67 [0.57, 0.77], or +0.20 [0.07, 0.33] with refitted thresholds. Re-reading the prefix instead of the unread text gives +0.001 [−0.053, 0.055] (Table 30).
  • Second dataset. The +2.20 on N24News above is a same-recipe gain.

So Read does not need our adapter, and it works on top of an unmodified MoRA model. Whether it helps when trained jointly under MoRA's own recipe is still open.

W3. Table 1 and the "23 of 27" summary

We followed your suggestion.

  • Table 1 now includes our reproduced DPL and TLP's published results.
  • The abstract and Sec. 5.2 now say that DPL has higher fixed-threshold F1 than DRA in all nine MM-IMDb settings. DRA's advantage is in AP (nine of nine) and in F1 with per-label validation thresholds (eight of nine; DPL is 0.35 higher at text-missing 90%; Table 32).
  • "23 of 27" is no longer in the abstract. Sec. 5.2 reports it as a recipe-level comparison, not an estimate of Read: every setting also changes the training recipe, and Read is used only on MM-IMDb.
  • The two rules put the models at different operating points. At the fixed 0.5 cutoff (both-missing 70%), DPL predicts 14.4% positives and DRA 9.0%, against a label rate of 11.4%. With a single threshold fitted on validation, DRA has the higher mean in all nine settings, and six of the nine intervals exclude zero (Table 23).
  • We claim no superiority over TLP or MCULoRA. TLP's column is copied from its paper (70% missing rate). TLP adapts a frozen source model for five steps on unlabeled test batches, a different protocol from ours, so we list it for reference only. We have no same-protocol reproduction of MCULoRA, which uses other datasets and encoders; the paper discusses it next to our state-private adapter control (Table 3c), which is not a reproduction.

In short, the paper now stands on less than before, and we believe all of it is supported. With the inputs and the model fixed, a missing image raises the value of unread text on two datasets. A one-matrix reader turns that into +0.50 F1 on MM-IMDb and +2.20 accuracy on N24News under a matched recipe.

Thanks for the response. I will maintain my score.

Reviewer_woiuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies parameter-efficient fine-tuning for multimodal classification with missing modalities. The authors identify the limited context length of CLIP’s text encoder as a source of information loss, as longer text inputs are truncated. To address this issue, they propose DRA, a method built on MoRA that allocates additional text-encoding resources only to samples with missing images. The reported experimental results show improvements in classification performance.

Critical strengths
  • The problem framing is clear, and the proposed method is lightweight.
  • Text truncation is a limitation of vanilla CLIP models.
Critical weaknesses
  1. The comparison with alternative approaches to text truncation is incomplete. Read addresses information loss caused by CLIP’s 77-token context window, but the paper does not compare it with existing long-context alternatives, such as Long-CLIP (ECCV 2024) and TULIP (ICLR 2025). Long-CLIP extends the context window to 248 tokens, while TULIP uses relative positional encodings to support longer inputs. Comparisons with these alternatives would help clarify the practical advantage of Read.
  2. The state-conditioned reading component is evaluated on only one dataset. Read is enabled only on MM-IMDb, whereas the experiments on Food101 and Hateful Memes use Adapt without Read because their text inputs fit within CLIP’s 77-token context window. Although these results provide evidence for the effectiveness of Adapt, the evidence for the generalizability of the state-conditioned reading strategy remains limited to a single dataset.
Questions for authors

NA

3 comments

Thank you for the review. Both comparisons you asked for are now in the paper: Read against Long-CLIP and TULIP, and Read on a second long-text dataset (N24News). In short: on N24News, whose articles run well past 248 positions, DRA is ahead of Long-CLIP; on MM-IMDb, where few plots do, the two are level; and Read can be added on top of a longer-context encoder. We claim that the two are complementary, not that DRA is better in general.

W1. Comparison with long-context encoders

N24News (both-missing 70%; three seeds; one adapter recipe for every row; Table 4a):

Text encoder / readingBody tokens per article (max)Accuracy
CLIP-77, no extra reading7471.53
Long-CLIP-24824572.66
CLIP-77 + DRA74; 74 + 222 if the image is missing73.73
Long-CLIP-248 + tail windows46774.59
CLIP-77 + tail windows for every article74 + 22275.09

Long-CLIP helps on this task (+1.13 over CLIP without reading), so it is a working baseline. DRA on the original CLIP is +1.07 [0.25, 1.88] above it while encoding 80.9% as many text positions (a count of positions, not a latency figure). This is a cross-backbone comparison with three seeds, so it also reflects the two text towers. After 248 positions, 86.5% of the articles with text still have some left, and adding tail windows on top of Long-CLIP gains +1.93 [1.08, 2.77]. That is why we call the two complementary. Reading tails for every article is the most accurate option.

MM-IMDb. Only 5.5% of rows with text have anything left after 248 positions, and the two routes are level. Within each backbone the gains are similar: Read adds +0.57 on CLIP (57.45 to 58.02), and going from 77 to 248 positions adds +0.69 on Long-CLIP (56.47 to 57.16). Across backbones, CLIP + DRA against Long-CLIP-248 is +0.85 [−0.44, 2.14], an interval that includes zero (Table 11). Two further results limit what we claim. Attaching our reader to Long-CLIP's 77-position prefix did not help (−0.08 [−1.01, 0.84]). On ViT-L/14, DRA is above TULIP-248 (59.94 vs 58.67; +1.26 [0.17, 2.36]), but CLIP L/14 without reading already scores 59.76, so most of that gap is the backbone; Read's own gain there is +0.17 [−0.66, 1.01] (Table 4b). All arms share one adapter recipe that we did not tune per backbone, so these compare encoders under a common recipe, not each at its best. What does transfer is the allocation rule: inside Long-CLIP, giving the 248-position window to image-missing rows instead of random rows improves F1 by +0.58 [0.15, 1.02] with the same number of long windows.

What Read offers in practice. It is an add-on to stock CLIP: one 512×512 matrix, with no re-pretrained text tower. It can be fitted onto an already-trained model with everything else frozen (+0.67 F1 [0.57, 0.77] on six frozen official MoRA models; Table 30). And it spends the extra encoding only where the image is missing.

W2. A second dataset

N24News gives three results (Table 4a, Table 7, App. A.1):

  • Read helps under one recipe. Official MoRA scores 70.76, our recipe without Read 71.53, and with Read 73.73. Of the +2.97 gap to MoRA, +2.20 [1.75, 2.64] comes from Read.
  • A missing image makes the unread text more valuable. On the same 1,824 articles with the model fixed, reading the tail improves accuracy by 7.40 points when the image is present and by 11.66 when it is hidden, a difference of +4.26 [2.60, 5.85].
  • Choosing by state beats choosing at random at the same budget: +0.66 [0.58, 0.74] with the reader held fixed, and +0.44 [0.18, 0.71] when each policy is trained separately (eight seeds each).

Two limits. N24News is one missingness setting. And after a 248-position prefix, choosing the remaining text by state or at random is indistinguishable (+0.07 [−0.19, 0.33]).

Cost

With B/16 on MM-IMDb, GPU forward time per row is 1.80 ms without Read, 1.96 ms for DRA and 2.13 ms for Long-CLIP-248 (Table 16). In a separate paired measurement, skipping empty windows halves Read's forward overhead, from 8.0% to 4.0%, without changing any validation decision (Table 17). On N24News the overhead is 17%, against 32% for reading every article (Table 19); the state rule reads 54% of the extra windows and keeps 62% of the gain from reading everything (random selection: 51%; App. A.1). With L/14, DRA's full online pipeline is slower than TULIP-248's (6.25 vs 4.80 ms/row); the CLIP L/14 pipeline without Read already takes 5.96 ms, so Read itself adds about 5% (Table 18).

Please tell us if either comparison is not what you had in mind; we can clarify before the discussion closes.

Bracketed ranges are 95% intervals over paired seeds (article bootstrap for the image-hiding test), not adjusted for multiple comparisons.

Thank you for the detailed rebuttal. The new experiments address the concerns raised in my review. However, they also clarify that, after the revision, the contribution reduces to a carefully measured but intuitive interaction effect, together with a reader whose gain comes mostly from reading more truncated text rather than from state-conditioned allocation.

  • State-conditioned allocation, the paper’s central idea, adds little beyond reading more text. On N24News, same-budget random reading reaches 73.35 versus 73.73 for DRA (Table 4a). After a 248-position prefix, state and random selection are indistinguishable.
  • Read’s gain is modest and backbone-dependent. It adds +0.50 F1 on MM-IMDb with B/16 but +0.17 with L/14.
  • The revised layout on page 9 is also overly crowded and poorly organized.

The core finding, that unread text becomes more valuable when the image is missing, is intuitive, and the proposed allocation adds limited benefit beyond reading more text. I therefore do not find the contribution significant enough for ICLR and will maintain my rating.

Thank you for confirming that the new experiments address the concerns in your original review; the two comparisons you asked for made the paper’s claims more precise. We agree that the state gate provides a modest additional gain over random reading on N24News. We respectfully disagree, however, that the contribution is exhausted by an intuitive interaction and the recovery of truncated text. We answer your points in order.

State allocation versus reading more text. The small gap and a large state effect are the same fact. With model and articles fixed, the unread tail is worth 7.40 accuracy points when the image is present and 11.66 when it is hidden, 58% more. At the rule’s budget, random selection already reads 53% of the image-missing articles, so switching to the state rule moves a read from a complete article to an image-missing one for only 16% of the test set. 0.16 × 4.26 ≈ 0.69, which is what eight fixed readers measure: +0.66 [0.58, 0.74] (App. A.1; Sec. 5.5 now states this). The 73.35-versus-73.73 gap between trained policies is of the same order. It is what a large per-article effect predicts under this benchmark’s mix, and the fixed-reader result separates it from retraining. The effect is not specific to N24News or to CLIP: on MM-IMDb the state rule adds +0.36 AP [0.22, 0.49] over random reading (Table 13), and inside Long-CLIP, giving the 248-position window to image-missing rows instead of random rows adds +0.58 F1 [0.15, 1.02] (Table 11).

Backbone dependence. The L/14 gain of +0.17 has an interval of [-0.66, 1.01] over three seeds, which contains the B/16 value of +0.50; the data do not separate the two backbones. Read touches only rows with a missing image and a truncated plot, 17% of MM-IMDb. On those rows it adds +3.97 AP [3.50, 4.44] with B/16 and +3.09 AP [1.45, 4.72] with L/14 (Table 15; post hoc groups).

Is the finding only intuitive, and only about truncated text? Two results say it is more than that. First, the intuition is not safe as a general rule: Prop. 1 shows that the same conditionally independent views can be substitutes under log loss and complements under classification error, so "the image is gone, read more text" can be the wrong priority depending on the loss. The analysis tells a practitioner when the priority should change. Second, the effect does not depend on truncation. On 6,814 Food101 samples whose titles fit entirely inside the prefix, removing the text raises the benefit of an extra visual view by +1.05 accuracy points [0.63, 1.46] (Table 29). This is an interaction, not a method gain, and it shows that the value of additional evidence shifts with what else is visible, for visual evidence as well as text.

Page 9. We have re-laid it out in the updated PDF. Page 9 now carries only Table 3, whose four panels are aligned in two columns under a shorter caption. Table 4 has moved to page 10, beside the text that discusses it, and separates the three-seed trained comparison, the Long-CLIP rows, the eight fixed-model allocation and the L/14 comparison. Sec. 5.5 is reordered as evidence value, allocation at a fixed budget, then backbone and context limits, with the Food101 result first. No result or table label changed.

With the additional experiments addressing the concerns regarding magnitude and backbone dependence, we believe the paper’s contribution is more clearly established. The loss-dependent characterization, combined with the controlled interventions on both text and visual evidence, demonstrates that our findings go well beyond a simple intuitive observation. We hope these clarifications and new results demonstrate the robustness of our method, and we respectfully invite you to reconsider your rating in light of these strengthened theoretical and empirical foundations.

Comments on the paper

1 comment

We thank the reviewers and the AC. This comment summarizes the revision; individual replies are posted under each review. New and modified text in the PDF is blue. The method, the theory (Prop. 1) and the DRA numbers reported in Table 1 are unchanged. What changed is that we ran the controls the reviewers asked for and narrowed the claims to what those controls support.

What the paper claims now

With the inputs and the model fixed, removing the image raises the value of the unread text, on two datasets (MM-IMDb and N24News); Prop. 1 characterizes when this happens. Read, a one-matrix residual reader, turns this into +0.50 macro-F1 on MM-IMDb and +2.20 accuracy on N24News under a matched recipe. The adapter (Adapt) and validation thresholds (Decide) are supporting pieces and are no longer presented as contributions. Margins over MoRA in Table 1 are full-recipe comparisons and are reported separately from the controlled effect of Read.

Main concerns and what we did

  1. Read was evaluated on one dataset (all three reviewers). We added N24News. Under one recipe, Read raises accuracy from 71.53 to 73.73 (+2.20 [1.75, 2.64]; official MoRA scores 70.76). Random reading at the same budget reaches 73.35; choosing by state adds +0.44 [0.18, 0.71] over random with eight seeds. With the model and the articles fixed, hiding the image raises the benefit of reading the tail by +4.26 points [2.60, 5.85], the same direction as on MM-IMDb (Table 4a, Table 7).

  2. No comparison with long-context encoders (woiu, Cezm). We added Long-CLIP and TULIP. On N24News, CLIP + DRA is +1.07 [0.25, 1.88] above Long-CLIP-248, and tail windows on top of Long-CLIP add a further +1.93 [1.08, 2.77]. On MM-IMDb the two are not distinguishable (+0.85 [−0.44, 2.14]). On L/14, DRA is above TULIP-248 by +1.26 [0.17, 2.36], but Read's own gain on that backbone is +0.17 [−0.66, 1.01]. We describe reading and longer context as complementary and claim no general superiority (Table 4, Table 11).

  3. Gains confounded with the training recipe; MoRA + Read dropped by 0.34 (cB7b, Cezm). A six-seed grid separates the factors on MM-IMDb (Table 13). The recipe moves F1 from 55.20 to 57.59, the adapter makes no measurable difference, and Read adds +0.50 [0.29, 0.72]. The −0.34 was a three-seed average; with six seeds it is −0.03 [−0.78, 0.72]. Training only the reader on frozen official MoRA models gives +0.67 [0.57, 0.77] at MoRA's fixed threshold and +0.20 [0.07, 0.33] with refitted thresholds (Table 30).

  4. Table 1 was selective (cB7b). Table 1 now includes reproduced DPL and published TLP. "23 of 27" is removed from the abstract, which now says that DPL has higher fixed-threshold F1 in all nine MM-IMDb settings, and that DRA leads in AP (nine settings) and in F1 with per-label validation thresholds (eight).

  5. Decide is standard; is the cost justified? (Cezm) Decide is no longer presented as a contribution. Read is one 512×512 matrix with no policy network. Its GPU forward overhead is 8.0% on MM-IMDb (4.0% when empty windows are skipped) and 17% on N24News, against 32% for reading every article (Table 17, Table 19).

Limits stated in the paper

  • The state rule is a simple prior, not the best selector. On MM-IMDb its F1 gain over random reading has an interval that includes zero (+0.23 [−0.04, 0.50]; AP +0.36 [0.22, 0.49]). On N24News, ranking by first-pass entropy at the same budget is better by +0.98 [0.90, 1.05]. Reading every eligible input is more accurate than either.
  • After a 248-position prefix on N24News, state-based and random selection are indistinguishable (+0.07 [−0.19, 0.33]).
  • Whether Read helps when trained jointly under MoRA's own recipe is unresolved, and we claim no superiority over TLP or MCULoRA.

Bracketed ranges are 95% intervals over paired seeds (bootstrap for the fixed-model tests), not adjusted for multiple comparisons.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.