acceptodds
Withdrawn from ICLR 2027

From Alignment Diagnosis to Joint Supervision in Multimodal Draft Training

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal draft training benefits from multimodal (MM) and text-only (TXT) supervision, yet the underlying alignment changes remain poorly understood. We study draft–target disagreement through an acceptance-related risk framework that decomposes distributional error into text-conditioned and visual-response components. Lower overall risk generally accompanies longer accepted sequences, while similar text-conditioned alignment can coexist with markedly different visual-response risk and acceptance. Controlled continuation training shows that high-risk groups outperform random and low-risk groups, and visual-response risk guides MM training more effectively than aggregate risk. However, useful within-modality rankings do not ensure effective cross-modal ranking: raw-score pooling overwhelmingly favors TXT examples in our candidate pools. We therefore compare risks relative to their modality-specific means rather than by absolute magnitude. Mean normalization preserves within-modality rankings and enables mixed continuation-set construction without predefined modality quotas. With an EAGLE draft for LLaVA-1.5-7B, mean-normalized joint construction improves macro-averaged accepted length across five benchmarks from 3.03 with uncalibrated raw-score pooling to 3.60, under the same 34K-example continuation budget and common warm-up.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Speculative decoding, Vision-language models, Risk decomposition, Draft-target alignment, Training data selection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_iSgAOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates how multimodal (MM) and text-only (TXT) supervision affect draft–target alignment in multimodal speculative decoding. The authors decompose distributional disagreement into text-conditioned and visual-response risks, and demonstrate that these component risks can guide training data selection. To combine MM and TXT samples, they propose a mean-normalized risk selection strategy (MNRS-Joint) that enables joint selection without predefined modality quotas. Experiments on LLaVA-1.5-7B and Qwen2.5-VL-3B show improvements in accepted length over several baselines.

Critical strengths
  1. The proposed decomposition provides insights into how different supervision modalities affect draft–target alignment. The experiments further demonstrate that visual-response risk can serve as a effective selection criterion.
  2. The paper includes controlled comparisons across different supervision settings, risk-based sampling strategies, and normalization methods. The consistent improvements across multiple benchmarks and two vision-language models provide empirical support for the proposed approach.
Critical weaknesses
  1. The proposed mean normalization is relatively simple, and its advantages over other potential cross-modal selection strategies are not fully established. Comparisons with percentile-based ranking or stronger modality-ratio baselines would better justify the proposed design.
  2. The experiments primarily report accepted length rather than actual inference latency or throughput. Moreover, the computational overhead of estimating the proposed risk scores is not quantified, making the practical efficiency trade-off unclear.
  3. The relationship between total variation distance and token rejection probability is established under stochastic speculative sampling, whereas the experiments employ greedy decoding with dynamic draft trees. The theoretical connection to the actual evaluation setting therefore requires further justification.
  4. Although the method is evaluated on two vision-language models, both experiments rely on EAGLE-style draft architectures and the same training data sources. Furthermore, the lack of results across larger models makes it difficult to assess the stability of the reported improvements.
Questions for authors
  1. How does MNRS-Joint compare with selecting high-risk samples under different fixed MM/TXT ratios or using percentile-based cross-modal rankings?
  2. Could the authors provide actual inference throughput or latency measurements, along with an analysis of the computational overhead introduced by risk-based data selection?
  3. Are the reported improvements consistent across multiple random seeds, particularly the gain of MNRS-Joint over the Fixed 1:1 High baseline?
Reviewer_Gx4EOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies EAGLE-style draft training on LLaVA-1.5-7B model with risk-guided curriculum. The proposed method isolates the effect of visual input by decomposing the distributional error into text-induced risk and the complement. The two risk components are further used to rank data samples to train the draft model for specualtive decoding. Experiments on serveral multimodal evaluation datasets show that the proposed method of using high-risk training samples improves the mean accepted token length and reduces the mean risk. Decomposed risk ranking on either modality also improves acceptance rate in the evaluations. The paper finishes with a caliberated ranking of training samples by their normalized risks in the respective modality, which resulted in the biggest gain in risk reduction.

Critical strengths
  1. The analysis on multimodal risk decomposition is grounded and convincing.
  2. The topic is timely and insights are interesting.
Critical weaknesses
  1. The paper only tests the proposed method on LLaVA-1.5-7B and Qwen2.5-VL-3B, which are obsolete and small. Does larger and more capable later models demonstrate the same behavior as shown in the paper? Does the proposed method work on those later/bigger models?
  2. The paper presents the risk supervision as a training sample selection mechanism, but for the proposed method to be meaningful to applications, it needs to work as a curriculum, where improvement come from ranking instead of selection. Would you show the results under a full-dataset training setup?
  3. This paper needs substantial revision in writing and narrative before publication to improve readability and delivery.
Questions for authors

Can you clarify how the diagnostics guide the joint supervision, as stated in the title?

Reviewer_bnniOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to choose text and image-text data to train a draft model that better matches a larger target model. It separates their disagreement into text-related and visual-response differences, then normalizes the scores to select useful examples from both sources. With the same training sample budget, this raises average accepted draft length from 3.03 with raw-score selection to 3.60, beating fixed-ratio high-risk selection at 3.45.

Critical strengths

Its main strength is turning draft–target disagreement into a practical way to choose training data, rather than just measuring it. It also combines text and image-text examples without fixing their proportions upfront, improving accepted draft length under the same sample budget.

Critical weaknesses
  1. The main experimental setup is old and fairly narrow. Most of the evidence comes from one target model and one draft architecture. A method that helps this setup might behave differently when the target understands images better or the draft is already well aligned. The Qwen results help, but a full set of experiments on stronger targets and another draft architecture would make the general claim more convincing.

  2. The biggest improvement comes against a baseline that selects almost no image-text data. Raw-score pooling picks only 18 image-text examples out of 34,000, so it is already badly imbalanced. Improving accepted length from 3.03 to 3.60 partly shows that fixing this imbalance matters. The more meaningful comparison is against fixed-ratio high-risk selection, where the gain is only 3.45 to 3.60. The authors should put more emphasis on that smaller, cleaner comparison.

  3. It is unclear whether the extra data-selection work pays for itself. Scoring requires comparing draft and target distributions, including with and without visual input for image-text samples. That adds inference work before training even begins. It could still be worthwhile, especially if the trained draft serves many requests, but the paper needs to show the scoring cost and how much deployment usage is needed to recover it.

  4. Longer accepted drafts do not automatically mean proportionally faster inference. The experiments mainly report accepted length, but users care about latency and throughput. Draft generation, verification, memory use, and batching all affect the final result. The authors should measure actual speed under several batch sizes and output lengths rather than treating accepted length as the whole efficiency story.

  5. Dividing by the mean is a useful heuristic, but it does not establish equal training value across modalities. A text sample at twice its pool’s average is not necessarily as useful as an image-text sample at twice its average. The authors acknowledge this, which is good. Still, comparisons with percentile-based selection, a sweep of modality ratios, and a stronger tuned mixing baseline would help show why this particular rule is a good choice.

Overall, the idea is sensible: diagnose the draft’s disagreement and spend training data where it needs help. The main concern is whether this is a broadly useful, cost-effective method, or a helpful selection rule for one relatively limited setup.

Questions for authors

see weakness

Reviewer_fgGDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how multimodal and text-only supervision affect draft–target alignment in multimodal speculative decoding. It decomposes distributional error into text-conditioned and visual-response components and investigates their usefulness for continuation-data selection. The proposed MNRS-Joint method normalizes modality-specific scores by their respective means before jointly selecting examples. Experiments with EAGLE-style drafts for LLaVA-1.5-7B and Qwen2.5-VL-3B show improvements in accepted length over raw-score pooling, fixed-ratio high-risk selection, and composition-matched random selection.

Critical strengths
  1. The controlled selection experiments provide useful evidence that component-wise disagreement contains training-relevant information. In particular, varying one modality while holding the other fixed, and comparing visual-response versus aggregate-risk selection at matched modality counts, help distinguish the contribution of the proposed signal from simply adding more data.

  2. The principal findings are reproduced on two target models. The composition-matched random baselines and additional budget and warm-up experiments strengthen the empirical case beyond comparison with the severely imbalanced Raw-Joint baseline.

Critical weaknesses
  1. The robustness of the incremental gains is insufficiently established. The paper reports a single random seed and no variability estimates. On LLaVA, the improvements over Fixed 1:1 High and RMS Joint are 0.15 and 0.11 in macro-averaged accepted length, respectively. Since all normalization methods preserve within-modality rankings, their selected sets differ through modality allocation. Mean and RMS normalization change the MM count by only 230 examples out of 34K, making repeated training runs particularly important for assessing whether the reported difference is reproducible. The current results support useful risk-based selection more strongly than a reliable advantage of mean normalization specifically.

  2. Practical acceleration remains insufficiently validated. Accepted length is a relevant proxy, but the paper does not report measured latency, throughput, or the preprocessing cost of scoring the candidate pools. Moreover, Appendix C.1 computes risks using unscaled softmax distributions and teacher-forced target features, whereas acceptance is evaluated with greedy dynamic-tree decoding. The paper acknowledges this distinction, but the single-token stochastic acceptance identity therefore does not directly characterize the evaluated decoding process. Direct timing measurements and evidence addressing the diagnostic-to-decoding mismatch would strengthen the practical significance.

Questions for authors
  1. Can you report repeated-run means and variability for MNRS-Joint, Fixed 1:1 High, Random-Matched, and RMS Joint, including independently sampled random subsets? Are the incremental gains stable across seeds?

  2. What are the measured end-to-end decoding latency and throughput under the stated configuration? What additional time and memory are required for candidate scoring, including the paired image-removed passes?

  3. How well do the proposed risks predict acceptance under actual draft-generated feature rollouts? Does the selection advantage persist under stochastic decoding where risk computation and decoding use matched distributions?

  4. Would a small validation-based sweep of fixed MM:TXT ratios with within-modality high-risk selection achieve comparable performance? This would help assess the benefit of the proposed allocation rule beyond the single 1:1 reference.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.