acceptodds
Withdrawn from ICLR 2027

Look Only When Needed: Certificate-Guided Visual Routing for Efficient Diffusion VLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Diffusion vision-language models (DVLMs) process their entire input at every denoising step. Since visual tokens make up 89-97% of this input on the benchmarks we study, most of the decoding computation is spent repeatedly processing the same image. Visual token pruning methods partially relieve this burden by removing part of the visual tokens, but they lose the removed information and still process the remaining image at every step. To measure how often the image is actually needed, we compare LLaDA-V's predictions with and without the image at every decoding step and find that removing the image changes a token being unmasked at only 22.8% of steps on average and at as few as 7.9%. We therefore propose LOWN, a decoding method that decides at every step whether to process the image and how many tokens to unmask. LOWN predicts the masked tokens without the image and processes the image only when this prediction disagrees with the last image-conditioned prediction. Agreement certifies the earlier prediction for reuse, and the number of certified tokens sets how many are unmasked in that step. We further observe that the need for the image differs across benchmarks, so LOWN's adaptive decision is needed rather than a fixed schedule. Without training or a tuned threshold, LOWN retains 99.6% of LLaDA-V's score on ten benchmarks at 35.3% of the decoding FLOPs on average. LOWN also generalizes across LLaDA-based DVLMs, composes orthogonally with visual token pruning, and runs under key-value caching for further efficiency gains.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Diffusion Language Models, Diffusion Vision-Language Models, Efficiency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7G6uOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes LOWN, a training-free decoding method for diffusion vision-language models that adaptively skips visual processing. Motivated by observations that FULL and TEXT-ONLY predictions frequently agree during denoising, LOWN maintains an image-conditioned prediction anchor and uses agreement with a fresh TEXT-ONLY prediction to determine whether another FULL forward is necessary. The same agreement mechanism controls adaptive token unmasking. On LLaDA-V, LOWN retains 99.6% of the baseline's normalized average benchmark score using 35.3% of decoding FLOPs. Experiments additionally demonstrate transfer to LaViDa and VidLaDA, compatibility with visual-token pruning, and further acceleration with approximate KV caching.

Critical strengths
  • A meaningful efficiency formulation. LOWN identifies image-access frequency as an optimization dimension distinct from visual-token retention. Its FULL/TEXT-ONLY agreement mechanism provides a concrete, adaptive implementation of this idea.
  • Strong empirical results. The proposed method largely preserves benchmark performance while achieving substantial reductions in decoding FLOPs and latency.
Critical weaknesses
  • Prior AR-VLM works (Reasoning with Fewer Eyes [1], DyVTE [2]) and diffusion-based D3ToM [3] have explored adaptive visual computation. The manuscript insufficiently clarifies what new diffusion-specific insight its prediction-agreement routing offers beyond these existing principles.
  • The 22.8% image-dependent-step observation lacks a validated mechanism. Prediction agreement may reflect language priors or dataset biases rather than genuine visual redundancy.
  • LOWN's efficiency varies substantially with visual dependence. Although the paper acknowledges reduced savings on document and chart understanding, its behavior on tasks requiring sustained fine-grained visual grounding remains insufficiently characterized.
  • Important implementation details and the complete LaViDa results are relegated to the appendix, making the main presentation less self-contained.
  • (Minor) Figure 1b's visual-token dominance is specific to the evaluated configurations. Its generality to longer reasoning sequences remains unclear, as text-only computation may become increasingly significant.

[1] A. Ramazzina et al., "Reasoning with Fewer Eyes: Efficient Visual Token Withdrawal for Multimodal Reasoning".

[2] Q. Wu et al., "Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings."

[3] S. Chang et al., "D3ToM: Decider-Guided Dynamic Token Merging for Accelerating Diffusion MLLMs".

Questions for authors
  • Could the authors provide a latency breakdown separating FULL forwards, TEXT-ONLY forwards, and routing overhead, particularly under larger batch sizes?
  • LaViDa also provides a Dream-based variant. Could the authors evaluate LOWN on this backbone to further examine its generalizability beyond LLaDA-based diffusion models?
  • Since visual dependence is often concentrated in early denoising steps, how does LOWN compare with a simple schedule that uses FULL forwards initially and then switches permanently to TEXT-ONLY decoding? Does certificate-guided routing provide substantial additional benefits?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 7G6u (1/3)

1. Diffusion-Specific Insight beyond Fewer Eyes, DyVTE, and D3ToM

Thank you for pointing us to Fewer Eyes and DyVTE. Comparing LOWN with these two papers let us explain more clearly where it departs from prior work, and the argument of the paper is fuller for it. Like D3ToM, both start from the view that visual tokens are not needed at every stage of inference, and they cut that computation, which is also the direction LOWN takes. But Fewer Eyes and DyVTE are built for autoregressive VLMs, and only D3ToM targets a DVLM as LOWN does. The main insights behind LOWN all come from how a DVLM is built.

We first observed that the inefficiency of a DVLM lies in how often it reads the image, more than in how many visual tokens it reads. An autoregressive VLM encodes the image once and reuses its keys and values. A DVLM cannot reuse them exactly unless the model is changed, so it pays the image cost again at every step. Yet removing the image changes the unmasked tokens at only 22.8% of the steps on average. DVLMs draw interest as an alternative to autoregressive VLMs because they generate the response in parallel, but as long as the image is reprocessed at every step, that advantage does not fully reach the inference cost. Autoregressive VLMs never had this obstacle, so Fewer Eyes and DyVTE do not address it, and D3ToM reduces the number of visual tokens but still processes the image at every step. LOWN reduces the repetition itself, without retraining.

The difficulty is that the steps that need the image differ across tasks and cannot be fixed in advance. In a DVLM, the model's own predictions can tell us. A DVLM predicts every masked position at each step, so its predictions with and without the image can be compared before any token is committed. An autoregressive model predicts only the next token and offers no such comparison. Fewer Eyes is the closest to LOWN in that it drops all visual tokens during generation, but it drops them once, after a preset number of generated tokens or when the reasoning ends, and does not bring them back. DyVTE removes them after a layer that a trained hyper-network picks for each input, and does not reverse that decision either. The removal point must then be set again for each task or model, and if it is wrong, the rest of the response is generated without the image. LOWN checks at every step and reads the image again when needed, so one wrong decision does not carry over to later steps, and LOWN applies to a new task or model without tuning. Q3 shows that a fixed schedule that reads the image only at the start scores lower.

The agreement signal from this comparison decides both whether to read the image and how many tokens to unmask at a step. D3ToM, also on a DVLM, decides which visual tokens to look at, but the number of tokens follows a preset schedule and the number of steps is unchanged. LOWN uses all visual tokens at the steps where it reads the image, so it does not trade visual information for efficiency, and the same signal also reduces the number of steps. D3ToM is already a baseline in Figure 5 and Table 3. On LaViDa it keeps 79.3% of the baseline score at 0.385× FLOPs, and LOWN keeps 99.7% at 0.174×.

We have added Fewer Eyes and DyVTE to Related Work in the revision.


2. Validation of the Visual Redundancy Observation

We agree that agreement between the two predictions does not by itself show that a position is independent of the image, so we tested this in two ways.

First, we tested whether agreement separates the positions that depend on the image from those that do not. At every position LOWN commits, we ran two FULL forwards on the same partial response, one with the sample's own image and one with the image of a randomly chosen other sample, and counted how often the top-1 prediction changes (200 samples per task).

Tokens changed by the swapMathVistaMMBenchDocVQALLaVA-Bench
Committed on TEXT-ONLY steps3.5%5.3%5.4%7.4%
Committed on FULL steps42.7%36.7%56.2%61.2%

An unrelated image changes only 3.5–7.4% of the tokens committed on TEXT-ONLY steps, where the predictions agreed, and 36.7–61.2% of the tokens committed on FULL steps, where they did not. The positions where the predictions agree are the ones whose prediction barely changes with the image, and cases where the image is needed but skipped are few.

AuthorsOpenReview ↗

Response to Reviewer 7G6u (2/3)

Conversely, not every token committed on a FULL step changes. Some positions where the predictions disagree are still decided independently of the image content, and some may respond to the presence of an image in the input more than to what it shows. We accept the point that agreement does not separate image-dependent positions perfectly. But the error is on the safe side for LOWN. Reading the image when it is not needed costs some computation and does not change the response. Only skipping the image when it is needed can change the response, and that case is far less frequent. We cannot label every token as image-dependent or not, but LOWN reads the image whenever it is unsure and still reduces FLOPs to 0.353×.

Second, we tested whether this agreement comes from a language prior or a dataset bias. A language prior and a dataset bias are both available without the image, so if agreement came from them, decoding without the image from the start would give similar responses. The scores of decoding with TEXT-ONLY forwards only are below.

MMBenchMathVistaDeCapBenchChartQADocVQAGQA
LLaDA-V71.659.133.278.284.153.4
No image19.232.63.49.07.530.4

Scores drop sharply without the image. Taken together, the two results say that once the content read from the image is written into the partial response, many of the remaining positions are determined by that text, and that the same response cannot be produced without that content. The redundancy we refer to is that, at many steps, processing the image again does not change LLaDA-V's prediction. We do not claim that the answer at those positions is unrelated to the image.

By construction, a token committed on a TEXT-ONLY step is the prediction of the last FULL forward, and the TEXT-ONLY forward only tests whether that prediction still holds. A language prior therefore cannot introduce a token that the FULL forward did not predict. In addition, a fresh FULL forward at the same step predicts a different token for 2.2–4.7% of them (Table 16).


3. Behavior under Sustained Fine-Grained Grounding

LOWN reads the image often on tasks that keep needing it and less on tasks that do not. The table below splits the denoising steps of LLaDA-V, in order, into four quarters. Each cell is the share of steps (%) in that quarter at which the image changes an unmasked token. The last column is LOWN's FLOPs.

BenchmarkSteps 0–25% (early)Steps 25–50%Steps 50–75%Steps 75–100% (late)LOWN FLOPs
DocVQA59.739.531.244.80.45×
MathVista22.518.415.415.00.33×

On MathVista, the image-dependent steps are most frequent early and become fewer as decoding proceeds. On DocVQA, the share stays at 31–60% in every quarter and rises again in the last one. The steps that need the image differ across tasks, and a fixed schedule that reads the image only at the start does not fit a task like DocVQA that needs it to the end (Q3). LOWN follows these differences without a per-task setting. The smaller saving on tasks that need sustained grounding is the result of reading the image where it is needed, and we see this self-adjustment to the task as a strength of LOWN. On DocVQA, LOWN still keeps 98% of LLaDA-V's score at 0.45× FLOPs. Rates by DocVQA question type are below (a question can have several types).

DocVQA question typeQuestionsFULL per stepFLOPs
Layout1,9810.420.44×
Table or list1,7800.430.45×
Form1,0210.460.48×
Free text7650.430.46×
Handwritten3190.450.47×
Figure or diagram2650.410.43×
Image or photo980.430.45×
Yes/No280.510.53×
Others2360.440.46×

4. Generality to Longer Responses

Even when the response length is extended to 512 tokens, visual tokens still make up at least 86% of the input. LOWN's FLOPs reduction shrinks slightly as the response grows, but it remains large. LOWN still runs at 0.52–0.60× FLOPs at 512 tokens (table below). Both tasks are captioning-style tasks that describe the image at length and so read it often, which makes them far from the most favourable case for LOWN (the average over the 10 benchmarks is 0.353×). Even at this longest response length, a TEXT-ONLY forward still costs only about 12–13% of a FULL forward.

Task, response length (tokens)Visual tokens (% of input)LLaDA-VLOWNLOWN FLOPs
DeCapBench, 12895%33.234.40.39×
DeCapBench, 25692%39.940.70.51×
DeCapBench, 51286%47.750.80.60×
LLaVA-Bench, 6496%58.257.40.45×
LLaVA-Bench, 25691%54.552.80.49×
LLaVA-Bench, 51286%45.349.50.52×
AuthorsOpenReview ↗

Response to Reviewer 7G6u (3/3)

5. Implementation Details and LaViDa Results in the Main Text

In the revised version, we have moved the important implementation details into the main text, including Algorithm 1, the rule of the repetition guard, and the response length and decoding settings of each benchmark. We have also moved the LaViDa results table there.


Q1. Latency Breakdown under Larger Batches

We replayed LOWN's logged per-sample schedules in batches, running a FULL forward only for the samples that need one, and timed each component. Shares of decoding time over seven tasks are below.

BatchFULL forwardsTEXT-ONLY forwardsRoutingToken commit
158–77%22–40%0.7–1.1%0.1–0.2%
478–90%9–21%0.7–1.3%0.1–0.2%
1682–95%4–17%0.6–1.1%0.1–0.2%
3281–96%4–18%0.6–0.9%0.1%

As the batch grows, the share of TEXT-ONLY forwards falls from 22–40% to 4–18%. We attribute this to input length. A TEXT-ONLY forward has no image tokens, so its input is short and it gains much from batching, whereas a FULL forward has a long input and runs only for the samples that need it, so it gains less. This is why the share of FULL forwards rises, and it is also why LOWN's throughput gain grows with the batch size (1.43–3.62× LLaDA-V's at batch 1 and 1.75–5.50× at batch 32). Routing, which covers the agreement test and the acceptance decision, stays near 1% of the time at every batch size.


Q2. Evaluation on a Dream-Based Backbone

On LaViDa-Dream as well, LOWN keeps 95.8–99.8% of the baseline score at 0.11–0.39× FLOPs on four benchmarks. The reference is the official decoder of LaViDa-Dream. Dream pads responses with the end-of-text token, so on Dream that token is exempt from the repetition guard.

TaskLaViDa-DreamLOWNLOWN FLOPs
ChartQA61.461.30.255×
MathVista42.740.90.175×
MMBench73.873.40.112×
DeCapBench32.832.50.390×

Q3. Comparison with a Front-Loaded Schedule

We thank the reviewer for this point. To measure the effect of certificate-guided routing more precisely, we ran the ablation the reviewer suggests.

The suggested front-loaded schedule runs FULL forwards on the first m steps and then decodes from TEXT-ONLY predictions only. m is matched to the average number of FULL forwards LOWN uses on each task, and every method unmasks k tokens per step so that only the timing of the image reads differs. Certificate places the FULL forwards with LOWN's agreement test.

We also include a decreasing schedule. The suggested schedule stops reading the image entirely after a point, and this abrupt switch could by itself decide the outcome. We added a fixed schedule that places the same m FULL forwards densely at first and more sparsely later, so that the image is still read now and then in the later steps (Appendix F.3 of the revision gives the exact placement). The table reports scores.

TaskmFront-loadedDecreasingCertificate (fixed k)
MathVista1557.349.057.6
ScienceQA469.770.577.4
ChartQA448.728.977.8
MME4979.3946.81624.6
MMBench442.032.469.4
DocVQA780.978.283.5
CapArena26−96.7−97.2−86.8
LLaVA-Bench1547.246.157.9
GQA436.537.152.5
DeCapBench2627.225.433.0

With the same number of FULL forwards, the front-loaded schedule scores lower than Certificate on all ten benchmarks. The decreasing schedule, which still reads the image in the later steps, is also lower on all ten. The gap therefore comes less from the permanent switch to TEXT-ONLY than from fixing in advance when the image is read. As the reviewer notes, this schedule comes close to the certificate on a task whose image dependence is concentrated early. MathVista is such a task (see the table in our answer to point 3), and there the gap is only 0.3 points. On the other tasks, tokens that depend on the image remain after the early steps, and switching to TEXT-ONLY permanently lowers the score sharply (42.0 against 69.4 on MMBench, 36.5 against 52.5 on GQA, 48.7 against 77.8 on ChartQA). Which tasks behave like MathVista is not known in advance. The certificate returns to a FULL forward whenever one is needed after TEXT-ONLY steps, so it keeps the score regardless of the task. This gap is the benefit that certificate-guided routing adds.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer,

We have posted our response to your comments. As the discussion period closes soon, we would like to check whether it addresses your concerns. We would be grateful if you could take a look when your schedule allows. If you have any further questions, we would be glad to address them before the discussion closes.

Thank you again for your time and thoughtful feedback!

Best regards,

Authors of LOWN

Reviewer_hj6NOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposed LOWN, a decoding method which starts from the observation that diffusion VLMs like LLaDA-V reprocess the full image at every denoising step, even though removing the image changes the token being unmasked on only about 23% of steps. At each step, LOWN runs a cheap TEXT-ONLY forward and compares its predictions to an anchor from the last image-conditioned (FULL) forward. If a long enough confidence-ordered prefix agrees, it skips the image and unmasks that whole prefix; if not, it runs a FULL forward and refreshes the anchor. It keeps 99.6% of LLaDA-V’s score across ten benchmarks at 0.35x the decoding FLOPs, and it transfers to LaViDa and VidLaDA.

Critical strengths

S1. The method is based on clear empirical insights. It provides a detailed full vs text-only run analysis for multiple benchmarks.

S2. The method is simple, training-free and needs no per-task tuning. It is plug-and-play and composes with existing visual token pruning and KV caching methods.

Critical weaknesses

W1. My major concern is how this differs from, and why it is needed over, prefix KV caching (Fast-dLLM [1], Elastic-Cache [2]). The motivation assumes the image is fully reprocessed at every step, but caching methods already avoid this by reusing the image and prompt K/V and refreshing them only periodically or when drift is detected. Under caching, skipping the image saves only the attention over the cached image keys, so the paper’s cost argument largely no longer applies. The paper does not clearly explain what LOWN offers beyond what prefix caching already gives.

W2. The efficiency gain is measured against uncached LLaDA-V, but KV caching is the standard way these models are run. The paper does not compare standalone LOWN against prefix-caching methods. Reading Table 2 against Table 1, Fast-dLLM alone appears both cheaper and better than standalone LOWN on the three shared benchmarks. Adding LOWN on top of Fast-dLLM lowers the score on all three, and only three of the ten benchmarks are evaluated. Elastic-Cache is cited but not compared at all. The authors should include caching methods as standalone baselines in Table 1.

W3. Closely related ideas, such as T-GATE [3], and VTW [4] are not properly discussed.


References

  1. Wu, Chengyue, et al. "Fast-dllm: Training-free acceleration of diffusion llm by enabling kv cache and parallel decoding." International Conference on Learning Representations. Vol. 2026. 2026.

  2. Nguyen-Tri, Quan, Mukul Ranjan, and Zhiqiang Shen. "Attention is all you need for kv cache in diffusion llms." International Conference on Learning Representations. Vol. 2026. 2026.

  3. Liu, Haozhe, et al. "Faster diffusion through temporal attention decomposition." Transactions on Machine Learning Research (2025).

  4. Lin, Zhihang, et al. "Boosting multimodal large language models with visual tokens withdrawal for rapid inference." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 5. 2025.

Questions for authors

please check weaknesses for questions.

3 comments
AuthorsOpenReview ↗

Response to Reviewer hj6N (1/2)

Thank you for the detailed comments on the relation to prefix KV caching. Comparing LOWN directly with the caching methods you mention let us show more clearly how LOWN differs from a cache and what it saves on top of one.

1. Comparison with Prefix KV Caching

A prefix cache and LOWN reuse different things. A cache stores the keys and values of the image and prompt and reuses them at every step. LOWN keeps the predictions of the last FULL forward and reuses them at the positions where a TEXT-ONLY forward agrees. The reviewer notes that, under caching, skipping the image saves only the attention over the cached image keys. But even with a KV cache, most of the FLOPs (90–99%) are spent on refreshes that reprocess the whole image, and LOWN reduces the number of these refreshes, from 16 to 6.2 per response on CapArena and from 13 to 3.9 on MathVista. A cost therefore remains for LOWN to reduce on top of a cache.

The FLOPs a cache saves come at a price, and the largest part of it is memory. A cache has to hold the keys and values of the image and prompt for every sample between steps, 1.9 GiB per sample for Fast-dLLM's KV cache and 3.7 GiB for Elastic-Cache. Each sample added to a batch raises the peak memory by 2.3 GiB with the KV cache and by 4.1 GiB with Elastic-Cache. At batch 8, the extra memory of the cache already exceeds the 16 GiB needed to load all of LLaDA-V's weights on the GPU. At batch 32 the KV cache needs about 90 GiB, and at batch 64 it no longer fits on a 140 GiB GPU. Elastic-Cache already needs about 82 GiB at batch 16 and exceeds 140 GiB at batch 32. LLaDA-V and LOWN run up to batch 256 on the same GPU (Appendix F.10 and Figure 11 of the revision), because LOWN keeps almost no state between steps and so uses the same memory as LLaDA-V. A comparison of FLOPs does not show this cost, but in serving it is memory that sets the batch size that can be processed.

The score of a cache also depends on its settings, and how far the refresh interval can be stretched differs across tasks. On MathVista the score holds up to a refresh every 4 forwards and falls sharply from 8, whereas on DeCapBench it barely changes even at 32. The Fast-dLLM paper reports a similar drop on LLaDA-V (56.6 against 59.2 of MathVista). The parallel decoding threshold of Fast-dLLM shows the same pattern. Raising it to four times the value in the released code lowers FLOPs by a further 44–51% on the four long-response tasks, but MathVista and DeCapBench drop by 1.6 and 1.4 points while CapArena and LLaVA-Bench drop by 8.5 and 11.1. A cache needs a setting that keeps its score, and the margin for it differs across tasks. LOWN has no value to tune and follows the same rule on all ten benchmarks.

As the reviewer suggests, we compared LOWN directly with the caching methods on all ten benchmarks. We did not tune the cache settings per task. Each setting is fixed to one value for all benchmarks, chosen so that the score does not drop much on average. The refresh interval is the longest one that the Fast-dLLM paper uses on LLaDA-V without a loss in accuracy, the parallel decoding threshold is the value in the released code, and Elastic-Cache also runs with a single refresh threshold that is not tuned per task. KV cache in the table is Fast-dLLM's cache, which stores and reuses the keys and values of the image and prompt. Cells are scores from the same judges as Table 1, with FLOPs relative to LLaDA-V in parentheses.

BenchmarkLLaDA-VLOWNKV cacheElastic-CacheKV cache + Parallel Decoding (Fast-dLLM)KV cache + LOWN
MathVista59.156.0 (0.325)58.2 (0.288)50.1 (0.252)57.9 (0.220)56.2 (0.110)
ScienceQA76.578.8 (0.243)77.9 (0.255)77.4 (0.242)79.0 (0.171)77.8 (0.094)
ChartQA78.278.4 (0.576)78.8 (0.298)78.3 (0.251)78.6 (0.178)77.9 (0.201)
MME17191655 (0.231)1726 (0.253)1735 (0.241)1749 (0.155)1650 (0.080)
MMBench71.670.6 (0.270)71.0 (0.254)71.5 (0.221)70.8 (0.160)70.9 (0.099)
DocVQA83.982.3 (0.452)83.5 (0.256)81.7 (0.297)83.5 (0.211)81.8 (0.146)
CapArena−88.7−85.7 (0.388)−87.8 (0.276)−91.0 (0.256)−86.5 (0.268)−86.2 (0.128)
LLaVA-Bench58.257.4 (0.454)56.9 (0.263)60.4 (0.244)56.4 (0.255)58.0 (0.141)
GQA53.353.2 (0.205)53.7 (0.254)53.8 (0.228)53.8 (0.150)53.1 (0.075)
DeCapBench33.234.4 (0.391)33.9 (0.275)33.4 (0.257)34.1 (0.267)34.6 (0.128)
Average100.0%99.6% (0.353)100.2% (0.267)98.7% (0.249)100.5% (0.203)99.5% (0.120)
AuthorsOpenReview ↗

Response to Reviewer hj6N (2/2)

Compared alone, the caching methods use fewer FLOPs than LOWN and their average scores are similar or higher, as the reviewer points out. That saving is paid for with the memory and the settings above, and LOWN keeps 99.6% of LLaDA-V's score at 0.353× FLOPs without paying either. Where a cache can be used, LOWN is added on top of it.

Adding LOWN to the cache lowers FLOPs further on every benchmark, to less than half of the cache's on average, and the average score stays at 99.5%. The cost a cache leaves is the refresh, and the refresh is what LOWN reduces. On the four long-response tasks (MathVista, CapArena, LLaVA-Bench, DeCapBench), KV cache + LOWN keeps 100.4% of LLaDA-V's score at 0.127× FLOPs, half the cost of Fast-dLLM's parallel decoding on the same cache (99.9% at 0.252×). Where a cache cannot be used, LOWN is a way to lower FLOPs with no extra memory, and where it can, LOWN gives the lowest average FLOPs on top of it.

We also compared the two at matched FLOPs (Appendix F.11 and Table 27 of the revision). When Fast-dLLM's parallel decoding on the KV cache is raised on each task to the largest threshold whose FLOPs are still not below those of KV cache + LOWN, it keeps 96.7% of LLaDA-V's score on average, against 99.5% for KV cache + LOWN at lower FLOPs. The gap comes from the long responses of CapArena, LLaVA-Bench and DeCapBench.


2. Discussion with T-GATE and VTW

Thank you for pointing us to these two papers. Comparing LOWN with them helped us state more clearly where it stands. T-GATE starts from the observation that the text condition is used mostly in the early denoising steps, and it skips the computation on the condition in the later steps. VTW starts from the observation that visual tokens are barely needed in deep layers, and it withdraws them from those layers on. Both methods assume that the conditioning input is not needed at every stage of inference and skip its computation, and in this respect they point in the same direction as LOWN.

The two methods differ from LOWN in what they target. T-GATE works on diffusion models that generate images from text, so the task is different. VTW works on autoregressive VLMs, so the type of model is different. LOWN targets diffusion VLMs, which read an image and generate text.

As a result, each method removes a different cost. An autoregressive VLM processes the image once and reuses its keys and values, so what VTW saves is the computation on visual tokens in deep layers. A diffusion VLM recomputes the whole image at every denoising step, and LOWN reduces this repeated cost. T-GATE reuses the output of the cross-attention layers in models where the condition enters through separate cross-attention layers. A diffusion VLM has no separate layer that handles only the image. Its image tokens are processed together with the question and the response as one input, so this approach cannot be applied as it is. Diffusion VLMs can generate a response in parallel and are developing quickly as an alternative to autoregressive VLMs, but because they have to reprocess the image at every step, this advantage does not easily turn into lower inference cost. LOWN reduces this structural cost without retraining the model, and we see its value in lowering one barrier to deploying diffusion VLMs in real systems.

The gate step of T-GATE and the layer of VTW are fixed in advance and applied to every sample in the same way, and once the computation is skipped it is not brought back. LOWN checks, for each sample and at each denoising step, whether the prediction made without the image agrees with the prediction made with it, and it can read the image again after a step that skipped it. The step at which the image is needed differs by task, so a fixed schedule that reads the image only in the first m steps scores lower on all ten benchmarks than one that decides the timing with the agreement test. The same agreement signal also sets how many tokens are unmasked in a step.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer,

We have posted our response to your comments. As the discussion period closes soon, we would like to check whether it addresses your concerns. We would be grateful if you could take a look when your schedule allows. If you have any further questions, we would be glad to address them before the discussion closes.

Thank you again for your time and thoughtful feedback!

Best regards,

Authors of LOWN

Reviewer_Us5uOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LOWN, an adaptive decoding method that reduces redundant visual-token processing in diffusion vision-language models. Motivated by the observation that many denoising steps produce similar predictions with and without visual inputs, LOWN uses agreement between text-only predictions and previously cached full-model predictions to decide when to access visual tokens and how many tokens to unmask. The paper also demonstrates that LOWN can maintain performance while reducing FLOPs and generation time across 10 LLaDA-V tasks.

Critical strengths
  1. The paper identifies the source of computational redundancy in diffusion VLMs. The paired FULL and TEXT-ONLY analysis provides empirical evidence that visual inputs do not substantially change selected unmasking decisions at many denoising steps.

  2. The proposed method is simple and intuitive, using prediction agreement to jointly control visual access and token acceptance. It does not require additional training or modifications to the underlying model architecture, and its adaptive design can allocate more computation to steps where visual information appears important.

  3. The experimental evaluation is reasonably comprehensive, covering multiple multimodal benchmarks and reporting both theoretical FLOPs and actual generation latency. The authors also provide budget-matched comparisons and ablations that help distinguish the benefits of adaptive visual access from those of reducing the total number of decoding steps.

Critical weaknesses
  1. It is a little bit unclear whether adaptive visual access significantly outperforms simpler fixed schedules. Since visual dependence is concentrated in early denoising steps, front-loaded or decreasing schedules may offer similar savings without token-level adaptivity. Periodic and random baselines do not fully address this, especially if token-acceptance policies differ. The authors may add comparisons controlling for decoding policies and compute budgets to better isolate the benefits of adaptivity.

  2. Some design choices of LOWN are not sufficiently justified by the ablations. In particular, the method accepts the longest agreeing prefix in confidence order rather than all agreeing candidates, and uses an additional repetition guard to avoid undesirable parallel unmasking behavior. It is unclear how much these choices contribute to quality and efficiency, or whether they introduce unnecessary FULL refreshes. Ablating the prefix-acceptance rule and repetition guard would help determine which components are essential to the reported improvements.

Questions for authors
  1. How strongly does disagreement between accepted tokens and fresh FULL predictions correlate with final-answer errors? Can the authors analyze whether disagreements involving answer-critical tokens are disproportionately responsible for the observed performance drops?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Us5u (1/2)

We thank the reviewer for the close reading of how LOWN decides when to read the image and which tokens to accept. The comparisons and ablations the reviewer suggested let us show more clearly where the agreement test helps.

1. Comparison with Fixed Image-Access Schedules

Following the suggestion, we ran front-loaded and decreasing schedules. Because LOWN accepts several agreeing tokens at once and so changes the decoding process itself, all three methods here unmask exactly k tokens per step and differ only in when FULL forwards run. The reference, Certificate, uses LOWN's agreement test to decide when to run a FULL forward but keeps the number of unmasked tokens fixed at k. The front-loaded schedule runs FULL forwards on the first m steps and then unmasks the top-k TEXT-ONLY predictions. The decreasing schedule is the same except that the m FULL forwards are placed densely at first and more sparsely later (the i-th is at step T(i/m)^2, rounded, for i = 0, …, m−1, and a response that ends early skips the later ones). m is matched to the average number of FULL forwards LOWN uses on the task. The table below reports the score of each schedule on the ten benchmarks.

TaskmFront-loadedDecreasingCertificate (fixed k)
MathVista1557.349.057.6
ScienceQA469.770.577.4
ChartQA448.728.977.8
MME4979.3946.81624.6
MMBench442.032.469.4
DocVQA780.978.283.5
CapArena26−96.7−97.2−86.8
LLaVA-Bench1547.246.157.9
GQA436.537.152.5
DeCapBench2627.225.433.0

Both fixed schedules are below Certificate on all ten benchmarks. A fixed schedule places its FULL forwards in advance, whether or not the remaining tokens depend on the image, whereas Certificate reads the image again at any step where the TEXT-ONLY prediction disagrees. The gap is large on tasks that keep needing the image, such as ChartQA (77.8 against 48.7) and MMBench (69.4 against 42.0).

We also compare the periodic and random schedules under two unmasking rules. Fixed k is the setting of Table 5 in the paper and unmasks k tokens per step, while LOWN acceptance gives the schedule LOWN's agreement test on its TEXT-ONLY steps so that agreeing tokens are accepted together. When fewer than k positions agree, a fixed schedule fills the step from the TEXT-ONLY prediction. The random schedule uses LOWN's FULL rate on each task. Cells are scores with FLOPs relative to LLaDA-V in parentheses, and Avg. is the mean score relative to LOWN.

ScheduleDecoding strategyMathVistaLLaVA-BenchDeCapBenchAvg.
LOWNLOWN acceptance56.0 (0.325)57.4 (0.454)34.4 (0.391)100.0% (0.390)
Period 2Fixed k57.0 (0.541)56.1 (0.516)33.4 (0.522)98.9% (0.526)
Period 4Fixed k52.0 (0.321)49.0 (0.274)27.8 (0.284)86.3% (0.293)
Period 8Fixed k48.5 (0.211)41.3 (0.154)19.1 (0.164)71.3% (0.176)
RandomFixed k53.4 (0.349)53.8 (0.456)30.5 (0.389)92.5% (0.398)
Period 2LOWN acceptance55.3 (0.318)54.1 (0.416)33.3 (0.375)96.6% (0.370)
Period 4LOWN acceptance50.8 (0.246)42.5 (0.254)24.5 (0.244)78.7% (0.248)
Period 8LOWN acceptance48.8 (0.182)38.9 (0.153)17.5 (0.156)68.6% (0.164)
RandomLOWN acceptance55.8 (0.373)55.4 (0.491)32.1 (0.422)96.5% (0.429)

As Table 5 of the paper already shows, with fixed k Period 4 uses similar FLOPs to LOWN on MathVista but scores 4.0 points lower, and the random schedule, matched to LOWN's FULL rate, is below LOWN on all three tasks. Likewise, with LOWN acceptance Period 4 and Period 8 remain lower on all three tasks, and the random schedule uses more FLOPs for a lower score. Fixing when FULL forwards run therefore falls short of LOWN even when the unmasking rule is the same. For the periodic schedules, LOWN acceptance also lowers FLOPs by 7–30% on average, but the average score drops with it.

AuthorsOpenReview ↗

Response to Reviewer Us5u (2/2)

2. Ablation of the Prefix Rule and the Repetition Guard

As the reviewer points out, the submission did not show how much the confidence-ordered prefix rule and the repetition guard each contribute. We thank the reviewer for raising this and ran additional experiments that remove each rule and both. No guard turns off only the repetition guard, and Accept all agreeing accepts every candidate whose TEXT-ONLY prediction agrees with the anchor instead of stopping at the first disagreement, with everything else as in LOWN of Table 1. We tested four benchmarks of different character, namely MathVista (long reasoning), MMBench (short multiple choice), DocVQA (fine-grained document reading) and DeCapBench (long captions). Cells are scores, with the difference from LOWN on the same examples in parentheses.

TaskLOWNNo guardAccept all agreeingBoth removed
MathVista56.055.7 (−0.3)54.3 (−1.7)55.4 (−0.6)
MMBench70.670.6 (+0.0)68.2 (−2.4)68.1 (−2.5)
DocVQA82.382.0 (−0.3)80.6 (−1.7)77.0 (−5.3)
DeCapBench34.430.4 (−4.0)23.0 (−11.4)7.2 (−27.2)
FLOPs vs. LOWN−9 to −19%−17 to −52%−32 to −87%

Without the repetition guard, FLOPs fall by 9–19% but DeCapBench drops by 4.0 points, and the share of responses that loop a phrase (e.g., "the the the the") rises from 0.8% to 28.5% on DeCapBench and from 4.6% to 18.2% on MathVista. MathVista scores only the extracted answer, so its score does not show this. The guard prevents the repetition that arises when several tokens are accepted in one step, and its effect appears in the score on long captions, where the whole response is judged.

Accepting every agreeing candidate instead of the confidence-ordered prefix lowers FLOPs by 17–52% but lowers the score on all four tasks, by 11.4 points on DeCapBench. The prefix rule treats the first disagreement as a sign that the anchor needs refreshing and does not accept the less confident positions after it, and without this limit too many tokens are accepted at once from an outdated anchor.

Removing both rules lowers FLOPs by 32–87% but costs 5.3 points on DocVQA and 27.2 on DeCapBench. The FULL forwards these two rules add are therefore what keeps caption and document quality.


3. Relation between Disagreements and Final Errors

LOWN accepts tokens on TEXT-ONLY steps without reading the image again, so at some positions a FULL forward at that step would have predicted a different token. To see whether such disagreements lead to final errors, we took about 300 samples per task from seven tasks and ran an extra FULL forward at every step where LOWN accepts tokens, comparing its prediction with the accepted token (this FULL forward is not used for decoding). 2.9% of the accepted tokens differ from the fresh FULL prediction. We then computed, for each sample, the share of its accepted tokens that differ, grouped the samples by this share, and compared LOWN's error rate in each group with LLaDA-V's on the same samples. If disagreements caused errors, LOWN's error rate would rise above LLaDA-V's as the share grows.

Share of accepted tokens that differSamplesLLaDA-V wrongLOWN wrong
01,34633.1%32.7%
up to 5%22940.6%44.5%
5–10%35430.8%30.8%
10–20%10132.7%34.7%
above 20%6536.9%35.4%

LOWN's error rate stays within 4 points of LLaDA-V's in every group and does not grow with the share.

To check whether disagreements on answer-critical tokens matter more, we took MME, MMBench, GQA and DocVQA, where the answer span can be located in the response, and compared the error rate of samples with a disagreement inside the span against those without. Disagreements inside the span show no clear association with errors either (the odds of an error are 1.23 times those without such a disagreement, 95% CI [0.72, 2.09]).

In the revision, the schedule comparisons of point 1 are in Appendix F.3, the ablation of point 2 in Appendix F.4, and the error analysis of point 3 in Appendix F.5.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer,

We have posted our response to your comments. As the discussion period closes soon, we would like to check whether it addresses your concerns. We would be grateful if you could take a look when your schedule allows. If you have any further questions, we would be glad to address them before the discussion closes.

Thank you again for your time and thoughtful feedback!

Best regards,

Authors of LOWN

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their time and thoughtful feedback. We are encouraged that the reviewers found LOWN to be a simple, training-free method built on a clear empirical insight, with strong results. Below we summarize the strengths noted across the reviews, and we address each weakness and question in our individual responses.

Strengths

  1. Clear insight into the redundancy of diffusion VLMs (Reviewers Us5u, hj6N, 7G6u). The reviewers noted that the paired FULL and TEXT-ONLY analysis shows that the image does not change the unmasked token at many denoising steps. Reviewer 7G6u described how often the image is read as an efficiency dimension distinct from how many visual tokens are kept.

  2. Simple, training-free method (Reviewers Us5u, hj6N). The reviewers found the method simple and intuitive, with one agreement signal deciding both when to read the image and how many tokens to unmask, and with no training, no change to the model and no per-task tuning.

  3. Complementary to existing acceleration (Reviewer hj6N). LOWN is plug-and-play and composes with visual-token pruning and KV caching.

  4. Comprehensive evaluation and strong results (Reviewers Us5u, 7G6u). LOWN largely preserves benchmark performance while substantially reducing decoding FLOPs and latency, and the evaluation includes budget-matched comparisons and ablations.

Revision

We uploaded a revised version of the paper, with the changes marked in blue. Following the reviews, it adds comparisons with fixed schedules and KV caching, further ablations and analyses, and results on longer responses, larger batches and LaViDa-Dream.

We hope that the additional experiments and clarifications in our responses address the reviewers' concerns, and we will do our best to resolve any remaining questions during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.