- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes LOWN, a training-free decoding method for diffusion vision-language models that adaptively skips visual processing. Motivated by observations that FULL and TEXT-ONLY predictions frequently agree during denoising, LOWN maintains an image-conditioned prediction anchor and uses agreement with a fresh TEXT-ONLY prediction to determine whether another FULL forward is necessary. The same agreement mechanism controls adaptive token unmasking. On LLaDA-V, LOWN retains 99.6% of the baseline's normalized average benchmark score using 35.3% of decoding FLOPs. Experiments additionally demonstrate transfer to LaViDa and VidLaDA, compatibility with visual-token pruning, and further acceleration with approximate KV caching.
Critical strengths
- A meaningful efficiency formulation. LOWN identifies image-access frequency as an optimization dimension distinct from visual-token retention. Its FULL/TEXT-ONLY agreement mechanism provides a concrete, adaptive implementation of this idea.
- Strong empirical results. The proposed method largely preserves benchmark performance while achieving substantial reductions in decoding FLOPs and latency.
Critical weaknesses
- Prior AR-VLM works (Reasoning with Fewer Eyes [1], DyVTE [2]) and diffusion-based D3ToM [3] have explored adaptive visual computation. The manuscript insufficiently clarifies what new diffusion-specific insight its prediction-agreement routing offers beyond these existing principles.
- The 22.8% image-dependent-step observation lacks a validated mechanism. Prediction agreement may reflect language priors or dataset biases rather than genuine visual redundancy.
- LOWN's efficiency varies substantially with visual dependence. Although the paper acknowledges reduced savings on document and chart understanding, its behavior on tasks requiring sustained fine-grained visual grounding remains insufficiently characterized.
- Important implementation details and the complete LaViDa results are relegated to the appendix, making the main presentation less self-contained.
- (Minor) Figure 1b's visual-token dominance is specific to the evaluated configurations. Its generality to longer reasoning sequences remains unclear, as text-only computation may become increasingly significant.
[1] A. Ramazzina et al., "Reasoning with Fewer Eyes: Efficient Visual Token Withdrawal for Multimodal Reasoning".
[2] Q. Wu et al., "Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings."
[3] S. Chang et al., "D3ToM: Decider-Guided Dynamic Token Merging for Accelerating Diffusion MLLMs".
Questions for authors
- Could the authors provide a latency breakdown separating FULL forwards, TEXT-ONLY forwards, and routing overhead, particularly under larger batch sizes?
- LaViDa also provides a Dream-based variant. Could the authors evaluate LOWN on this backbone to further examine its generalizability beyond LLaDA-based diffusion models?
- Since visual dependence is often concentrated in early denoising steps, how does LOWN compare with a simple schedule that uses FULL forwards initially and then switches permanently to TEXT-ONLY decoding? Does certificate-guided routing provide substantial additional benefits?
4 comments
Response to Reviewer 7G6u (2/3)
Conversely, not every token committed on a FULL step changes. Some positions where the predictions disagree are still decided independently of the image content, and some may respond to the presence of an image in the input more than to what it shows. We accept the point that agreement does not separate image-dependent positions perfectly. But the error is on the safe side for LOWN. Reading the image when it is not needed costs some computation and does not change the response. Only skipping the image when it is needed can change the response, and that case is far less frequent. We cannot label every token as image-dependent or not, but LOWN reads the image whenever it is unsure and still reduces FLOPs to 0.353×.
Second, we tested whether this agreement comes from a language prior or a dataset bias. A language prior and a dataset bias are both available without the image, so if agreement came from them, decoding without the image from the start would give similar responses. The scores of decoding with TEXT-ONLY forwards only are below.
| MMBench | MathVista | DeCapBench | ChartQA | DocVQA | GQA | |
|---|---|---|---|---|---|---|
| LLaDA-V | 71.6 | 59.1 | 33.2 | 78.2 | 84.1 | 53.4 |
| No image | 19.2 | 32.6 | 3.4 | 9.0 | 7.5 | 30.4 |
Scores drop sharply without the image. Taken together, the two results say that once the content read from the image is written into the partial response, many of the remaining positions are determined by that text, and that the same response cannot be produced without that content. The redundancy we refer to is that, at many steps, processing the image again does not change LLaDA-V's prediction. We do not claim that the answer at those positions is unrelated to the image.
By construction, a token committed on a TEXT-ONLY step is the prediction of the last FULL forward, and the TEXT-ONLY forward only tests whether that prediction still holds. A language prior therefore cannot introduce a token that the FULL forward did not predict. In addition, a fresh FULL forward at the same step predicts a different token for 2.2–4.7% of them (Table 16).
3. Behavior under Sustained Fine-Grained Grounding
LOWN reads the image often on tasks that keep needing it and less on tasks that do not. The table below splits the denoising steps of LLaDA-V, in order, into four quarters. Each cell is the share of steps (%) in that quarter at which the image changes an unmasked token. The last column is LOWN's FLOPs.
| Benchmark | Steps 0–25% (early) | Steps 25–50% | Steps 50–75% | Steps 75–100% (late) | LOWN FLOPs |
|---|---|---|---|---|---|
| DocVQA | 59.7 | 39.5 | 31.2 | 44.8 | 0.45× |
| MathVista | 22.5 | 18.4 | 15.4 | 15.0 | 0.33× |
On MathVista, the image-dependent steps are most frequent early and become fewer as decoding proceeds. On DocVQA, the share stays at 31–60% in every quarter and rises again in the last one. The steps that need the image differ across tasks, and a fixed schedule that reads the image only at the start does not fit a task like DocVQA that needs it to the end (Q3). LOWN follows these differences without a per-task setting. The smaller saving on tasks that need sustained grounding is the result of reading the image where it is needed, and we see this self-adjustment to the task as a strength of LOWN. On DocVQA, LOWN still keeps 98% of LLaDA-V's score at 0.45× FLOPs. Rates by DocVQA question type are below (a question can have several types).
| DocVQA question type | Questions | FULL per step | FLOPs |
|---|---|---|---|
| Layout | 1,981 | 0.42 | 0.44× |
| Table or list | 1,780 | 0.43 | 0.45× |
| Form | 1,021 | 0.46 | 0.48× |
| Free text | 765 | 0.43 | 0.46× |
| Handwritten | 319 | 0.45 | 0.47× |
| Figure or diagram | 265 | 0.41 | 0.43× |
| Image or photo | 98 | 0.43 | 0.45× |
| Yes/No | 28 | 0.51 | 0.53× |
| Others | 236 | 0.44 | 0.46× |
4. Generality to Longer Responses
Even when the response length is extended to 512 tokens, visual tokens still make up at least 86% of the input. LOWN's FLOPs reduction shrinks slightly as the response grows, but it remains large. LOWN still runs at 0.52–0.60× FLOPs at 512 tokens (table below). Both tasks are captioning-style tasks that describe the image at length and so read it often, which makes them far from the most favourable case for LOWN (the average over the 10 benchmarks is 0.353×). Even at this longest response length, a TEXT-ONLY forward still costs only about 12–13% of a FULL forward.
| Task, response length (tokens) | Visual tokens (% of input) | LLaDA-V | LOWN | LOWN FLOPs |
|---|---|---|---|---|
| DeCapBench, 128 | 95% | 33.2 | 34.4 | 0.39× |
| DeCapBench, 256 | 92% | 39.9 | 40.7 | 0.51× |
| DeCapBench, 512 | 86% | 47.7 | 50.8 | 0.60× |
| LLaVA-Bench, 64 | 96% | 58.2 | 57.4 | 0.45× |
| LLaVA-Bench, 256 | 91% | 54.5 | 52.8 | 0.49× |
| LLaVA-Bench, 512 | 86% | 45.3 | 49.5 | 0.52× |
Response to Reviewer 7G6u (3/3)
5. Implementation Details and LaViDa Results in the Main Text
In the revised version, we have moved the important implementation details into the main text, including Algorithm 1, the rule of the repetition guard, and the response length and decoding settings of each benchmark. We have also moved the LaViDa results table there.
Q1. Latency Breakdown under Larger Batches
We replayed LOWN's logged per-sample schedules in batches, running a FULL forward only for the samples that need one, and timed each component. Shares of decoding time over seven tasks are below.
| Batch | FULL forwards | TEXT-ONLY forwards | Routing | Token commit |
|---|---|---|---|---|
| 1 | 58–77% | 22–40% | 0.7–1.1% | 0.1–0.2% |
| 4 | 78–90% | 9–21% | 0.7–1.3% | 0.1–0.2% |
| 16 | 82–95% | 4–17% | 0.6–1.1% | 0.1–0.2% |
| 32 | 81–96% | 4–18% | 0.6–0.9% | 0.1% |
As the batch grows, the share of TEXT-ONLY forwards falls from 22–40% to 4–18%. We attribute this to input length. A TEXT-ONLY forward has no image tokens, so its input is short and it gains much from batching, whereas a FULL forward has a long input and runs only for the samples that need it, so it gains less. This is why the share of FULL forwards rises, and it is also why LOWN's throughput gain grows with the batch size (1.43–3.62× LLaDA-V's at batch 1 and 1.75–5.50× at batch 32). Routing, which covers the agreement test and the acceptance decision, stays near 1% of the time at every batch size.
Q2. Evaluation on a Dream-Based Backbone
On LaViDa-Dream as well, LOWN keeps 95.8–99.8% of the baseline score at 0.11–0.39× FLOPs on four benchmarks. The reference is the official decoder of LaViDa-Dream. Dream pads responses with the end-of-text token, so on Dream that token is exempt from the repetition guard.
| Task | LaViDa-Dream | LOWN | LOWN FLOPs |
|---|---|---|---|
| ChartQA | 61.4 | 61.3 | 0.255× |
| MathVista | 42.7 | 40.9 | 0.175× |
| MMBench | 73.8 | 73.4 | 0.112× |
| DeCapBench | 32.8 | 32.5 | 0.390× |
Q3. Comparison with a Front-Loaded Schedule
We thank the reviewer for this point. To measure the effect of certificate-guided routing more precisely, we ran the ablation the reviewer suggests.
The suggested front-loaded schedule runs FULL forwards on the first m steps and then decodes from TEXT-ONLY predictions only. m is matched to the average number of FULL forwards LOWN uses on each task, and every method unmasks k tokens per step so that only the timing of the image reads differs. Certificate places the FULL forwards with LOWN's agreement test.
We also include a decreasing schedule. The suggested schedule stops reading the image entirely after a point, and this abrupt switch could by itself decide the outcome. We added a fixed schedule that places the same m FULL forwards densely at first and more sparsely later, so that the image is still read now and then in the later steps (Appendix F.3 of the revision gives the exact placement). The table reports scores.
| Task | m | Front-loaded | Decreasing | Certificate (fixed k) |
|---|---|---|---|---|
| MathVista | 15 | 57.3 | 49.0 | 57.6 |
| ScienceQA | 4 | 69.7 | 70.5 | 77.4 |
| ChartQA | 4 | 48.7 | 28.9 | 77.8 |
| MME | 4 | 979.3 | 946.8 | 1624.6 |
| MMBench | 4 | 42.0 | 32.4 | 69.4 |
| DocVQA | 7 | 80.9 | 78.2 | 83.5 |
| CapArena | 26 | −96.7 | −97.2 | −86.8 |
| LLaVA-Bench | 15 | 47.2 | 46.1 | 57.9 |
| GQA | 4 | 36.5 | 37.1 | 52.5 |
| DeCapBench | 26 | 27.2 | 25.4 | 33.0 |
With the same number of FULL forwards, the front-loaded schedule scores lower than Certificate on all ten benchmarks. The decreasing schedule, which still reads the image in the later steps, is also lower on all ten. The gap therefore comes less from the permanent switch to TEXT-ONLY than from fixing in advance when the image is read. As the reviewer notes, this schedule comes close to the certificate on a task whose image dependence is concentrated early. MathVista is such a task (see the table in our answer to point 3), and there the gap is only 0.3 points. On the other tasks, tokens that depend on the image remain after the early steps, and switching to TEXT-ONLY permanently lowers the score sharply (42.0 against 69.4 on MMBench, 36.5 against 52.5 on GQA, 48.7 against 77.8 on ChartQA). Which tasks behave like MathVista is not known in advance. The certificate returns to a FULL forward whenever one is needed after TEXT-ONLY steps, so it keeps the score regardless of the task. This gap is the benefit that certificate-guided routing adds.
Gentle Reminder
Dear Reviewer,
We have posted our response to your comments. As the discussion period closes soon, we would like to check whether it addresses your concerns. We would be grateful if you could take a look when your schedule allows. If you have any further questions, we would be glad to address them before the discussion closes.
Thank you again for your time and thoughtful feedback!
Best regards,
Authors of LOWN
Response to Reviewer 7G6u (1/3)
1. Diffusion-Specific Insight beyond Fewer Eyes, DyVTE, and D3ToM
Thank you for pointing us to Fewer Eyes and DyVTE. Comparing LOWN with these two papers let us explain more clearly where it departs from prior work, and the argument of the paper is fuller for it. Like D3ToM, both start from the view that visual tokens are not needed at every stage of inference, and they cut that computation, which is also the direction LOWN takes. But Fewer Eyes and DyVTE are built for autoregressive VLMs, and only D3ToM targets a DVLM as LOWN does. The main insights behind LOWN all come from how a DVLM is built.
We first observed that the inefficiency of a DVLM lies in how often it reads the image, more than in how many visual tokens it reads. An autoregressive VLM encodes the image once and reuses its keys and values. A DVLM cannot reuse them exactly unless the model is changed, so it pays the image cost again at every step. Yet removing the image changes the unmasked tokens at only 22.8% of the steps on average. DVLMs draw interest as an alternative to autoregressive VLMs because they generate the response in parallel, but as long as the image is reprocessed at every step, that advantage does not fully reach the inference cost. Autoregressive VLMs never had this obstacle, so Fewer Eyes and DyVTE do not address it, and D3ToM reduces the number of visual tokens but still processes the image at every step. LOWN reduces the repetition itself, without retraining.
The difficulty is that the steps that need the image differ across tasks and cannot be fixed in advance. In a DVLM, the model's own predictions can tell us. A DVLM predicts every masked position at each step, so its predictions with and without the image can be compared before any token is committed. An autoregressive model predicts only the next token and offers no such comparison. Fewer Eyes is the closest to LOWN in that it drops all visual tokens during generation, but it drops them once, after a preset number of generated tokens or when the reasoning ends, and does not bring them back. DyVTE removes them after a layer that a trained hyper-network picks for each input, and does not reverse that decision either. The removal point must then be set again for each task or model, and if it is wrong, the rest of the response is generated without the image. LOWN checks at every step and reads the image again when needed, so one wrong decision does not carry over to later steps, and LOWN applies to a new task or model without tuning. Q3 shows that a fixed schedule that reads the image only at the start scores lower.
The agreement signal from this comparison decides both whether to read the image and how many tokens to unmask at a step. D3ToM, also on a DVLM, decides which visual tokens to look at, but the number of tokens follows a preset schedule and the number of steps is unchanged. LOWN uses all visual tokens at the steps where it reads the image, so it does not trade visual information for efficiency, and the same signal also reduces the number of steps. D3ToM is already a baseline in Figure 5 and Table 3. On LaViDa it keeps 79.3% of the baseline score at 0.385× FLOPs, and LOWN keeps 99.7% at 0.174×.
We have added Fewer Eyes and DyVTE to Related Work in the revision.
2. Validation of the Visual Redundancy Observation
We agree that agreement between the two predictions does not by itself show that a position is independent of the image, so we tested this in two ways.
First, we tested whether agreement separates the positions that depend on the image from those that do not. At every position LOWN commits, we ran two FULL forwards on the same partial response, one with the sample's own image and one with the image of a randomly chosen other sample, and counted how often the top-1 prediction changes (200 samples per task).
An unrelated image changes only 3.5–7.4% of the tokens committed on TEXT-ONLY steps, where the predictions agreed, and 36.7–61.2% of the tokens committed on FULL steps, where they did not. The positions where the predictions agree are the ones whose prediction barely changes with the image, and cases where the image is needed but skipped are few.