acceptodds
Under review as a conference paper at ICLR 2027

PerceptionDLM: Parallel Region Perception with Multimodal Diffusion Language Models

Abstract

Multimodal large language models (MLLMs) have achieved remarkable progress in visual understanding tasks. However, most existing MLLMs rely on autoregressive generation, which limits their efficiency for perception tasks that require captioning multiple regions. In this work, we introduce , a stronger multimodal diffusion language baseline that integrates diffusion language models (DLMs) with a visual encoder, achieving state-of-the-art performance among open-source diffusion-based MLLMs. Then, by leveraging the parallel decoding nature of DLMs, we further propose an efficient prompting mechanism that enables simultaneous perception of multiple masked regions, allowing the model to generate region descriptions in parallel at both the sequence and token levels. This design significantly improves inference efficiency compared with existing approaches that process regions sequentially. To better evaluate the parallelism property of visual perception capability for DLMs, we construct a new llel etailed ocalized aptioning Benchmark () by scaling the DLC-Bench to include multiple region masks per image, enabling joint evaluation of both caption quality and inference speed. Experiments demonstrate that PerceptionDLM maintains competitive region captioning performance while achieving substantial speed improvements for multi-region perception tasks. Our results highlight the potential of diffusion-based multimodal language models for efficient parallel visual perception. Code and model will be released.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Diffusion Language Models, Multimodal Large Language Models, Vision-Language Models, Region-Level Perception, Parallel Generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bfwNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PerceptionDLM, a diffusion-based multimodal model for parallel multi-region captioning. It combines RoI-aligned features, region prompting, and structured attention masking to generate descriptions for multiple regions within a single denoising process. The paper also introduces ParaDLC-Bench and shows that PerceptionDLM substantially outperforms prior diffusion VLMs while achieving better efficiency than sequential region-captioning pipelines.

Critical strengths
  1. Clear use of diffusion parallelism. The paper identifies a natural setting where diffusion decoding is beneficial and demonstrates real wall-clock speedups for multi-region perception.
  2. Strong empirical support. The method is evaluated against both diffusion and AR baselines, with useful ablations showing that region prompting and structured attention are important for avoiding cross-region interference
Critical weaknesses

Efficiency claims could be better controlled. The AR baselines are evaluated mainly with sequential per-region inference, so the current results do not fully establish an advantage over optimized batched or joint AR multi-region decoding.

Questions for authors
  1. Can the authors compare against a stronger AR baseline using batched or shared-feature multi-region inference?
  2. Can the authors provide a controlled sequential-vs-parallel comparison using the same PerceptionDLM, reporting both quality and latency?
  3. How does performance and memory scale beyond the region counts used during training and current experiments?
4 comments

Thank you for recognizing the motivation and the role of the region-aware components. Below we provide the batched AR comparison, controlled same-checkpoint group scaling, and region scaling analyses.

Q1. Batched or shared-feature AR.

We benchmark batched GAR-8B (B=2B=2 and B=4B=4) with FlashAttention 2 and per-sequence KV caching on a single H100 GPU (full experimental details and table are provided in our response to Reviewer uxvi, Q1). Even at B=4B=4 (228.25 s228.25\text{ s}), batched GAR-8B remains slower than PerceptionDLM with step distillation at single-image batching (191 s191\text{ s} at τ=0.5\tau=0.5 and 149 s149\text{ s} at τ=0.9\tau=0.9). Regarding shared-feature AR inference: unlike general VQA models that encode an image once for multiple text queries, dedicated AR region captioners (e.g., GAR-8B and DAM) use mask-conditioned visual encoding where each region mask is injected directly into the vision backbone. Because the visual tokens themselves change with every target mask, a single vision encoding cannot be reused across regions without altering the model architecture, leaving tensor batching and KV caching as the applicable optimizations (both enabled in our B=2/B=4B=2/B=4 baseline). In contrast, PerceptionDLM decouples global image encoding from sequence-level region binding, naturally sharing a single global image encoding across all regions in one forward pass.


Q2. Same-checkpoint sequential versus parallel inference.

Thank you for requesting a comparison that jointly reports caption quality and latency.

While Fig. 1(c) in the main paper illustrates latency scaling across group sizes, we evaluate both caption quality, wall-clock latency, and peak VRAM on the exact same PerceptionDLM checkpoint across the complete ParaDLC-Bench using native region group sizes G∈{1,2,4}G \in \{1, 2, 4\} (where G=1G=1 decodes regions sequentially one by one, and G=4G=4 decodes four regions concurrently per pass). All other settings are identical (L=32,S=32L=32, S=32, single H100, BF16, Qwen3.5-27B judge, three runs):

Region group size (GG)Avg (%)Full-benchmark inference time (s, mean ±\pm SD)Peak allocated VRAM (GiB)
1176.2076.20743.15±3.27743.15 \pm 3.2723.6723.67
2274.5274.52440.16±2.56440.16 \pm 2.5623.8523.85
4473.7073.70281.26±1.56281.26 \pm 1.5624.2124.21

This controlled comparison establishes three key findings:

  1. Minimal accuracy trade-off: Moving from sequential inference (G=1G=1) to 44-region parallel inference (G=4G=4) changes Avg accuracy by only 2.502.50 percentage points (76.2076.20\\% to 73.7073.70\\%), confirming that our structured attention mask effectively isolates concurrent regions during denoising.
  2. Substantial wall-clock speedup: G=2G=2 and G=4G=4 achieve 1.69×1.69\times and 2.64×2.64\times speedups over G=1G=1, with G=4G=4 saving 62.262.2\\% of total inference time.
  3. Near-constant memory overhead: Peak VRAM increases by only 0.54 GiB0.54\text{ GiB} (+2.3+2.3\\%) from G=1G=1 (23.67 GiB23.67\text{ GiB}) to G=4G=4 (24.21 GiB24.21\text{ GiB}). Because the shared global image tokens dominate sequence length, appending three additional 3232-token caption slots per pass incurs negligible memory cost. In conclusion, parallel inference on the same checkpoint delivers a 2.64×2.64\times speedup with virtually flat memory usage and minimal quality degradation.

Q3. Scaling beyond the training region count.

Because PerceptionDLM is trained with up to six learnable region prompts per sequence, scaling to denser scenes raises two practical questions: (1) How robust is the model if multiple regions are forced to share the same prompt identity? and (2) How do accuracy, latency, and memory scale when an image contains more than six regions?

  1. Robustness Under Prompt Collisions (Table 10 in the main paper): To test the worst-case failure mode where distinct region prompts are unavailable, Table 10 in the main paper evaluates a stress test in which all masks in an image reuse the exact same visual prompt (Qwen3.5-27B judge):
Visual prompt assignmentPos (%)Neg (%)Avg (%)
Standard assignment (unique prompts)59.459.487.587.573.573.5
Same visual prompt for all masks52.852.884.484.468.668.6
  1. Practical Scaling and Memory Behavior for Dense Scenes: In practical deployment, scenes with many regions (e.g., 1212 or 2020 masks) do not require reusing prompts or packing all regions into a single long sequence. Instead, we partition the masks into bounded parallel groups (e.g., splitting 1212 masks into 33 passes of G=4G=4). As shown in Q2, each G=4G=4 group uses only 24.21 GiB24.21\text{ GiB} peak VRAM on one H100 and runs 2.64×2.64\times faster than sequential decoding, avoiding both prompt collisions and quadratic attention memory growth. When additional compute and GPU memory is available, multiple G=4G=4 groups can also be processed in a tensor batch. Finally, for applications that specifically require >6> 6 simultaneous regions in a single sequence, the learnable visual-prompt table can be expanded with corresponding training.

Thank you for your response.

I will maintain my rating of 3. Unfortunately, I am not very familiar with this research area, so my confidence score is 2.

Dear Reviewer,

Thank you very much for taking the time to review our paper and read our rebuttal, as well as for your positive rating! Your questions helped us clarify the empirical analysis, which will make the revised manuscript much clearer to a broader readership.

Best regards,

The Authors

Reviewer_XjLDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes PerceptionDLM, a multimodal diffusion language model for efficient parallel region captioning. Built upon PerceptionDLM-Base, the method incorporates region prompting, RoI-aligned features, and structured attention masking to generate multiple region descriptions simultaneously. Experiments demonstrate competitive captioning quality and improved inference efficiency.

Critical strengths

S1. Meaningful motivation and practical efficiency gains. The paper identifies multi-region captioning as a suitable application for diffusion language models, effectively leveraging their parallel decoding capabilities. The proposed method achieves substantial inference speedups as the number of regions increases, highlighting its practical value.

S2. Comprehensive experimental evaluation. The paper provides extensive comparisons with existing AR and diffusion-based VLMs, along with detailed ablation studies. The construction of ParaCaption-5.7M and ParaDLC-Bench further strengthens the empirical contributions and supports systematic evaluation of multi-region perception.

Critical weaknesses

W1. Limited methodological novelty and generalizability. PerceptionDLM-Base largely follows existing diffusion VLM architectures and training strategies, while the proposed parallel perception framework combines established techniques such as RoI-aligned features, learnable region embeddings, and structured attention masking. Moreover, the proposed mechanism is only validated on region captioning, making its broader applicability unclear.

W2. Insufficient comparison with parallel AR baselines. The reported efficiency gains mainly come from comparisons with AR models that process regions sequentially. However, AR models can also generate captions for independent regions concurrently through batched inference. Without such comparisons, it is difficult to determine whether the speedups are specific to diffusion decoding or simply result from parallelizing region-level generation. Additionally, PerceptionDLM exhibits a non-trivial accuracy gap compared with specialized AR-based models (62.4% vs. 69.5% on ParaDLC-Bench), raising concerns about the quality-efficiency trade-off.

Questions for authors

Q1.Could the authors clarify the key methodological novelty beyond combining existing components? Furthermore, can the proposed parallel decoding framework be generalized to other multi-output multimodal tasks beyond region captioning?

Q2. Could the authors compare PerceptionDLM against AR-based region captioning models using batched or parallel inference under matched hardware settings? A more comprehensive accuracy-latency comparison would help establish whether the proposed method provides a favorable quality-efficiency trade-off over optimized AR baselines.

3 comments

Thank you for the positive assessment of the motivation and evaluation. We clarify our methodological contribution and present the batched AR comparison below.

W1/Q1. Methodological contribution.

While PerceptionDLM-Base builds on a multimodal diffusion backbone and adopts RoI feature replay from prior single-region work (Section 3.2 in the main paper), directly applying a diffusion VLM to multi-region perception fails. Unlike sequential AR models that process one region per forward pass, a parallel diffusion model denoises multiple region descriptions simultaneously within a single sequence. Without explicit target binding and attention isolation, features from different regions interfere with one another during denoising, causing severe attribute leakage and identity confusion across targets.

Our methodological contribution is a joint region-binding and denoising-isolation mechanism designed for concurrent multi-region diffusion. Specifically, we spatially fuse learnable region embeddings with masked visual tokens to anchor each target's identity, while our structured attention mask restricts each caption span to attend only to the global image context, the shared instruction, its own RoI features, and its own caption tokens. The controlled DAM-only ablations in Table 7 of the main paper demonstrate why this joint design is necessary: the full model achieves 53.753.7\\%, whereas removing region prompting causes the model to lose target correspondence and collapse to 1.11.1\\%, replacing our structured mask with full attention degrades accuracy by 6.26.2 points (to 47.547.5\\%), and removing RoI replay reduces accuracy by 2.42.4 points (to 51.351.3\\%).

In conclusion, our novelty lies not in proposing RoI pooling or attention masks in isolation, but in formulating a coupled binding-and-isolation mechanism that makes parallel multi-region denoising effective within a shared visual context.

Discussion on Generalizability. In this paper, we validate our framework on dense multi-region captioning as a representative multi-output generation task. More broadly, our parallel decoding design applies to multi-output vision-language tasks that satisfy two structural conditions: (1) multiple queries share the same global visual context, and (2) each output slot maps to an individually addressable target, such as multi-region attribute QA, dense object grounding/categorization, and multi-box scene text recognition. Conversely, tasks that require explicit reasoning across output slots (e.g., pairwise visual relationship detection or joint scene-graph generation) cannot use strict inter-slot isolation throughout the entire generation process. In such settings, our structured attention mask can be relaxed in later denoising steps to allow controlled cross-slot communication after initial target binding. We will revise the manuscript to explicitly discuss these applicability conditions and boundaries.


W2/Q2. Optimized AR baselines and quality–latency trade-off.

We evaluate batched GAR-8B under matched hardware and precision. To avoid repetition across responses, we report the full setup and latency table in our response to Reviewer uxvi (Q1). Briefly, while batching reduces GAR-8B's runtime from 309.06 s309.06\text{ s} (B=2B=2) to 228.25 s228.25\text{ s} (B=4B=4), PerceptionDLM with step distillation remains 1.53×1.53\times faster even at single-image batching (191 s191\text{ s} at τ=0.5\tau=0.5 and 149 s149\text{ s} at τ=0.9\tau=0.9, Table 14 in the main paper), and image-level batching can also be applied to PerceptionDLM. Regarding the accuracy–latency trade-off, specialized AR models such as GAR-8B achieve higher accuracy (69.569.5\\% vs. 62.462.4\\% for native PerceptionDLM) by leveraging mature autoregressive LLM pretraining and causal token-by-token decoding, whereas prior diffusion VLMs struggle on dense region captioning. Thus, GAR-8B and PerceptionDLM serve complementary operating regimes: GAR-8B favors peak accuracy at higher latency, while PerceptionDLM establishes a new state of the art among diffusion VLMs and offers a substantially faster solution for latency-sensitive multi-region perception.

Thank you for the rebuttal and for addressing my concerns. I will maintain my score of 3.

Dear Reviewer,

Thank you very much for reading our rebuttal and for your positive support of our work! We are glad that our responses addressed your concerns, and we will incorporate all these additional analyses and clarifications into the revised manuscript.

Best regards,

The Authors

Reviewer_uxviOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PerceptionDLM, a multimodal diffusion language model for parallel region captioning. The authors first develop PerceptionDLM-Base through large-scale multimodal training, then introduce region prompting, RoI-aligned features, and structured attention masking to generate descriptions for multiple regions simultaneously. They also introduce ParaDLC-Bench to evaluate caption quality and inference efficiency. Experiments show substantial speed improvements over sequential region captioning methods while maintaining competitive caption quality.

Critical strengths
  1. Clear motivation and practical value. The paper addresses the efficiency limitations of multi-region captioning. Using diffusion language models to generate multiple region descriptions simultaneously is an interesting direction with practical value.
  2. Effective architecture and extensive experiments. The proposed approach achieves meaningful speedups and demonstrates strong performance among diffusion VLMs. The ablation studies support the effectiveness of region prompting and structured attention. The large-scale training data and ParaDLC-Bench are also useful contributions.
Critical weaknesses
  1. The efficiency comparison with AR models is not sufficiently convincing. The main comparisons use AR models that process regions sequentially. However, AR models can also generate captions for multiple regions using batch decoding. Without a batched AR baseline and matched output lengths, it is difficult to determine whether the speed advantage comes from diffusion decoding or simply from parallelizing region requests. Moreover, PerceptionDLM still performs worse than specialized AR models in caption quality.
  2. The evaluation and performance attribution need further support. ParaDLC-Bench contains only 100 images and relies on LLM-based evaluation. Although the authors test multiple judges, agreement with human judgments is not reported. Furthermore, PerceptionDLM-Base and the final model use substantial additional training data, making it difficult to separate the contributions of training scale and architectural design.
Questions for authors
  1. Could the authors compare PerceptionDLM with AR region captioning models using optimized batch decoding and matched output lengths? This would help establish the actual efficiency advantage of diffusion-based generation.
  2. How does the speed and quality trade-off change when generating longer captions or processing more regions? It would be helpful to evaluate latency under comparable caption quality and output lengths.
  3. Could the authors provide human evaluation results or agreement statistics to further validate the reliability of the LLM judges used in ParaDLC-Bench?
  4. Could the authors further clarify how much of the performance improvement comes from the proposed architecture rather than the large-scale training data? A controlled comparison under the same training data would strengthen the conclusions.
4 comments

Thank you for recognizing the practical motivation and the effectiveness of the region-aware design. We address each of your questions below.

Q1. Optimized AR decoding and matched lengths. To evaluate optimized AR decoding, we benchmark GAR-8B with independent region-mask sequences at batch sizes of 22 and 44 (B=2B=2 and B=4B=4) across all 100100 images / 299299 regions. Each configuration runs three times on a single H100 GPU. Under natural stopping, GAR-8B generates an average of 4545 tokens per region, matching the length regime of our 3232-token setting. End-to-end generation time drops by 26.126.1\\% from 309.06±5.79 s309.06 \pm 5.79\text{ s} (B=2B=2) to 228.25±3.86 s228.25 \pm 3.86\text{ s} (B=4B=4):

ConfigurationWall Block Time (s)
GAR-8B, B=2B=2309.06309.06
GAR-8B, B=4B=4228.25228.25
PerceptionDLM + Distill, B=1B=1, τ=0.5\tau=0.5 (Table 14 in paper)191191
PerceptionDLM + Distill, B=1B=1, τ=0.9\tau=0.9 (Table 14 in paper)149149

Analysis and conclusion: Even when GAR-8B uses batching (B=4B=4, 228.25 s228.25\text{ s}), PerceptionDLM with step distillation remains faster at single-image batching (B=1B=1), finishing in 191 s191\text{ s} (τ=0.5\tau=0.5) and 149 s149\text{ s} (τ=0.9\tau=0.9, 1.53×1.53\times faster). This speedup stems from a fundamental difference in how computation is parallelized: AR batching stacks separate region sequences along the batch dimension and still decodes tokens sequentially one step at a time. In contrast, PerceptionDLM shares a single global image encoding and denoises both multiple regions and multiple token positions simultaneously within one sequence. Because sequence-level parallelism is orthogonal to batching, PerceptionDLM can also batch multiple images when GPU memory permits, confirming that our efficiency advantage comes from multi-region diffusion decoding rather than unbatched AR baselines.


Q2. Longer captions and more regions.

Tables 9 and 11 in the main paper evaluate how caption length (LL) and denoising steps (SS) affect the quality–latency trade-off under the Qwen3.5-27B judge. At L=32L=32, varying S∈{16,32,48,64}S \in \{16, 32, 48, 64\} yields Avg scores of 69.069.0\\%, 73.573.5\\%, 70.570.5\\%, and 67.567.5\\% at 138138, 276276, 411411, and 549 s549\text{ s}. At fixed S=32S=32, increasing LL from 3232 to 6464 lowers Avg from 73.573.5\\% to 68.068.0\\% at the same latency because longer captions require proportionally more denoising steps to converge. These results explain why L=32,S=32L=32, S=32 serves as our default operating point: it achieves the highest accuracy among tested settings without paying for redundant denoising steps.

To evaluate scaling across more regions, we further test native region group sizes G∈{1,2,4}G \in \{1, 2, 4\} in ParaDLC-Bench:

Region group size (GG)Avg (%)Full-benchmark time (s)TPF
1176.2076.20743.15±3.27743.15 \pm 3.271.001.00
2274.5274.52440.16±2.56440.16 \pm 2.561.731.73
4473.7073.70281.26±1.56281.26 \pm 1.562.772.77

Conclusion: Increasing concurrency from G=1G=1 to G=4G=4 raises tokens per forward (TPF) from 1.001.00 to 2.772.77 and delivers a 2.64×2.64\times wall-clock speedup (from 743.15 s743.15\text{ s} to 281.26 s281.26\text{ s}) with only a 2.502.50-point difference in Avg accuracy, demonstrating that PerceptionDLM scales to multiple concurrent regions with minimal quality loss (see Reviewer bfwN Q2 for VRAM analysis).


Q3. Human validation and reliability of the judge.

Beyond human verification of the benchmark questions (Appendix D.2) and cross-judge evaluations with Gemini-3.1-Pro and Qwen3.5-27B (Table 8 in the main paper), we conduct a blinded human audit to directly measure human–judge agreement. Two independent human annotators score 3030 randomly sampled images (one region per image, retaining all 259259 questions) on PerceptionDLM (G=4G=4) captions. Both annotators answer every question independently with model identities and Qwen3.5-27B judgments hidden. Confidence intervals use 10,00010{,}000 paired bootstrap draws clustered by image:

ComparisonExact option agreement95% image-cluster CI
Annotator 1–Qwen86.4986.49\\% (224/259224/259)[82.17[82.17\\%, 90.51\\%]
Annotator 2–Qwen86.1086.10\\% (223/259223/259)[81.30[81.30\\%, 90.57\\%]
Annotator 1–Annotator 291.8991.89\\% (238/259238/259)[88.43[88.43\\%, 95.06\\%]

Conclusion: The LLM judge reaches >86> 86\\% exact agreement with both human annotators, closely approaching the 91.8991.89\\% human inter-annotator agreement. This high consistency is enabled by ParaDLC-Bench's reference-free, discriminative multiple-choice protocol: the judge checks whether objective visual attributes are present, omitted, or contradicted rather than scoring open-ended fluency or style. Together with Table 8 in the paper, this audit confirms that the LLM judge provides reliable and reproducible evaluations.

Q4. Architecture versus training data.

Tables 7 and 5 in the main paper disentangle the contributions of our architecture and training data through two controlled comparisons:

  1. Controlled architecture ablation on identical data (Table 7 in the main paper): When all variants are trained exclusively on the DAM dataset, the full PerceptionDLM design achieves 53.753.7\\% Avg on ParaDLC-Bench. Removing structured attention (reverting to full attention) drops accuracy by 6.26.2 points to 47.547.5\\% due to cross-region interference; removing RoI feature replay reduces accuracy by 2.42.4 points to 51.351.3\\%; and removing region prompting causes multi-region generation to collapse to 1.11.1\\% (−52.6-52.6 points) because the model cannot bind caption slots to their corresponding target regions.
  2. Controlled data scaling under the same design (Table 5 in the main paper): Holding the full architecture fixed (under the same design), expanding the training data from DAM-only (53.753.7\\%) to DAM + COCONut (57.757.7\\%) and further to DAM + COCONut + SAM (62.462.4\\%) yields a +8.7+8.7-point improvement.

Conclusion: These controlled comparisons confirm that our architectural design and scaled dataset play distinct, complementary roles: architectural design is essential to enable parallel multi-region generation without cross-region confusion, while dataset scaling further improves visual diversity and coverage.

Thanks for the rebuttal. It addressed my concerns. I will raise my rating to 3 accordingly.

Dear Reviewer uxvi,

Thank you very much for reading our rebuttal and for raising your rating! We are glad that our responses and additional experiments addressed your concerns, and we will incorporate all these results and clarifications into the revised manuscript.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.