- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes PerceptionDLM, a diffusion-based multimodal model for parallel multi-region captioning. It combines RoI-aligned features, region prompting, and structured attention masking to generate descriptions for multiple regions within a single denoising process. The paper also introduces ParaDLC-Bench and shows that PerceptionDLM substantially outperforms prior diffusion VLMs while achieving better efficiency than sequential region-captioning pipelines.
Critical strengths
- Clear use of diffusion parallelism. The paper identifies a natural setting where diffusion decoding is beneficial and demonstrates real wall-clock speedups for multi-region perception.
- Strong empirical support. The method is evaluated against both diffusion and AR baselines, with useful ablations showing that region prompting and structured attention are important for avoiding cross-region interference
Critical weaknesses
Efficiency claims could be better controlled. The AR baselines are evaluated mainly with sequential per-region inference, so the current results do not fully establish an advantage over optimized batched or joint AR multi-region decoding.
Questions for authors
- Can the authors compare against a stronger AR baseline using batched or shared-feature multi-region inference?
- Can the authors provide a controlled sequential-vs-parallel comparison using the same PerceptionDLM, reporting both quality and latency?
- How does performance and memory scale beyond the region counts used during training and current experiments?
4 comments
Q3. Scaling beyond the training region count.
Because PerceptionDLM is trained with up to six learnable region prompts per sequence, scaling to denser scenes raises two practical questions: (1) How robust is the model if multiple regions are forced to share the same prompt identity? and (2) How do accuracy, latency, and memory scale when an image contains more than six regions?
- Robustness Under Prompt Collisions (Table 10 in the main paper): To test the worst-case failure mode where distinct region prompts are unavailable, Table 10 in the main paper evaluates a stress test in which all masks in an image reuse the exact same visual prompt (Qwen3.5-27B judge):
| Visual prompt assignment | Pos (%) | Neg (%) | Avg (%) |
|---|---|---|---|
| Standard assignment (unique prompts) | |||
| Same visual prompt for all masks |
- Practical Scaling and Memory Behavior for Dense Scenes: In practical deployment, scenes with many regions (e.g., or masks) do not require reusing prompts or packing all regions into a single long sequence. Instead, we partition the masks into bounded parallel groups (e.g., splitting masks into passes of ). As shown in Q2, each group uses only peak VRAM on one H100 and runs faster than sequential decoding, avoiding both prompt collisions and quadratic attention memory growth. When additional compute and GPU memory is available, multiple groups can also be processed in a tensor batch. Finally, for applications that specifically require simultaneous regions in a single sequence, the learnable visual-prompt table can be expanded with corresponding training.
Thank you for your response.
I will maintain my rating of 3. Unfortunately, I am not very familiar with this research area, so my confidence score is 2.
Dear Reviewer,
Thank you very much for taking the time to review our paper and read our rebuttal, as well as for your positive rating! Your questions helped us clarify the empirical analysis, which will make the revised manuscript much clearer to a broader readership.
Best regards,
The Authors
Thank you for recognizing the motivation and the role of the region-aware components. Below we provide the batched AR comparison, controlled same-checkpoint group scaling, and region scaling analyses.
Q1. Batched or shared-feature AR.
We benchmark batched GAR-8B (B=2 and B=4) with FlashAttention 2 and per-sequence KV caching on a single H100 GPU (full experimental details and table are provided in our response to Reviewer uxvi, Q1). Even at B=4 (228.25 s), batched GAR-8B remains slower than PerceptionDLM with step distillation at single-image batching (191 s at τ=0.5 and 149 s at τ=0.9). Regarding shared-feature AR inference: unlike general VQA models that encode an image once for multiple text queries, dedicated AR region captioners (e.g., GAR-8B and DAM) use mask-conditioned visual encoding where each region mask is injected directly into the vision backbone. Because the visual tokens themselves change with every target mask, a single vision encoding cannot be reused across regions without altering the model architecture, leaving tensor batching and KV caching as the applicable optimizations (both enabled in our B=2/B=4 baseline). In contrast, PerceptionDLM decouples global image encoding from sequence-level region binding, naturally sharing a single global image encoding across all regions in one forward pass.
Q2. Same-checkpoint sequential versus parallel inference.
Thank you for requesting a comparison that jointly reports caption quality and latency.
While Fig. 1(c) in the main paper illustrates latency scaling across group sizes, we evaluate both caption quality, wall-clock latency, and peak VRAM on the exact same PerceptionDLM checkpoint across the complete ParaDLC-Bench using native region group sizes G∈{1,2,4} (where G=1 decodes regions sequentially one by one, and G=4 decodes four regions concurrently per pass). All other settings are identical (L=32,S=32, single H100, BF16, Qwen3.5-27B judge, three runs):
This controlled comparison establishes three key findings: