acceptodds
Withdrawn from ICLR 2027

SPROUT: Syntax-Progressive Denoising for Detailed Captioning in Diffusion VLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models (VLMs) are increasingly expected to produce long, detailed image descriptions, yet additional detail often comes with increased hallucination, as attributes, relations, and secondary objects are generated from linguistic priors rather than visual evidence. We attribute part of this failure to flat caption supervision, which treats sentence skeletons and fine-grained modifiers as an undifferentiated sequence. We introduce SPROUT (Syntax-PRogressive Ordered Unmasking Training), a syntax-progressive framework for detailed captioning in masked diffusion VLMs that turns the denoising axis into a within-example curriculum. We construct a dataset in which each human-written caption is decomposed into five nested levels, yielding a syntactic tier for every word. During training, a tier-conditioned corruption schedule masks fine-detail tokens earlier than skeleton tokens, shifting detail supervision toward contexts where the ground-truth sentence skeleton is visible. SPROUT further incorporates schedule-consistent weighting, same-tier block masking, and termination supervision, while requiring no architectural changes or additional inference-time computation. Across hallucination benchmarks (CHAIR, AMBER, THRONE, and OpenCHAIR) and detailed-captioning benchmarks (Image Paragraph, IIW-400, LN-COCO, and DOCCI-test), SPROUT improves fine-grained faithfulness over both the pretrained backbone and a same-data uniform-SFT baseline while preserving coverage and descriptive quality. It also retains general multimodal capability on MME and MMBench. Tier-stratified analysis further reveals emergent skeleton-first generation under the unchanged confidence decoder.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Diffusion Vision-Language Models, Image Captioning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HZUyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SPROUT, a training framework for improving the faithfulness of detailed image captions generated by masked diffusion vision-language models. The authors construct five levels of caption complexity from DOCCI through progressive simplification and assign each word a tier. A tier-dependent masking schedule and corresponding loss weighting shift supervision of detail tokens toward contexts in which the ground-truth sentence skeleton is more likely to be visible. The method also incorporates same-tier block masking and termination supervision, while retaining the backbone’s original confidence-based decoding procedure. Experiments on LLaDA-V compare SPROUT with the pretrained backbone, same-data uniform SFT, data-ordering curricula, and several masking and tier-construction variants. The results show improvements on multiple hallucination and detailed-captioning metrics. An analysis of token commitment order also shows a stronger skeleton-before-detail tendency after training.

Critical strengths
  1. SPROUT makes effective use of the partially observed contexts available in masked diffusion training. Proposition 1 distinguishes supervision allocation from simply reducing the loss weight on detail tokens: the integrated coefficient is preserved while the surrounding context changes.

  2. Relative to same-data uniform SFT, SPROUT improves AMBER coverage from 59.8 to 67.2 and THRONE F1 from 72.0 to 75.9, while reducing OpenCHAIR from 13.7 to 10.5 (Table 1). This supports a contribution from the schedule beyond block masking alone. The caption-quality results in Table 2 provide complementary evidence.

Critical weaknesses
  1. Proposition 1 establishes greater visibility of the ground-truth skeleton during training. It does not establish increased reliance on visual evidence or ensure that the skeleton is correct during free generation. Similarly, Figure 4 shows average commitment order across tiers, but does not by itself establish that generating the skeleton earlier causes fewer hallucinations in subsequent details. The paper lists tier-stratified object- and attribute-level hallucination evaluation among its contributions, yet I could not find the corresponding error rates, coverage measurements, or uncertainty estimates in the supplied manuscript. The reversed-schedule control mainly reports commitment order rather than the associated faithfulness–coverage trade-off. This leaves the connection between the observed ordering and the performance gains unresolved.

  2. The hallucination improvements are also not uniform: in Table 1, SPROUT has a CHAIR_s of 28.0, compared with 18.8 for uniform SFT. Overall, the evidence more strongly supports improvements on several aggregate metrics than the proposed fine-grained mechanism.

  3. Severe degradation in several ablations remains insufficiently explained, and important controls and implementation details are missing. In Table 4, combining the tier-conditioned schedule with per-token masking reduces AMBER coverage to 22.5 and the average object count to 0.7. Shuffled, frequency-based, and POS-based tiers also lead to substantial under-description. These results reveal sensitivity to the design choices, but describing them as evidence of strong component coupling does not rule out premature termination, optimization instability, or configuration effects.

  4. Section 3.2 states that termination supervision is ablated separately, but I could not locate that result. It is also unclear whether uniform SFT uses exactly the same termination-token filling and training canvas.

Questions for authors
  1. Could you provide the tier-stratified object and attribute hallucination evaluation described in the paper, including coverage, for SPROUT, uniform scheduling with block masking, and the reversed schedule? How are tiers assigned to generated captions in Figure 4, how many examples are evaluated, and how stable are the differences in commitment order?

  2. Do all controlled variants in Tables 1, 4, and 5 share the same termination-token filling, loss normalization, training canvas, and decoding settings? What are the results of the separate termination-supervision ablation mentioned in Section 3.2? For the severely degraded variants, do output lengths and premature-termination rates explain the behavior, and does it recur across random seeds?

  3. The reverse transition in Appendix A.1 uses tier-dependent (r_{k,i}), and joint unmasking depends on the span partition. Both are derived from the target caption. Should Equation (4) therefore be understood as a NELBO for a model conditioned on these tiers and partitions? If it is intended to describe a model conditioned only on the image and prompt, how are the additional variables defined or marginalized?

Reviewer_UbsGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the faithfulness gap in detailed captioning by masked diffusion vision-language models: under uniform corruption, fine-grained modifier tokens are often supervised while the subject-verb-object skeleton they depend on remains masked, so the model learns from local regularities rather than from an established scene structure. It proposes SPROUT, which replaces the shared masking probability with a tier-conditioned forward schedule in which fine-detail tokens are corrupted earlier in forward time and revealed later in reverse time, while skeleton tokens become visible earlier. Experimental results show that SPROUT improves fine-grained faithfulness.

Critical strengths
  1. SPROUT addresses a concrete limitation of uniform masking, with theoretical guarantees on skeleton visibility and supervision allocation.
  2. The appendix provides formal justification.
  3. The hierarchical annotations are supported by constrained LLM simplification, quality checks, and selection-bias analysis.
  4. Experiment results show that it improves faithfulness metrics across benchmarks.
Critical weaknesses
  1. The method is evaluated only on LLaDA-V. The effectiveness on other masked diffusion models is unclear, which weakens its effectiveness.
  2. Despite the stated RQ4, the paper lacks ablations on schedule exponent ratios, making their selection appear heuristic.
  3. The tier hierarchy is constructed from a relatively small, English-only DOCCI dataset, and comparisons with inference-time hallucination mitigation methods are missing.
  4. The training time and computational overhead relative to uniform SFT are not reported.
Questions for authors
  • Can the authors report SPROUT trained on at least one additional diffusion backbone?
Reviewer_2f44OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on errors in detailed image descriptions generated by diffusion models. The authors divide words into five levels and change how often words at each level are hidden during training. The aim is to help the model learn details with more of the basic sentence visible. The experiments use LLaDA-V and compare the method with standard training on the same data and other ways of organizing training.

Critical strengths
  1. The paper measures both errors and the amount of detail in image descriptions. This helps check whether a model avoids errors simply by saying less.

  2. The method only changes training. The model uses the same steps to generate descriptions afterward.

  3. The main comparison uses models trained on the same data. The authors also test different ways of grouping and hiding words. Some results favor SPROUT: on AMBER, it makes fewer errors about objects and mentions more relevant objects than standard training.

Critical weaknesses
  1. The paper does not yet give enough evidence for its main explanation: seeing more of the basic sentence during training leads to fewer errors in the details. Most reported scores concern objects or overall description quality. They do not clearly show fewer mistakes about colors or how objects relate to each other.

  2. Table 4 shows a large drop when the proposed rule hides words separately instead of hiding groups together. The model mentions only 0.7 objects per description, compared with 9.3 for the full method. Its THRONE F1 score is also much lower: 37.1 versus 75.9. It is unclear whether this version ends its descriptions too early or has problems during training. Without a clear explanation, this large gap alone tells us little about why hiding words in groups helps.

  3. In Table 1, CHAIR_i falls from 7.8 to 6.0 compared with uniform SFT, but CHAIR_s rises from 18.8 to 28.0. The average object count changes only from 9.1 to 9.3. More descriptions contain errors, which makes the overall benefit unclear.

  4. Several ideas used here already appear in earlier work. DiffusionBERT [1] varies hiding probabilities across words, TreeDiff [2] hides groups based on code structure, and the f′/ff'/f weight follows from MDLM [3]. SPROUT differs in its task and design. Its main addition seems to be the five levels built by deleting text and their use for image descriptions. How much this adds beyond the earlier methods remains unclear.

Questions for authors
  1. Figure 4 shows that reversing the schedule changes the order of generation. Does it also change the error rate or description quality? This would help assess whether generating the basic sentence first contributes to the improvement.

  2. The five word levels and their hiding rules are central to the method. For the manual check in Appendix A.5, how many captions were reviewed, what counted as a wrong label, and how often were errors found? Appendix A.3 explains the overall scale of γ=(4.5,3,1.5,0.9,0.6)\gamma=(4.5, 3, 1.5, 0.9, 0.6). How were the relative values chosen, and do small changes give similar results?

  3. Where is the separate test of stopping behavior mentioned in Section 3.2? Did all versions in Table 4 use the same training for stopping?

[1] Zhengfu He et al. DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models. ACL, 2023.

[2] Yiming Zeng et al. TreeDiff: AST-Guided Code Generation with Diffusion LLMs. arXiv:2508.01473, 2025.

[3] Subham Sekhar Sahoo et al. Simple and Effective Masked Diffusion Language Models. arXiv:2406.07524, 2024.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.