acceptodds
Withdrawn from ICLR 2027

DIFFSCHED: Closed-Loop Semantic Computation Scheduling for Efficient Compositional Text-to-Image Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Text-to-image diffusion models still struggle with prompts that combine multiple objects, attributes, counts, and relations. Regional control can improve compositional accuracy, but fixed schedules may waste computation on resolved requirements while providing insufficient intervention for difficult ones. We propose , a closed-loop scheduling framework that adapts regional computation and semantic review during generation. Before sampling, role-specialized agents decompose the prompt into objects, relations, spatial regions, and an initial generation plan. During sampling, a lightweight trigger tracks the magnitude and recent trend of disagreement between regional and global denoising predictions. At eligible checkpoints, a low trigger score stops the current regional interventions without invoking semantic review; otherwise, a Review Agent diagnoses unresolved requirements and the Scheduler updates the next intervention interval. The residual guides execution rather than certifying semantic correctness, while high-risk requirements retain mandatory checks. On GenEval 2 and T2I-CompBench++, achieves a better quality–computation trade-off than fixed scheduling heuristics, using approximately 42–56% fewer denoiser evaluations while improving compositional scores. These results show that selectively allocating regional computation and semantic feedback can improve compositional generation without uniformly increasing inference cost.

Primary area:
generative models
Keywords:
Text-to-Image Generation, Diffusion Models, Compositional Generation, Adaptive Inference, Inference-Time Scheduling, Multi-Agent Systems

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fvy2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary
  • DiffSched is an inference-time controller that plans the prompt into regions before sampling and, during sampling, uses a cheap residual to decide how much regional computation and how many VLM reviews each semantic group receives.

  • The claimed contribution is this joint, cost-gated scheduling, and it comes with a conditional optimality argument.

  • The validation runs on small benchmark subsets against heuristic, static and agentic baselines and reports large denoiser savings at higher GenEval 2 scores.

  • The design, the ablation protocol and the code release are all in good order.

  • However, the evidence does not yet carry the claims.

Critical strengths

A clear, well-motivated decomposition that is new relative to the closest prior work.

  • The two separations the method is built on, cheap signal from expensive observation and allocation bounded by the next checkpoint, are sensible design choices.

  • Gating the cost of its own feedback, jointly per group and per interval, is what none of the positioned prior works does.

Careful claim scoping and controlled, paired internal comparisons, with code released.

  • The paper is explicit about what its residual, its cost metric and its theory do not claim.

  • The internal comparisons are paired on everything but the component under test, and costs are reported at several levels.

Critical weaknesses

The headline claims outrun the evidence: one benchmark does not show the stated gain, the other is a small single-seed subset without variance.

  • On T2I-CompBench++ the tables show the heuristic baselines matching or beating DiffSched, not the improvement the abstract claims. Specifically, the abstract claims improved compositional scores on both benchmarks. Yet in Tables 1 and 5 Visual-Complexity beats DiffSched on T2I-CompBench++.

  • On GenEval 2 the margins are small, single-seed, single-subset, and carry no uncertainty estimate. Specifically, on GenEval 2 the margins over the heuristics are +1.42–1.55 GM and +0.48–1.28 AM. They come from one 100-prompt subset and seed 42. There is no confidence interval or paired test, so their stability across prompts and seeds is unknown.

End-to-end efficiency is not established against the baselines the 42–56% figure targets, and Fig. 4a does not match Table 1.

  • The saving is measured in denoiser evaluations against two heuristics whose end-to-end cost is never reported, so no wall-clock advantage over them is shown. Specifically, the 42–56% saving in denoiser evaluations is measured against Region-Size and Visual-Complexity. Table 2 reports end-to-end time for DiffSched and three external methods only. Those heuristics, Static-Semantic-Schedule, the backbone and MCCD have none. So no wall-clock advantage over the direct baselines is shown.

  • The Pareto figure misplaces the two heuristics and silently omits a method that would sit on the frontier. Specifically, Fig. 4a draws Region-Size and Visual-Complexity far left of their Table 1 NFE. It omits BiDPO (20.00 NFE, 56.70 AM) without saying which rows its frontier covers.

The anonymous repository carries local paths that may identify the authors.

Specifically, the given link of https://anonymous.4open.science/r/DIFFSCHED-E1BB/, in the introduction section, reveals the identity information: config/marginal_utility_calibration_v3.json, fields source.scores and source.adaptive_records. It may violate the paper submission requirement.

Questions for authors

See weaknesses.

Reviewer_AhgQOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DIFFSCHED, a training-free framework that adaptively schedules regional interventions and semantic reviews during compositional text-to-image generation. A residual-based trigger determines when regional computation should stop and when semantic feedback is needed. The authors report improved compositional scores with fewer denoiser evaluations on GenEval 2 and T2I-CompBench++.

Critical strengths
  • The paper explicitly measures computational costs, including denoiser evaluations, GPU time, and end-to-end latency.
  • The authors evaluate multiple LLM backbones for the semantic controller.
Critical weaknesses
  • Limited novelty and conceptual contribution: The method combines established components, including prompt decomposition, regional interventions, and feedback-guided generation. RPG already employs planning and a closed loop, while MCCD explores multi-agent scene parsing and hierarchical compositional diffusion. Although DIFFSCHED introduces adaptive scheduling based on a residual trigger, the paper does not establish a sufficiently substantial conceptual advance or provide new insights into compositional generation. The contribution feels primarily engineering-oriented.

  • Limited evaluation and unconvincing qualitative results: Although the paper provides quantitative evaluation on SDXL, the qualitative evidence is extremely limited, with only three SDXL examples and two SD3.5-M examples in the main paper. Moreover, some results exhibit visible artifacts: for example, the zebra in Figure 3 has malformed legs, while Figure 5 in the supplementary material appears to show more visual leakage in DIFFSCHED's results than in the baseline. These examples raise concerns about visual fidelity and whether the reported compositional scores consistently translate into perceptually convincing results.

  • Hyperparameter and scheduling complexity: Although DIFFSCHED aims to reduce unnecessary computation, it introduces several configuration choices and relies on LLM-guided rescheduling. The paper describes the available scheduling actions and the information provided to the Scheduler, but it is unclear how the LLM determines the specific schedule values. If these choices require trial and error, how can the method identify effective values quickly enough to avoid excessive computation?

  • Presentation: Some figures are not explicitly referred to in the text, and Figure 2 is particularly dense and difficult to follow. Overall, the presentation could be clearer.

  • Reproducibility: The paper describes the roles and expected outputs of its LLM agents, but I could not find their actual instruction prompts in the paper or supplementary material. Since the method relies on LLM-based agents, providing the prompts is important for reproducibility.

Questions for authors

How does the Scheduler determine the specific intervention durations and control strengths? Does the process involve trial and error, and if so, how does it find effective schedules efficiently?

Reviewer_RzFNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DIFFSCHED is a training-free, closed-loop scheduler for compositional text-to-image generation. Before sampling, MLLM agents split the prompt into requirements, semantic groups and soft masks. During sampling, regional branches are fused with the global prediction in the MultiDiffusion style. A normalized regional/global residual then decides whether to stop the regional branches or call a VLM Reviewer, and a Scheduler revises the plan for the next interval. A value-of-information analysis gives conditional certificates for the trigger. On SDXL, the paper reports gains on GenEval 2 and T2I-CompBench++ while using 42–56% fewer NFEs than heuristic schedulers.

Critical strengths

1.It clearly frames compositional control as deciding where to spend computation and when to request semantic review for each requirement.

Critical weaknesses
  1. Limited novelty. Agentic decomposition (MCCD, RPG), masked regional fusion (MultiDiffusion), intermediate VLM feedback (PromptLoop, Inference Intervention, SLD) and adaptive regional computation (RAS, SDiT) already exist. The new element is mainly a threshold-based early-stopping gate. Propositions 1 and 2 restate marginal-gain stopping and the non-negativity of information value. Theorem 1 relies on envelopes (A11–A12) that are never estimated. Appendix B.9 says λD, λR, τ0, calibration sizes and violation rates are required, yet none are reported.
  2. Fragile evidence. GenEval 2 is evaluated on 100 prompts with one seed, and T2I-CompBench++ on 15 images per category. No confidence intervals are given. The GM gain over the backbone (9.61 to 10.71) is small. DIFFSCHED does not beat Region-Size or Uniform Regional on 2D, non-spatial or numeracy.
  3. Missing baselines. The 42–56% saving is measured against heuristics the authors designed, not against SDiT or RAS. The MCCD row in Table 1 matches the "MCCD-style proxy" in Table 6 exactly, which suggests the official method was never run. The paper lacks quantitative comparisons with:
    • RAS, SDiT, verifier-guided noise search (Ma et al., 2025), SLD and LMD.
    • The agentic system GenArtist [a] and the reflection-based inference-scaling method Reflect-DiT [b].
    • Stronger backbones such as FLUX.1 [c] and SD3.5. For SD3.5, only two qualitative examples are shown.
  4. Overstated efficiency. E2E latency (42.7 s) is about 15× the backbone's diffusion time (2.93 s). E2E latency is not reported for the backbone or the heuristic schedulers.
  5. Inconsistencies.
    • Reviews per image at β=1 are 1.40 in Tables 2–3 but 2.06 in Table 7.
    • Tokens for the w/o Trigger variant are 8810.5 in Table 2 but 8914.4 in Table 5.
    • Table 7 gives identical GM/AM/NFE for β∈{0.5, 0.75, 1.0} even though review counts differ. Its GM (≈8.7) also differs from 10.71 with no explanation.
    • The w/o Trigger variant reaches a higher GM than the full method.
    • The residual compares predictions made under different conditions, and its correlation with requirement satisfaction is never measured.

[a] Wang, Zhenyu, et al. "Genartist: Multimodal llm as an agent for unified image generation and editing." In NeurIPS, 2024. [b] Li, Shufan, et al. "Reflect-dit: Inference-time scaling for text-to-image diffusion transformers via in-context reflection." In ICCV, 2025. [c] Black Forest Labs. FLUX.1 (official model release). 2024.

Questions for authors

See above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.