acceptodds
Withdrawn from ICLR 2027

TrajSP: Trajectory-Grounded Structured Prompting for Feedback-Guided Text-to-Image Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

While structured prompt enhancement enables image generation to benefit from visual knowledge and increased prompter capacity, existing distillation-based approaches require a separate training pipeline for an external LLM enhancer that lacks direct visual access at inference. We study whether unified multimodal models (UMMs) can eliminate the need for this separately trained enhancer by directly observing intermediate images and refining structured conditioning during denoising. Our framework, TrajSP, first applies a lightweight noisy understanding adaptation to enhance the understanding branch's capacity to interpret noisy intermediate images. During sampling, the adapted branch serves as a trajectory-grounded Polisher: at an early checkpoint, it processes the intermediate visual state jointly with the user prompt to predict a structured prompt comprising global scene fields, per-object elements with normalized bounding boxes, and inter-object relations. By grounding this representation in the evolving visual state, TrajSP provides targeted conditioning for subsequent denoising steps, with the aim of avoiding redundant prompt expansion. Joint understanding-generation finetuning optimizes shared parameters for both schema prediction from noisy states and schema-conditioned image generation. An optional late-stage Validator is designed to provide targeted feedback for subsequent refinement. Experimental results demonstrate improved semantic alignment and greater fidelity in object-level details, particularly attribute binding, without a standalone prompt enhancer. 15:52

Primary area:
generative models
Keywords:
text-to-image generation, unified multimodal models, diffusion models, noisy visual understanding, structured prompting, compositional generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_g6KeOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TRAJSP, a structured prompt enhancement method based on visual feedback from the generation trajectory, aiming to address the issue that traditional prompt enhancement methods cannot perceive intermediate states during generation. Built on unified multimodal models, the method uses an early-stage Polisher to convert visual information from noisy images into structured generation conditions, and a late-stage Validator to check generation results, revising conditions and regenerating when necessary. In addition, the authors construct the GSP-30K dataset and adopt a two-stage training strategy to enhance the model's noisy visual understanding and structured condition generation capabilities. Experiments show that the method achieves performance improvements over base models on multiple benchmarks such as GenEval, WISE, and GenAI-Bench.

Critical strengths

1.The research problem is practically meaningful, and the framework design is reasonable. This paper addresses the issue that traditional prompt enhancement methods cannot leverage intermediate visual states during generation, which is of research value. By reusing the unified multimodal model's own visual understanding capability, TRAJSP introduces structured condition updates based on the generation trajectory into the denoising process, without requiring an additional independent prompt enhancement model at inference time. This provides a practical way to coordinate understanding and generation capabilities within unified multimodal models.

2.Performance improvements are achieved on multiple benchmarks, with corresponding ablation validation. TRAJSP outperforms the SenseNova-U1 baseline on multiple evaluation benchmarks, including GenEval, WISE, and GenAI-Bench. Ablation experiments further show that, compared with ordinary fine-tuning, the Polisher brings clear performance gains, while the Validator provides additional improvements on some metrics. These results provide experimental evidence for the effectiveness of the proposed framework.

Critical weaknesses
  1. Verification of the independent contribution of Trajectory-Grounded Feedback remains insufficient. The paper shows that the Polisher can significantly improve performance over ordinary fine-tuning, but current experiments still cannot distinguish whether the gains come from "structured prompting itself" or from "visual feedback in the current generation trajectory." In particular, given that VD-SP has already demonstrated that structured prompting alone is highly effective, the paper lacks a controlled experiment directly comparing text-only structured prompting with trajectory-grounded structured prompting under the same model, schema format, and training settings. Therefore, the paper's core Trajectory Grounding contribution has not yet been adequately isolated and validated.

  2. The Polisher's ability to correct erroneous intermediate states lacks sufficient analysis. The schemas in GSP-30K are constructed as descriptions of actual image content, so even if the generated image does not fully satisfy the original prompt, the corrected schema still remains consistent with the current image. Consequently, the Polisher's supervision more directly encourages the model to understand and describe the current generation state, rather than explicitly modeling "the deviation between the current state and the target prompt." When the intermediate image already contains errors in count, attributes, or relations, whether such image-grounded schemas can help the model correct errors, or mainly reinforce the existing composition, has not been systematically verified.

Questions for authors
  1. The paper's core contribution is to use intermediate generation states to achieve trajectory-grounded structured prompting. Table 2 already shows that the Polisher brings clear improvements over ordinary fine-tuning, but this result still does not fully disentangle the effects of structured prompting itself and trajectory-grounded visual feedback. Can the authors compare, under the same model, schema format, training data, and inference settings, a method that generates schemas based only on the original prompt with one that generates schemas based on the original prompt plus intermediate states? This comparison would more directly validate the independent contribution of intermediate visual trajectory information.

  2. The schemas in GSP-30K are corrected to be consistent with the actual image content, so the Polisher more directly learns to understand and describe the current generation state. Can the authors further analyze the Polisher's behavior when the intermediate state has already deviated from the original prompt, for example, when incorrect object counts, attributes, or spatial relations appear? Specifically, can these image-grounded schemas help subsequent denoising recover from errors, or are they mainly used to refine and consolidate content that has already been formed? Reporting recovery effects separately on "semantically correct" and "semantically incorrect" intermediate states would help clarify the functional boundary between the Polisher and the Validator.

  3. Can the authors report the Validator's automatic trigger rate, error detection accuracy, and the proportion of cases that actually improve after regeneration is triggered? Figure 8 currently uses forced regeneration, so it cannot directly reflect the true effectiveness of the automatic Validator.

If these concerns can be convincingly addressed through additional controlled experiments or analyses in the rebuttal, I would be open to revising my evaluation upward.

Reviewer_MDn2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TrajSP adapts SenseNova-U1 to inspect the current noisy image and update the prompt used for generation. An early Polisher produces a structured prompt schema listing objects, attributes, positions, and relations. Generation then continues along the same trajectory using this schema. An optional later Validator can revise the schema and restart from fresh noise once. The authors build the image–prompt–schema dataset GSP-30K and report improvements on several generation benchmarks.

Critical strengths
  1. The improvements over U1 are encouraging, including counting and following the prompt. Table 2 shows that most gains come from the Polisher, before another generation attempt. This supports the usefulness of the overall method.

  2. TrajSP reuses the model's image-understanding pathway, avoiding a separate prompt-enhancement model at inference. The noisy-input MME results show that adaptation improves understanding of corrupted images. Refreshing the KV cache lets generation use the new schema while continuing along the current trajectory.

Critical weaknesses
  1. The Polisher adds access to the current noisy image, while training also teaches the model to produce and use structured schemas. It is unclear whether the prompt-only fine-tuning baseline has the same schema training. Section 4.3.3 attributes the gains mainly to seeing the image, but the reported setup leaves this attribution unclear.
Questions for authors
  1. What training is included in the "+ Finetune" row of Table 2? Does it include schema prediction and schema-conditioned generation, or only generation from the original prompts?

  2. Which generator checkpoint is used for VD-SP in Table 2? Is its generator trained to use structured schemas in the same way as TrajSP's generator? This affects whether the comparison tests the prompting method or also changes the generator.

  3. How is schema accuracy in Figure 7(b) measured? What reference annotations and scoring rule are used? This would help interpret the evidence for prompt masking and Validator timing.

Reviewer_WRYsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TrajSP uses a unified multimodal model to inspect intermediate denoising states and produce structured prompts for subsequent generation. An early Polisher updates conditioning, while a later Validator can revise the schema and trigger regeneration. Noisy-understanding adaptation and joint training support this pipeline. Experiments report improvements on several generation benchmarks.

Critical strengths
  1. The framework reuses the model’s understanding capability to provide structured guidance during generation without a separate inference-time prompt enhancer.
  2. The Polisher configuration improves several metrics over the backbone and prompt-only finetuning, suggesting practical value beyond the regeneration stage.
Critical weaknesses
  1. Incorrect intermediate interpretations may reinforce generation errors. Early noisy states can produce mistaken object, attribute, or relation predictions. Turning these predictions into conditioning may solidify errors rather than correct them. The experiments do not quantify this failure mode or establish whether the Validator reliably resolves it.
  2. The contribution of trajectory visual feedback is insufficiently isolated. Gains over prompt-only finetuning do not distinguish the benefits of observing intermediate states from structured conditioning and joint adaptation. Same-model controls using text-only schemas or mismatched intermediate observations are missing.
Questions for authors
  1. How frequently are intermediate schemas incorrect, and how does conditioning on incorrect schemas affect final prompt alignment? Please report improvement and degradation rates separately for correct and incorrect schemas.
  2. Under matched training and inference settings, how does trajectory-conditioned schema generation compare with text-only schema generation and mismatched-state controls?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.