acceptodds
Under review as a conference paper at ICLR 2027

StateWise: From Noise Awareness to State-Calibrated MLLM Conditioning for Unified Generation

Abstract

Unified multimodal models aim to improve image generation and editing through visual understanding. Connecting a pretrained multimodal large language model (MLLM) to a diffusion generator allows its understanding capabilities to guide image synthesis. However, reusing a fixed condition throughout denoising prevents the MLLM from adapting its guidance to the evolving latent. Adapting this condition requires identifying the current latent state, which may depart from the state expected at the input timestep during finite-step sampling.We introduce StateWise, a state-calibrated MLLM conditioning framework that reads noisy latents directly in the generator's native VAE space without an auxiliary visual encoder on this pathway. During training, we vary each latent's noise level while keeping its input timestep fixed, teaching the MLLM not only to perceive the latent's noise state but also to estimate how much noisier or cleaner it is than expected at that timestep. These estimates guide adjustments to the condition supplied to the diffusion generator at each denoising step, while an additional constraint discourages changes when the observed and expected noise levels agree. Using only publicly available training data, achieves 0.90 on GenEval and 77.4 on GenEval2, alongside competitive image editing. Code will be publicly released.

Primary area:
generative models
Keywords:
unified multimodal models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_cE2AOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper identifies the mismatch between nominal denoising timestep and the actual state of a sampled latent in unified multimodal models. It introduce StateWise, a state-calibrated conditioning framework for unified multimodal image generation and editing. The key idea of the method is to set the nominal noise level specified by the input timestep apart from the actual noise level of the evolving latent. A state head predicts a signed noise-level discrepancy relative to the nominal timestep and a gated adapter uses this prediction to update the DiT conditioning at each denoising step. An identity constraint suppresses unnecessary updates when the latent matches the timestep. The VAE and MLLM remain frozen, while the latent tokenizer, conditioning interface, calibration adapter, SANA-1.5 DiT, and editing-specific source-conditioning modules are trained.

As for the contributions, the paper’s main contribution is an explicit training scheme for teaching an MLLM to recognize latent-state deviations. Besides, it also provides a gated condition-calibration mechanism and integrates it into a unified generation-and-editing model using native VAE latents.

For about the validation, the final model achieves 0.90 on GenEval, 77.4 on GenEval2, and 86.22 on DPGBench. It obtains 4.28 overall on ImageEdit and 7.38 overall on GEdit-Bench-EN. In the controlled ablation, native-VAE noise-aware conditioning, noise-level augmentation, and state calibration improve GenEval from 0.86 to 0.87, 0.88, and 0.89, respectively; the final calibration step further improves counting from 82.81 to 84.38 and position from 82.25 to 84.00. Fixed-timestep trajectory tests show that the state head consistently distinguishes earlier, current, and later latent states.

In conclusion, I lean toward weak acceptance because the method is generally coherent, the controlled ablation provides a consistent signal, and the main concerns appear experimentally addressable without changing the core method.

Critical strengths
  1. The paper identifies and isolates a specific conditioning failure mode and explicits state–time discrepancy modeling. StateWise keeps the nominal timestep fixed while varying the latent’s actual noise level, trains a state query to predict this discrepancy, and uses a gated adapter to update the DiT condition. The identity constraint further suppresses updates for matched states. This is important and valuable because it distinguishes state calibration from generic noisy-latent conditioning and makes the proposed mechanism directly testable.

  2. The ablation studies of the paper provide evidence that the proposed training components contribute to generation quality, while the framework is integrated into a unified model that supports both generation and editing. Within the ablation setting, native-VAE noise-aware conditioning improves GenEval from 0.86 to 0.87, noise-level augmentation raises it to 0.88, and state calibration raises it to 0.89; the final step improves counting from 82.81 to 84.38 and position from 82.25 to 84.00. The final system also obtains 0.90 on GenEval, 77.4 on GenEval2, 4.28 overall on ImageEdit, and 7.38 overall on GEdit-Bench-EN.

Critical weaknesses
  1. The contribution of state calibration is not isolated at the final model scale. The proposed final system uses Qwen3-VL-4B, three-stage training, and a Stage 3 mixture with editing data. However, the component ablation in Table 4 evaluates InternVL3-2B with only Stage 1/2 data and a shorter training schedule. Although Table 4 includes a no-calibration control, it is available only in this smaller setting. Therefore, I have a concern that the reported 0.88-to-0.89 improvement does not establish that state calibration drives the final model’s gains rather than other changes in the model and training setup.

  2. The evidence for the proposed calibration mechanism is indirect. In D.1, the forward-noised and recursively sampled latents share text conditions but not the same clean latent or noise realization. Therefore, the 10.4–10.8% gap in noise-prediction norms does not isolate a state mismatch. In D.2, the final model only shows that the state head responds monotonically to earlier and later latents at a fixed timestep. It does not measure prediction accuracy or the resulting condition correction, and it does not evaluate the final model without calibration on generation and editing. The paper also does not report the extra inference cost of recomputing the Qwen3-VL-4B conditioning path at every sampling step.

Questions for authors

Clarification Requests:

  1. It would be appreciated if the authors can provide a final-scale matched ablation using Qwen3-VL-4B and SANA-1.5, with the same three-stage data and optimization budget, comparing no calibration, noise-level augmentation only, full calibration, and separate identity/state-loss ablations to eliminate my concerns.
  2. What is the state head’s held-out prediction error on synthetic offset pairs and on actual sampled trajectories? Please report MAE/RMSE or correlation, the distribution of gate values and correction norms, and an oracle-delta_u* or shuffled-state control.
  3. Can the forward-noised versus sampled-latent diagnostic be repeated with the final SANA-1.5 model at 1024 resolution using paired prompts/images and a direct measure of effective state or SNR discrepancy?
  4. What are the end-to-end latency, memory, FLOPs, and throughput costs under the reported 20-step CFG sampling procedure, compared with the corresponding no-calibration model? Please also clarify whether the conditional and unconditional MLLM paths are both recomputed at every step.

Suggestion for future work: A useful follow-up would compare StateWise with latent correction, time-shift sampling, and noise-awareness guidance under a matched end-to-end inference budget, including the cost of recomputing the MLLM condition.

Reviewer_3342OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents StateWise, a state-calibrated MLLM conditioning framework for unified image generation and editing. The authors identify two limitations of existing approaches: the reuse of fixed conditions throughout denoising and the lack of explicit modeling of discrepancies between the current latent state and the nominal timestep in existing noise-aware conditioning methods. To address these limitations, the authors introduce two main designs.

First, a lightweight native-VAE tokenizer directly maps the generator's current noisy latents into input tokens for a frozen MLLM. This allows the MLLM to produce dynamic conditions at each denoising step based on the text prompt, current latent, and timestep, without introducing an auxiliary pretrained visual encoder on the noisy-latent pathway. Second, the authors decouple the latent construction noise level from the input timestep to create training examples with noise-level deviations. A state prediction head is supervised to estimate the relative discrepancy between these levels, and the predicted discrepancy is used to control a learned condition residual, thereby calibrating the condition supplied to the DiT.

The authors adopt a three-stage training strategy consisting of large-scale text-to-image pretraining, high-quality generation refinement, and joint image generation and editing training. Experiments are conducted on GenEval, GenEval2, DPGBench, ImageEdit, and GEdit-Bench-EN. The final model achieves scores of 0.90 on GenEval and 77.4 on GenEval2, alongside competitive image-editing performance.

Critical strengths

The paper addresses the potential mismatch between latent states and nominal timesteps during finite-step diffusion sampling and introduces StateWise, which extends dynamic noise-aware MLLM conditioning with explicit state calibration. By fixing the nominal timestep while varying the latent construction noise level, the authors construct a state discrepancy supervision scheme that avoids the timestep shortcut. The predicted discrepancy is then used to control a learned condition residual, enabling dynamic condition calibration while retaining the original flow-matching objective and sampling procedure.

Experimental results show that progressively introducing the proposed components improves GenEval from 0.86 to 0.89. Using only publicly available training data, the final model achieves scores of 0.90 on GenEval, 77.4 on GenEval2, and 86.22 on DPGBench, alongside competitive performance on image-editing tasks.

Critical weaknesses

1. Insufficient justification for the proposed MLLM-based conditioning architecture.

Prior works such as RepFusion and TUNA have already explored dynamic generation conditioning by allowing MLLMs to read evolving noisy latents. StateWise presents the use of a lightweight native-VAE tokenizer in place of existing visual encoding pathways as one of its contributions. However, the paper lacks direct comparisons with existing visual encoding pathways and does not sufficiently demonstrate the specific advantages of the proposed tokenizer in terms of generation performance, computational efficiency, or representation quality.

More importantly, the newly projected noisy VAE embeddings are fed into a frozen MLLM, whose pretrained representations were originally developed for text inputs and visual inputs processed by its native visual encoder. The paper does not verify whether these embeddings can effectively leverage the multimodal semantic knowledge acquired during MLLM pretraining, or whether this pathway primarily serves as a more complex nonlinear transformation. This leaves the necessity of using an MLLM insufficiently justified. Furthermore, since the proposed approach requires repeatedly running the MLLM at every denoising step, the paper also lacks comparisons with alternative architectures under matched computational budgets.

2. Insufficient evidence that explicit state discrepancy estimation is responsible for the generation improvements.

The core contribution of StateWise is to explicitly predict the discrepancy between the current latent's noise state and the nominal timestep, and to use the predicted Δu^\widehat{\Delta u} to calibrate the DiT condition, thereby improving generation during actual sampling. However, the proposed module simultaneously introduces state prediction supervision, a residual adapter, and a gate. Although Table 4 demonstrates performance improvements from the complete module, the existing ablations do not sufficiently distinguish whether these gains arise from explicit state discrepancy estimation or from the additional capacity for learned condition adaptation.

Appendix D further demonstrates statistical differences between sampled latent states and forward-noised references, as well as the state head's ability to identify relative denoising progress. However, these results do not establish a direct connection between state discrepancy estimation, targeted condition calibration, and the resulting improvements in generation performance. Therefore, the independent value of explicit state discrepancy estimation over general dynamic condition adaptation remains insufficiently validated.

Questions for authors

Q1. Advantages of the Native-VAE Tokenizer

The paper presents the use of a lightweight native-VAE tokenizer without an auxiliary visual encoder as one of its main contributions. Could the authors provide direct comparisons with existing visual encoding pathways under comparable training settings and quantify its advantages in generation quality, computational cost, or representation capability? What are the substantive benefits of this design over existing approaches?

Q2. Computational Budget Allocation for the MLLM

StateWise requires running a large MLLM at every denoising step. Could the authors report the corresponding inference computational cost and analyze whether allocating the same computational budget to a larger DiT or additional generator computation could achieve comparable or even better results? If so, what would justify the use of the MLLM?

Q3. Utilization of Pretrained Multimodal Knowledge in the MLLM

A primary motivation for using an MLLM may be to leverage its rich pretrained knowledge and capabilities. However, noisy VAE latents are passed through a newly trained tokenizer and fed into a frozen MLLM that was not pretrained on such representations. How do the authors verify that this process effectively utilizes the MLLM's pretrained multimodal semantic knowledge? If this knowledge is not effectively utilized, does the MLLM mainly serve as an additional computationally expensive nonlinear transformation?

Q4. Practical Effectiveness and Causal Role of State Calibration

The existing experiments show that the state head can identify the relative denoising progress of latents and that the complete calibration module improves generation scores. However, how do the authors demonstrate that these improvements actually result from identifying state discrepancies during sampling and performing targeted condition calibration?

Reviewer_3mahOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes using the VAE latent with a frozen MLLM, predicts its noise-level deviation from the timestep, and uses that prediction to adjust diffusion conditioning

Critical strengths

The paper builds upon previous work of MetaQuery, Tuna and Repfusion and studies the important topic of MLLM generation.

Critical weaknesses

(-) the paper's motivation says "During sampling, however, latents are recursively updated using model predictions, and accumulated prediction and numerical integration errors can cause them to depart from the states expected at those timesteps. A diagnostic with a frozen Sana model shows a late-stage difference in noise-prediction norms between sampled latents and forward-noised references at matched timesteps". This is a critical claim and motivation of the paper but (1) SANA is not using a MLLM but a DiT, yet the paper's central theme is MLLM, why SANA? (2) It should be tested that such difference is widespread instead of just one model SANA to motivate and justify the research

(-) there are rather complicated and weird training steps and experiments (3 stages). The paper's main theme and motivation is prove that "state calibrated" design is useful, but the experiments are complicated and hard to follow. It also does not provide evidence to support the central theme that "state calibrated" design is useful. The only data point as I can find is 0.88 -> 0.89 on GenEval in Table 4? All other results are NOT testing what the paper's motivation and claims are? 0.88 -> 0.89 improvement is likely just noise, as the evaluation of GenEval itself is noisy (if one looks at the generated image vs GenEval claimed scores, many of the evals are not correct or ambiguous to begin with)

(-) Novelty is very incremental compared to RepFusion/TUNA. The only novelty is adding the "state calibration", but it is not supported by experiments

(-) Much more evaluation on "state calibration" is needed, including more models, more datasets, potentially human evaluation to support the usefulness and strength of having "state calibration". The theme of the paper should be focused on "state calibration" instead of having other factors complicate the experiment result. The headline / main results are NOT showing the effect of state calibration as the way the paper currently writes

Questions for authors

n/a

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.