acceptodds
Under review as a conference paper at ICLR 2027

Self-correction Optimization for Interleaved Multimodal Generation

Abstract

Multimodal large language models (MLLMs) have made significant progress in visual understanding and generation. However, generating interleaved image–text content remains challenging, as it requires tightly integrated multimodal understanding and generation capabilities. Although existing MLLMs provide promising solutions, most rely on additional training with augmented data, which is computationally expensive and remains limited in preserving visual subjects, temporal consistency, and physical plausibility. In this work, we propose self-correction optimization (SCO), an effective training-free method for consistent interleaved generation. SCO treats the classifier-free guidance update as a reference and performs minimal self-correction under two complementary constraints, including new-event and state-preserving constraints. Specifically, the new-event constraint promotes temporal consistency across image–text sequences, while the state-preserving constraint maintains the coherence of visual subjects throughout subsequent generation steps. Experiments on challenging interleaved multimodal generation benchmarks demonstrate significant improvements in temporal coherence and visual-subject preservation. Futhermore, SCO can be extended to video generation and improves the modeling of physically grounded processes, including robot manipulation and long-horizon handcrafting.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Interleaved Image–Text Generation, Multimodal Generation, Subject Coherence, Temporal Consistency, Classifier-Free Guidance

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_1uJBOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Self-Correction Optimization (SCO), a training-free test-time guidance method for interleaved image-text generation and video generation. At each denoising step up to tau_end = 0.5, instead of only predicting the unconditional and full conditional velocities (v_u and v_f) as in standard CFG, SCO predicts two additional branches in the latent velocity space: an event-off counterfactual branch v_counter and a history-off branch v_event. By taking the differences a_new = v_f - v_counter and a_state = v_f - v_event, the method isolates an explainable new-event direction and a state-preserving direction, and solves a 2-variable bounded quadratic program (Section 3.4) to fuse them into the CFG update. Experiments across modular/unified image-text generators (OpenING and ISG-Bench) and video generators (Video-CraftBench and Interleaved-VLA) demonstrate consistent improvements in visual quality, subject preservation, and background consistency.

Critical strengths
  1. Intuitive, explainable, and effective test-time guidance Isolating the new-event direction (v_f - v_counter) and the state-preserving direction (v_f - v_event) in the latent velocity space and fusing them into the CFG update is both intuitive and explainable. As shown in Figure 4 (as well as Figures 2, 6, 9, and 10), background consistency and subject appearance are preserved very well across sequential editing, object removal, and embodied video generation tasks.
  2. Plug-and-play improvements across diverse backbones. Because SCO operates purely at test time without parameter updates, it can be directly plugged into a wide variety of modular and unified image-text models (e.g., MM-Interleaved, Show-o2, BAGEL, SenseNova-U1, DuoGen) and video models (VideoWorld2, Cosmos-2.5, Cosmos-3) with consistent quantitative gains.
Critical weaknesses
  1. Computationally heavy at inference time with unreported latency/memory overhead. Up to tau_end = 0.5, SCO predicts four velocities (v_u, v_f, v_counter, v_event) instead of two at each denoising step, which doubles the per-step forward passes during the first half of sampling. Despite this noticeable extra computation, the paper does not report actual wall-clock inference latency or GPU memory overhead compared to standard CFG.
  2. Missing prompt-rewriting details in the text and appendix. Figure 7 in the appendix shows that external LLMs are used to rewrite the text prompts to construct the counterfactual (event-off) and event-only (history-off) branches. However, this prompt-rewriting step is not explained in Sections 3.2-3.3 of the main paper, and although the Reproducibility Statement (Page 10) claims that "the detailed process of text prompt rewriting is displayed in the appendix," the actual prompt templates and rewriting procedure are missing from the appendix text.
  3. Lack of a fixed-weight fusion baseline in the ablation study (Table 3). While Table 3 evaluates adding +a_state, +a_new, and +Both, it does not include a simple fixed-weight fusion baseline (similar to standard composable CFG, where a_new and a_state are combined with constant scalar weights) to isolate how much improvement specifically comes from the 2D QP solver in Section 3.4.
Questions for authors
  1. Wall-clock latency and memory overhead: Since predicting four velocities instead of two at each denoising step (up to tau_end = 0.5) doubles the per-step forward passes, could you report the actual wall-clock inference latency and peak GPU memory overhead of SCO compared to standard CFG?
  2. Details of LLM prompt rewriting: Could you provide the missing prompt-rewriting details promised in the Reproducibility Statement—specifically, which LLM and prompt templates are used to generate the counterfactual and event-only prompts shown in Figure 7?
  3. Comparison with fixed-weight fusion: In Table 3, how does the 2D QP solver in Section 3.4 compare against simply fusing a_new and a_state with fixed constant weights (as in standard composable CFG)?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 1uJB (1/2)

We thank the reviewer for the positive assessment and also for the constructive comments.

W1: Inference costs.

This concern is also shared by Reviewers fPE9 and ufvw. The reviewer is correct that SCO doubles the forward passes in the first half of sampling, which amounts to 1.5× the denoising passes of standard CFG overall. Table R3 reports the measured cost of three backbones on two NVIDIA H100 GPUs. Including the rewriter calls, SCO increases the per-sample latency by 25.0–46.0% and the peak GPU memory by 7.7–13.3 GB across MM-Interleaved, SenseNova-U1, and BAGEL. The memory increase arises because the additional branches are evaluated in the same batch as the CFG branches, and can be avoided by evaluating them sequentially at the cost of higher latency. The rewriter Qwen3.6-Plus is accessed through an API and uses no local GPU memory. Given that SCO is training-free and consistently improves generation quality, we consider this overhead acceptable.

Table R3: Inference cost with and without SCO. Latency includes text generation, image generation, and the rewriter calls (Qwen3.6-Plus).

ModelLatency (s/sample) ↓LLM queries / sampleThroughput (images/min) ↑Peak memory (GB) ↓
MM-Interleaved27.781.005.4718.99
+ SCO40.56 (+46.0%)2.533.74 (−31.6%)26.69 (+40.5%)
SenseNova-U159.901.002.5438.98
+ SCO80.30 (+34.1%)2.531.89 (−25.6%)52.24 (+34.0%)
BAGEL73.301.002.0735.40
+ SCO91.60 (+25.0%)2.531.66 (−19.8%)47.75 (+34.9%)

W2: Missing prompt-rewriting details.

This concern is also shared by Reviewers fPE9 and ufvw. In the revision, we describe the prompt rewriting in detail in Sec. 3.2–3.3 and add an appendix subsection (Appendix A.4) with the procedure and complete template. The added content is summarized below.

Rewriting procedure. Besides the full condition cfullc_{\mathrm{full}} of standard CFG, SCO requires an event-off condition ccounterc_{\mathrm{counter}}, which differs from cfullc_{\mathrm{full}} only in the current event (Eq. (4)), and a history-off condition ceventc_{\mathrm{event}}, which differs from cfullc_{\mathrm{full}} only in the absence of the inherited history (Eq. (6)). The rewriter is Qwen3.6-Plus, a commercial LLM queried through an API. As illustrated in Fig. 7, for the mm-th image slot, the rewriter takes the cached history Hm\mathcal H_m (all previous text segments and images) and the full prompt as input, e.g., “A woman wearing a red skirt is standing in a kitchen beside a wooden table. She is holding a whole apple and cuts it into two pieces with a knife.”, and returns two prompts in one JSON object:

  • Event-off (ccounterc_{\mathrm{counter}}): keeps all content of the full prompt that precedes the current event and replaces the event with the state immediately before it, e.g., “… She is holding a whole apple before cutting it.” If this state cannot be described explicitly, the event clause is removed and “The states of all visual subjects and backgrounds remain stable and unchanged.” is appended.
  • History-off (ceventc_{\mathrm{event}}): describes only the current event, using generic references and no inherited attributes, e.g., “A person cuts an apple into two pieces with a knife.”

Because the event-off prompt is a minimal edit of the full prompt, anew=vfull−vcountera_{\mathrm{new}} = v_{\mathrm{full}} - v_{\mathrm{counter}} captures the event rather than wording differences. The full and event-off branches keep the history images as visual context, the history-off branch removes both history text and history images, and the unconditional branch uses an empty prompt. The rewriter produces only text conditions and never accesses the generator's latents or parameters.

AuthorsOpenReview ↗

Response to Reviewer 1uJB (2/2)

W3: Lack of a fixed-weight fusion baseline in the ablation study.

Fixed-weight fusion is equivalent to SCO with constant coefficients: δ=δref+μnewanew+μstateastate\delta = \delta_{\mathrm{ref}} + \mu_{\mathrm{new}} a_{\mathrm{new}} + \mu_{\mathrm{state}} a_{\mathrm{state}}, where (μnew,μstate)(\mu_{\mathrm{new}}, \mu_{\mathrm{state}}) are held fixed across all sampling steps and all interleaved generation tasks, whereas the bounded quadratic program in Sec. 3.4 adapts them at each step. We compare the two on OpenING, using SenseNova-U1 as the generator. All variants share the same four branches, schedule (τend=0.5\tau_{\mathrm{end}} = 0.5), and hyper-parameters, and differ only in how (μnew,μstate)(\mu_{\mathrm{new}}, \mu_{\mathrm{state}}) are obtained. For the fixed-weight baseline, the coefficients are set to the mean (μˉnew,μˉstate)(\bar\mu_{\mathrm{new}}, \bar\mu_{\mathrm{state}}) or the median (μ^new,μ^state)(\hat\mu_{\mathrm{new}}, \hat\mu_{\mathrm{state}}) of the coefficients produced by SCO over all sampling steps and samples, so that the overall correction strength matches that of SCO. Table R1 reveals that SCO reduces FID from 67.2 to 55.9, whereas the mean fixed weights reduce it only to 64.7. Moreover, with mean weights, CLIP-T, GPT-J1, CbC_b, and GPT-J2 fall below the baseline, and with median weights, all metrics fall below the baseline. We attribute this to the constant correction being applied even at steps where the CFG update already satisfies both constraints, which over-corrects the update.

We additionally include a sequential variant that solves the two constraints one after the other, ignoring the cross term Gnew,stateG_{\mathrm{new,state}}. As shown in Table R1, both orders are clearly inferior to SCO (FID 65.1/60.4 vs. 55.9). The new →\rightarrow state variant attains a marginally higher CsC_s (0.36 vs. 0.35), but falls below the baseline on CLIP-T and GPT-J1. Because the two directions are not orthogonal, the second correction changes the projection onto the first one, so a sequential scheme cannot reliably satisfy both constraints. This confirms the analysis in Sec. 3.4 and motivates solving the two constraints jointly.

Table R1: Necessity of joint optimization, evaluated with SenseNova-U1 on OpenING. All variants use the same four branches and settings, and differ only in how (μnew,μstate)(\mu_{\mathrm{new}}, \mu_{\mathrm{state}}) are obtained. Fixed: the median (μ^new,μ^state)(\hat\mu_{\mathrm{new}}, \hat\mu_{\mathrm{state}}) or mean (μˉnew,μˉstate)(\bar\mu_{\mathrm{new}}, \bar\mu_{\mathrm{state}}) of the SCO coefficients over all steps and samples. Sequential: the two constraints are solved one after the other in the indicated order.

MethodFID ↓CLIP-I ↑CLIP-T ↑GPT-J1 ↑CsC_s ↑CbC_b ↑GPT-J2 ↑
SenseNova-U1 (Diao et al., 2026)67.20.3250.4710.6580.290.370.574
+ Fixed (μ^new,μ^state)(\hat\mu_{\mathrm{new}}, \hat\mu_{\mathrm{state}})72.10.3070.4590.6250.270.330.530
+ Fixed (μˉnew,μˉstate)(\bar\mu_{\mathrm{new}}, \bar\mu_{\mathrm{state}})64.70.3300.4630.6510.300.350.561
+ Sequential (new →\rightarrow state)65.10.3370.4550.6180.360.380.587
+ Sequential (state →\rightarrow new)60.40.3580.5030.6790.260.350.563
+ SCO (joint optimization)55.90.3760.5070.6940.350.400.602
Reviewer_1uJBOpenReview ↗

Thanks for detailed responses

I think responses have fully resolved my concerns. Specially, I agree with author's claim that the inference cost is accepatble given the quality improved. Also, the additional ablations strengthen the importance of joint optimization. I raised my score :).

Reviewer_ufvwOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper introduces Self-Correction Optimization (SCO), a training-free guidance strategy designed to enhance classifier-free guidance (CFG) for interleaved image–text generation. The authors observe that conditioning on the full context inherently entangles elements that should remain invariant with those that must be modified, often leading to either under-realized events or subject/scene drift. To address this, SCO introduces two auxiliary branches during the early steps of the sampling trajectory: (1) a counterfactual branch, which retains the context history while replacing the instruction with a pre-event state description, and (2) an event-only branch, which omits the history entirely. The differences between these branches and the full-condition output yield two explicit guidance vectors: a "new-event" direction and a "state-preserving" direction. SCO then computes a minimal adjustment to the standard CFG update to ensure sufficient movement along both directions under bounded projection constraints, which reduces to an analytically solvable two-variable optimization problem. The proposed method is evaluated across seven modular and unified models on OpenING and ISG-Bench benchmarks.

Critical strengths
  • Clear Motivation & Novel Guidance Formulation: The paper tackles a core limitation in interleaved image–text generation: the conflation of persistent historical context and incremental instruction changes in nominal CFG. Formulating event realization and context preservation as two decoupled directional constraints with an exact, closed-form two-variable solution is elegant and mathematically sound.
  • Plug-and-Play Generality: The training-free formulation allows SCO to be integrated directly into diverse generative frameworks (both unified and modular architectures) at inference time without requiring parameter updates or fine-tuning.
  • Extensive Empirical Evaluation: The method is evaluated across multiple benchmarks (OpenING and ISG-Bench) over seven generative models, demonstrating broad empirical applicability.
Critical weaknesses

Reliance on external LLMs/rewriters and unmeasured inference overhead

While SCO is positioned as a training-free and parameter-free inference-time guidance strategy, its execution critically depends on LLM-generated condition prompts at every generation step: specifically, a pre-event state description, an isolated event-only description, and a rewritten full-condition prompt (as indicated in Fig. 7). However, the manuscript omits crucial details regarding this pipeline: the exact model utilized (Fig. 7 merely references an unspecified "Commercial models" box), the system prompts, decoding parameters, and how pre-event states are defined for tasks lacking explicit temporal transitions (e.g., travel or design domains in OpenING). Additionally, the video experiments mandate an external GPT-5 planner as a prerequisite preprocessing stage. These omissions introduce three major issues:

  • Confounding Factors in Performance Gains: In Fig. 7, the rewritten full condition explicitly summarizes historical context. It is therefore unclear whether the reported gains stem from the proposed optimization dynamics or simply from prompt enrichment and LLM-based event planning. Crucially, the paper does not state whether the baseline models were evaluated using identical rewritten prompts and decomposed sub-goals. Fair ablation against prompt-augmented baselines is essential.

  • Inference Latency & Computational Overhead: During the active correction window (τ\tau), SCO requires four forward diffusion passes per step instead of the standard two in nominal CFG. Beyond diffusion passes, the pipeline introduces non-trivial latency and monetary cost from upstream LLM calls. Please report the wall-clock latency, total LLM query counts, and throughput comparisons against standard CFG across the evaluated backbones.

Number discrepancies between OpenING and ISG-Bench tables

There appear to be severe inconsistencies and potential copy-paste errors across the quantitative results:

  1. Table 3 vs. Table 1 & 2: Table 3 is presented as an ablation study on the OpenING benchmark. However, the reported metrics for the MM-Interleaved Baseline and +Both rows are identical metric-for-metric to the MM-Interleaved results on ISG-Bench in Table 2 (e.g., FID of 94.1 and 81.3), rather than the OpenING numbers in Table 1 (FID 76.2 and 73.6). Paradoxically, the SenseNova-U1 column in Table 3 aligns with Table 1.
  2. Fig. 3 Sensitivity Analysis: Sec. 4.4 states that the sensitivity analysis in Fig. 3 evaluates SenseNova-U1 on OpenING. However, reading the values at the default configuration and at τend=0\tau_{\text{end}} = 0 (which corresponds to the unguided baseline) yields FID≈55.9\text{FID} \approx 55.9 and ≈67.2\approx 67.2, along with the corresponding GPT-J1 and GPT-J2 scores. These values exactly match the SenseNova-U1 entries on ISG-Bench from Table 2, contradicting the reported OpenING baseline of FID 49.7 / 52.1 in Table 1.

Please clarify the exact data source for each table/figure, resolve these contradictions, and verify the integrity of the experimental results.

Questions for authors
  • Which model rewrites the prompts, and do the baselines receive the same rewritten prompts? If not, how does a baseline given the rewritten prompts but without SCO perform?
  • How do the results change with other or smaller rewriters?
3 comments
AuthorsOpenReview ↗

Response to Reviewer ufvw (1/2)

We thank the reviewer for the positive assessment and for the constructive comments on the ablation analysis in performance gains.

W1: (1) Rewriter details.

The rewriter is Qwen3.6-Plus, an external LLM that we query through an API; it uses no local GPU memory. For the mm-th image slot, the rewriter takes the cached history Hm\mathcal H_m (previous texts and images) and the full prompt as input and returns two prompts in one JSON object. The event-off prompt (ccounterc_{\mathrm{counter}}, Eq. (4)) is a minimal edit of the full prompt in which the current event is replaced by the state immediately before it. The history-off prompt (ceventc_{\mathrm{event}}, Eq. (6)) describes only the current event with generic references and no inherited attributes.

For tasks without an explicit temporal transition, such as travel or design, the pre-event state often cannot be described explicitly. In this case, the event clause is removed and the sentence “The states of all visual subjects and backgrounds remain stable and unchanged.” is appended. The event-off condition then describes the inherited state without change, so anewa_{\mathrm{new}} captures the content introduced by the current instruction. More details of the prompt rewriting are given in Appendix A.4.

W1: (2) Confounding factors in performance gains.

Prompt enrichment. In all experiments, each baseline and its SCO counterpart receive the same rewritten full prompt. They therefore share the same full condition cfullc_{\mathrm{full}} and differ only in the SCO correction applied to δref\delta_{\mathrm{ref}}. The event-off and history-off prompts are used solely to compute anewa_{\mathrm{new}} and astatea_{\mathrm{state}} (Eqs. (4)–(8)). The gains in Tables 1 and 2 are therefore attributable to SCO rather than to prompt enrichment.

Optimization dynamics. To isolate the effect of the joint optimization, we compare SCO with fixed-weight and sequential variants on OpenING using SenseNova-U1 (Table R1). All variants use identical prompts, conditional branches, directions, schedule (τend=0.5\tau_{\mathrm{end}} = 0.5), and hyper-parameters, and differ only in how (μnew,μstate)(\mu_{\mathrm{new}}, \mu_{\mathrm{state}}) are obtained. With the mean SCO coefficients held fixed, FID improves only from 67.2 to 64.7, compared with 55.9 for SCO. In the sequential variant, the two constraints are enforced one after the other, so the cross term Gnew,stateG_{\mathrm{new,state}} is ignored. Its results depend on the order, and both orders are clearly inferior to SCO (FID 65.1/60.4 vs. 55.9). Because anewa_{\mathrm{new}} and astatea_{\mathrm{state}} are not orthogonal, the second correction changes the projection onto the first direction and can violate the constraint that was already enforced. Since all other factors are fixed, these differences come solely from how the coefficients are computed, which confirms that the gains of SCO stem from the joint optimization.

Table R1: Necessity of joint optimization, evaluated with SenseNova-U1 on OpenING (Zhou et al., 2025). All variants use the same four branches and settings, and differ only in how (μnew,μstate)(\mu_{\mathrm{new}}, \mu_{\mathrm{state}}) are obtained. Fixed: the median (μ^new,μ^state)(\hat\mu_{\mathrm{new}}, \hat\mu_{\mathrm{state}}) or mean (μˉnew,μˉstate)(\bar\mu_{\mathrm{new}}, \bar\mu_{\mathrm{state}}) of the SCO coefficients over all steps and samples. Sequential: the two constraints are solved one after the other in the indicated order.

MethodFID ↓CLIP-I ↑CLIP-T ↑GPT-J1 ↑CsC_s ↑CbC_b ↑GPT-J2 ↑
SenseNova-U1 (Diao et al., 2026)67.20.3250.4710.6580.290.370.574
+ Fixed (μ^new,μ^state)(\hat\mu_{\mathrm{new}}, \hat\mu_{\mathrm{state}})72.10.3070.4590.6250.270.330.530
+ Fixed (μˉnew,μˉstate)(\bar\mu_{\mathrm{new}}, \bar\mu_{\mathrm{state}})64.70.3300.4630.6510.300.350.561
+ Sequential (new →\rightarrow state)65.10.3370.4550.6180.360.380.587
+ Sequential (state →\rightarrow new)60.40.3580.5030.6790.260.350.563
+ SCO (joint optimization)55.90.3760.5070.6940.350.400.602

LLM-based event planning for video experiments. The planner is applied identically to each baseline and its SCO counterpart. For each sample, the GPT-5-based planner runs once before sampling, decomposes the prompt into temporally ordered events, and assigns frames to them. It does not interact with the generator and only defines the frame-aligned conditions. Since the baselines and SCO share identical inputs and differ only in the guidance update, the comparison in Table 5 is fair, and the improvements are attributable to SCO rather than to the planner. We have clarified this in Sec. 4.7.

AuthorsOpenReview ↗

Response to Reviewer ufvw (2/2)

W1: (3) Inference Latency & Computational Overhead.

Table R3 reports the wall-clock latency (including the rewriter calls), the number of LLM queries, throughput, and peak GPU memory with and without SCO, measured on two NVIDIA H100 GPUs. In practice, SCO increases the per-sample latency by 25.0–46.0% across the three backbones. Peak memory increases by 7.7–13.3 GB because the additional branches are evaluated in the same batch as the CFG branches. This can be avoided by evaluating them sequentially at the cost of higher latency. Regarding LLM queries, the baseline calls the rewriter once per sample to rewrite the full prompt, and SCO requires 1.53 additional calls per sample on average (2.53 in total). Given that SCO is training-free and consistently improves generation quality, we consider this overhead acceptable.

Table R3: Inference cost with and without SCO, measured on two NVIDIA H100 GPUs. A sample is one interleaved generation task and contains 2.53 generated images on average. Latency is the wall-clock time to complete one sample, including text generation, image generation, and the rewriter calls (Qwen3.6-Plus). LLM queries is the number of rewriter calls per sample. Throughput is the number of images generated per minute. Peak memory is the maximum GPU memory allocated per GPU during inference.

ModelLatency (s/sample) ↓LLM queries / sampleThroughput (images/min) ↑Peak memory (GB) ↓
MM-Interleaved27.781.005.4718.99
+ SCO40.56 (+46.0%)2.533.74 (−31.6%)26.69 (+40.5%)
SenseNova-U159.901.002.5438.98
+ SCO80.30 (+34.1%)2.531.89 (−25.6%)52.24 (+34.0%)
BAGEL73.301.002.0735.40
+ SCO91.60 (+25.0%)2.531.66 (−19.8%)47.75 (+34.9%)

W2: Number discrepancies between OpenING and ISG-Bench tables.

We sincerely thank the reviewer for carefully cross-checking our results. The reviewer is correct that the submission contains these inconsistencies. They stem from a labeling error on our side; the experimental results themselves are unaffected. Specifically, the benchmark names in the captions of Tables 1 and 2 were swapped: Table 1 reports results on ISG-Bench, and Table 2 reports results on OpenING. The benchmark-specific descriptions in Sec. 4.3 followed the incorrect captions and were swapped as well. With the corrected labels, all entries are consistent:

  • Table 3 (ablation). The MM-Interleaved ablation was conducted on OpenING; its Baseline and +Both rows (FID 94.1→81.394.1 \to 81.3) match the results in Table 2 (OpenING). The SenseNova-U1 ablation was conducted on ISG-Bench; its Baseline and +Both rows (FID 52.1→49.752.1 \to 49.7) match the results in Table 1 (ISG-Bench). This is consistent with the caption of Table 3, which states that the two ablations were conducted on OpenING and ISG-Bench, respectively.
  • Fig. 3 (sensitivity analysis). As stated in Sec. 4.4, this analysis evaluates SenseNova-U1 on OpenING. The values at the default configuration (FID 55.9) and at τend=0\tau_{\mathrm{end}} = 0 (FID 67.2, the unguided baseline), together with the corresponding GPT-J1 and GPT-J2 scores, match the SenseNova-U1 entries of Table 2.

Q2: Performance change with some other rewriters.

Thank you for the constructive suggestion. We compare different LLM rewriters on OpenING, keeping the generator, the template, and all hyper-parameters fixed (Table R4). With Qwen3.6-27B or stronger rewriters, SCO consistently improves over the baseline (FID 67.2), indicating that it does not rely on a specific LLM. A 9B rewriter is insufficient, as it less reliably produces minimal edits of the full prompt.

Table R4: Results of SCO with different LLM rewriters on OpenING, using SenseNova-U1 as the generator.

RewriterFID ↓CLIP-I ↑CsC_s ↑GPT-J2 ↑
Qwen3.5-9B71.80.3300.290.558
Qwen3.6-27B60.70.3590.330.595
Qwen3.6-Plus55.90.3760.350.602
GPT5.6-Luna57.40.3830.340.621
Claude Opus 4.853.10.3790.350.615
Reviewer_ufvwOpenReview ↗

Response to authors' rebuttal

Thank you for this detailed rebuttal and the experiments!

My questions and concerns, such as the cost analysis and the performance comparison of different rewriter are addressed. I think the results make sense to me. Overall, I think I lean towards 4 and willing to raise my scores. Though 9B rewriter is not very good. But as the VLM backbone gets stronger. I believe this method has more potential in the future. Please add the related discussion in the paper. Good luck!

Reviewer_fPE9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Self-Correction Optimization, a training-free guidance method for interleaved image–text generation and video generation. At selected sampling steps, it evaluates four conditional branches to obtain a new-event direction and a state-preserving direction. When either direction is not sufficiently supported, it corrects the CFG update by solving a small bounded quadratic program. Experiments cover seven image–text generators on OpenING and ISG-Bench, and three video generators.

Critical strengths
  1. The problem is practical. Interleaved generation must realize new events while keeping earlier visual states. SCO splits the CFG update into two matching directions, which is intuitive and requires no training.

  2. The method is applied to seven image–text models and three video models, with mostly consistent improvements. The authors also report the failure case on MiniGPT-5.

Critical weaknesses
  1. The need for the joint projection is not tested. The paper argues that the two directions should be corrected jointly, since separate corrections may cancel each other. However, the ablation only adds or removes directions within SCO. It does not compare with a simpler use of the same four branches, such as adding the two directions with fixed weights, similar to multi-condition CFG in image editing. Without this comparison, it is unclear whether the quadratic program is needed.

  2. The evaluation makes the gains hard to interpret. The paper states that OpenING follows its official protocol, but the tables report custom metrics, and the official metrics of neither benchmark are given. The video experiments add a GPT-5 planner, and it is unclear whether the baselines use it. The image–text pipeline relies on an unnamed commercial model to rewrite prompts, and the templates are not provided. Inference cost is also not reported, although SCO doubles the forward passes in the first half of sampling.

Questions for authors
  1. How does SCO compare with adding the two directions with fixed weights, using the same four branches?

  2. What are the results on the official metrics of OpenING and ISG-Bench?

I am willing to raise my score if the authors can convincingly address the concerns.

4 comments
AuthorsOpenReview ↗

Response to Reviewer fPE9 (1/3)

We thank the reviewer for the constructive feedback on the necessity of joint optimization, the detailed process of prompt rewriting, and inference costs. We clarify these concerns below.

W1: The necessity of joint optimization on two complementary constraints

Combining the two directions with fixed weights is equivalent to SCO with constant coefficients: δ=δref+μnewanew+μstateastate\delta = \delta_{\mathrm{ref}} + \mu_{\mathrm{new}} a_{\mathrm{new}} + \mu_{\mathrm{state}} a_{\mathrm{state}}, where (μnew,μstate)(\mu_{\mathrm{new}}, \mu_{\mathrm{state}}) are held fixed across all sampling steps and all interleaved generation tasks. SCO instead obtains the coefficients at each step from the bounded joint problem in Eqs. (9)–(12). As a result, (i) no correction is applied when the CFG update already satisfies a constraint, (ii) the correction scales with the actual shortfall, and (iii) the interaction between the two directions is accounted for through their Gram matrix.

To test whether the joint optimization is necessary, we compare SCO with two groups of alternatives on OpenING, using SenseNova-U1 as the generator (Table R1, Sec. 4.6). All variants share the same four branches and hyper-parameters, and differ only in how (μnew,μstate)(\mu_{\mathrm{new}}, \mu_{\mathrm{state}}) are obtained.

  • Fixed weights: the coefficients are set to the mean (μˉnew,μˉstate)(\bar\mu_{\mathrm{new}}, \bar\mu_{\mathrm{state}}) or the median (μ^new,μ^state)(\hat\mu_{\mathrm{new}}, \hat\mu_{\mathrm{state}}) of the SCO coefficients over all sampling steps and samples, so that the overall correction strength matches that of SCO.
  • Sequential correction: the two constraints are enforced one after the other, in either order. Each constraint is solved on the update produced by the preceding correction, so the cross term Gnew,stateG_{\mathrm{new,state}} in Eq. (10) is ignored.

SCO achieves the best result on six of the seven metrics. Fixed weights recover only a small part of its gain: mean weights reduce FID from 67.2 to 64.7 (vs. 55.9 for SCO) and fall below the baseline on four metrics, while median weights fall below the baseline on all metrics. This is because a constant correction is also applied at steps where the CFG update already satisfies both constraints. Sequential correction depends on the order and is inferior to SCO in both orders (FID 65.1/60.4 vs. 55.9). The new →\rightarrow state variant attains a slightly higher CsC_s (0.36 vs. 0.35) but falls below the baseline on CLIP-T and GPT-J1, preserving the state at the expense of the new event. Since anewa_{\mathrm{new}} and astatea_{\mathrm{state}} are not orthogonal, the second correction can violate the constraint enforced by the first (Sec. 3.4), which the joint optimization avoids.

 

Table R1: Necessity of joint optimization, evaluated with SenseNova-U1 on OpenING (Zhou et al., 2025). All variants use the same four branches and settings, and differ only in how (μnew,μstate)(\mu_{\mathrm{new}}, \mu_{\mathrm{state}}) are obtained. Fixed: the median (μ^new,μ^state)(\hat\mu_{\mathrm{new}}, \hat\mu_{\mathrm{state}}) or mean (μˉnew,μˉstate)(\bar\mu_{\mathrm{new}}, \bar\mu_{\mathrm{state}}) of the SCO coefficients over all steps and samples. Sequential: the two constraints are solved one after the other in the indicated order.

MethodFID ↓CLIP-I ↑CLIP-T ↑GPT-J1 ↑CsC_s ↑CbC_b ↑GPT-J2 ↑
SenseNova-U1 (Diao et al., 2026)67.20.3250.4710.6580.290.370.574
+ Fixed (μ^new,μ^state)(\hat\mu_{\mathrm{new}}, \hat\mu_{\mathrm{state}})72.10.3070.4590.6250.270.330.530
+ Fixed (μˉnew,μˉstate)(\bar\mu_{\mathrm{new}}, \bar\mu_{\mathrm{state}})64.70.3300.4630.6510.300.350.561
+ Sequential (new →\rightarrow state)65.10.3370.4550.6180.360.380.587
+ Sequential (state →\rightarrow new)60.40.3580.5030.6790.260.350.563
+ SCO (joint optimization)55.90.3760.5070.6940.350.400.602
AuthorsOpenReview ↗

Response to Reviewer fPE9 (2/3)

W2: (1) Choice of metrics and official metrics.

SCO targets two specific properties of interleaved generation: realizing the newly requested event and preserving the visual state inherited from earlier rounds. The official protocols of OpenING (Zhou et al., 2025) and ISG-Bench (Chen et al., 2025) assess overall response quality and report aggregated judgments. In these judgments, cross-image consistency is entangled with other criteria and is not reported separately. They can therefore indicate whether a response improves overall, but not which of the two properties is responsible. In our work, we thus report metrics where each one targets one aspect of SCO:

  • New-event realization: CLIP-T and GPT-J1, which measure the prompt fidelity of each generated image;
  • State preservation: CsC_s and CbC_b, which measure the similarity of DINOv2/CLIP features of the subject and background across images;
  • Overall plausibility: GPT-J2, which rates the factual consistency and physical plausibility of the sequence;
  • Image quality: FID and CLIP-I.

For completeness, we also report the official metrics. Table R2 gives the image-level scores on ISG-Bench. SCO raises the average score of both generators (0.179→0.2170.179 \to 0.217 for MM-Interleaved and 0.378→0.4080.378 \to 0.408 for SenseNova-U1) and improves seven of the eight tasks for each. On OpenING, SCO raises the IntJudge win rate of SenseNova-U1 from 85.36% to 89.48%, computed from pairwise comparisons against MiniGPT-5 under the w/ Tie (.5) setting. These results show that the gains of SCO are not an artifact of our choice of metrics. Both results have been added to the Appendix A.7.

Table R2: Official image-level evaluation on ISG-Bench (Chen et al., 2025). Tasks are grouped by the dominant output modality. Image: Style Transfer (Style), Progressive Transformation (Prog.), 3D Scene Transformation (3D), and Image Decomposition (Dec.). Image–Text: Image–Text Complementation (I-T C.), Temporal Prediction (Temp.), and Visual Story Telling (VST). Text: Visual Question Answering (VQA). Avg. is the mean over the eight tasks.

ModelAvg.StyleProg.3DDec.I-T C.Temp.VSTVQA
MM-Interleaved0.1790.2210.1270.1700.0160.1080.5250.0540.213
MM-Interleaved + SCO0.2170.2930.1420.1590.0850.1500.5470.0920.270
SenseNova-U10.3780.4320.3370.2590.4710.3490.4130.5070.255
SenseNova-U1 + SCO0.4080.4700.3540.3230.4840.3960.4470.4760.314

 

W2: (2) Planner in the video experiments.

All baselines in Table 5 use the same planner outputs as baselines + SCO. The GPT-5-based planner runs once per sample before sampling. It decomposes the prompt into temporally ordered events and assigns frames to them (Appendix A.1). It does not interact with the generator and only defines the frame-aligned conditions required by the per-frame SCO directions (Eqs. (14)–(15)). Each baseline is therefore standard CFG under the same planned condition cfullc_{\mathrm{full}}, which is exactly SCO with μnew=μstate=0\mu_{\mathrm{new}} = \mu_{\mathrm{state}} = 0 (equivalently, τend=0\tau_{\mathrm{end}} = 0). Since the baselines and SCO share identical inputs and differ only in the guidance update, the improvements in Table 5 are attributable to SCO rather than to the planner. We have clarified this in Sec. 4.2.

AuthorsOpenReview ↗

Response to Reviewer fPE9 (3/3)

W2: (3) Prompt rewriter and template.

The rewriter is Qwen3.6-Plus, a commercial LLM that we query through an API. It uses no local GPU memory and only produces text conditions, without access to the generator's latents or parameters. The baseline and SCO use the same rewritten full prompt; SCO additionally uses the rewriter to construct its two auxiliary conditions. In the revision, we give the rewriting procedure and template in the Appendix (Lines 961–1012).

Rewriting procedure. For the mm-th image slot, the rewriter takes the cached history Hm\mathcal H_m (previous texts and images) and the full prompt as input, and returns two prompts in one JSON object:

  • Event-off (ccounterc_{\mathrm{counter}}, Eq. (4)): a minimal edit of the full prompt in which the current event is replaced by the state immediately before it, while all preceding content is kept. If this state cannot be described explicitly, the event clause is removed and “The states of all visual subjects and backgrounds remain stable and unchanged.” is appended.
  • History-off (ceventc_{\mathrm{event}}, Eq. (6)): a one-sentence description of the current event alone, using generic references and no inherited attributes.

For example, the full prompt “… She is holding a whole apple and cuts it into two pieces with a knife.” yields “… She is holding a whole apple before cutting it.” (event-off) and “A person cuts an apple into two pieces with a knife.” (history-off). Because the event-off prompt is a minimal edit, anew=vfull−vcountera_{\mathrm{new}} = v_{\mathrm{full}} - v_{\mathrm{counter}} isolates the event rather than wording differences. The full and event-off branches keep the history images, the history-off branch drops both history text and images, and the unconditional branch uses an empty prompt.

Template:

You are given the previous texts ⟨text₁⟩, …, ⟨textₘ₋₁⟩ and images of an interleaved sequence, and the full prompt ⟨text⟩ for the next image. Return a JSON object with two fields. "event-off": Copy the full prompt verbatim, but replace the current event with the visual state immediately before it occurs, so that all content preceding the event is retained. If this state cannot be described explicitly, delete the event description and append: “The states of all visual subjects and backgrounds remain stable and unchanged.” "history-off": In one sentence, describe only the current event, using generic references (e.g., “a person”, “an object”) and no information from the history. Rules: Use only information present in the inputs. Describe visual content only. Output valid JSON only.

W2: (4) Inference cost.

Table R3 reports the wall-clock latency, number of LLM queries, throughput, and peak GPU memory of three backbones with and without SCO, measured on two NVIDIA H100 GPUs. All latencies include the rewriter calls.

SCO increases the per-sample latency by 46.0%, 34.1%, and 25.0% for MM-Interleaved, SenseNova-U1, and BAGEL, respectively. The batched evaluation also causes the increase in peak memory (+7.7 to +13.3 GB), which can be avoided by evaluating the branches sequentially at the cost of higher latency. Both the baseline and SCO use Qwen3.6-Plus through API calls. The baseline calls it once per sample to rewrite the full prompt. SCO requires 1.53 additional calls per sample on average to construct the two auxiliary conditions. Given that SCO is training-free and substantially improves generation quality, we consider this overhead acceptable. Table R3 has been added to Appendix A.5.

Table R3: Inference cost with and without SCO, measured on two NVIDIA H100 GPUs. A sample is one interleaved generation task and contains 2.53 generated images on average. Latency is the wall-clock time to complete one sample, including text generation, image generation, and the rewriter calls (Qwen3.6-Plus). LLM queries is the number of rewriter calls per sample. Throughput is the number of images generated per minute. Peak memory is the maximum GPU memory allocated per GPU during inference.

ModelLatency (s/sample) ↓LLM queries / sampleThroughput (images/min) ↑Peak memory (GB) ↓
MM-Interleaved27.781.005.4718.99
+ SCO40.56 (+46.0%)2.533.74 (−31.6%)26.69 (+40.5%)
SenseNova-U159.901.002.5438.98
+ SCO80.30 (+34.1%)2.531.89 (−25.6%)52.24 (+34.0%)
BAGEL73.301.002.0735.40
+ SCO91.60 (+25.0%)2.531.66 (−19.8%)47.75 (+34.9%)

Thank you for the clarifications and additional experiments. The new ablations support joint optimization, and the official metrics, implementation details, and cost reporting sufficiently address my main concerns. I am raising my score to 3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers and the Area Chair

We sincerely thank all reviewers and the Area Chair for their time and effort in evaluating our submission. We are encouraged by the reviewers' positive assessment of our work and grateful for their insightful comments, which have helped us substantially improve the paper.

Core contributions.

  1. We identify that, in interleaved multimodal generation, each new image must both realize the newly requested event and preserve the visual state inherited from the history, whereas the standard CFG update enforces neither requirement explicitly.
  2. We propose Self-Correction Optimization (SCO), a training-free guidance method that corrects the CFG update along a new-event direction and a state-preserving direction, with their coefficients obtained at each sampling step from a bounded joint optimization.
  3. SCO is plug-and-play across interleaved generators and extends to video generation, consistently improving image quality and cross-image consistency on OpenING and ISG-Bench.

Summary of revisions. Based on the reviewers' feedback, we have made the following revisions:

  • Necessity of joint optimization (fPE9, 1uJB). We add fixed-weight and sequential-correction variants (Table 4, Sec. 4.6). SCO outperforms both, showing that its gains stem from the adaptive coefficients of the joint optimization.
  • Prompt rewriting (fPE9, ufvw, 1uJB). We name the rewriter, describe the rewriting in Sec. 3.2–3.3, add the procedure and complete template to Appendix A.4. We further show that SCO remains effective with different rewriters of sufficient capability (Table 9, Appendix A.8).
  • Inference cost (fPE9, ufvw, 1uJB). We report the latency, LLM queries, throughput, and peak memory of three backbones (Table 6, Appendix A.5); SCO increases the per-sample latency by 25.0–46.0%.
  • Official metrics (fPE9). We report the official ISG-Bench scores and the OpenING IntJudge win rate (Table 8, Appendix A.7), both consistent with our main results.
  • Fairness of comparisons (fPE9, ufvw). We clarify that each baseline shares the same rewritten full prompt and planner outputs with SCO, so the two differ only in the guidance update (Sec. 4.2 and 4.7).
  • Table labels (ufvw). We correct the swapped benchmark names in the captions of Tables 1 and 2 and in Sec. 4.3; the experimental results are unaffected.

All revisions are highlighted in orange in the revised paper. Detailed responses are provided in our reply to each reviewer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.