acceptodds
Under review as a conference paper at ICLR 2027

Reallocate, Rescale, Retain: Test-Time Latent Control for Image Generation

Abstract

Optimizing continuous text and image latents at test time can improve image generation in unified multimodal large language models without updating pretrained parameters. Yet these latent states evolve during optimization, while the policy controlling their refinement often remains fixed. This mismatch can waste updates on locally saturated positions, ignore differences in update reliability, and discard better intermediate images by returning the final iterate. We introduce TRILO (Triple-Control Latent Optimization), a state-dependent method that controls where to refine, how to scale update signals, and which evaluated image to return. Reallocate uses predictive entropy to shift a fixed-size text-latent refinement window away from local saturation. Rescale adjusts update signals within the selected window, applying more conservative scaling at less confident positions. Retain returns the highest-reward evaluated image if the stopping criterion remains unmet after budget exhaustion, without additional generation or reward evaluation. Together, these controls adapt how a fixed optimization budget is deployed rather than increasing it. Local theoretical analysis and controlled diagnostics support these design principles. Experiments on GenEval, T2I-CompBench, and WISE demonstrate consistent improvements in compositional and knowledge-intensive generation on Janus-Pro-1B and Janus-Pro-7B. With benchmark evaluators as online rewards and matched maximum update budgets, TRILO achieves higher scores and 12.8–24.0% lower measured end-to-end runtime than a fixed-policy baseline across all three benchmarks on the 7B model.

Primary area:
generative models
Keywords:
Multimodal Image Generation, Test-Time Optimization

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_caApOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper explores adaptive test-time latent optimization for frozen unified multimodal models. Building on MILR’s joint reasoning optimization for text and image vector representations, TRILO adds three controls: it selects a high-entropy text window (Reallocate), scales that window’s update signals inversely with entropy (Rescale), and keeps the best image seen so far if the budget runs out before the stopping criterion is met (Retain). Image latents follow a fixed-prefix policy. Theoretical analysis and component ablations support these design choices. Experiments on GenEval, T2I-CompBench, and WISE show the effectiveness of TPILO, surpassing MILR and reducing runtime by 12.8–24.0%.

Critical strengths

(1) Clear presentation. The paper is well organized around Reallocate, Rescale and Retain. The clear structure makes the motivation and the three controls easy to follow.

(2) Detailed experimental and theoretical analysis. The empirical evidence is strong, supported by component ablations, alternative online rewards, and matched MILR comparisons. The theoretical proof is precise and well-scoped.

Critical weaknesses

The current design restricts adaptive allocation and scaling to text latents, while image latents rely on fixed-prefix updates. However, the paper does not analyze the impact of this asymmetry, leaving it unclear how this specific design choice influences overall performance.

Questions for authors

(1) The text-window length is evaluated at a fixed image refinement ratio in appendix. Whether the optimal text ratio remains stable when the image ratio varies, particularly for tasks that are highly sensitive to visual details, such as counting problems and space relations.

(2) The current design restricts adaptive allocation and scaling to text latents, while image latents rely on fixed-prefix updates. The underlying motivation for this asymmetry isn't fully fleshed out for me. A controlled comparison (e.g., text-only vs. image-only vs. joint adaptive refinement) needs to be conducted to whether this asymmetry design will affect the text image balance for the unified multimodal models.

4 comments
AuthorsOpenReview ↗

Response to Reviewer caAp (Part 1 of 2) — Q1: Joint text–image refinement ratios for spatial relations and counting

We thank the reviewer for the suggestions on joint allocation and modality control. We address both with a joint ratio sweep and a four-way modality ablation.

Q1. Does the preferred text-window ratio remain stable when the image refinement ratio varies, particularly for counting and spatial relations?

A1. The highest-scoring text ratio changes with the image allocation on Spatial, while Counting favors 0.4 at all three tested image ratios. We evaluated the nine combinations below on 300 T2I-CompBench Spatial prompts (TS300) and 80 GenEval counting prompts (GC80), using one generation seed.

TS300 Spatial score

Text ratioImage 0.01Image 0.02Image 0.05
0.10.52900.53710.5192
0.20.54680.54130.5257
0.40.54270.53070.5327

GC80 counting accuracy

Text ratioImage 0.01Image 0.02Image 0.05
0.188.75%91.25%85.00%
0.286.25%90.00%85.00%
0.491.25%92.50%88.75%

On Spatial, the best tested text ratio shifts from 0.2 to 0.4 when the image ratio increases from 0.02 to 0.05. At the default image ratio of 0.02, increasing the text ratio to 0.4 improves Counting but reduces Spatial performance. The shared default of 0.2 follows the separate quality–runtime analysis in Appendix C.6. The joint sweep identifies task-specific alternatives. Section 5.3 summarizes the findings, and Appendix C.6.1, Table 19 reports the full grid.

AuthorsOpenReview ↗

Response to Reviewer caAp (Part 2 of 2) — Q2 and Critical Weakness: Why use adaptive text refinement with a fixed image prefix?

Q2. What motivates adaptive control on text with fixed-prefix image refinement, and how does this choice affect the balance between modalities? This also addresses the Critical Weakness.

A2. Adaptive text control with fixed-prefix image refinement performs best in the four-way ablation below. Text adaptivity is motivated by the observed weak local gradients at saturated text positions. The comparison tests whether extending the same controls to image latents also helps.

Each executed update refines both text and image latents in all four configurations. “Fixed” uses a fixed prefix with unit scaling. “Adaptive” applies Reallocate and Rescale. The four configurations share the backbone, learning rate, refinement ratios, clipping bounds, 20-update limit, feedback, and Retain.

ConfigTextImageT300GC80
A00FixedFixed0.609869/80 (86.25%)
A10 (TRILO)AdaptiveFixed0.621972/80 (90.00%)
A01FixedAdaptive0.594061/80 (76.25%)
A11AdaptiveAdaptive0.596167/80 (83.75%)

T300 averages 50 prompts from each of six T2I-CompBench categories. GC80 contains 80 GenEval counting prompts.

Adding image adaptivity to TRILO lowers T300 from 0.6219 to 0.5961 and counting accuracy from 90.00% to 83.75%, supporting the tested text-adaptive, image-prefix design. Moving image windows reset their Adam state, so the comparison evaluates the complete control policy.

Section 5.3 summarizes the comparison. Appendix C.3.5 reports the full ablation in Table 10 and describes the optimizer-state handling.

Thanks for authors' response. I will keep my score positive but will not raise score to 4. While the paper is well organized and demonstrates empirical effectiveness, the proposed idea and method are relatively straightforward and offer limited new insights.

Thank you for your thoughtful response. We are pleased to have had the opportunity to address your questions and provide further clarification.

TRILO’s advance lies in adapting refinement to the evolving latent state. Our diagnostics show that refinement opportunities vary across positions and steps, motivating adaptive update locations and signal weighting. Together with retention of the best evaluated image, these controls improve generation quality and reduce measured runtime compared with MILR under matched benchmark-aligned feedback and the same maximum update budget.

Thank you again for your constructive suggestions and careful evaluation of our work.

Reviewer_fQmvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TRILO for test-time latent optimization in unified multimodal image generation. Building on MILR, it uses predictive entropy to select text-latent refinement windows and rescale update signals, and retains the highest-reward evaluated image after budget exhaustion. Experiments are conducted on Janus-Pro-1B and 7B across three benchmarks.

Critical strengths
  1. The method is simple and requires no additional model training or increase in the maximum refinement budget.
  2. The matched experiments show consistent improvements over MILR across three benchmarks, supported by component ablations and three generation seeds.
Critical weaknesses
  1. Limited methodological novelty. TRILO mainly adds entropy-based window selection, signal rescaling, and best-iterate selection to MILR. The technical advance appears incremental.

  2. Evaluation is limited to Janus-Pro-1B and 7B, which share the same architecture. It is unclear whether the entropy-based controls transfer to other unified multimodal generators.

  3. No sensitivity analysis is provided for the inverse-entropy clipping bounds (0.5, 2.0). Since these bounds control the strength of Rescale, their influence on performance should be examined."

Questions for authors

See the weaknesses above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer fQmv (Part 1 of 2) — W1: TRILO’s methodological contribution beyond MILR

We thank the reviewer for recognizing the matched comparisons and fixed refinement budget. We address the contribution, generator transfer, and clipping sensitivity below.

W1. The methodological advance over MILR appears incremental.

Response to W1. The contribution we claim is to make a fixed refinement budget responsive to the evolving latent state. MILR updates a fixed text prefix with uniform signal scaling, although those positions need not remain useful throughout optimization. TRILO adapts the update location and relative weighting while keeping the reward objective, optimization variables, refinement size, and maximum update budget unchanged.

The Janus-Pro-7B ablation in Table 2(a) shows how the optimization controls perform before best-iterate selection is introduced.

MethodT2I-CompBenchWISE
Matched MILR0.53990.6098
Reallocate + Rescale0.57640.6730
TRILO0.62070.7105

Reallocate and Rescale improve both benchmarks before Retain is introduced. The separately evaluated full configuration achieves the highest aggregate scores on both benchmarks. The improvement therefore extends beyond selecting a better intermediate image. We have clarified this contribution in Sections 3.2–3.3 and highlighted the controlled ablation in Section 5.3.

AuthorsOpenReview ↗

Response to Reviewer fQmv (Part 2 of 2) — W2–W3: Transfer beyond Janus-Pro and sensitivity to clipping bounds

W2. Janus-Pro-1B and Janus-Pro-7B share an architecture. Do the controls transfer to other unified multimodal generators?

Response to W2. We evaluated TRILO on OneCAT-3B and Show-o2-1.5B, which use multi-scale autoregressive and flow-matching image generation, respectively. On both generators, Reallocate selects a window of conditional prompt embeddings using entropy from the frozen language head, and Rescale weights the update signals within that window. Both models keep their pretrained weights and native image samplers.

We evaluated both generators on the full GenEval, T2I-CompBench, and original WISE benchmarks. Each entry shows the aggregate score for the backbone’s initial images → the score for the images returned by TRILO on the same prompts.

BackboneGenEval (553 prompts)T2I-CompBench (1,800 prompts)WISE (1,000 prompts)
OneCAT-3B0.8855 → 0.89060.5554 → 0.57730.8791 → 0.9037
Show-o2-1.5B0.5992 → 0.63920.4780 → 0.51150.4267 → 0.5443

Aggregate scores improve in all six backbone–benchmark comparisons. On T2I-CompBench, 427 of 1,800 prompts improve on OneCAT and 476 improve on Show-o2. The largest gain occurs on Show-o2 WISE, where WiScore rises by 0.1176, with strict improvements on 537 of 1,000 prompts.

These results show that TRILO’s text-side refinement and output selection transfer to both generator families under benchmark-aligned feedback. Retain selects from the evaluated candidates, including the initial image, and Show-o2 draws fresh noise for each candidate. Section 5.3 summarizes the findings, and Appendix C.12 describes the adaptation and reports the full results in Table 34.

W3. How sensitive is Rescale to the inverse-entropy clipping bounds of (0.5, 2.0)?

Response to W3. Across the five tested clipping settings, the T300 macro score varies by 0.0061, while Spatial shows greater sensitivity. We varied one bound at a time on the same 300 prompts, with 50 per T2I-CompBench category. All runs use Janus-Pro-7B, text/image refinement ratios of 0.2/0.02, and a maximum of 20 updates.

Raw boundsT300 macroSpatial
(0.25, 2.00)0.61580.4078
(0.50, 1.50)0.61800.4231
(0.50, 2.00)0.62190.4546
(0.50, 4.00)0.61630.4176
(0.75, 2.00)0.62120.4431

The default (0.5, 2.0) gives the highest observed macro and Spatial scores. The bounds clip raw inverse-entropy weights before unit-mean normalization. We summarize the sensitivity analysis in Section 5.3, report the full comparison in Appendix C.3.4, Table 9, and clarify the clipping and normalization order in Section 3.2.

Reviewer_zsjYOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a reallocate, rescale, and retain method to improve the optimization process of latentseek-like reasoning. It achieves 0.99 overall performance on the GenEval benchmark, and achieves 0.6207 on the T2I Compbench benchmark. The ablation studies demonstrate the effectiveness of the three proposed mechanisms.

Critical strengths
  1. The proposed mechanisms largely improve performance, and the ablation studies clearly state the source of the improvement.
  2. The ablation clearly demonstrates the performance of each mechanism.
  3. Though the methods are mostly engineering improvements, they are very useful.
Critical weaknesses
  1. The writing should be polished. Especially in the theoretical part, it would be better to present the paragraphs in a theorem-like form. Figure 6 isn't a vector graph.
  2. The reported performance in the main table uses the benchmark-aligned reward
Questions for authors
  1. I can't find the precise iteration budget for TRILO. Can the authors point it out to me?
  2. Line 376: what is the meaning of the mean normalized score? What is score?
3 comments
AuthorsOpenReview ↗

Response to Reviewer zsjY — Theory and Figure 6, benchmark feedback, update budget, and score definition

We thank the reviewer for the careful reading of our paper and for highlighting the practical value of the three controls and their ablations.

W1. The writing should be polished, particularly by presenting the theoretical results in a theorem-like form. Figure 6 is not a vector graphic.

Response to W1. We have revised Section 4 to present the theoretical results as Propositions 4.1–4.3 with explicit assumptions. Appendix A provides the full proofs. In the revised Figure 6, the heatmaps, reward curve, axes, and labels in panels (b–d) are vector graphics. Panel (a) shows the corresponding generated images.

W2. The reported performance in the main table uses the benchmark-aligned reward.

Response to W2. Indeed, the primary results use the benchmark evaluator as online feedback. In our matched TRILO–MILR comparison, both methods receive the same feedback and maximum refinement budget, allowing us to compare their control policies under common conditions. To evaluate performance with a separate online critic, we use UnifiedReward-Qwen-7B to guide refinement and the official benchmark evaluators only for final scoring. The table compares frozen Janus-Pro-7B with TRILO under this feedback (Appendix C.7, Table 21).

Official metricJanus-Pro-7B+ TRILO
T2I-CompBench0.39210.5201
WISE0.35000.5326

We have clarified the online feedback and final evaluator for each setting in Section 5.1 and Appendix C.7.

Q1. I cannot find the precise iteration budget for TRILO. Can the authors point it out?

A1. In the standard Janus-Pro experiments, TRILO allows at most 20 joint text–image latent updates per prompt. We evaluate the initial image before optimization and one image after each update, so a full-budget run evaluates 21 images in total. Runs stop earlier when reward exceeds the task-specific threshold. If the threshold remains unmet after 20 updates, Retain returns the highest-reward evaluated image, including the initial candidate. We have made the update budget and candidate count explicit in Section 5.1.

Q2. Line 376: What does “mean normalized score” mean? What is the score?

A2. “Score” denotes the reward-weighted gradient norm of the greedy token’s log probability with respect to its text latent. For each prompt and step, each norm is divided by the mean across text positions, plus a small numerical-stability constant. Figure 4(a) averages these normalized values within each entropy quintile (Appendix C.4.1, Eqs. 25–26).

We have replaced “mean normalized score norm” with “mean normalized reward-weighted gradient norm” and clarified the definition in Section 5.3 and the caption of Figure 4(a).

Thank you for the response. My concerns have been addressed, and I will raise my score to 4. I like the paper, and it is a clear advance in test-time latent reasoning.

One suggestion for the final version, independent of my score: I recommend citing this overview of test-time latent reasoning, as it covers the line of work this paper builds on: https://latentreasoning.github.io/test-time-latent-reasoning/

We are pleased to hear that our response has addressed your concerns. We sincerely appreciate your careful consideration of our work and your decision to raise your rating.

Thank you also for recommending this overview. We will cite it in the Related Work section of the final version, as it will help us place TRILO in the broader context of test-time latent reasoning.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.