- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper explores adaptive test-time latent optimization for frozen unified multimodal models. Building on MILR’s joint reasoning optimization for text and image vector representations, TRILO adds three controls: it selects a high-entropy text window (Reallocate), scales that window’s update signals inversely with entropy (Rescale), and keeps the best image seen so far if the budget runs out before the stopping criterion is met (Retain). Image latents follow a fixed-prefix policy. Theoretical analysis and component ablations support these design choices. Experiments on GenEval, T2I-CompBench, and WISE show the effectiveness of TPILO, surpassing MILR and reducing runtime by 12.8–24.0%.
Critical strengths
(1) Clear presentation. The paper is well organized around Reallocate, Rescale and Retain. The clear structure makes the motivation and the three controls easy to follow.
(2) Detailed experimental and theoretical analysis. The empirical evidence is strong, supported by component ablations, alternative online rewards, and matched MILR comparisons. The theoretical proof is precise and well-scoped.
Critical weaknesses
The current design restricts adaptive allocation and scaling to text latents, while image latents rely on fixed-prefix updates. However, the paper does not analyze the impact of this asymmetry, leaving it unclear how this specific design choice influences overall performance.
Questions for authors
(1) The text-window length is evaluated at a fixed image refinement ratio in appendix. Whether the optimal text ratio remains stable when the image ratio varies, particularly for tasks that are highly sensitive to visual details, such as counting problems and space relations.
(2) The current design restricts adaptive allocation and scaling to text latents, while image latents rely on fixed-prefix updates. The underlying motivation for this asymmetry isn't fully fleshed out for me. A controlled comparison (e.g., text-only vs. image-only vs. joint adaptive refinement) needs to be conducted to whether this asymmetry design will affect the text image balance for the unified multimodal models.
4 comments
Response to Reviewer caAp (Part 2 of 2) — Q2 and Critical Weakness: Why use adaptive text refinement with a fixed image prefix?
Q2. What motivates adaptive control on text with fixed-prefix image refinement, and how does this choice affect the balance between modalities? This also addresses the Critical Weakness.
A2. Adaptive text control with fixed-prefix image refinement performs best in the four-way ablation below. Text adaptivity is motivated by the observed weak local gradients at saturated text positions. The comparison tests whether extending the same controls to image latents also helps.
Each executed update refines both text and image latents in all four configurations. “Fixed” uses a fixed prefix with unit scaling. “Adaptive” applies Reallocate and Rescale. The four configurations share the backbone, learning rate, refinement ratios, clipping bounds, 20-update limit, feedback, and Retain.
| Config | Text | Image | T300 | GC80 |
|---|---|---|---|---|
| A00 | Fixed | Fixed | 0.6098 | 69/80 (86.25%) |
| A10 (TRILO) | Adaptive | Fixed | 0.6219 | 72/80 (90.00%) |
| A01 | Fixed | Adaptive | 0.5940 | 61/80 (76.25%) |
| A11 | Adaptive | Adaptive | 0.5961 | 67/80 (83.75%) |
T300 averages 50 prompts from each of six T2I-CompBench categories. GC80 contains 80 GenEval counting prompts.
Adding image adaptivity to TRILO lowers T300 from 0.6219 to 0.5961 and counting accuracy from 90.00% to 83.75%, supporting the tested text-adaptive, image-prefix design. Moving image windows reset their Adam state, so the comparison evaluates the complete control policy.
Section 5.3 summarizes the comparison. Appendix C.3.5 reports the full ablation in Table 10 and describes the optimizer-state handling.
Thanks for authors' response. I will keep my score positive but will not raise score to 4. While the paper is well organized and demonstrates empirical effectiveness, the proposed idea and method are relatively straightforward and offer limited new insights.
Thank you for your thoughtful response. We are pleased to have had the opportunity to address your questions and provide further clarification.
TRILO’s advance lies in adapting refinement to the evolving latent state. Our diagnostics show that refinement opportunities vary across positions and steps, motivating adaptive update locations and signal weighting. Together with retention of the best evaluated image, these controls improve generation quality and reduce measured runtime compared with MILR under matched benchmark-aligned feedback and the same maximum update budget.
Thank you again for your constructive suggestions and careful evaluation of our work.
Response to Reviewer caAp (Part 1 of 2) — Q1: Joint text–image refinement ratios for spatial relations and counting
We thank the reviewer for the suggestions on joint allocation and modality control. We address both with a joint ratio sweep and a four-way modality ablation.
Q1. Does the preferred text-window ratio remain stable when the image refinement ratio varies, particularly for counting and spatial relations?
A1. The highest-scoring text ratio changes with the image allocation on Spatial, while Counting favors 0.4 at all three tested image ratios. We evaluated the nine combinations below on 300 T2I-CompBench Spatial prompts (TS300) and 80 GenEval counting prompts (GC80), using one generation seed.
TS300 Spatial score
GC80 counting accuracy
On Spatial, the best tested text ratio shifts from 0.2 to 0.4 when the image ratio increases from 0.02 to 0.05. At the default image ratio of 0.02, increasing the text ratio to 0.4 improves Counting but reduces Spatial performance. The shared default of 0.2 follows the separate quality–runtime analysis in Appendix C.6. The joint sweep identifies task-specific alternatives. Section 5.3 summarizes the findings, and Appendix C.6.1, Table 19 reports the full grid.