acceptodds
Withdrawn from ICLR 2027

FLAT: Resampling Image and Text into 1D Flexible-Length Aligned Transmodal Tokens for Retrieval and Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Traditional multimodal representation learning and generation are two stages: a contrastive or self-supervised visual encoder is trained first, followed by a separate downstream generative model. This setup bottlenecks generative performance behind frozen embeddings. To bridge this gap, we revisit joint multimodal representation learning and generation to produce linearly interpolatable embeddings that are directly consumable by generative decoders. We present FLAT (Flexible-Length Aligned Transmodal representations), a representation pre-training framework that jointly optimizes a shared multimodal encoder alongside downstream text-to-image (T2I) and image-to-text (I2T) decoders. By combining contrastive alignment with bidirectional cross-modal generative objectives, FLAT ensures its representations function as both discriminative semantic descriptors and generative conditions. Architecturally, FLAT maps visual and textual inputs into a unified continuous 1D sequence space, applying nested dropout over prefix-K tokens to enable dynamic output lengths. A single pre-training stage allows FLAT to perform cross-modal retrieval and generation across variable prefix K, especially with a T2I GenEval score of 71.1. Task-specific fine-tuning further aligns model performance with state-of-the-art baselines: 83.1 GenEval on T2I generation; 40.5 BLEU-4 and 138.6 CIDEr on MS-COCO image captioning; and Recall@5 scores of 88.4 (I2T) / 77.9 (T2I) on MS-COCO alongside 98.3 (I2T) / 93.6 (T2I) on Flickr30K. Finally, qualitative evaluations demonstrate that FLAT representations natively support linear interpolation, latent space arithmetic, and zero-shot composed retrieval.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal representation learning, flexible-length representations, cross-modal generation and retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eeb5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

FLAT jointly trains a multimodal encoder and image-to-text and text-to-image decoders. Shared register tokens yield image and text representations with a variable-length prefix, aligned by contrastive learning and used for generation. The paper evaluates retrieval, captioning, and image synthesis; its fine-tuned image generator reaches about 0.83 on GenEval.

Critical strengths

The shared ordered-token design gives a concrete mechanism for using the same representation across retrieval and both generation directions.

Loss ablations and token-length sweeps reveal a useful distinction: retrieval is relatively stable at short lengths, whereas compositional image generation needs more tokens.

Critical weaknesses

The best retrieval, captioning, and image-generation results use separate task-specific fine-tuned checkpoints. They should not be read as the performance of a single unchanged model.

The value of joint training remains difficult to isolate because the main comparisons change backbone, data, or training budget; pretraining uses 65.7 million image–text pairs.

Flexible length has a substantial quality cost for generation: fine-tuned GenEval rises from 0.49 at one token to 0.83 at 16 tokens. Measured latency and peak-memory trade-offs are needed to assess its practical value.

Questions for authors

Can you report a backbone-, data-, and compute-matched comparison with separately trained or two-stage retrieval and generation models?

For each token length, what are measured latency, peak memory, retrieval, captioning, and image-generation results?

Which headline results come from the shared pretrained checkpoint, and which require task-specific fine-tuning?

Reviewer_hyVkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes to learn a unified image-text representation to be applied for multimodal retrieval, i2t and t2i generation. The unified representation could have flexible token length for different tasks.

Critical strengths
  1. The research question is well-motivated and meaningful.
  2. The proposed method is effective in the experiments.
Critical weaknesses
  1. The authors should emphasis what's original in this paper. There are plenty papers research on 1D visual resampling [1,2], nested dropout [3,4], contrastive + generative learning [5,6].
  2. Ablation is needed to verify that each component (learnable registers, nested dropout, cross-modal alignment, training objectives) has positive influence on the performance.
  3. In Table 10, why increasing token decrease the retrieval performance?

[1] Yu, Qihang, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen. "An image is worth 32 tokens for reconstruction and generation." NeurIPS 2024. [2] Bachmann, Roman, Jesse Allardice, David Mizrahi, Enrico Fini, Oğuzhan Fatih Kar, Elmira Amirloo, Alaaeldin El-Nouby, Amir Zamir, and Afshin Dehghan. "Flextok: Resampling images into 1d token sequences of flexible length." ICML 2025. [3] Rippel, Oren, Michael Gelbart, and Ryan Adams. "Learning ordered representations with nested dropout." ICML 2014. [4] Kusupati, Aditya, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder et al. "Matryoshka representation learning." NeurIPS 2022. [5] Li, Junnan, Dongxu Li, Caiming Xiong, and Steven Hoi. "Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation." ICML 2022. [6] Yu, Jiahui, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. "Coca: Contrastive captioners are image-text foundation models." 2022.

Questions for authors

See weakness.

Reviewer_YPmtOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper identifies the separation of representation learning and generation as a limitation of existing multimodal pipelines. Approaches that attach generative models to frozen pretrained encoders such as CLIP and SigLIP constrain generative performance by the fixed representations and require realignment during generative model training. Conversely, recent unified models eliminate explicit representation spaces, making it difficult to obtain contrastive embeddings that support linear interpolation. To address these limitations, the authors propose FLAT, which resamples images and text into shared registers to produce a single variable-length 1D representation. Contrastive, captioning, and image generation losses are jointly applied to this representation.

Critical strengths

The topic is interesting. Exploring whether a single representation can serve both retrieval and bidirectional generation, while retaining an explicit embedding space, is a relevant question given the current trend toward unified models.

Critical weaknesses
  1. The baselines are inconsistently selected across key comparisons, and the claimed advantages depend on this selection. (a) Generation: Table 1 includes models evaluated with prompt rewriting, and the paper claims that FLAT outperforms all compared baselines, including two 7B models using this technique. Having adopted prompt-rewriting systems as valid baselines, the paper omits BAGEL, which reports 0.88 under the same setting and exceeds FLAT’s 0.83. The claim of surpassing “every baseline we compare against” therefore holds only because of this selection. (b) Backbone: FLAT uses Qwen3.5 as its backbone, whereas MetaQuery, a comparison method from the same model family, uses Qwen2.5-VL. Thus, the advantage reported in Table 1 cannot be attributed solely to the proposed method, as it may partly reflect improvements in the underlying backbone. A controlled comparison using the same backbone, where feasible, would help isolate the contribution of the proposed method. (c) Retrieval: Table 3 does not report comparisons against established baselines such as CLIP, SigLIP, or BLIP-2. Notably, despite being fine-tuned on COCO, FLAT (with an encoder of at least 2B parameters) achieves only 65.8/50.5 I→T/T→I R@1, marginally outperforming the much smaller SigLIP 2 B/32 (63.7/47.2) while falling substantially short of SigLIP 2 So400m (71.5/55.1), even though the latter results are obtained in a zero-shot setting. Although these results are not directly comparable due to differences in training protocols, they raise concerns about FLAT's retrieval effectiveness and do not establish a clear advantage over existing approaches.

  2. The paper’s central claim that retrieval and generation are mutually reinforcing is contradicted by its own ablation (Table 5). Under a comparable training budget, retrieval improves when all three losses are combined, but adding only the captioning or generation loss actually degrades T2I retrieval. GenEval shows no improvement from adding the other losses, while captioning performs best with the captioning loss alone. These results directly contradict the paper’s statement that “the three objectives provide complementary signals and make positive marginal contributions across prefix lengths.” This is not a matter of missing baselines; the core claim itself needs to be revised.

  3. The presentation lacks clarity on key design choices and training details. Section 3 describes nested dropout but does not explain why variable-length representations are necessary. The Introduction discusses 1D visual tokenization as motivation, but its role in FLAT is not clearly established. Furthermore, the paper does not explain in the main text or Figure 1 that the encoder and text decoder share the same Qwen backbone, nor does it specify which parameters are trained at each stage. These details appear only in Appendix Tables 6 and 7, making the overall framework difficult to understand. The paper also insufficiently distinguishes FLAT from prior work on joint representation learning and generation, such as SEED (ICLR 2024) and UniLIP (ICLR 2026).

Questions for authors
  1. How does FLAT differ from prior work that jointly addresses retrieval and generation, such as SEED (ICLR 2024) and UniLIP (ICLR 2026)? Please clarify the specific novelty of each component, including the shared encoder, variable-length representations, and joint training.

  2. If retrieval performance remains nearly unchanged as K increases from 1 to 256 in Table 3, what benefits do additional tokens and variable-length representations provide for retrieval? What specific claim is this table intended to support? Additionally, why does retrieval performance deteriorate as K increases for the pretrained checkpoint in Table 10?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.