acceptodds
Under review as a conference paper at ICLR 2027

CoRe: Jointly Learning Contextual Representations for Understanding and Generation

Abstract

Understanding and generation place distinct demands on learned representations, yet these demands need not conflict. We introduce CoRe which jointly learns contextual representations that support understanding, provide conditioning, and serve as the space for generation. Both images and text are generated through continuous flows in the representations used for understanding. Trained from scratch on CC12M, the 383M-parameter model matches same-data contrastive controls on global understanding and reaches COCO-30K FID 10.10, comparable to much larger generators with pretrained components. Stronger language-pretrained systems retain an advantage in prompt following and caption fidelity. Matched interventions show that generating directly in the representation used for understanding improves compositional generation over a separate projected space, even as the latter reconstructs more accurately. A more flexible readout also recovers most of the dense-prediction gap. Together, these results show that understanding and generation can coexist in one contextual representation, with their interaction governed largely by how information is retained, organized, and made accessible.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Joint Understanding and Generation, Continuous Generative Models, Multimodal Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4W9KOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CoRe learns image and text encoders jointly with two continuous denoisers. Each modality's contextual representation supports contrastive understanding, conditions generation of the other modality, and supplies the latent space for generating its own modality. Pixel and token losses anchor the moving representations, while a detached representation-regression target supports sampling. The 383M-parameter system is initialized from scratch and trained on recaptioned CC12M, with pretrained VGG16/DINOv2 used for training supervision. It preserves global understanding relative to same-data, same-step contrastive controls, supports both generation directions, and allows few-step distillation. The paper further studies whether sharing the generation space improves compositional generation and whether the loss in linear dense-prediction performance reflects reduced accessibility rather than wholesale loss of information.

Critical strengths
  1. A concrete, effective realization of the shared-representation hypothesis. Equations (1)-(7) place understanding and both generation directions in jointly learned contextual spaces, with a clear observable-space anchor for moving targets. At the reported scale, CoRe achieves ImageNet zero-shot accuracy of 50.63 versus 49.51 for the contrastive control and retrieval mR of 70.34 versus 69.75, while producing images with COCO-30K FID 10.10 and captions with SPICE 13.53 (Tables 1-3). These establish a useful feasibility result even though the model does not lead the stronger prompt-following or caption-fidelity baselines.

  2. The experimental accounting makes the conclusions substantially more credible. The paper includes frozen-contrastive and no-alignment controls, matched Gram/iBOT controls, a shared-versus-separate continuation, a readout ladder, and distillation comparisons. Appendices A-C report training exposure, compute, inherited supervision, and sampler selection; Appendix D.4 reports benchmark-template overlap in the fine-tuning mixture. Figures 6-7 show unsuccessful samples and verbatim caption defects. This transparency is particularly valuable for a unified model whose capabilities can otherwise be conflated across checkpoints or evaluation recipes.

Critical weaknesses
  1. The shared-space experiment does not yet isolate the benefit of using exactly the understanding representation from the benefit of directly aligning the generation target. In Section 5, Table 4, and Appendix D.3 (pp. 27-28, Table 15), the separate arm generates through a second projection that does not receive its own alignment loss, whereas the shared target is directly shaped by alignment. The difference of .066 GenEval and 5.39 DPG is convincing for this particular intervention, including the reported prompt bootstrap and three sampling seeds, but these checks do not distinguish semantic regularization of the target from a benefit that requires identical spaces. The study also uses one training seed and a 16K-step continuation from an already jointly trained checkpoint, rather than a complete independent training comparison. The authors acknowledge both limitations on p. 28; nevertheless, the abstract and conclusion give the result greater generality than this control warrants. A separate projection with its own matched alignment/readout objective, repeated across training seeds, would address the main alternative explanation. Alternatively, scope the claim explicitly to this unconstrained separate-projection continuation rather than implying a general advantage over separate semantically organized spaces.

  2. The evidence for recovering the dense-prediction deficit is narrower than the main-task deficit it is used to explain. The primary full-resolution ADE20K evaluation shows a 5.62 mIoU loss on the contextual representation, or 7.94 with matched Gram/iBOT regularization (Tables 1 and 17). The near-recovery result instead uses a twofold-downsampled package, only 8K training images, and one probe run per cell (Appendices B.1/H.1, Figure 3, Table 28). Within this diagnostic the MLP comparison is useful evidence of recoverable information, but the contrastive MLP itself scores below its best linear readout: 27.21 versus 27.91, and 29.76 versus 31.26 with Gram/iBOT. Comparing the best reported readout of each model therefore leaves gaps of 1.69 and 2.75 mIoU, rather than the highlighted .99 and 1.25. Most of the original diagnostic gap is still recovered; this does not invalidate that observation. It does limit extrapolation to the full-resolution task. Repeat the flexible-readout comparison under the primary ADE20K protocol, report probe variability, and specify the dense MLP architecture and optimizer settings precisely. Until then, frame the retention conclusion as evidence from the reduced-resolution diagnostic.

Questions for authors

Assessment-changing clarifications:

  1. Is there evidence from an independently aligned separate generation projection, or a longer/multiple-training-seed version of Appendix D.3, that separates the benefit of identical spaces from the benefit of aligning the generation target? If unavailable, can the claim be narrowed to the intervention actually tested? Additional experiments would strengthen the interpretation, but are not presumed to already exist.
  2. What are the exact architecture, learning rate, regularization, and selection protocol for the dense MLP probe in Table 28? Are full-resolution flexible-probe results available, and does the near-recovery conclusion hold when each representation is compared using its best validated readout? If such results are unavailable, the manuscript should state that the full-resolution deficit remains unexplained by a matched flexible-probe experiment.

Future-work suggestion, not a prerequisite for this assessment: A FLOP-matched contrastive control would help determine whether joint training offers an understanding-quality advantage per unit compute. The present same-step controls already answer the narrower coexistence question, and the paper explicitly discloses their compute mismatch.

4 comments

We thank the reviewer for the careful reading and for recognizing CoRe as “a concrete, effective realization of the shared-representation hypothesis.” We have added the proposed aligned separate-space control, repeated the continuation across training seeds, and conducted independent training from scratch.

Short answer. We ran the control you propose: a second projection with its own alignment loss, matched to CoRe's, on two training seeds and from scratch. It closes the gap and is never better than CoRe. The benefit therefore does not require identical spaces as such: it requires that understanding shape the space in which generation runs, together with generation. One representation is the simplest design with this property, and a second space that has it converges to the first.

Matched 16K-step continuations of one 24K-step checkpoint (shorter than the 125K-step model, hence below its .461); Δ is against the CoRe run of the same seed and node shape.

generation latentshaped by understandingGenEvalΔ GenEvalDPGPSNR (dB)
CoRe (single representation), 3 runs✓.428 / .428 / .427reference71.3 / 70.6 / 71.123.4
second latent with its own alignment loss, 2 seeds (new)✓.423 / .411−.004 / −.01770.1 / 70.023.2
separate latent (no alignment loss; the submission's variant), 2 seeds✗.362 / .349−.066 / −.07965.9 / 65.425.0
  1. "Does not yet isolate … identical spaces from … aligning the generation target." The new control isolates it: what matters is that understanding and generation shape the representation space jointly and that the generation trajectory runs in it; a second space with this property is no better.

    • Evidence: the aligned latent is shaped as CoRe's Z is: by the same alignment loss through the same pooling, and by generation. It is within .004 and .017 GenEval of CoRe and never ahead in GenEval or DPG. Without the alignment loss, the gap is .066 and .079.
    • What it means: a generation space needs semantic organization. Retaining the image is not what limits generation: the unaligned latent is the better autoencoder (25.0 vs. 23.4 dB) and the worse generation space.
  2. "One training seed and a 16K-step continuation … rather than a complete independent training comparison." The continuation now has two training seeds, and the comparison is repeated from scratch.

    • Evidence: trained from scratch with CoRe's recipe and schedule, the aligned two-head design is behind CoRe at 24K steps on both training seeds (GenEval .341 vs. .372 and .328 vs. .369) and at 48K steps (.397 vs. .411; zero-shot 41.8 vs. 43.4). The unaligned design is far behind (.284 vs. .411 at 48K steps). The aligned latent becomes nearly a linear function of the understanding latent (R² .89 / .95; at most .29 without alignment).
    • What it means: a second space that understanding shapes converges to the first: it is a copy, not an alternative. The continuation and the from-scratch runs give the same order.
  3. "The abstract and conclusion give the result greater generality than this control warrants." Both now state exactly what the controls show.

    • Evidence: the abstract restricts the comparison to "a second space that understanding does not shape". Section 5 reports the aligned control next to it: such a space "generates nearly as well as the shared representation and never better". The conclusion no longer claims an advantage over separate spaces in general.
  4. "A general advantage over separate semantically organized spaces." Both kinds of semantically organized separate space are measured, and neither is better than one jointly shaped representation.

    • Evidence: (i) organized by its own alignment loss and by generation, the second space converges to the shared one and adds nothing (items 1 and 2). (ii) Organized by understanding alone and then frozen, with the generator trained afterwards (the order RAE follows), it is worse at equal data (GenEval .400 vs. .461, COCO FID 12.44 vs. 10.10; Table 3).
    • What it means: understanding organizes a space, and generation makes it retain the observation; a generation space needs both properties. One jointly shaped representation has both with no added head, and is never behind either alternative in GenEval or DPG.

The resulting claim is that understanding and generation can jointly learn one contextual representation that serves both. The controls support complementary roles for semantic organization and recoverable detail, and show that a second generation space is unnecessary in the tested setting. This is the representation-level contribution of CoRe: learning the space through both objectives, and generating within that same learned space.

AuthorsOpenReview ↗

W2, Q2 and the compute suggestion: dense prediction at full resolution

Thank you for specifying the protocol precisely, and we followed it. Full-resolution flexible-probe results are now available, with the data, resolution and iterations of the primary ADE20K protocol and three probe seeds (settings in item 2). On Core's joint understanding and generation representation space Z, CoRe's linear-probe deficit to the contrastive-only control shrinks under an MLP probe, and under the attentive probe CoRe leads (+1.18 mIoU). The paper now makes its dense-prediction statement under the full-resolution protocol.

  1. "Repeat the flexible-readout comparison under the primary ADE20K protocol." On Z the gap closes with probe capacity and reverses at the best probe. ADE20K mIoU, frozen Z, learning rate 1e-3; CI: paired bootstrap over validation images:
probecontrastive onlyCoRedifference [95% CI]contrastive + Gram/iBOTCoRe + Gram/iBOT
linear30.1524.69−5.45 [−5.77, −4.75]33.1325.45
MLP, mean of 3 probe seeds35.6134.12−1.49 [−1.87, −0.92]38.3035.61
attentive, mean of 3 probe seeds41.2042.38+1.18 [+0.53, +1.89]43.3643.54

A linear probe reads what is linearly arranged, while a stronger probe reads what is there. The reversal shows that generation adds information to Z that understanding alone does not keep. Revised PDF: Section 5 (Figure 4); Appendix E.1.

  1. "Report probe variability, and specify the dense MLP … precisely." Probe seeds vary little. MLP seeds differ by at most 0.35 mIoU; attentive seeds have an s.d. of 0.08 (contrastive only) and 0.47 (CoRe). Every probe starts from frozen features with Dropout2d(0.1) and BatchNorm. The linear probe adds a 1×1 conv (as in DINOv3), the MLP 1×1 convs C → 2048 → 150 with GELU, the attentive probe two pre-norm transformer blocks of width 512. All use AdamW, learning rate 1e-3, weight decay 1e-3 and 40K iterations, and report the last iteration, with no selection on validation images. Revised PDF: Appendix B, probe settings.

  2. "When each representation is compared using its best validated readout." The attentive probe is the strongest of the three probes for both models, and there CoRe leads by +1.18. Learning rate 1e-3 gives every model a higher linear-probe score than the submission's 1e-4, with the same order; the revised paper reports 1e-3 throughout. For the reduced-resolution diagnostic, Appendix H.1 now states the best-vs-best gaps of 1.69 and 2.75.

  3. "7.94 with matched Gram/iBOT regularization." With its best probe, the matched Gram/iBOT pair is level. With the linear probe, CoRe + Gram/iBOT is behind its matched control. With the attentive probe, the best probe of both models, they reach 43.5 and 43.4 mIoU (three probe seeds each).

  4. Why the linear probe drops at Z, and why this is not lost information. The gap is specific to Z: on the encoder features h it is under one point with the linear probe and absent with the MLP and attentive probes (Appendix E.1). It comes from the generation gradient through the noised representation. Stopping this gradient raises the linear probe (15.5 → 22.3, reduced resolution) but lowers GenEval (.427 → .404) and reconstruction (22.2 → 19.5 dB, 256 images). The information is present in Z: the attentive probe reads it, and CoRe leads there (item 1).

  5. Compute (the compute-matched suggestion). Our controls match training steps and data exposure. The paper's claim is that the understanding and generation coexist at equal steps, which, as the reviewer notes, the same-step controls answer. A CoRe step costs about 2.5× a contrastive-only step (Appendix A); the additional compute trains the two generators. At equal steps, CoRe is ahead in zero-shot accuracy at all six checkpoints (+0.6 to +1.3 points). It is also ahead in few-shot classification on the pooled representation g (+1.9 / +1.7 / +1.5 at 1 / 5 / 10 shots) and as input to a frozen LLM (VQAv2 65.6 vs. 64.9; Appendix E.10). A second training seed of both models gives the same order (zero-shot 37.8 vs. 36.3 at 24K steps), and on the long descriptions of DOCCI, image-to-text R@1 is 83.4 vs. 81.6 (+1.8, 95% CI [+0.7, +2.8]). Joint training therefore does not cost global understanding.

Thank you for your rebuttal. I will keep my score.

Thank you for your positive assessment. We especially appreciate your recognition of the shared-representation contribution and the transparency of our experimental setup. We are grateful for your careful reading and constructive feedback, which have helped strengthen the paper.

Reviewer_1NEuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a novel architecture and learning framework for multimodal image-text models that supports image-to-text generation, text-to-image generation, as well as image understanding (image classification, semantic segmentation). Both modalities are generated using continuous flow modeling. Image understanding is achieved via linear or shallow non-linear readouts.
Experiments are conducted using 383M parameter sized models trained from scratch on the CC12M dataset. The model yields COCO FID comparable to that of much larger models, but it is not as strong in terms of prompt following. It trails MiniT2I significantly despite using the same training data (however, MiniT2I uses a retrained text encoder and cannot perform text captioning). Image understanding performance is comparable to that of a control baseline trained with CLIP-like loss only, but trails quite far behind SOTA vision encoders like DINOv3 (imagenet linear probe 88.1 vs 74.3 accuracy, ADE segmentation 55.9 vs 37.3 mIoU), understandably so given the small data and model scale.

Critical strengths
  • A unified architecture is presented capable of taking text and images as input and output. By not relying on existing pre-trained components meaningful internal comparisons are enabled, where data and compute are controlled.

  • In principle the setup is reproducible due to the use of open data. If the authors will release checkpoints plus training and inference code (unclear from the manuscript), it would lead to fully reproducible results.

  • A range of tasks is considered, including Image understanding (classification as well as semantic segmentation). Image-to-text generation (captioning), and text-to-image generation.

  • Despite studying a relatively small model, the observed performance is competitive with larger models such as DREAM and FLUID (both around 5B params) in terms of COCO FID.

  • The architectural design is relatively simple and mostly symmetric across modalities.

Critical weaknesses
  • The paper studies unified multimodal models, but does not include analysis of text understanding (text in / text out). Instead, the paper focusses only on text-to-image, image-to-text, and image understanding. It was not clear to me why the authors chose not to include this important capability for unified models in this study.

  • Evaluations on VQA are not included, thereby missing another important capability of multimodal systems. VQA could be added either at pertaining or in a fine-tuning stage, using, eg, Cambrian 10M data. It was not clear to me why the authors chose not to include this important capability for unified models in this study.

  • The study misses existing, or reimplemented, baselines that leverage a strong VAE image auto encoder (such as the one from Flux 2), and one with an image understanding encoder for visual generation. See for example [a], which uses cross-entropy next-token prediction on the text output, and diffusion / flow matching on the visual tokens. With a RAE [b] decoder that maps (generated) image representations to RGB pixel images. Such baselines, with controlled same data, compute, and model capacity, would help to understand the advances made by the current paper wrt the state of the art recipes.

[a] https://arxiv.org/abs/2603.03276 [b] https://arxiv.org/abs/2510.11690

  • The comparisons that are drawn to related work in Tables 1,2,3 are useful, but compare mostly at a systems level. Detailed ablations are missing to isolate individual components one-by-one. For example:

    • what happens if we plugin strong pre-trained text and/or image encoders used in prior work,
    • what happens if more common discrete autoregressive text generation is used,
    • what happens if cross-conditioning is done with only the local or only global representations,
    • what happens if the “self conditioning” path is not used in training,
  • The scale of the models (380M) and data (12M image-text pairs) considered are small by today’s standards, which makes it unclear how the findings carry over to larger models with stronger performance.

  • It is not clear whether the latent-to-observation decoders are trained on clean or noisy latents.

  • The text-to-image performance is significantly below that of MiniT2I despite using the same data. It is not clear why the presented model is worse. Ablations that use the same captions, and replace the learned text-encoder with FLAN could shed light on this.

Questions for authors
  • Why aren’t all metrics reported for the finetuned CoRe model in Table 2 ?

  • The COCO captioning performance is very poor compared to prior work. It is unclear where this originates from. Could it make sense to add fine-tuning for this task, to instill the coco caption style on this model, similar to the image generation fine-tuning.

  • The distinction between understanding and conditioning was not obvious to me in the first sections of the paper. I’d suggest to give examples of image understanding in the introduction, so that any ambiguity is avoided.

  • The different conditionings (self, cross, null) used in training (Lines 166-177) would be clearer if written explicitly in section 3.

  • The model architecture is discussed only in appendix A, not in the main text. I’d suggest to give a brief description of a couple of lines in the main text.

  • Would it be possible to report the CC12M-50K FID for MiniT2I by sampling the released checkpoint ?

  • Can you add results with distillation from the “+finetuning (final)” checkpoint in Table 3? This seems more relevant than distillation from the worse performing checkpoint that excludes the fine-tuning stage.

7 comments
AuthorsOpenReview ↗

W1: text understanding

Thank you for a detailed review and concrete suggestions. We ran every experiment requested. The table gives the outcome of each, and the replies follow in the same order.

your pointstatusresult in one line
W1 text understandingaddedon par with CLIP's text encoder with 33× fewer pairs (SICK-R 69.0 vs. 69.1); the same representation also regenerates text
W2 VQAaddedgood input for a language model: VQAv2 65.6 with a frozen LLM (DINOv2-B 64.6); 70.0 vs. 69.3 with the LLM finetuned
W3 VAE / RAE baselinesaddedat equal data, CoRe is ahead of the RAE-style recipe (GenEval .461 vs. .386, FID 10.10 vs. 11.02); among nine generation spaces, VAE and DINOv2 + RAE included, CoRe's has the best FID
W4 four ablationsaddedall four run: CoRe's text encoder conditions as well as FLAN-T5-Large; token and pooled conditions both contribute; the self route gives reconstruction; an autoregressive decoder also profits from CoRe's encoder
W5 scalediscussedthis scale makes every comparison controlled; the findings hold across checkpoints on a second seed
W6 clean or noisy latentsclarifiedno separate decoder: the pixel head is trained at every noise level (Section 3)
W7 comparison to MiniT2Icomparedsame captions and protocol: better FID on COCO and CC12M (10.10 vs. 13.42; 4.33 vs. 9.51) with no pretrained language model; GenEval .795 with only the image denoiser fine-tuned
Q1 Table 2 metricsfilledall entries of the fine-tuned model
Q2 COCO captioningaddedfine-tuning the text denoiser on COCO: SPICE 13.5 → 20.0, CIDEr-D 9.1 → 96.1, hallucinated objects 11.2% → 5.1%
Q3–Q5 presentationdoneexamples of understanding; conditionings in Section 3; architecture in the main text
Q6 MiniT2I CC12M-50K FIDadded9.51, against 4.33 for CoRe
Q7 distillation from the fine-tuned modeladdedGenEval .792 with 9 denoiser evaluations, at the level of its teacher (.795 with 257); .806 with self-ranking

W1. Text understanding (text in / text out). CoRe is a study of representation learning: by understanding we mean what can be read from a frozen representation, zero-shot or with a probe. Text is treated like images (the symmetry the reviewer notes): its representation is used for understanding, and text is generated in it.

  • Already in the submission: the text representation is evaluated wherever language is involved. It enters every zero-shot, retrieval and SugarCrepe result of Table 1, and the captions of Table 2 are generated in it.
  • New, following the reviewer's suggestion: the frozen text representation evaluated on its own, at three levels.
  1. Sentence level: on par with CLIP's text encoder, with 33× fewer training pairs. Two standard tests on the pooled output of the frozen text encoder, with no training:
    • SICK-R (9,927 sentence pairs): Spearman correlation between the cosine similarity of two sentence embeddings and the human relatedness rating.
    • Caption-to-caption retrieval: every test image has five human captions; each caption queries all other captions of the test set (25K for COCO, 5K for Flickr30k). R@1 is the fraction of queries whose nearest neighbor describes the same image.
frozen text encodertraining pairsSICK-R (Spearman)caption→caption R@1, COCOcaption→caption R@1, Flickr30k
CLIP ViT-B/32400M67.935.061.3
CLIP ViT-B/16400M69.135.060.9
CoRe12M69.035.562.5

CoRe is level with CLIP on sentence relatedness and above it on caption retrieval.

  1. Token level: the representation is contextual. We take six polysemous words in COCO captions and label the sense of each occurrence by unambiguous cue words in its caption (e.g. trunk with elephant). The sense of an occurrence is predicted from its nearest neighbor among the other occurrences of the same word. We use the word's token in Z_t, CoRe's text representation. Accuracy is 98.8%. Chance is 47%, which is also what a context-free word embedding gives, since it is the same for every occurrence.

  2. Text in, text out: the same representation generates text. Conditioned only on Z_t of a caption (no image), the text denoiser regenerates the sentence. On 5K COCO test captions, BLEU-4 against the input is 62, and 95% of the regenerated sentences retrieve their source among 25K captions. One representation thus serves both to understand text and to generate it, as for images; a contrastive text encoder offers no route from text back to text.

CoRe has seen no text-only corpus: its language is learned together with vision, from 12M captions, and is not inherited from a language model. Scaling this joint training to more data, text-only data included, is the future work we are most interested in.

AuthorsOpenReview ↗

W2, W5, W6: VQA, scale, and what the pixel head is trained on

W2. VQA. We add VQA in the form that measures the representation, for two reasons.

  1. In the systems the reviewer refers to, the answer is written by a language model, which can hide the quantity we measure. [a] trains its own language model on about 520B text tokens, besides its image–text data, and then fine-tunes on Cambrian-7M. Cambrian-1 notes that language "can act as a shortcut, compensating for the deficiencies in learning effective visual representations".
  2. A pretrained LLM inside CoRe would remove the control the paper rests on. Every comparison in the paper is between models trained from scratch on the same 12M pairs, which the reviewer lists as a strength. With an LLM pretrained on orders of magnitude more text, a gain could no longer be attributed to the representation.

VQA by LLM probing (new). We therefore keep the LLM outside the model and frozen, a stricter form of the protocol that Cambrian-1 uses "as an evaluation protocol for various visual representations": frozen encoder → 2-layer MLP projector → frozen Qwen2.5-1.5B-Instruct, trained for one epoch on COCO captions and one on VQAv2. Only the projector is trained, identically for every encoder, so differences between rows reflect the visual features.

frozen image encoder (ViT-B)training dataVQAv2yes/nonumberother
none (question only)—32.9———
DINOv2 ViT-B/14142M images64.685.045.554.1
CLIP ViT-B/16 (SLIP release)CC12M, same images, original captions54.678.137.541.0
SLIP ViT-B/16CC12M, same images, original captions61.883.042.150.8
contrastive only (CoRe's encoders, alignment loss only)CC12M, same data64.985.145.354.4
CoReCC12M, same data65.685.746.055.4
  1. Same data, with and without generation: generation makes the encoder a better input for an LLM. CoRe is above the contrastive-only control on every answer type (65.6 vs. 64.9), and the order holds with a 7B frozen LLM (70.0 vs. 68.7). A frozen LLM is a strong readout: as with the attentive probe on ADE20K, it reaches what generation adds to the representation.
  2. Same images, other training objectives: CoRe is above both (SLIP, CLIP). SLIP and CLIP trained on CC12M reach 61.8 and 54.6.
  3. 12 times more data: CoRe is above DINOv2-B, the standard self-supervised encoder of its size. DINOv2-B reaches 64.6. It is the default encoder of RAE [b] (it is also a perceptual loss in CoRe's training, with 0.1% of the encoder's gradient; Appendix A).
  4. Only a shared representation allows this: the language model reads what CoRe generates, before any pixels. CoRe generates an image as a representation Z, so the projector and frozen LLM of the table, trained only on real images, read a generated Z as it is. For images generated from 1,000 COCO captions (CoRe + Gram/iBOT), the LLM's description of the generated Z reaches CIDEr 93 against the human captions of the source image; rendering the image and encoding it again gives 82 (Appendix E).
  5. With the language model trained as well (LoRA), CoRe remains the best input. CoRe reaches 70.0, against 69.8 for DINOv2-B and 69.3 for the same-data control. A trained LLM narrows the differences between encoders; the frozen protocol measures them directly.

W5. Scale. The scale is chosen so that every finding is a controlled comparison: all models are trained from scratch on identical data, which billion-pair runs do not allow. The findings are not tied to one operating point:

  • the understanding gain of the joint model holds at all checkpoints of the matched, equal-step comparison and on a second training seed, and the shared-vs-separate gap holds at 24K and 48K steps from scratch;
  • the 512² model keeps understanding and generation (zero-shot 51.5; GenEval .795 after image-denoiser-only fine-tuning).

Scaling this joint training is the direction we are most interested in, and this study gives it a clean, controlled reference.

W6. Clean or noisy latents: CoRe has no separate decoder, and the pixel head is trained at every noise level. The image denoiser is a transformer with two output heads on its features (Section 3): a representation head, which predicts the clean representation and drives sampling, and an observable head, which predicts pixels (a convolutional head of 1.4M parameters). Both heads read the denoiser's features at every noise level during training, and the generation loss is applied to the observable prediction. The pixel head is thus trained on the states it meets at sampling time. Please refer to Figure1 and Figure2, as well as Section 3.

AuthorsOpenReview ↗

W3: baselines

W3. Baselines with a VAE, and with an understanding encoder + RAE decoder ([a], [b]). Both baselines train a generator in a given, frozen space. Figure 1 (left) places them: a VAE latent is shaped by reconstruction (first row); RAE [b] and [a] use a representation shaped by understanding, then frozen (second row). CoRe addresses the orthogonal question of how such a space is learned: it is shaped by understanding and generation jointly (third row). We ran both baselines, on our data and with the released pretrained spaces.

  1. At equal data, CoRe is ahead of the RAE recipe. Same data, architecture and training steps as CoRe (Table 3; Appendix D.5):
same data, fully trainedGenEvalCOCO FID
the RAE order: shaped by understanding, then frozen; generator trained afterwards.40012.44
the same, with a separate decoder trained by RAE's recipe.38611.02
shaped jointly (CoRe).46110.10

The decoder is a ViT-B trained on the same data with RAE's losses and schedule (L1, LPIPS and adversarial; 20.5M image presentations, as in RAE). It improves the FID of the frozen space, not its prompt following; the jointly shaped representation is ahead on FID and on prompt following.

  1. The baselines asked for, "with controlled same data, compute, and model capacity" for the generator. In every row, the image denoiser (newly initialized), the text conditioning, the data (CC12M) and the schedule (16K steps at batch size 8192; about twice the presentations of Scale-RAE's text-to-image runs) are identical; only the generation space differs. Values are therefore below the fully trained model's (GenEval .461).
frozen generation spacespace trained onGenEvalDPGCOCO FID
pixels (no representation)—.20050.239.77
SD-VAE latentexternal data.31962.520.09
FLUX.1 VAE latentexternal data.27659.624.19
FLUX.2 VAE latentexternal data.41572.113.20
DINOv2-B + released RAE decoder142M images.45671.513.80
SigLIP 2 So400M + released RAE decoder ([a])~10B pairs.41274.313.45
contrastive only (shaped by understanding)CC12M.34367.814.31
contrastive + Gram/iBOTCC12M.32268.413.88
CoRe + Gram/iBOT (shaped by understanding and generation)CC12M.35569.611.68
  • At equal data, the jointly shaped space is the best generation space on all three metrics (bold), as in the fully trained pair of item 1.
  • CoRe's space has the best FID of all rows, including every pretrained space.
  • Better reconstruction does not by itself give better generation. In this setting, the SD-VAE and FLUX.1 latents reconstruct better than CoRe's representation (25.0 and 31.2 dB vs. 24.0 dB) and are behind on every generation metric. This agrees with [a] and [b]: [b] reports 8.0% ImageNet linear-probe accuracy for SD-VAE latents against 84.5% for DINOv2-B (its Table 1 (d)), and CoRe's Z, learned from scratch on 12M pairs, reaches 63.7% on its tokens and 73.5% pooled.
  • Pretrained spaces (DINOv2-B on 142M images, SigLIP 2 on about 10B pairs, the FLUX.2 VAE) are ahead in prompt following at this short schedule and behind in FID. They inherit their data, so these rows are not data-controlled. They show what a given space contributes, which is orthogonal to how a space is learned. Learning CoRe's space on data of that scale is the direction we are most interested in; the controlled setting comes first, because it is where the mechanism can be isolated.
  1. The two approaches compose, and CoRe needs no separate decoder. An encoder trained with CoRe can be used in the RAE recipe like any pretrained encoder: a separate ViT-B decoder trained on the frozen CoRe representation reaches GenEval .430 and FID 10.90. The denoiser's own pixel head is better (.457 and 10.02).

VAE-based systems trained on CC12M are compared in Table 3: DREAM-L and FLUID-L (SD-VAE latents, about 5.5B parameters) reach COCO FID 10.0 and 9.62, against 10.10 for CoRe with 383M parameters. The autoregressive text side of [a] is item 2 of our reply on W4.

In short. RAE and VAE-based generators study how to generate well in a given space; CoRe studies how the space itself should be learned, shaped by understanding and generation jointly (Figure 1, left). The baselines are therefore reference points for a different question. At equal data, the jointly shaped space is the best one to generate in, and it can be used inside the RAE recipe (item 3).

Reviewer_vsq2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CoRe, a model that jointly learns image and text contextual representations for three purposes: understanding, conditioning generation of the other modality, and serving as the space for generation of the same modality. It combines contrastive image–text alignment with continuous-flow generation and trains the model from scratch on CC12M. The authors evaluate image understanding, image-to-text generation, and text-to-image generation. CoRe performs similarly to a same-data contrastive baseline on global understanding and achieves competitive image-generation fidelity, while its prompt following and caption quality lag behind some pretrained systems. An ablation reports better compositional generation when generation uses the understanding representation rather than a separate projected space; a nonlinear readout also narrows the reported dense-prediction gap.

Critical strengths
  1. A focused representation-level experiment. The shared-versus-separate generation-space comparison finds better compositional generation with a shared representation, despite better reconstruction in the separate-space variant. This is the paper’s most consequential evidence because it addresses why sharing a representation might matter, beyond showing that both tasks can run in one model. The comparison does not fully isolate sharing from other architectural differences.

  2. A reasonably broad demonstration without pretrained inference components. CoRe supports understanding and generation in both directions while retaining global understanding performance close to a same-data contrastive control. This makes the proposed design more credible as a workable system, even if its individual components and overall conclusion are not especially novel.

Critical weaknesses
  1. Limited conceptual novelty. Prior work has already shown that understanding and generation can be combined. CoRe’s more specific contribution is generating in the contextual representations used for understanding, but the architecture largely assembles established alignment and generative components. This limits the significance of the paper’s broad conclusion, which does not appear substantially different from earlier findings.

  2. The key causal claim is not fully isolated. The shared-space variant improves compositional generation over a separate-space variant, but the two spaces also differ in representation rank and alignment. The comparison is a single, relatively short continuation rather than a full training run. Thus, the result is promising but does not firmly establish that sharing the understanding representation itself causes the gain. This matters because that claim is the paper’s strongest potential contribution.

Questions for authors
  1. What conclusion about jointly supporting understanding and generation does CoRe establish that prior unified models have not? Clarifying the contribution beyond this general finding would materially affect my assessment of novelty.

  2. In the shared-versus-separate generation-space comparison, how do the authors distinguish the effect of sharing the understanding representation from the effects of differences in representation rank or alignment? This distinction is central to assessing the paper’s main representation-level claim.

4 comments
AuthorsOpenReview ↗

W1 / Q1: what does CoRe establish that prior works have not?

Thank you for asking us to clarify this distinction. CoRe asks how a representation space should be learned, and answers with one that understanding and generation shape together, from scratch, and in which the generation trajectory itself runs (Figure 1). To our knowledge, it is the first such model, for images and for text. In prior work the space is learned for one purpose and then given: by reconstruction or by understanding, then frozen (e.g. RAE); DREAM trains understanding and generation jointly, but its generator predicts fixed VAE latents. Many unified models keep separate encoders or tokenizers for the two tasks (e.g. Janus, TokenFlow), since semantics and detail are reported to conflict in one visual representation. CoRe's view is that a good representation organizes and retains: to understand an observation is to organize what it semantically contains, to generate it is to retain what it contains. In a same-data, same-architecture comparison, a controlled reference for pretraining at larger scale, CoRe establishes:

  1. One representation can be learned by understanding and generation together. One 383M-parameter model with no pretrained component in the inference model reaches the zero-shot accuracy of CC12M encoders (50.6) and the COCO FID of generators with ~5.5B parameters (10.10).
  2. Sharing is not a compromise: understanding and generation help each other. Against the same representation shaped by understanding, then frozen, generation improves, text-to-image and image-to-text (GenEval .400 → .461, COCO FID 12.44 → 10.10; caption SPICE 10.1 → 13.5; Figure 1, right), and so does understanding (zero-shot 49.5 → 50.6; VQAv2 read by a frozen LLM 64.9 → 65.6).
  3. Why they are compatible: understanding organizes a representation, generation makes it retain the observation. Without the alignment loss, GenEval is .193: the space must be organized for text-to-image generation. Shaped by understanding alone, it reconstructs an image at 19.3 dB; shaped jointly, at 24.1 dB. Dense information is retained and arranged differently: a linear-probe deficit on ADE20K becomes a lead under an attentive probe (42.4 vs. 41.2).
  4. A second space is not needed. A separate generation space of the same encoder is worse when understanding does not shape it (GenEval .362 vs. .428) and the same when it does (reply on W2 / Q2).
  5. The tools of understanding act on generation. Because sampling runs the same representation space Z learned jointly by understanding and generation, probes trained on real images read a generation in progress and steer it toward a layout sketch (83% of pixels vs. 50%), and the model could rank and select its own samples (GenEval .436 → .501).
  6. Language and vision are symmetric, and generation is fast. Images and text are each generated by continuous diffusion in their own representation, so both could be distilled to a few steps (GenEval .792 with 9 denoiser evaluations; teacher .795 with 257).

"Prior work has already shown that understanding and generation can be combined." Combined, and even trained jointly, yes; but not at the level of the representation space: understanding and generation did not together shape one representation in which the generation trajectory also runs. Unified multimodal models share an architecture, a sequence model or a task interface on top of given spaces. CoRe is not presented as another unified model: it studies how the representation is learned. Figure 1 (left) orders them by what shapes the representation:

representation shaped bygeneration runs inthat space learned jointlytext is generated
latent diffusion (LDM)reconstructionautoencoder latentno—
RAEunderstandingfrozen pretrained featuresno—
UniTokcontrastive + reconstructiontokenizer codesnoautoregressively (LLM)
CoReunderstanding + generationthe representation jointly trained with understandingyes, from scratchby diffusion in the text representation

"The architecture largely assembles established alignment and generative components." The components are standard by design, so that the comparisons isolate the objectives; what is new is where generation runs, what shapes that space, and that this works. This has been tried: REPA-E reports that training a latent with the diffusion loss alone is ineffective. Two choices make it work in CoRe: the generation loss is applied to the observation (pixels and tokens); and the alignment loss organizes the same space (without it, GenEval is .193).

We thank the reviewer for naming the alternative explanations. Our answer in one sentence: the gain is caused by understanding shaping the space in which the generation trajectory lies, and sharing is the simplest design that provides this. Three new controls establish it.

The three designs are compared twice: as matched 16K-step continuations of one 24K-step checkpoint (same starting function, data, steps and understanding path; two training seeds), and as requested, we add results trained from scratch for 48K steps. Only the space in which images are generated differs.

The new evidence supports a more precise conclusion: generation benefits when understanding and generation jointly shape its space; sharing provides a simple, sufficient construction, while a separately aligned space offers no measured advantage.

generation runs inshaped by understandingGenEval, continuation (2 seeds)GenEval, from scratch (48k steps)PSNR (dB)
Z (CoRe)✓.427 / .428.41123.4
a second latent with its own alignment loss (new)✓.423 / .411.39723.2
a second latent without alignment loss✗.362 / .349.28425.0
  1. "The two spaces also differ in … alignment." Alignment is the cause, and the new variant isolates it.

    • Evidence: rows 2 and 3 differ only in the alignment loss on the second latent. With it, the generation gap closes (.423 and .411 against .427 and .428) and the reconstruction advantage disappears (23.2 against 25.0 dB).
    • What it means: a space is good for cross-modal (text-to-image) generation when understanding shapes it. This is the difference between Z and the separate latent of the submission.
  2. "… differ in representation rank." Rank is an effect of what shapes the latent, not a second cause.

    • Evidence: We measure rank in two complementary ways: participation ratio, which reflects how broadly variance is distributed across dimensions, and the number of principal components needed to explain 99% of the variance. (a) with alignment, the second representation becomes nearly a linear function of Z (R² up to .95); without it, it drifts away (at most .40). (b) Rank does not order the variants: the participation ratio ranks the separate latent higher (32.8 vs. 21.8), the number of components for 99% of the variance ranks Z higher (91 vs. 39). (c) The submission gave two clean tests, a rank-matched separate space or one with its own alignment loss; item 1 is the second, and it closes the gap.
    • What it means: the separate latent is the better autoencoder and the worse generation space. What it lacks is not information about the image but semantic organization.
  3. "A single, relatively short continuation rather than a full training run." The order holds on two training seeds and from scratch.

    • Evidence: the two-seed rows above. Trained from scratch with CoRe's full recipe, the unaligned two-head design is behind at 48K steps (GenEval .284 vs. .411); the aligned one is not ahead at 24K steps (.341 vs. .372) or at 48K steps (.397 vs. .411), and its second latent is again nearly a linear function of the first (R² .89 / .95).
    • What it means: a second space that understanding shapes converges to the first, and longer training does not change the order.
  4. "Does not firmly establish that sharing … itself causes the gain." The cause is that understanding shapes the generation space together with generation; one representation is the minimal design with this property.

    • Evidence: three spaces lack joint shaping, each in a different way, and each is behind CoRe: a latent that understanding does not shape (.362 vs. .428); a representation shaped by understanding and then frozen, with the generator trained afterwards, the order RAE follows (.400 vs. .461, and .386 with a separate decoder trained by RAE's recipe; Table 3, Appendix D.5); and generation losses without the alignment loss (.193).
    • What it means: understanding organizes the space, and generation makes it retain the observation. A space shaped by understanding and generation serves both tasks. "Shared or separate" is then not the real choice: a separate space that understanding shapes becomes the shared one, and one that it does not shape is worse. Section 5 now states the conclusion in this form:

A generation space benefits from being jointly shaped by understanding and generation. One contextual representation per modality provides this directly; introducing a second space does not improve generation in the tested settings, and largely recovers the shared model’s performance only when that second space also receives semantic alignment.

Reviewer_vsq2OpenReview ↗

Post-Rebuttal Assessment: Update to Weak Acceptance

Thank you for the detailed clarification and additional controlled experiments. The aligned separate-space baseline, two continuation seeds, and from-scratch results substantially address my main concern about the original shared-versus-separate comparison. I particularly appreciate the revised conclusion: the evidence supports jointly shaping the generation space with understanding and generation, rather than establishing that sharing itself is uniquely beneficial.

The clarification also makes the representation-level contribution more concrete than the general claim that understanding and generation can be unified. I remain cautious about the stronger mechanistic statements: the rank analyses do not rule out rank-related effects, and high linear predictability does not establish equivalence between the spaces. Please keep these interpretations appropriately qualified in the revision.

Overall, the new evidence and more precise framing alleviate my principal concerns. I am updating my rating from 2 (Weak rejection) to 3 (Weak acceptance).

Thank you for the careful reassessment and for raising your rating. We have revised the interpretation accordingly:

  1. Rank. The rank measurements are reported as observations. Appendix D.3 now states that neither control varies rank alone explicitly.
  2. Linear predictability. With its own alignment loss, the second space is largely linearly predictable from the understanding representation (R2R^2 .89 / .95). Appendix D.3 now states that this does not by itself make the two spaces equivalent, and we removed the wording that the second space "converges to" the shared one. What remains is what was measured: such a space generates nearly as well as the shared one and never better.
  3. Conclusion. Section 5 now emphasizes that jointly shaping the generation space with understanding and generation is beneficial in our controlled setting, and that one shared representation suffices. We do not claim that identical spaces are uniquely beneficial or that the geometric mechanism has been isolated.

Thank you again for helping us sharpen both the representation-level contribution and the scope of the evidence.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (to all reviewers and the area chair)

We thank the reviewers for their thoughtful feedback. The revision sharpens CoRe’s representation-level contribution and strengthens the controlled evidence supporting it.

The question. Can understanding and generation jointly shape a representation space that supports both? CoRe learns one contextual representation per modality from scratch, shaped by understanding and generation together: understanding reads it, it conditions generation of the other modality, and generation trajectory itself unfolds in that same representation space. To our knowledge, CoRe is the first model to jointly learn this construction both images and text. CoRe is trained from scratch and uses no pretrained model at inference; every finding below is measured against controls with the same data, architecture and training steps, which gives a clean reference for pretraining at larger scale.

Main results.

  1. One contextual representation space can be learned by both understanding and generation objectives, from scratch, and serve both tasks. One 383M-parameter model reaches ImageNet zero-shot 50.6, level with CC12M encoders (49.7 to 51.4), and COCO-30K FID 10.10, level with DREAM-L and FLUID-L (10.0 and 9.62; ~5.5B parameters).

  2. Understanding and generation help each other. Against the same encoder shaped by understanding and then frozen (Figure 1, right):

    • Generation improves, text-to-image and image-to-text: GenEval .400 → .461, COCO FID 12.44 → 10.10, caption SPICE 10.1 → 13.5.
    • Understanding improves as well: ImageNet zero-shot 49.5 → 50.6; ADE20K with an attentive probe 41.2 → 42.4 mIoU; VQAv2 read by a frozen LLM 64.9 → 65.6.
  3. Understanding organizes the representation, generation makes it retain the observation, and the jointly learned representation space has both properties.

    • Organization. The image–text alignment loss arranges the representation by meaning. Text-to-image generation depends on it: without this loss, prompt following collapses (GenEval .193 vs. .461).
    • Retention. Shaped by understanding alone, the representation reconstructs an image at 19.3 dB; shaped jointly, at 24.1 dB.
    • They are different axes. A linear probe reads what is linearly arranged; a stronger probe reads what is there. On ADE20K, CoRe trails the understanding-only encoder under a linear probe (24.7 vs. 30.2 mIoU) and leads under an attentive probe: generation adds information that understanding alone does not keep.
  4. A second generation space is not needed (Vsq2, 4W9K). Generating in a separate space of the same encoder is worse when understanding does not shape that space (GenEval .362 vs. .428), and no better when it does (.423 / .411 vs. .427 / .428, two seeds).

  5. Generation runs in the representation that understanding reads, so the tools of understanding work on generation. A frozen probe reads the class of a generation after one denoiser call (54% correct), a frozen segmentation probe steers sampling toward a layout sketch (83% of pixels vs. 50%), and the model ranks its own samples (GenEval .436 → .501).

  6. Language and vision are symmetric, and generation could be accelerated. Each modality has its own contextual representation and is generated in it by continuous flow, so images and text both distill to a few steps: GenEval .792 with 9 denoiser evaluations (teacher: .795 with 257) and caption SPICE 13.7 with 9 (teacher: 13.5 with 128).

New experiments, by concern.

concernnew evidence
Sharing, or aligning the generation space? (Vsq2, 4W9K)an aligned second space, on two training seeds and from scratch: no better than sharing
Dense prediction (4W9K)linear, MLP and attentive probes, three probe seeds; a second training seed
VQA and text understanding (1NEu)a frozen LLM reads CoRe's encoder better than DINOv2-B (VQAv2 65.6 vs. 64.6); the text encoder matches CLIP's with 33× fewer pairs
Baselines and ablations (1NEu)every requested baseline and component ablation, in the controlled setup; among generation spaces learned from the same data, CoRe's generates best

Revision. Additions are in purple: a first-page figure, new controls and probes in Section 5, and the requested baselines and ablations in the appendix.

AuthorsOpenReview ↗

Final Summary of Discussion and Revisions

We thank the Area Chair and Reviewers vsq2, 1NEu, and 4W9K—every question sharpened our controlled evidence or claims.

1. Core insight and contributions. Prior work combines understanding and generation—even jointly—yet treats the generation space as given: fixed by reconstruction (LDM/DREAM), frozen after understanding (RAE), or split across encoders (Janus/TokenFlow) assuming semantics and detail conflict. CoRe instead learns the space itself, establishing two results (Fig. 1, Sec. 3–5):

  • First representation space jointly shaped by understanding and generation—for images and text: To our knowledge, CoRe is the first model to learn one contextual representation Z per modality from scratch that understanding reads, cross-modal generation conditions on, and in which the generation trajectory unfolds—generating images in a jointly learned patch space (not VAEs or frozen features) and text in its contextual token space (not lookup or separate latents). Where diffusion-only latent training was reported ineffective (REPA-E), CoRe succeeds in one run: alignment organizes Z semantically, while flow matching noises Z and predicts pixels/tokens via the denoiser’s head—with no separately trained decoder (1NEu, W6)—so Z retains detail (19.3 → 24.1 dB). Because generation runs inside Z, frozen probes steer layouts without training (83% vs. 50% pixels), CoRe self-ranks samples (.436 → .501), and symmetric flows distill both to 9 evaluations (.797 vs. .806 GenEval; 13.7 vs. 13.5 SPICE; 1NEu, Q7).
  • Understanding and generation improve each other in a clean setup: Trained from scratch on open CC12M in one 383M model (no pretrained model at inference), matching data, architecture, and steps:
    • Generation improves understanding (vs. understanding-only encoder): ImageNet zero-shot 49.5 → 50.6 (matching CC12M encoders), ADE20K attentive probe 41.2 → 42.4 mIoU, and frozen-LLM VQAv2 64.9 → 65.6 (above DINOv2-B’s 64.6);
    • Understanding and joint shaping improve text-to-image: alignment lifts GenEval .193 → .461, and joint shaping beats freezing the understanding encoder (GenEval .400 → .461; COCO FID 12.44 → 10.10, matching ~5.5B DREAM-L / FLUID-L: 10.0 / 9.62);
    • Understanding and joint shaping improve image-to-text: joint shaping beats that frozen encoder on SPICE (10.1 → 13.5; 20.0 COCO-tuned), and alignment lifts TrueMatch (49.0 → 53.3, above LLaVA-NeXT-34B).

2. Discussion summary. Reviewers valued the clean from-scratch setup (1NEu, vsq2, 4W9K), transparent accounting (4W9K), representation-level focus (vsq2), and modal symmetry (1NEu):

  • Reviewer vsq2 stated an update from 2 to 3 (Weak acceptance): new controls "substantially address my main concern" and make "the representation-level contribution more concrete," with cautions on rank and predictability incorporated.
  • Reviewer 4W9K (3, Weak acceptance) recognized "a concrete, effective realization of the shared-representation hypothesis" whose accounting "makes the conclusions substantially more credible," guiding our aligned-space and probe controls.
  • Reviewer 1NEu wrote our response "resolves many of my concerns, and I am willing to improve my rating," citing "the additional results and reproducibility"; we followed up on W6 and design simplicity.

3. Key resolutions (revisions in purple).

  • Sharing vs. alignment (vsq2, 4W9K; Sec. 5, App. D.3): On two seeds and from scratch (48K), an unaligned second space reconstructs better (25.0 vs. 23.4 dB) yet generates worse (.362 vs. .428 GenEval; .284 vs. .411); aligning that space closes the gap (.423 / .411; .397 vs. .411) without beating CoRe: generation benefits when understanding and generation jointly shape its space, and one space suffices.
  • Dense retention (4W9K; Sec. 5, App. B, E.1): On full-resolution ADE20K (3 probe seeds), CoRe trails understanding-only under a linear probe (24.7 vs. 30.2 mIoU), but reverses the gap on both models’ best readout: under an attentive probe, CoRe (joint understanding + generation) beats understanding-only (42.4 vs. 41.2, +1.18 mIoU; CoRe 43.5 vs. 43.4 with Gram/iBOT)—a linear probe reads what is linearly arranged; a stronger probe reads what is there.
  • Baselines & text probes (1NEu; Table 3, App. D–E): Frozen text features match CLIP on SICK-R at 33× fewer pairs (69.0 vs. 69.1), resolve polysemy (98.8%), and equal FLAN-T5-Large (.355 vs. .346); across nine spaces (SD/FLUX VAEs, DINOv2/SigLIP2 + RAE), CoRe’s space has the best FID (11.68) and beats same-data RAE (.461 vs. .386).

4. Scope and future impact. Deliberately controlled at 383M parameters and CC12M, CoRe lays a foundation for scaling unified pretraining without frozen codecs or dual encoders, multimodal and text-only pretraining on LLM benchmarks via continuous text flows (as 1NEu suggested), and latent trajectory steering and verification before rendering pixels or tokens. Thank you again.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.