- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
CoRe learns image and text encoders jointly with two continuous denoisers. Each modality's contextual representation supports contrastive understanding, conditions generation of the other modality, and supplies the latent space for generating its own modality. Pixel and token losses anchor the moving representations, while a detached representation-regression target supports sampling. The 383M-parameter system is initialized from scratch and trained on recaptioned CC12M, with pretrained VGG16/DINOv2 used for training supervision. It preserves global understanding relative to same-data, same-step contrastive controls, supports both generation directions, and allows few-step distillation. The paper further studies whether sharing the generation space improves compositional generation and whether the loss in linear dense-prediction performance reflects reduced accessibility rather than wholesale loss of information.
Critical strengths
-
A concrete, effective realization of the shared-representation hypothesis. Equations (1)-(7) place understanding and both generation directions in jointly learned contextual spaces, with a clear observable-space anchor for moving targets. At the reported scale, CoRe achieves ImageNet zero-shot accuracy of 50.63 versus 49.51 for the contrastive control and retrieval mR of 70.34 versus 69.75, while producing images with COCO-30K FID 10.10 and captions with SPICE 13.53 (Tables 1-3). These establish a useful feasibility result even though the model does not lead the stronger prompt-following or caption-fidelity baselines.
-
The experimental accounting makes the conclusions substantially more credible. The paper includes frozen-contrastive and no-alignment controls, matched Gram/iBOT controls, a shared-versus-separate continuation, a readout ladder, and distillation comparisons. Appendices A-C report training exposure, compute, inherited supervision, and sampler selection; Appendix D.4 reports benchmark-template overlap in the fine-tuning mixture. Figures 6-7 show unsuccessful samples and verbatim caption defects. This transparency is particularly valuable for a unified model whose capabilities can otherwise be conflated across checkpoints or evaluation recipes.
Critical weaknesses
-
The shared-space experiment does not yet isolate the benefit of using exactly the understanding representation from the benefit of directly aligning the generation target. In Section 5, Table 4, and Appendix D.3 (pp. 27-28, Table 15), the separate arm generates through a second projection that does not receive its own alignment loss, whereas the shared target is directly shaped by alignment. The difference of .066 GenEval and 5.39 DPG is convincing for this particular intervention, including the reported prompt bootstrap and three sampling seeds, but these checks do not distinguish semantic regularization of the target from a benefit that requires identical spaces. The study also uses one training seed and a 16K-step continuation from an already jointly trained checkpoint, rather than a complete independent training comparison. The authors acknowledge both limitations on p. 28; nevertheless, the abstract and conclusion give the result greater generality than this control warrants. A separate projection with its own matched alignment/readout objective, repeated across training seeds, would address the main alternative explanation. Alternatively, scope the claim explicitly to this unconstrained separate-projection continuation rather than implying a general advantage over separate semantically organized spaces.
-
The evidence for recovering the dense-prediction deficit is narrower than the main-task deficit it is used to explain. The primary full-resolution ADE20K evaluation shows a 5.62 mIoU loss on the contextual representation, or 7.94 with matched Gram/iBOT regularization (Tables 1 and 17). The near-recovery result instead uses a twofold-downsampled package, only 8K training images, and one probe run per cell (Appendices B.1/H.1, Figure 3, Table 28). Within this diagnostic the MLP comparison is useful evidence of recoverable information, but the contrastive MLP itself scores below its best linear readout: 27.21 versus 27.91, and 29.76 versus 31.26 with Gram/iBOT. Comparing the best reported readout of each model therefore leaves gaps of 1.69 and 2.75 mIoU, rather than the highlighted .99 and 1.25. Most of the original diagnostic gap is still recovered; this does not invalidate that observation. It does limit extrapolation to the full-resolution task. Repeat the flexible-readout comparison under the primary ADE20K protocol, report probe variability, and specify the dense MLP architecture and optimizer settings precisely. Until then, frame the retention conclusion as evidence from the reduced-resolution diagnostic.
Questions for authors
Assessment-changing clarifications:
- Is there evidence from an independently aligned separate generation projection, or a longer/multiple-training-seed version of Appendix D.3, that separates the benefit of identical spaces from the benefit of aligning the generation target? If unavailable, can the claim be narrowed to the intervention actually tested? Additional experiments would strengthen the interpretation, but are not presumed to already exist.
- What are the exact architecture, learning rate, regularization, and selection protocol for the dense MLP probe in Table 28? Are full-resolution flexible-probe results available, and does the near-recovery conclusion hold when each representation is compared using its best validated readout? If such results are unavailable, the manuscript should state that the full-resolution deficit remains unexplained by a matched flexible-probe experiment.
Future-work suggestion, not a prerequisite for this assessment: A FLOP-matched contrastive control would help determine whether joint training offers an understanding-quality advantage per unit compute. The present same-step controls already answer the narrower coexistence question, and the paper explicitly discloses their compute mismatch.
4 comments
W2, Q2 and the compute suggestion: dense prediction at full resolution
Thank you for specifying the protocol precisely, and we followed it. Full-resolution flexible-probe results are now available, with the data, resolution and iterations of the primary ADE20K protocol and three probe seeds (settings in item 2). On Core's joint understanding and generation representation space Z, CoRe's linear-probe deficit to the contrastive-only control shrinks under an MLP probe, and under the attentive probe CoRe leads (+1.18 mIoU). The paper now makes its dense-prediction statement under the full-resolution protocol.
- "Repeat the flexible-readout comparison under the primary ADE20K protocol." On Z the gap closes with probe capacity and reverses at the best probe. ADE20K mIoU, frozen Z, learning rate 1e-3; CI: paired bootstrap over validation images:
| probe | contrastive only | CoRe | difference [95% CI] | contrastive + Gram/iBOT | CoRe + Gram/iBOT |
|---|---|---|---|---|---|
| linear | 30.15 | 24.69 | −5.45 [−5.77, −4.75] | 33.13 | 25.45 |
| MLP, mean of 3 probe seeds | 35.61 | 34.12 | −1.49 [−1.87, −0.92] | 38.30 | 35.61 |
| attentive, mean of 3 probe seeds | 41.20 | 42.38 | +1.18 [+0.53, +1.89] | 43.36 | 43.54 |
A linear probe reads what is linearly arranged, while a stronger probe reads what is there. The reversal shows that generation adds information to Z that understanding alone does not keep. Revised PDF: Section 5 (Figure 4); Appendix E.1.
-
"Report probe variability, and specify the dense MLP … precisely." Probe seeds vary little. MLP seeds differ by at most 0.35 mIoU; attentive seeds have an s.d. of 0.08 (contrastive only) and 0.47 (CoRe). Every probe starts from frozen features with Dropout2d(0.1) and BatchNorm. The linear probe adds a 1×1 conv (as in DINOv3), the MLP 1×1 convs C → 2048 → 150 with GELU, the attentive probe two pre-norm transformer blocks of width 512. All use AdamW, learning rate 1e-3, weight decay 1e-3 and 40K iterations, and report the last iteration, with no selection on validation images. Revised PDF: Appendix B, probe settings.
-
"When each representation is compared using its best validated readout." The attentive probe is the strongest of the three probes for both models, and there CoRe leads by +1.18. Learning rate 1e-3 gives every model a higher linear-probe score than the submission's 1e-4, with the same order; the revised paper reports 1e-3 throughout. For the reduced-resolution diagnostic, Appendix H.1 now states the best-vs-best gaps of 1.69 and 2.75.
-
"7.94 with matched Gram/iBOT regularization." With its best probe, the matched Gram/iBOT pair is level. With the linear probe, CoRe + Gram/iBOT is behind its matched control. With the attentive probe, the best probe of both models, they reach 43.5 and 43.4 mIoU (three probe seeds each).
-
Why the linear probe drops at Z, and why this is not lost information. The gap is specific to Z: on the encoder features h it is under one point with the linear probe and absent with the MLP and attentive probes (Appendix E.1). It comes from the generation gradient through the noised representation. Stopping this gradient raises the linear probe (15.5 → 22.3, reduced resolution) but lowers GenEval (.427 → .404) and reconstruction (22.2 → 19.5 dB, 256 images). The information is present in Z: the attentive probe reads it, and CoRe leads there (item 1).
-
Compute (the compute-matched suggestion). Our controls match training steps and data exposure. The paper's claim is that the understanding and generation coexist at equal steps, which, as the reviewer notes, the same-step controls answer. A CoRe step costs about 2.5× a contrastive-only step (Appendix A); the additional compute trains the two generators. At equal steps, CoRe is ahead in zero-shot accuracy at all six checkpoints (+0.6 to +1.3 points). It is also ahead in few-shot classification on the pooled representation g (+1.9 / +1.7 / +1.5 at 1 / 5 / 10 shots) and as input to a frozen LLM (VQAv2 65.6 vs. 64.9; Appendix E.10). A second training seed of both models gives the same order (zero-shot 37.8 vs. 36.3 at 24K steps), and on the long descriptions of DOCCI, image-to-text R@1 is 83.4 vs. 81.6 (+1.8, 95% CI [+0.7, +2.8]). Joint training therefore does not cost global understanding.
Thank you for your rebuttal. I will keep my score.
Thank you for your positive assessment. We especially appreciate your recognition of the shared-representation contribution and the transparency of our experimental setup. We are grateful for your careful reading and constructive feedback, which have helped strengthen the paper.
W1 / Q1
We thank the reviewer for the careful reading and for recognizing CoRe as “a concrete, effective realization of the shared-representation hypothesis.” We have added the proposed aligned separate-space control, repeated the continuation across training seeds, and conducted independent training from scratch.
Short answer. We ran the control you propose: a second projection with its own alignment loss, matched to CoRe's, on two training seeds and from scratch. It closes the gap and is never better than CoRe. The benefit therefore does not require identical spaces as such: it requires that understanding shape the space in which generation runs, together with generation. One representation is the simplest design with this property, and a second space that has it converges to the first.
Matched 16K-step continuations of one 24K-step checkpoint (shorter than the 125K-step model, hence below its .461); Δ is against the CoRe run of the same seed and node shape.
"Does not yet isolate … identical spaces from … aligning the generation target." The new control isolates it: what matters is that understanding and generation shape the representation space jointly and that the generation trajectory runs in it; a second space with this property is no better.
"One training seed and a 16K-step continuation … rather than a complete independent training comparison." The continuation now has two training seeds, and the comparison is repeated from scratch.
"The abstract and conclusion give the result greater generality than this control warrants." Both now state exactly what the controls show.
"A general advantage over separate semantically organized spaces." Both kinds of semantically organized separate space are measured, and neither is better than one jointly shaped representation.
The resulting claim is that understanding and generation can jointly learn one contextual representation that serves both. The controls support complementary roles for semantic organization and recoverable detail, and show that a second generation space is unnecessary in the tested setting. This is the representation-level contribution of CoRe: learning the space through both objectives, and generating within that same learned space.