- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper extends CoBit to multimodal generation through binary representations and partial denoising. Separate models are trained for MNIST-Sum, speech-text, and protein sequence-structure modeling, with each checkpoint supporting joint and conditional generation. The experiments demonstrate that the same bitstream diffusion protocol can capture different forms of cross-modal dependence.
Critical strengths
-
A reusable generative interface across modalities is a meaningful research direction. Supporting multiple generation directions with one checkpoint per setting demonstrates useful flexibility.
-
The speech and protein experiments provide evidence of cross-modal consistency, supported by sampler analyses, codec reconstruction checks, and direct refolding of generated protein sequences.
Critical weaknesses
-
The additional empirical contribution beyond CoBit is not sufficiently established. The authors acknowledge inheriting the core method, and cross-domain reuse is useful. However, matched comparisons that isolate the binary interface are missing, leaving its quality or computational benefits over existing shared-token approaches unclear. The sampler sweeps do not resolve this question. Consequently, the results demonstrate feasibility more convincingly than comparative value.
-
The image-text evaluation is limited to four MNIST digits paired with a templated equation. Although appropriately described as a controlled proof of concept, this narrow task provides limited evidence about realistic visual-semantic relationships. Without a substantive image-text baseline or a richer benchmark, near-perfect accuracy adds little evidence about the framework’s comparative value in this domain.
Questions for authors
Under matched codecs, data, backbone capacity, task mixtures, and training budgets, what quality or computational benefit does the binary interface offer over categorical/discrete diffusion? Controlled evidence on a substantive domain would materially affect my assessment.
2 comments
Response to Reviewer CjUM (Cont.)
W2. Image–text evaluation
The image-text evaluation is limited to four MNIST digits paired with a templated equation. Although appropriately described as a controlled proof of concept, this narrow task provides limited evidence about realistic visual-semantic relationships. Without a substantive image-text baseline or a richer benchmark, near-perfect accuracy adds little evidence about the framework's comparative value in this domain.
Response. We have substantially expanded the image–text experiments beyond MNIST-Sum. On zero-shot MS-COCO, CoBit-CC3M and CoBit-CC12M achieve FID-30K of 18.48 and 16.59, respectively, using only 20 and 32 denoiser evaluations (Section 4.2, Table 2). These results use 462M- and 633M-parameter denoisers trained from scratch on 2.9M and 11.7M image–caption pairs, without a pretrained language model or semantic text encoder. Images are represented using a frozen pretrained Open-MAGVIT2 codec. (Sampling settings were selected through exploratory evaluation-set sweeps; Appendix B.1 gives the protocol.)
| Model | Pairs | Parameters | Steps / NFE | FID-30K |
|---|---|---|---|---|
| CoBit-CC3M | 2.9M | 462M | 10 / 20 | 18.48 |
| CoBit-CC12M | 11.7M | 633M | 16 / 32 | 16.59 |
For context, LAFITE reports zero-shot COCO FID of 26.94 after CC3M training, compared with 18.48 for CoBit-CC3M. This is a published comparison rather than a matched experiment; the model and evaluation differences are discussed in Appendix B.8, Table 12.
The same checkpoints also perform image captioning and unconditional image–text co-generation through the shared bitwise head and objective. The image results therefore show that the common formulation can support natural-image generation at a practical sampling budget, while retaining both the reverse and joint generation directions. The evaluation covers 30,000 generated images and includes captioning on the 5,000-image Karpathy test split, GenEval, codec reconstruction checks and uncurated samples (Appendix B). These natural-image experiments now provide the main evidence for the image–text setting; MNIST-Sum remains a control with exactly checkable outputs.
Q. Comparison with discrete diffusion
Under matched codecs, data, backbone capacity, task mixtures, and training budgets, what quality or computational benefit does the binary interface offer over categorical/discrete diffusion? Controlled evidence on a substantive domain would materially affect my assessment.
Response.
To address your concerns, we trained a small 50.38M-parameter CoBit and bit-MDLM controls from scratch. Both use the same complete bidirectional denoiser, initial weights and self-conditioning policy, trained on co-design only, without sequence replay. The frozen structure tokenizer ensures a shared clean 18-bit representation grouped into one transformer position per residue. CoBit receives Gaussian-corrupted bits whereas bit-MDLM receives masked bits.
Both models completed 200k updates on the same 216,237 paired proteins, with 10k warmup, batch 256 under the same training seed. We get the 200k EMA checkpoint on the same 4,098 held-out pairs. We evaluated 1,500 original co-designed pairs per model, 300 at each length 100/200/300/400/500, with 127 actual denoiser evaluations including self-conditioning with direct ESMFold refolding (no ProteinMPNN redesign)
| Metric | CoBit | bit-MDLM |
|---|---|---|
| Valid amino-acid sequences | 100.0% | 87.6% |
| RMSD Å | 26.60% | 20.53% |
| scTM | 48.60% | 22.60% |
| Both criteria (strict) | 21.93% | 14.33% |
| Distinct sequences among strict successes | 210 | 179 |
Recall that strict success requires both all-C RMSD Å and scTM . CoBit gains 7.6 percentage points in aggregate strict success (53% relative). The paired sampling 95% CI is +5.3–9.9 points (it measures sampling uncertainty, not training-seed variation). Exact-sequence deduplication gives 17.3% more distinct successful sequences and most gains concentrate at 200–300 residues. The gain comes at similar measured sampling cost (see Reviewer 6THc, W3, for timings and training costs). The separate submitted-model temperature experiment is reported in Reviewer 6THc, W2.
Response to Reviewer CjUM
We thank the reviewer for the comments. We have added natural-image experiments to address the limited scope of the original image–text evaluation. All section and table references below refer to the revised manuscript.
W1. Contribution beyond CoBit
Response. The diffusion objective and entropy-derived schedule are inherited from CoBit. The contribution here is their extension to joint and conditional multimodal generation, and the evidence that the same formulation works across three quite different relationships. Speech contains acoustic and speaker information absent from its transcript; protein sequence and structure are aligned residue by residue; natural images and captions have a much looser correspondence. In each setting, one checkpoint learns the joint distribution and the conditional directions included in training, using a shared input projection, transformer and bitwise head.
The binary representation lets very large vocabularies share a compact input and output interface. This architectural advantage becomes particularly useful as more modalities are added, each bringing its own vocabulary. It provides a natural basis for extending CoBit toward models that combine text, images, audio and video, including future multimodal world models. The codecs and layouts remain domain-specific, while the denoiser uses the same prediction head and objective across modalities.
The speech checkpoint jointly generates speech and text with 3.5% cross-modal WER, while direct refolding of the protein model's generated sequences gives mean scTM 0.911. The new natural-image experiments reach zero-shot COCO FID-30K of 18.48 and 16.59, with the same checkpoints also supporting captioning and joint generation. These results demonstrate the applicability of this interface across domains, but do not by themselves establish a generation-quality or end-to-end computational advantage over categorical diffusion; we address the controlled comparison below.