acceptodds
Under review as a conference paper at ICLR 2027

Multimodal Co-Generation with Continuous Bitstream Diffusion

Abstract

We show that continuous bitstream diffusion (CoBit), originally developed for language modeling, provides a unified framework for joint and conditional multimodal generation using the same training and sampling protocol. Across image–text, speech–text and protein sequence–structure modeling, models are trained separately from scratch, but each uses a single input projection, transformer backbone, bitwise prediction head and binary score-matching objective shared across its modalities. Within each setting, a single checkpoint supports joint generation and the conditional directions included in training by changing which parts of the bitstream are observed. We first validate the approach on MNIST-Sum, a controlled image–text task pairing raw binary images with equations encoded as text, achieving near-perfect conditional accuracy and high joint consistency without learned tokenizers. We then apply the same protocol at larger scale to speech–text and protein sequence–structure modeling. Trained without text pretraining, a single speech–text model performs speech synthesis, recognition, continuation, and unconditional co-generation. Jointly generated speech and text achieve 3.5% word error rate between the generated text and automatic transcriptions of the generated speech. In protein modeling, a single CoBit checkpoint supports forward folding, inverse folding, and joint sequence–structure generation. For jointly generated proteins spanning 100–500 residues, computational refolding of CoBit's sequences yields structures that agree with the generated backbones at a mean self-consistency TM-score of 0.911. Together, these results show that CoBit provides a common generative framework for substantially different modalities and forms of cross-modal dependence, without introducing a domain-specific objective or sampling procedure.

Primary area:
generative models
Keywords:
continuous diffusion, bitstream diffusion, multimodal generation, joint generation, binary representations, entropy-based noise schedule, speech-text generation, protein sequence-structure generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_CjUMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper extends CoBit to multimodal generation through binary representations and partial denoising. Separate models are trained for MNIST-Sum, speech-text, and protein sequence-structure modeling, with each checkpoint supporting joint and conditional generation. The experiments demonstrate that the same bitstream diffusion protocol can capture different forms of cross-modal dependence.

Critical strengths
  1. A reusable generative interface across modalities is a meaningful research direction. Supporting multiple generation directions with one checkpoint per setting demonstrates useful flexibility.

  2. The speech and protein experiments provide evidence of cross-modal consistency, supported by sampler analyses, codec reconstruction checks, and direct refolding of generated protein sequences.

Critical weaknesses
  1. The additional empirical contribution beyond CoBit is not sufficiently established. The authors acknowledge inheriting the core method, and cross-domain reuse is useful. However, matched comparisons that isolate the binary interface are missing, leaving its quality or computational benefits over existing shared-token approaches unclear. The sampler sweeps do not resolve this question. Consequently, the results demonstrate feasibility more convincingly than comparative value.

  2. The image-text evaluation is limited to four MNIST digits paired with a templated equation. Although appropriately described as a controlled proof of concept, this narrow task provides limited evidence about realistic visual-semantic relationships. Without a substantive image-text baseline or a richer benchmark, near-perfect accuracy adds little evidence about the framework’s comparative value in this domain.

Questions for authors

Under matched codecs, data, backbone capacity, task mixtures, and training budgets, what quality or computational benefit does the binary interface offer over categorical/discrete diffusion? Controlled evidence on a substantive domain would materially affect my assessment.

2 comments
AuthorsOpenReview ↗

Response to Reviewer CjUM

We thank the reviewer for the comments. We have added natural-image experiments to address the limited scope of the original image–text evaluation. All section and table references below refer to the revised manuscript.

W1. Contribution beyond CoBit

The additional empirical contribution beyond CoBit is not sufficiently established. The authors acknowledge inheriting the core method, and cross-domain reuse is useful. However, matched comparisons that isolate the binary interface are missing, leaving its quality or computational benefits over existing shared-token approaches unclear. The sampler sweeps do not resolve this question. Consequently, the results demonstrate feasibility more convincingly than comparative value.

Response. The diffusion objective and entropy-derived schedule are inherited from CoBit. The contribution here is their extension to joint and conditional multimodal generation, and the evidence that the same formulation works across three quite different relationships. Speech contains acoustic and speaker information absent from its transcript; protein sequence and structure are aligned residue by residue; natural images and captions have a much looser correspondence. In each setting, one checkpoint learns the joint distribution and the conditional directions included in training, using a shared input projection, transformer and bitwise head.

The binary representation lets very large vocabularies share a compact input and output interface. This architectural advantage becomes particularly useful as more modalities are added, each bringing its own vocabulary. It provides a natural basis for extending CoBit toward models that combine text, images, audio and video, including future multimodal world models. The codecs and layouts remain domain-specific, while the denoiser uses the same prediction head and objective across modalities.

The speech checkpoint jointly generates speech and text with 3.5% cross-modal WER, while direct refolding of the protein model's generated sequences gives mean scTM 0.911. The new natural-image experiments reach zero-shot COCO FID-30K of 18.48 and 16.59, with the same checkpoints also supporting captioning and joint generation. These results demonstrate the applicability of this interface across domains, but do not by themselves establish a generation-quality or end-to-end computational advantage over categorical diffusion; we address the controlled comparison below.

AuthorsOpenReview ↗

Response to Reviewer CjUM (Cont.)

W2. Image–text evaluation

The image-text evaluation is limited to four MNIST digits paired with a templated equation. Although appropriately described as a controlled proof of concept, this narrow task provides limited evidence about realistic visual-semantic relationships. Without a substantive image-text baseline or a richer benchmark, near-perfect accuracy adds little evidence about the framework's comparative value in this domain.

Response. We have substantially expanded the image–text experiments beyond MNIST-Sum. On zero-shot MS-COCO, CoBit-CC3M and CoBit-CC12M achieve FID-30K of 18.48 and 16.59, respectively, using only 20 and 32 denoiser evaluations (Section 4.2, Table 2). These results use 462M- and 633M-parameter denoisers trained from scratch on 2.9M and 11.7M image–caption pairs, without a pretrained language model or semantic text encoder. Images are represented using a frozen pretrained Open-MAGVIT2 codec. (Sampling settings were selected through exploratory evaluation-set sweeps; Appendix B.1 gives the protocol.)

ModelPairsParametersSteps / NFEFID-30K ↓\downarrow
CoBit-CC3M2.9M462M10 / 2018.48
CoBit-CC12M11.7M633M16 / 3216.59

For context, LAFITE reports zero-shot COCO FID of 26.94 after CC3M training, compared with 18.48 for CoBit-CC3M. This is a published comparison rather than a matched experiment; the model and evaluation differences are discussed in Appendix B.8, Table 12.

The same checkpoints also perform image captioning and unconditional image–text co-generation through the shared bitwise head and objective. The image results therefore show that the common formulation can support natural-image generation at a practical sampling budget, while retaining both the reverse and joint generation directions. The evaluation covers 30,000 generated images and includes captioning on the 5,000-image Karpathy test split, GenEval, codec reconstruction checks and uncurated samples (Appendix B). These natural-image experiments now provide the main evidence for the image–text setting; MNIST-Sum remains a control with exactly checkable outputs.

Q. Comparison with discrete diffusion

Under matched codecs, data, backbone capacity, task mixtures, and training budgets, what quality or computational benefit does the binary interface offer over categorical/discrete diffusion? Controlled evidence on a substantive domain would materially affect my assessment.

Response.

To address your concerns, we trained a small 50.38M-parameter CoBit and bit-MDLM controls from scratch. Both use the same complete bidirectional denoiser, initial weights and self-conditioning policy, trained on co-design only, without sequence replay. The frozen structure tokenizer ensures a shared clean 18-bit representation grouped into one transformer position per residue. CoBit receives Gaussian-corrupted bits whereas bit-MDLM receives masked bits.

Both models completed 200k updates on the same 216,237 paired proteins, with 10k warmup, batch 256 under the same training seed. We get the 200k EMA checkpoint on the same 4,098 held-out pairs. We evaluated 1,500 original co-designed pairs per model, 300 at each length 100/200/300/400/500, with 127 actual denoiser evaluations including self-conditioning with direct ESMFold refolding (no ProteinMPNN redesign)

MetricCoBitbit-MDLM
Valid amino-acid sequences100.0%87.6%
RMSD ≤2\leq2 Å26.60%20.53%
scTM ≥0.9\geq0.948.60%22.60%
Both criteria (strict)21.93%14.33%
Distinct sequences among strict successes210179

Recall that strict success requires both all-C α\alpha RMSD ≤2\leq2 Å and scTM ≥0.9\geq0.9. CoBit gains 7.6 percentage points in aggregate strict success (53% relative). The paired sampling 95% CI is +5.3–9.9 points (it measures sampling uncertainty, not training-seed variation). Exact-sequence deduplication gives 17.3% more distinct successful sequences and most gains concentrate at 200–300 residues. The gain comes at similar measured sampling cost (see Reviewer 6THc, W3, for timings and training costs). The separate submitted-model temperature experiment is reported in Reviewer 6THc, W2.

Reviewer_6THcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper applies continuous bitstream diffusion to three modality pairs—image-text, speech-text, and protein sequence-structure—using a shared binary input/output interface, denoising objective, and sampling protocol. In each setting a single checkpoint supports both joint and conditional generation by masking observed bitstream regions.

Critical strengths
  • This paper is easy to follow.

  • The paper investigates whether the same bitwise denoising interface can represent markedly different kinds of cross-modal dependence.

  • The evaluations include both cross-modal consistency and task-specific measures, with reported integration steps, denoiser evaluations, sampling settings, and substantial supplementary protocol details.

Critical weaknesses
  • The novelty and technical contributions of this paper can be further improved. The main contribution is modality layouts, task masks, and empirical transfer across domains.

  • The results show feasibility more clearly than competitive capability. Speech recognition and thematic continuation remain weak, and protein co-generation has a substantial gap between fold-level and strict coordinate agreement.

  • Some generation directions require hundreds or thousands of denoiser evaluations, yet the paper provides no matched analysis of latency, throughput, or compute cost against the alternative systems.

Questions for authors

See the weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 6THc

We thank the reviewer for the comments. The revised manuscript adds natural-image experiments and improved speech-recognition results. We address the contribution, the weaker metrics and sampling cost separately below.

W1. Novelty and technical contribution

The novelty and technical contributions of this paper can be further improved. The main contribution is modality layouts, task masks, and empirical transfer across domains.

Response. The contribution is the multimodal extension and its experimental validation; the underlying diffusion process and schedule come from the original CoBit. A shared bitwise denoiser learns both joint generation and the conditional directions included in training, with the observed mask specifying the task. This works across image–text, speech–text and protein sequence–structure data without changing the loss or adding modality-specific prediction heads. Each setting is trained separately, and its codecs and layouts remain domain-specific.

The revised image experiments strengthen the empirical case: CoBit-CC3M and CoBit-CC12M achieve zero-shot COCO FID-30K of 18.48 and 16.59, while also supporting captioning and joint generation (Table 2). In speech, one checkpoint obtains 3.5% cross-modal WER for joint generation and 5.3% WER with UTMOS 4.0 for text-to-speech, alongside recognition and continuation (Table 3). For proteins, direct refolding of generated sequences gives mean scTM 0.911 (Table 34).

W2. Recognition, continuation and protein precision

The results show feasibility more clearly than competitive capability. Speech recognition and thematic continuation remain weak, and protein co-generation has a substantial gap between fold-level and strict coordinate agreement.

Response. The performance is uneven across tasks. Speech generation and fold-level protein consistency are stronger than recognition, thematic continuation and coordinate-level precision. The new recognition results and the comparisons below help clarify these differences.

Speech recognition. Changing the score temperature to τ=0.3\tau=0.3 reduces CoBit-MLS WER from 8.2% to 7.9% on test-clean and from 22.0% to 21.7% on test-other (Table 19). Temperature was selected on development data. The checkpoint and sampling budget are unchanged: both temperatures use 1,022 CFG branch evaluations (Appendix C.1). The gains are modest, but require no additional training or denoiser calls.

The remaining gap should be considered alongside the training regimes. CoBit-MLS has 633M parameters and is trained from scratch on approximately 45K speech hours. SpiritLM and Moshi use 7B text-pretrained backbones and report approximately 570K speech hours and 7M audio hours, respectively. Their ASR results also use different task adaptations and evaluation settings. (SpiritLM's quoted 6.0/11.0 WER is for Base + ASR/TTS with 10-shot prompting; Moshi's 5.7 WER is its published test-clean result. See Appendix C.9 and the SpiritLM and Moshi papers.) GPA-v1.5 remains stronger in our evaluation (3.4/6.3 WER), but the cited GPA paper does not specify that checkpoint's training data. These comparisons do not isolate the effects of pretraining or data scale; Appendix C.9 now makes the differences explicit.

Thematic continuation. Table 20 shows that CoBit-MLS produces good acoustic continuations but often fails to preserve the topic. Its mean GPT-4o relevance score is 1.44, compared with 1.57 for Flow-SLM-L and 3.42 for TASLM-Embed. This is a substantive limitation. At the same time, relative to the 1.25B-parameter, text-initialized Flow-SLM-L trained on MLS-English, CoBit has higher UTMOS (3.89 versus 3.48), lower FSD-wlm (2,386 versus 3,476), lower FSD-e2v (13.0 versus 35.4), and lower perplexity of the transcribed continuations (83.6 versus 131.7). Speaker similarity is lower (0.34 versus 0.41). The same 633M checkpoint provides these generation capabilities together with ASR, TTS and joint speech–text generation; improving semantic continuation remains an open problem for this model.

AuthorsOpenReview ↗

Response to Reviewer 6THc (Cont.)

Protein precision. Fold recovery and coordinate recovery should be distinguished. For the 100 reported pairs, directly refolding CoBit's own sequences gives mean scTM 0.911; 69% satisfy scTM ≥0.9\geq0.9, while 41% satisfy all-C α\alpha RMSD ≤2\leq2 Å (Table 34). These are separate success rates. The results support sequence–structure consistency at the fold level, with a clear loss of geometric precision, especially at longer lengths.

To address your questions further, we ran a protein-temperature experiment on the submitted 396.23M-parameter checkpoint selected at 245k updates (please check a separate experiment on smaller co-design-only control in Reviewer CjUM's answers).

We also evaluated τ∈{1,0.7,0.5,0.3}\tau\in\{1,0.7,0.5,0.3\} on the same submitted checkpoint, using 900 fresh pairs per temperature: three sampling seeds (43–45), each with 60 draws at every length 100/200/300/400/500. Each temperature was fixed across lengths, with 99 generation NFEs per draw. The tables compare against fresh τ=1\tau=1 reruns, not the historical submission's success rates.

Recall that direct refolding uses CoBit's own sequence. MPNN-1 uses the first ProteinMPNN candidate; Here we add MPNN-8: it tests the nested pool of eight, requiring at least one candidate to satisfy both strict thresholds. ProteinMPNN's sampling temperature stays at 0.1. Redesign and refolding add work beyond the 99 CoBit generation calls.

Strict success relative to a fresh τ=1\tau=1 baseline (900 pairs per temperature).

Routeτ\tauτ=1\tau=1NewΔ\Delta pp [95% CI]Relative Δ\Delta (%) [95% CI]
Direct0.738.4%52.2%+13.8 [+10.1, +17.3]+35.8 [+25.1, +47.8]
MPNN-10.738.4%51.4%+13.0 [+9.0, +17.1]+33.8 [+22.0, +47.4]
MPNN-80.753.0%63.4%+10.4 [+6.6, +14.3]+19.7 [+12.0, +28.4]
Direct0.538.4%47.6%+9.1 [+5.1, +13.1]+23.7 [+12.6, +36.2]
MPNN-10.538.4%44.7%+6.2 [+2.0, +10.6]+16.2 [+4.9, +29.3]
MPNN-80.553.0%59.7%+6.7 [+2.4, +10.9]+12.6 [+4.5, +21.5]
Direct0.338.4%61.4%+23.0 [+19.6, +26.4]+59.8 [+48.3, +73.1]
MPNN-10.338.4%49.9%+11.4 [+7.8, +15.1]+29.8 [+19.1, +41.9]
MPNN-80.353.0%59.9%+6.9 [+3.7, +10.2]+13.0 [+6.5, +20.1]

Direct strict success by length: fixed τ=0.3\tau=0.3 versus τ=1\tau=1.

Residuesτ=1\tau=1τ=0.3\tau=0.3Δ\Delta pp [95% CI]Relative Δ\Delta (%) [95% CI]
100109/180133/180+13.3 [+4.4, +22.8]+22.0 [+6.4, +41.2]
200137/180178/180+22.8 [+16.7, +29.4]+29.9 [+20.4, +41.7]
30070/180144/180+41.1 [+31.7, +50.0]+105.7 [+70.4, +154.4]
40024/18097/180+40.6 [+31.7, +49.4]+304.2 [+183.3, +543.7]
5006/1801/180-2.8 [-7.7, +2.4]-83.3 [-99.9, +232.1]

The tables report 95% intervals from 10,000 paired bootstrap resamples within fixed seed–length strata, except at 500 residues, where sparse successes require conservative Bonferroni–Clopper–Pearson bounds. Intervals describe sampling uncertainty for this checkpoint, not training-seed or temperature-selection uncertainty.

Like in the speech experiment, the strongest aggregate direct result is at τ=0.3\tau=0.3; both redesign routes peak at τ=0.7\tau=0.7. Descriptively, the best direct settings are length-dependent among those tested are 0.7 at 100 residues, 0.3 at 200–400 and 0.5 at 500 (13/180 successes). Lowering τ\tau to 0.3 reduces unique sequences from 646 to 373 and distinct sequences among direct strict successes from 128 to 127; The caveat is that the higher per-draw success brings better downstream success on the quality of structures, but it does not imply more distinct protein designs.

AuthorsOpenReview ↗

Response to Reviewer 6THc (Cont.)

W3. Computational cost

Some generation directions require hundreds or thousands of denoiser evaluations, yet the paper provides no matched analysis of latency, throughput, or compute cost against the alternative systems.

Response. Several conditional tasks work well at much lower sampling budgets than the settings highlighted in the original paper. The new text-to-image results use 10 guided steps (20 NFE) for CC3M and 16 guided steps (32 NFE) for CC12M. This does not extend to every direction: captioning and joint image–text alignment benefit from longer sampling.

For speech, Table 23 compares step counts at τ=1\tau=1 under the same sweep protocol:

WER (%)32 nominal steps512 nominal steps
Text-to-speech5.75.3
Speech-to-text, clean8.38.2
Speech-to-text, other22.722.2

The conditional tasks therefore retain most of their quality with 16 times fewer nominal steps. These sweep results are distinct from the full-test temperature results above; the 7.9/21.7 WER has not been measured at 32 steps. Joint speech generation continues to benefit substantially from additional evaluations (Table 22).

The reviewer is correct that an NFE comparison does not establish a wall-clock advantage. The controlled protein comparison described in our response to Reviewer CjUM is intended to provide matched timings. We will report training GPU-hours, sampling latency, throughput and peak memory on the same hardware, counting self-conditioning and guidance calls and separating generation from codec decoding and external refolding. Those measurements are still pending. Distillation is a possible further reduction in cost, but was not used in these experiments.

The 50.38M CoBit and bit-MDLM rebuttal controls achieve 21.93% and 14.33% aggregate direct strict success at 127 NFEs each. Please check our answer to Reviewer CjUM: it gives their shared inputs, training protocol and quality table. Across 20 matched A100-80GB/FP32 settings (lengths 100–500; batches 1/4/8/16; two warmups and five measured batches), mean generation-plus-codec time differs by at most 5.5%, and peak allocated generation memory by less than 0.1%. At length 300/batch 16, generation alone averages 2.22 s per batch for CoBit and 2.25 s for bit-MDLM. This supports higher aggregate strict refolding success at similar measured sampling cost.

Training forward-example counts match at 76.80M, including self-conditioning. Committed training costs are 380.7/365.9 A100-GPU-hours (CoBit/bit-MDLM). CoBit's entropy calibration adds 0.21 GPU-hours. These measurements exclude startup. CoBit used 32 GPUs; bit-MDLM resumed from 32 to 16 with gradient accumulation. Warm sampling measurements exclude loading, calibration and external refolding, hence total pipeline cost is not claimed equal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.