acceptodds
Under review as a conference paper at ICLR 2027

SyncAudio: Parallel and Synchronized Text-Speech Decoding for Spoken Dialogue

Abstract

Extending text-based LLMs to spoken dialogue requires integrating speech generation without sacrificing the linguistic and knowledge capabilities of the underlying model. We study this problem in a shared-backbone architecture with parallel text and speech generation, where the two modalities are decoded through separate pathways but interact through a common language-model context. A central challenge is how the speech pathway interacts with the semantic structure inherited from the text LLM. Such interaction can introduce degradation in two complementary ways. First, temporal mismatch between text and speech can mix states from different stages of the response, weakening the semantic coherence of the shared context. Second, the speech pathway depends on backbone representations whose cross-modal relevance varies across depth, so conditioning on a single layer may either miss complementary information or emphasize representations that interfere with semantic preservation. We introduce SyncAudio, which addresses these two sources of cross-modal interference through dynamic text-speech synchronization and sparse multi-depth aggregation. For temporal coordination, SyncAudio uses an audio-guided synchronization mechanism to determine when the evolving speech state should be committed and fed back into the shared context. For representational interaction, it sparsely selects and aggregates representative hidden states from different backbone depths to provide richer conditioning for speech generation, while leaving the standard text-decoding pathway unchanged. Experiments show that dynamic synchronization substantially improves text-speech alignment while preserving natural and intelligible speech generation. More importantly, SyncAudio consistently outperforms an otherwise matched fixed-synchronization baseline on spoken question answering, suggesting better preservation of the semantic and knowledge capabilities inherited from the underlying text LLM. Further ablations show that sparse multi-depth representation aggregation provides more effective conditioning for the speech pathway than relying on the final backbone layer alone.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Spoken Dialogue Models, Parallel Text-Speech Decoding, Dynamic Text-Speech Synchronization, Sparse Multi-depth Aggregation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LA9gOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SyncAudio introduces parallel text-speech generation with two components: an audio-state-conditioned boundary predictor for dynamically grouping speech tokens, and sparse aggregation of backbone representations from different depths to condition the audio decoder. The model is evaluated on spoken QA, text-speech consistency, ASR/TTS, and full-duplex interaction. Compared with a matched fixed-group baseline, the reported results show substantially better synchronization and improved average spoken QA performance.

Critical strengths

The synchronization comparison is informative. Audio-guided boundaries reduce the reported MAD from 3.26 to 0.22 without the intelligibility degradation observed with text-guided prediction. This supports conditioning boundary decisions on the evolving audio state.

FixedAudio is matched in backbone, training data, and training recipe, providing a useful control beyond comparisons with heterogeneous existing systems. The separate depth-aggregation ablation also helps assess the second component.

Critical weaknesses

The central supervision signal is insufficiently specified. Section 3.3 and Appendix B assume reference audio-group boundaries but do not explain how speech frames are assigned to individual LLM text tokens to construct these labels. This is essential to both training and the synchronization evaluation. It is also unclear whether MAD is measured under teacher forcing or free-running generation. The boundary threshold and exact aggregation-layer indices are not provided either, leaving important parts of the method difficult to reproduce.

The capability-preservation interpretation is not well isolated. Table 7 compares the original backbone with text input against adapted models with speech input, which conflates capability retention with speech understanding. The spoken QA gains in Table 1 are largely driven by AlpacaEval, while factual QA results are mixed—for example, spoken WebQ decreases from 34.9 to 33.1. The key QA ablations also lack uncertainty estimates; sparse aggregation exceeds dense aggregation by only 0.38 points in average spoken QA. These results support a useful system improvement more clearly than the broader preservation claim.

Questions for authors

How are token-level reference audio boundaries constructed? Please specify the alignment procedure, boundary threshold, and selected backbone layers, and clarify whether synchronization MAD uses reference or generated histories.

Can you provide matched text-input evaluations of the adapted models to distinguish capability retention from improved speech understanding? Please also report uncertainty for the paired QA comparisons, particularly sparse versus dense aggregation.

Reviewer_otEzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SyncAudio, a spoken dialogue model that performs parallel text and speech generation over a shared LLM backbone. The paper focuses on two forms of cross-modal interaction that may affect the preservation of the underlying text LLM’s semantic and knowledge capabilities. First, it introduces audio-guided dynamic synchronization, which predicts when a variable-length audio-token group should terminate based on the evolving hidden state of the autoregressive audio decoder, rather than using a fixed text-speech ratio. Second, it proposes sparse multi-depth aggregation, which combines hidden states from lower, middle, and upper backbone layers to condition the speech-generation pathway while keeping the standard final-layer text decoder unchanged.

The model is built on a frozen Qwen3-4B backbone with LoRA adaptation, a Qwen3-0.6B audio head, and a CosyVoice2 tokenizer. The authors validate the method on synchronization quality, speech naturalness and intelligibility, spoken question answering, text-speech consistency, ASR/TTS, and full-duplex interaction. The main comparison is against a matched fixed-synchronization baseline, and ablations compare different synchronization mechanisms and multi-depth aggregation strategies.

Critical strengths
  1. A well-controlled study of synchronization in parallel text-speech generation.

The strongest part of the paper is the controlled comparison between Fixed, Text-Guided, Native, and Audio-Guided synchronization under an otherwise matched generation framework. The results show a meaningful distinction between simply predicting accurate boundaries and using a synchronization mechanism that also preserves speech quality. Audio-Guided achieves the lowest synchronization deviation while also obtaining the best UTMOS and WER among the tested variants. This makes the paper’s main empirical claim substantially more convincing than a comparison against unrelated external systems alone.

  1. Broad evaluation connecting synchronization to semantic preservation, rather than only acoustic quality.

The paper evaluates not only speech naturalness and intelligibility, but also spoken QA, text-speech consistency, text-only decoding, ASR/TTS, and full-duplex interaction. In particular, the comparison with the matched FixedAudio baseline shows improvements in both textual and spoken QA outputs, while the text-speech response gap also decreases, including larger gains on long-response subsets. This gives useful evidence that synchronization affects the shared semantic context, not merely the timing of acoustic generation.

Critical weaknesses
  1. The central claim that audio-guided synchronization specifically improves semantic and knowledge preservation is not sufficiently isolated. The main downstream evidence compares SyncAudio only against FixedAudio with a single fixed group size (G = 4). Although Text-Guided and Native synchronization are introduced as alternative dynamic strategies, they are evaluated only on synchronization deviation, UTMOS, and WER, but not on the spoken-QA or text-speech consistency benchmarks that support the paper’s central semantic-preservation claim. Moreover, there is no sweep over fixed group sizes or comparison under matched average group size / decoding latency. As a result, the current experiments do not establish whether the downstream gains come specifically from the proposed audio-guided boundary decision, from adaptive grouping in general, or simply from a more favorable decoding schedule. This is consequential because audio-guided dynamic synchronization is presented as one of the paper’s main methodological contributions.

  2. The supervision used by the core synchronization mechanism is insufficiently specified, making both the method and its primary synchronization metric difficult to assess. The boundary predictor is trained using “reference audio-group boundaries,” and synchronization deviation (MAD) is also computed against these boundaries. However, the paper does not explain how these reference boundaries are constructed: e.g., whether they are obtained from forced alignment, TTS duration information, an external alignment model, or another heuristic, nor how text tokens are mapped to the 25-Hz audio-token sequence. This is not a minor implementation detail, since the proposed synchronization mechanism directly learns from these targets and its strongest quantitative result is measured against the same targets. Without this information, it is difficult to determine what supervision the method actually requires, whether the proposed boundary is a well-defined target, and whether the synchronization results would generalize beyond the particular boundary-construction procedure.

Questions for authors
  1. How exactly are the “reference audio-group boundaries” used to train the Audio-Guided synchronization head and compute MAD constructed? Please specify how text tokens are aligned with the 25-Hz audio-token sequence, whether an external forced aligner, TTS duration information, or another model is used, and whether the same procedure is applied across all training data. The answer would materially affect my assessment because both the proposed synchronization objective and the main synchronization evaluation depend on these targets.

  2. Can the authors report downstream spoken-QA and text-speech consistency results for all synchronization variants (Fixed, Text-Guided, Native, and Audio-Guided), preferably also including tuned fixed group sizes or configurations matched in average group size / latency? In the current paper, the alternative dynamic mechanisms are compared only using MAD, UTMOS, and WER. Therefore, it remains unclear whether the semantic gains attributed to Audio-Guided synchronization are specific to its proposed synchronization signal or would also arise from other forms of adaptive grouping.

Reviewer_DaTNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SyncAudio studies parallel text–speech generation over a shared LLM backbone. It replaces fixed-size audio groups with an audio-guided boundary predictor that determines when generated speech is fed back to the shared context. It also aggregates representative hidden states from different backbone depths to condition the audio head, while retaining the standard final-layer text interface. Experiments report improved synchronization, text–speech consistency, and average spoken-QA performance over a matched fixed-group baseline, with an additional real-time duplex extension.

Critical strengths
  1. Useful controlled synchronization comparisons. FixedAudio shares the architecture, data, and training recipe, while the synchronization study also considers text-guided prediction and an end-of-group-token alternative. Audio-guided synchronization improves both boundary accuracy and speech metrics, rather than relying on alignment accuracy alone.

  2. Measurable text–speech consistency improvements. Overall text–speech WER decreases from 13.2% to 11.9%, and the long-response subset's response gap decreases from 9.3% to 5.9%. These results provide practical evidence that adaptive grouping can help limit mismatch during longer outputs.

Critical weaknesses
  1. The main QA comparison does not establish the benefit over a simple dynamic alternative. Fixed-size grouping is a relatively restrictive synchronization rule when speech duration varies. The paper appropriately includes Native termination through an end-of-group token in the synchronization/TTS study, but does not report its results in the main spoken-QA comparison. Consequently, it remains unclear whether the claimed semantic benefit requires the proposed boundary head or would largely follow from ordinary dynamic termination. This missing comparison limits the demonstrated incremental value of the synchronization design.

  2. The downstream gains are modest or uneven, and the advantage of sparse aggregation is small. Relative to FixedAudio, spoken LLaMAQ improves from 56.0 to 58.5, TriviaQ changes from 28.7 to 28.9, and WebQ decreases from 34.9 to 33.1. Much of the average improvement is driven by AlpacaEval, whose score is rescaled for aggregation; this does not demonstrate uniformly stronger knowledge QA. Sparse aggregation also improves average spoken performance only slightly over dense aggregation (49.93 versus 49.55), with mixed per-benchmark results. Without uncertainty estimates for these comparisons, the evidence supports the usefulness of dynamic grouping and multi-layer conditioning more clearly than the specific design choices or the broader capability-preservation interpretation.

Questions for authors
  1. How does Native end-of-group termination perform on the same QA and text–speech consistency evaluations? Does the separate boundary predictor retain a meaningful advantage under the matched training setup?

  2. Are the QA gains and sparse-versus-dense aggregation differences robust to generation/training variability or paired uncertainty analysis? Given the uneven benchmark results, how narrowly should the semantic and knowledge preservation claim be interpreted?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.