acceptodds
Under review as a conference paper at ICLR 2027

Language Directed Mono-to-Stereo Music Upmixing

Abstract

The same mono song can become many valid stereo mixes. A producer may keep the kick centered, spread the claps across the speakers, or widen a vocal only in the chorus; none of these choices is determined by the mono recording alone. We introduce Width as a Command, a language-directed system that makes such choices explicit. It turns a user's request into an editable plan for the width of individual musical elements, grounds that plan in the recording, and generates a stereo mix while preserving the original mono content. Our key representation describes width through rapid left–right motion, independent of an element's overall position. A generative model then realizes a plausible motion for each requested width without requiring the user to specify an exact stereo trajectory. Experiments on public music examine both automatic upmixing and authored width control; ablations indicate that left–right motion provides a more useful width description than side-channel energy. Making stereo imaging describable and editable offers an interface for music production and a path toward spatially annotated data for future music generation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Mono-to-stereo upmixing, music production, language-directed audio, controllable generation, spatial scripts, stereo imaging, computational creativity

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_r2UeOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper redefines monaural-to-stereo upmixing as language-driven stereo imaging creation rather than a sole reconstruction problem. The authors propose "Width as a Command": language directors translate user requests into element-level width and translation scripts, monaural source separation routes scripts to time-frequency regions, classification-based parametric stereo skeleton predicts IID/IC, conditional diffusion models generate locally centered IID motions, and a fixed renderer outputs stereo while preserving monaural channels. The experiments and ablation studies verify that local IID motions outperform side-channel energy as width descriptors.

Critical strengths
  1. The formulation of the task essence demonstrates profound insight, since it explicitly identifies the one-to-many uncertainty caused by downmixing and the mean regression issue of point-wise regression.
  2. The localized centering IID oscillation decouples "width" and "global translation" with clear acoustic semantics and facilitates editing, while the mid-channel algebraic preservation and post-translation design provide controllability and content fidelity guarantees, preventing generative models from arbitrarily altering mono-channel content.
  3. The experiments effectively demonstrate the role of diffusion and width representations.
Critical weaknesses
  1. Human experiments are missing. ABX, MOS, or professional mixing evaluations are absent.
  2. Training and evaluation data are limitted, where the universality of the model remains unknown, especially for long audio.
  3. Metric interpretations show potential conflicts, for example, Schroeder's Side-FAD is minimal while Fréchet-PS remains high, maybe indicating the proposed evaluation system has not yet demonstrated consistency with perceived quality.
Questions for authors
  1. Could you provide more evaluation results in subjective/expert experiment or universality experiment (e.g., long audio)?
  2. Could you provide more detailed explanation of the conflicts between the results of different evaluation metrics?
1 comment

Thank you for identifying the task formulation and the centered-swing design as the core of the work; that is exactly the framing the revision now foregrounds (Section 1). Section, table, and appendix numbers below refer to the revised PDF, changed text is set in blue.

1. Human evaluation and audio length. We have added a blinded listening test of the system's instruction following (Appendix D). It is a two-alternative forced choice in the style of the Artificial Analysis Music Arena: each question shows the written instruction and two renders of the same mono excerpt from the same frozen model, differing only in the width map, and the listener picks the one that follows the instruction. Twelve listeners answered ten questions each (120 judgments, 100 songs, loudness-matched, order randomized). The instructed mix was chosen on 97 of 120 questions (81%, Wilson 95% CI 73-87%, p < 1e-11), and every listener scored at least 6 of 10. To address length, the stimuli were rendered at three durations, 5 s, 15 s, and 30 s; accuracy was 78%, 83%, and 81% respectively (chi-square p = .83), so longer excerpts neither helped nor hurt, and the 30 s renders already exercise the long-form inference path of Section 5.4 (windowed generation with overlap merging, Appendix A.5). Per-element accuracy ranged from 70% (hi-hat/cymbals) to 94% (lead vocal). We are currently recruiting additional listeners with five or more years of music experience and will extend the table as results arrive.

2. Conflicts between metrics. The two distributional metrics measure different necessary conditions, and we have added Appendix C.1 to state this with pseudocode. Both are Frechet distances between Gaussians fitted to pooled per-frame vectors over all 300 excerpts; they differ only in the vector considered. Side-FAD embeds the side channel alone with VGGish, so it tests whether the side channel sounds like the side channel of real music, and it never sees the left and right channels together. Frechet-PS uses the six-band IID and IC of each frame, so it tests whether the stereo image has the statistics of real stereo. Schroeder is the case that pulls them apart: its side channel is the mid passed through all-pass filters, so it contains the right instruments with the right spectral envelope and scores the best Side-FAD (0.56), while its IID is near zero and its IC follows a fixed comb that never varies with the music, so its Frechet-PS is high (31.13). Combing both metrics we read Schroeder as 'fake' stereo. So the metrics are not inconsistent with each other; a good upmix has to satisfy both.

Reviewer_WNcbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a language-directed framework for mono-to-stereo music upmixing, enabling users to adjust the stereo width and pan positions of individual musical components via natural language instructions. A width representation grounded in locally centered inter-channel intensity difference (IID) is devised to decouple stereo width from spatial location. The complete system assembles an LLM-based spatial director, source separation module, categorical prediction backbone, conditional diffusion model, and parametric stereo renderer. Evaluations of automatic and width-conditioned upmixing are performed on public music datasets, alongside ablation studies examining the efficacy of the proposed width representation and generative modeling approach.

Critical strengths

(1) This work targets an intriguing, practically relevant research problem: language-guided stereo upmixing delivers an intuitive, flexible interface for music production. Supporting per-element manipulation of stereo width and panning constitutes a valuable advancement compared with conventional automatic upmixing pipelines.

(2) The locally centered IID representation presents a fresh viewpoint for disentangling stereo width and pan position. Additionally, benchmark comparisons against prior approaches and ablation analyses verify the advantages brought by this proposed width representation

Critical weaknesses

(1) Unclear technical focus.

The work describes a full-fledged system consisting of many tightly coupled modules, giving the manuscript a tone closer to a technical report rather than a focused research article. Although individual components are elaborated in detail, the core methodological innovation is under-emphasized and hard to separate from auxiliary engineering implementations. Readers may struggle to pinpoint the primary technical contribution, raising the cognitive load required to understand the overall framework. Reframing the manuscript to foreground the central insight while streamlining peripheral technical descriptions will enhance clarity and readability.

(2) Incomplete validation of the end-to-end framework.

Despite adopting a multi-stage pipeline, the experimental evaluation concentrates largely on the proposed width representation and diffusion module. The roles of other critical components — especially the language director and source-aware control routing — are not independently isolated or assessed. Furthermore, key quantitative results partially depend on oracle width conditions, which obscures the real performance of the full language-to-audio pipeline. Additional end-to-end benchmarking and perceptual listening tests are needed to solidify the empirical support.

Questions for authors

(1) Clarification of the core contribution.

Could the authors clarify the central methodological innovation of this work and better distinguish it from the supporting system components? Which design choices are essential to the proposed approach?

(2) Necessity of the complex pipeline.

Given the number of interconnected modules, could the authors explain why such a multi-stage architecture is necessary? In particular, would a simpler conditional generative model achieve comparable results?

(3) End-to-end effectiveness.

Could the authors provide more comprehensive evaluations of the complete language-to-audio pipeline, particularly regarding instruction-following accuracy, robustness to source separation errors, and perceptual audio quality?

1 comment

Thank you for your questions. Section, table, and appendix numbers below refer to the revised PDF, changed text is set in blue.

1. Core contribution. We have tightened the introduction and the contribution claims (Section 1) , and conclusion (Section 8) to foreground the central insight: the core of the work is the width-map representation, an element-wise description of the stereo image that is coarse enough for language to write and precise enough for an acoustic model to realize. Everything else is organized around it. On one side is the acoustic model trained with the map (Sections 3-5, evaluated in Section 6); on the other is the language upmixing director that writes the map from a request (Section 7, evaluated in Section 7.4). The LLM protocol, ownership masks, and compiler are supporting components that connect language to the map.

2. Why not a simpler conditional generative model. This question describes how the system came to be designed this way. A simple conditional generative model was the first thing we tried, and we spent considerable effort on it: several architectures (spectrogram DiT, latent flow, waveform GANs), objectives, and targets, all conditioned directly on parametric stereo, which is a far richer condition than the later width map. The revision adds Appendix E (Table 8) to document part of these attempts. Every learned model produced a side channel worse on both metrics than simply rendering the six-band parametric stereo through the fixed decoder, let alone the per-bin parametric stereo; the failure modes were collapse toward mono or noisy side texture. That gap is why we moved to generating parametric-stereo parameters rather than audio, which keeps the mid exact and leaves the generative model only a stereo trajectory to produce. We committed to the current design knowing the simpler alternatives had failed, and Appendix E now says so.

3. End-to-end effectiveness. The revision adds experiments that isolate the language director (Section 7.4, Table 2): request interpretation (300/300 valid plans, 289/300 exact semantics; Appendix B.6), element-specific width following (100/100 excerpts widen the target, +3.05 IID-dB; Appendix B.5, Figure 10), and element-specific pan following (100/100 correct direction), with spillover onto unmentioned elements measured in both cases. End-to-end results come from a blinded 2AFC listening test (Appendix D: 12 listeners, 120 judgments, 5-30 s excerpts; the instructed mix was chosen 81% of the time, 95% CI 73-87%), and we are recruiting more listeners. Source-aware routing uses soft mono-derived ownership masks (Section 7.2); the spillover and listening results are measured with these masks in the loop. On perceptual audio quality, we found it hard to design a meaningful absolute-quality test: our results have no obvious artifacts in our judgment, and because the mid is preserved exactly, quality differences are confined to the side channel. We believe the listening test speaks about quality indirectly: a low quality mix would destroy a judgment as subtle as 'element-wise width perception'.

Reviewer_t2wuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this work, the authors propose a language directed mono-to-stereo music upmixing method. The method works by inputting user commands to change the panning position and width of individual elements, converting this into an intermediate editable representation of panning and width adjustment across frequency bands via a deterministic “control compiler”, and then a diffusion model inputs the width control and outputs left-right intensity control params to apply to the output short-time Fourier transform together with the overall pan position controls from the control compiler output. Evaluation focuses on automatic upmixing and width controls using reference music from 100 songs (e.g. reconstruction of the ground truth widths).

Critical strengths
  1. Editable plan with separate pan and width controls. I like the idea of the editable plan with separate controls on pan position and width. This is an appealing aspect of the proposed interface and a potential source of value beyond automatic upmixing. The related work is heavily rooted in music and audio engineering, with less on LLMs as controllers, but is good overall.

  2. Demonstration of width control. The experiments show that the proposed width control can work very well with oracle width conditioning and illustrate trade-offs between control and reconstruction quality. The non-oracle method is also competitive, although it is not uniformly better than the baselines.

Critical weaknesses
  1. Evaluation does not establish the main language-directed capability. The introduction and abstract motivate general LLM control over a mono-to-stereo upmixer, but most evaluation seeks to reconstruct the stereo width of reference songs. It does not evaluate the language-directed part of the work. There is no real evaluation of the language control, control compiler, individual instrument/element control, or panning-position control. This kind of control is exceptionally hard from a mono mix, so the focus on width makes sense, but it contrasts with the overall motivation and introduction. Appendix B describes evaluation of the director and round trip, but I could not find the results or main takeaway. There is no listening study or subjective/qualitative commentary on audio quality either. I would love to see clear evaluation that separates pan control, width control, and LLM interpretation of human commands. This is important because much of the potential value is in the language-directed part of the work. Low-level sentence structure is good, but the evaluation section is presented before the final parts of the method, which seems strange and makes the connection between the method and its evaluation harder to follow.

  2. Unclear benefit and scientific insight without oracle conditioning. When the oracle-conditioned results are removed, the proposed method is competitive but not uniformly better than the baselines. It seems to offer a trade-off for width-control methods rather than a clear improvement overall. I struggle to identify the core reproducible scientific insights beyond strong oracle-width results and the trade-offs between control and reconstruction quality. More evaluation around the claimed contributions on page 2 would help clarify the benefit.

Questions for authors
  1. Clarification of results for the central interface. Where are the director and round-trip evaluation results described in Appendix B, and what are their main takeaways? What evidence separately establishes language interpretation, element-specific pan control, width control, and perceived audio quality?

  2. Clarification on non-oracle conclusions. What is the main benefit demonstrated without oracle conditioning? Should the results be understood as a trade-off for width-control methods rather than a general improvement over the baselines?

  3. Clarification on role of the upmixing backend. Could the proposed director/compiler be applied to the method of Serra et al., with some adaptation? This is a question about the scope of the contribution rather than a request to complete that additional experiment for clarification.

4 comments

Thank you for the valuable comments. Section, table, and appendix numbers below refer to the revised PDF, changed text is set in blue.

1. Evaluation of the language director. We agree that the submitted evaluation of the director was thin and unfocused. The revision adds experiments that isolate each ability separately, with objective tests in Section 7.4 (Table 2) and a detailed case study for each in the appendix cited there: (i) interpreting the request (300 prompts, 300/300 valid plans, 289/300 exact semantics; Appendix B.6), (ii) element-specific width following (100 excerpts, target widens on 100/100, +3.05 IID-dB on commanded cells; Appendix B.5, Figure 10), and (iii) element-specific pan following (100/100 correct direction, 16.4 dB left-right contrast), with spillover onto unmentioned elements measured in both cases. We also added a blinded 2AFC listening test (Appendix D: 12 listeners, 120 judgments, 5-30 s excerpts; target chosen 81%, 95% CI 73-87%) and are currently recruiting more listeners with 5+ years of music experience. Generally these show instructed upmixing largely follows the intent, with small losses on some elements and some paraphrases. We believe the listening test speaks about quality indirectly as well: a low quality mix would destroy a judgment as subtle as 'element-wise width perception'.

2. Why the auto mode is evaluated. Users might keep instructions minimal, for example specifying a stereo state for one element only. We therefore train the acoustic model with command dropout, so that it upmixes unspecified regions automatically and the instruction overrides only the requested elements. So we report both auto and oracle performance. We've added this explanation opens Section 6.

3. Role of the upmixing backend. No, the method of Serra et al. cannot serve as the backend of this system without retraining: (i) it is conditioned on mono only and has no input for an abstract width description, and (ii) its IID and IC are discrete tokens on a 1024-sample hop, at which Appendix A.1 shows that only about 9% of the width swing the command names survives.

In addition, we have tightened the introduction and the contribution claims (Section 1) to make the paper easier to follow. The core is the width-map representation; on one side is the acoustic model trained with it, and on the other the language upmixing director that writes it, each evaluated against that map written by order in the main tex.

Thanks for the revision. The new director evaluation (Sec. 7.4 / Table 2) addresses some of my concern albeit given the core contribution it's a relatively small scoped evaluation. The reorganized method and evaluation sections are much easier to follow. The Serra et al. answer is clear too.

A few questions remain:

  1. How were the ground-truth plans for the 300 prompts defined, and how varied are the prompts?
  2. Following WNcb's point, the event projection changes the requested map a lot (TV .529 to .152). How much of the user's request survives it?
  3. The 2AFC shows the instruction is audible, but it compares two outputs of the same model, so it says little about quality against Serra et al. or Schroeder. A small MOS or MUSHRA comparison would help.
  4. Please say plainly that auto mode is competitive with the baselines but not better, and that the main benefit is control.

I'm keeping my score for now, but open to discuss more.

Thanks for the revision. The new director evaluation (Sec. 7.4 / Table 2) is the right kind of test, and the reorganized sections are much easier to follow.

However, the evaluation is too limited to support the language-directed claim: 300 prompts for interpretation and 100 targets each for width and pan, with near-ceiling scores and no detail on how varied the prompts are or who defined the ground-truth plans. The event projection also changes the requested map substantially (TV .529 to .152), so it's unclear how much of the user's request survives. The 2AFC shows the instruction is audible but doesn't compare quality against Serra et al. or Schroeder, and auto mode is competitive with the baselines but not better.

I'm keeping my score. I think the controllable-width idea is promising, and a larger, more varied prompt evaluation with a quality comparison against baselines would make a much stronger paper.

Thank you for the follow-up and for the positive feedback on the reorganization.

1. Ground-truth plans and prompt variety. Each of the authored cases is defined by a structured specification: the target element and level, a band, time interval, pan direction, or relation where relevant. The expected plan follows directly from this specification. The 300 prompts render these cases with controlled paraphrases (all distinct, 9 to 27 words). One example per category:

CategoryPromptsExample requestExpected plan
Absolute80"I want a narrow stereo image for the lead vocal. Keep all other elements as they are."Lead vocal L1; nothing else changes
Band80"Only in 300 to 800 Hz, make the lead vocal narrow; keep its other frequencies and the other elements unchanged."Lead vocal L1 in 300-800 Hz only
Time80"Only from 1 to 2 seconds, make the lead vocal medium-width; leave the other times and elements unchanged."Lead vocal L2 from 1 to 2 s only
Pan30"Place the lead vocal clearly left at medium width; leave other elements alone."Lead vocal L2, panned left
Relative30"Keep the lead vocal at medium width and make the accompaniment exactly one width level wider; keep their pan unchanged."Lead vocal L2, accompaniment L3

We think these should cover the most common natural languages for upmixing request?

2. How much of the request survives the projection. The projection changes where width is placed, not what the request asks for. In ground-truth width maps/in real music, representative width is brief rather than sustained, based on elements 'main events': only 0.4% of band tracks hold L2 or above for one second, and the highest level occurs mostly at salient moments. The projection follows this pattern, so most of the large TV change comes from returning cells to lower width levels, as in real maps. The direction of the request always survives: in all 100 narrow/very-wide pairs the requested element widens (+3.05 IID-dB on commanded cells ), and all 100 left/right pairs pan correctly (Section 7) (already shown). On the previous new projection test for 10,000 songs, the projected map keeps 78% of the target's width change from the unprojected map (2.85 vs. 3.64 IID-dB), while change on other elements falls from 0.61 to 0.11 IID-dB. In addition, the listening test results show the request is largely followed.

3. 'Quality' comparison. We agree that the 2AFC test shows whether a request is followed, not overall quality. PS-MTM, Schroeder, and our model all keep the input mono as the mid channel and differ only in the side signal, their mixing styles differ, Schroeder would sound just like the mid with some post effects, while our model's results show a more dynamic side, 'quality' test here, could not be controlled, we still believe the instruction following is the main focus and rely on objective metrics for quality comparison; We expect the cross-model rating to be close to chance. DiffStereo is the exception: it regenerates the mid and adds unbearable noise, which the objective metrics already show (mid LSD 11.16, Table 2). We therefore designed the listening test around the focus of this work, control. It also checks quality indirectly: an artifact-laden mix would make the requested change hard to hear, yet listeners identified the target in 97 of 120 questions.

4. Phrasing. We agree, we will modify the corresponding part.

Comments on the paper

2 comments

We thank all reviewers for their careful reviews and valuable suggestions, which led to a clearer and stronger revision. We hope our revisions and responses address the concerns raised. We would also like to note the broader motivation. Much recent progress in music generation and understanding comes from enriching audio with structured representations such as lyric timings, beats, MIDI notes, and chords. More abstract attributes, such as the stereo image, singing-voice character, or mixing style, are still largely missing from these representations. This paper takes a step toward closing that gap for stereo width: a width map that can be extracted from existing mixes, rendered into audio, and written from language. We believe making such abstract attributes explicit is a meaningful direction, and could become a standard ingredient of future controllable music generation and of music understanding in audio language models.

AuthorsOpenReview ↗

Discussion Summary

We thank the reviewers for the constructive discussion. Below, we summarize the main concerns and where the revision addresses them, then refocus on the core innovation.

  • Evaluation of the language director: Section 7 now evaluates the path from a request to stereo output, including request interpretation and paired width and pan control.
  • Role of the event projection and the compiler: width map projection from a request is now in the main text (Section 4.2) moved from appendix, with a distribution analysis on 10,000 unseen songs (Table 1) and a step-by-step example (Appendix Figure 10).
  • Human evaluation and audio length: Appendix D reports a blinded listening test on 5- to 30-second excerpts.

In addition, we also clarified why auto model is evaluated while the trained acoustic model is used with width condition mainly, and clarified some metrics, elaborated the failure of end2end model (from our attempts).

Refocus on core innovation. Our main contribution is the width map: an explicit, describable representation of stereo imaging for music. It is extracted from real stereo to train the acoustic model and written from a natural language request at inference, so users can direct music upmixing by element-level width and pan while the input mono is preserved. A broader impact is that we hope it helps make stereo imaging an explicit attribute for future music generation and understanding.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.