acceptodds
Under review as a conference paper at ICLR 2027

SWIFT: Sequence-aware Wasserstein Informed Flow Transport for Conditional Audio Generation

Abstract

Existing methods for coupling audio sequences with noise sequences in flow matching use a frame-level Euclidean ground metric that treats each sequence as a flat vector, entirely ignoring both the temporal structure of acoustic events and the semantic content of the conditioning signal. We propose SWIFT (Sequence-aware Wasserstein Informed Flow Transport), a flow matching training framework for audio generation built on a novel ground metric called the Sliced Wasserstein Sequence Distance (SWSD). SWSD lifts each audio and noise sequence to a probability measure on its time-augmented path embedding, encoding temporal ordering explicitly into the geometry of the ground metric, and computes sliced Wasserstein distances using a semantic-aware slicing measure that concentrates projections along the acoustic subspace indicated by the text or visual conditioning vector. We validate SWIFT on two audio generation tasks: text-to-audio and video-to-audio. SWIFT outperforms baselines across evaluation metrics and inference budgets, with the largest improvements at low numbers of function evaluations, consistent with the design motivation that a semantically informed coupling produces straighter flow paths consistent with the test-time prior. We further demonstrate that a model pretrained with SWIFT provides a stronger initialisation for the reflow procedure, amplifying the benefit of trajectory straightening and achieving lower Fréchet Audio Distance at a fixed inference budget.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
flow matching, audio generation, sliced wasserstein

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xHFCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SWIFT, a flow matching training framework for conditional audio generation. It replaces the frame-level Euclidean cost in minibatch OT with a new ground metric, SWSD. SWSD prepends a time coordinate to each frame to encode temporal order. It then samples slicing directions from a von Mises Fisher distribution centered on the projected conditioning vector to compute a sliced Wasserstein distance. The Sinkhorn algorithm uses this distance to couple audio and noise. The paper evaluates SWIFT on text-to-audio generation with AudioCaps and video-to-audio generation with VGGSound. SWIFT outperforms existing coupling methods on most metrics, with larger gains at low NFE, and provides a better initialization for reflow.

Critical strengths
  1. The paper addresses the failure of Euclidean OT coupling in conditional audio generation. Data-noise coupling for flow matching in this setting has received little systematic attention.
  2. Time augmentation and vMF condition-directed slicing are combined into a single ground cost. This adds only training-time cost with no inference overhead, which makes the method practical.
  3. Results cover text-to-audio and video-to-audio, with fairly consistent FAD gains at low NFE. The ablations cover the main components, and SWIFT also gives a better initialization for reflow.
Critical weaknesses

Major weaknesses:

  1. L258–L261 state that "the resulting interpolation paths are straighter." This reads as an empirical conclusion and needs further verification. From the description of the method, SWSD projects each frame, sorts the projected values, and then compares them, so frame correspondence is determined by sorting rather than by time index. It is therefore unclear why an SWSD-based coupling should straighten the Euclidean interpolation paths. Moreover, Table 2 shows that SWIFT does not achieve better straightness than MinibatchOT or LOOM-CFM, and the gap widens at NFE = 2. This challenges the core theoretical claim of SWIFT.

  2. According to Sec. 3.1 and 3.2, SWSD's robustness to temporal shifts comes from the sorting and transport in sliced Wasserstein, and plain SW without the time coordinate is already fully invariant to frame permutations. The time coordinate binds frames back to their positions to guarantee injectivity, which makes the distance more sensitive to shifts. Preserving temporal order and tolerating temporal shifts are therefore a trade-off rather than complementary properties, as the paper claims. Their joint benefit should depend on the scale of the time coordinate, but the paper provides no corresponding analysis. I found this part confusing and difficult to follow.

  3. The conclusion at L418 (Table 3) does not fully match the authors' claim. With SWIFT, IS is higher in absolute terms, but its drop after reflow is larger. I suggest discussing this in the Limitations section of the appendix to help readers better understand SWIFT. In addition, the paper should include subjective evaluation by human experts to improve the credibility of the generation results. Audio and video generation ultimately serves users, not just metric improvements.

Minor weaknesses:

Some defensive writing in the paper weakens the presentation. The authors could reorganize the logic of certain sections to better highlight the contributions of SWIFT.

Questions for authors

Please review the content in the Critical Weaknesses; if it clears up most of my confusion, I'm willing to raise my score.

Reviewer_dEq1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SWIFT, a data–noise coupling method for conditional audio generation based on flow matching. It introduces Sliced Wasserstein Sequence Distance (SWSD), which incorporates temporal information and conditioning embeddings into the transport cost, and evaluates the approach on text-to-audio and video-to-audio generation.

Critical strengths
  1. Clear motivation and targeted methodological design. The paper identifies meaningful limitations of Euclidean-based data–noise coupling for conditional audio generation and proposes a principled alternative that incorporates temporal information and conditioning signals into the transport cost.

  2. Consistent empirical improvements with useful ablations. SWIFT achieves improvements over several coupling baselines on text-to-audio and video-to-audio generation, particularly at low inference budgets. The component ablations and reflow experiments provide additional evidence of the method's effectiveness.

Critical weaknesses
  1. Insufficient justification of the semantic-aware projection. A central component of SWIFT is to use the conditioning embedding to guide the slicing directions in the acoustic latent space. However, the paper does not sufficiently explain why a fixed random projection of the text or visual embedding should identify semantically meaningful directions in the acoustic feature space. Preserving geometric relationships among conditioning embeddings does not necessarily establish semantic alignment with acoustic features. Although the ablations show that condition-directed slicing improves generation performance, they do not directly validate the claimed semantic interpretation of the projection. Additional analysis of the semantic alignment of the resulting couplings, or experiments with different projection strategies, would strengthen the justification for this design.

  2. Lack of human evaluation. The paper evaluates generated audio using objective metrics such as FAD, KL, IS, and CLAP, but does not conduct human listening evaluations. For conditional audio generation, these metrics alone may not adequately capture perceptual audio quality, semantic alignment with the conditioning signals, or temporal coherence. Human evaluations are usually necessary in these cases.

  3. Evaluation across additional audio generation tasks. The experiments are restricted to text-to-audio and video-to-audio generation, while the proposed data–noise coupling method is intended for conditional audio generation more broadly. Both evaluated tasks primarily involve generating sound effects from semantic conditions, leaving its effectiveness on other audio generation tasks, such as text-to-speech, music generation, and audio editing, unexplored. In particular, it remains unclear whether the proposed method generalizes to tasks with different temporal alignment requirements. Broader evaluation would strengthen the claim of general applicability.

Questions for authors
  1. Statistical reliability of the reported improvements. Section 4.1 states that all results are computed five times and averaged, but the main tables report only mean values. Could the authors clarify whether these repetitions correspond to independent training runs or repeated evaluations of the same checkpoint, and report standard deviations or confidence intervals? This is particularly relevant for metrics such as CLAP, where some improvements are relatively small. The standard deviations in Table 2 appear to reflect variation across the AudioCaps test set rather than across repeated runs.

  2. Training efficiency. According to Appendix D, Table 7, SWIFT increases MeanAudio's per-iteration training time from 0.736 s to 1.113 s, approximately a 51% increase over independent coupling. Could the authors discuss the trade-off between the additional training cost and the improvements in generation quality or inference efficiency, particularly compared with other OT-based coupling methods?

Reviewer_kHACOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SWIFT, a new data–noise coupling for flow-matching audio generation. Standard minibatch OT uses a frame-wise Euclidean cost. SWIFT replaces it with a sliced Wasserstein distance between sequences. Each frame gets a time coordinate appended, and the projection directions are sampled around a direction derived from the text or video condition. The coupling is then solved with Sinkhorn. Experiments on text-to-audio and video-to-audio show small gains over baselines, mostly at low NFE. It also shows that SWIFT gives a better starting point for reflow.

Critical strengths
  1. The coupling problem is meaningful, and the method is easy to use.
Critical weaknesses
  1. The method is a simple extension that plugs a time-augmented sliced Wasserstein distance with vMF slicing into minibatch OT, and each piece already exists. The gains are also small, while training is much slower.

  2. The coupling is between audio and i.i.d. Gaussian noise, but the noise frames have no temporal structure; the rationale behind applying time shifts to noise is not clear.

  3. The condition direction comes from a frozen random matrix P, which has no link to which acoustic subspace is relevant to the text, so it is just a fixed random direction for each condition.

Questions for authors
  1. The noise frames are i.i.d., so why should the audio–noise cost be robust to time shifts? As the time coordinate only encodes frame position, not acoustic events, how could the time-augmented cost affect the temporal structure of the generated audio?

  2. How does the projection P point to a text-relevant acoustic subspace? What happens if P is replaced with a different random matrix, or with a random direction per sample that ignores the condition?

  3. Is MeanAudio trained with its original MeanFlow objective or with standard flow matching? How do the numbers of Meanaudio compare to those in the original MeanAudio paper?

  4. Could the authors report the mean and variance over several training seeds?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.