acceptodds
Under review as a conference paper at ICLR 2027

UniAVC: Unified Audio-Video Creation via Progressive Training and Asymmetric Scheduling

Abstract

Unified audio-video creation requires a single model capable of text-conditioned joint synthesis, bidirectional cross-modal generation, and instruction-guided editing across both modalities. Existing methods rely heavily on paired corpora, which are scarce for joint generation and even rarer for editing. Consequently, they are spent learning single-branch generative and editing capabilities that abundant unimodal data could readily teach, rather than on the cross-modal coupling. Furthermore, prevailing systems apply a single shared noise schedule to both modalities, neglecting two critical limitations. First, due to the significant discrepancy in information density, video requires prolonged high noise to organize spatial and motion structure, whereas the sparser audio signal can settle earlier during the synthesis process, the dynamics a uniform schedule might compromise. Second, lockstep denoising leaves asymmetric workflows, such as conditioning on one modality to synthesize or edit another, untrained. We present **UniAVC**, which addresses both obstacles. It first trains each branch on unimodal dataset to full generative and editing competence, and subsequently dedicates paired data to exploit cross-modal correspondence. Modality-decoupled schedulers meanwhile warp the noise schedule independently from a shared timestep, while subtasks with one clean modality directly supervise asymmetric configurations. These settings respectively enable intent-aware editing, which preserves one modality intact while enlisting it to constrain the other, and post-inference self-refinement, which regenerates only an unsatisfactory branch against an accepted one. UniAVC thus spans multiple creation tasks from a single set of weights without task-specific adapters. Extensive experiments demonstrate that UniAVC achieves versatile audio-video generation and editing with competitive performance across different operating modes.

Primary area:
generative models
Keywords:
Audio-Video Editing, Audio-Video Generation, Unified Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_FtdYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

UniAVC unifies audio-video generation and editing in one model. It first trains unimodal branches, then freezes them while learning cross-modal bridges from paired data. Asymmetric noise schedules and clean-branch supervision support joint generation, bidirectional cross-modal generation, selective editing, and branch refinement. Experiments on JAVEditBench, InsAVE-80K, Verse-Bench, and VGGSound report competitive performance across these tasks, with gains on several editing and cross-modal generation metrics. These results illustrate the breadth of the unified framework.

Critical strengths
  1. The system addresses a consequential workflow problem with broad capability coverage. One checkpoint supports joint generation, editing, and both cross-modal generation directions. In Table 2, UniAVC improves JAVEditBench instruction compliance and video fidelity to 4.42 and 4.48, respectively, and achieves an InsAVE-80K FVD of 150.86. Table 3 also reports strong cross-modal generation results. This makes the contribution practically interesting even without leading every individual metric.

  2. The separation of unimodal learning from paired alignment, together with explicit clean-branch supervision, is well motivated. Training the bridges on paired data after establishing unimodal competence is a plausible way to use limited paired supervision. Pinning the preserved branch also provides a concrete mechanism for selective editing, rather than relying on approximate reconstruction of that branch. Table 4 provides initial evidence that the progressive recipe and specialized inference modes improve editing quality.

Critical weaknesses
  1. The central scheduling claim is not sufficiently isolated by the ablations. Table 4 compares the asymmetric schedule (2,9) against (1,1), changing both asymmetry and the amount of schedule shifting. It does not establish whether the gain comes from asymmetry rather than a stronger shared shift. Moreover, clean-branch supervision directly trains the boundary conditions required for selective generation, but its contribution is not separated from the interior schedule. The single-stage/progressive comparison also needs a clearer account of matched data, updates, and trainable parameters. These distinctions matter because the paper attributes its benefits specifically to asymmetric scheduling and the staged allocation of paired supervision.

  2. The empirical protocol needs reconciliation before the comparative and data-efficiency claims are reliable. The MiniMax-H3 aggregate JAVEditBench results differ between Tables 2 and A2: for example, AVS is 0.2307 versus 0.3009, and AVQ is 3.61 versus 3.85. Appendix A1.1 also describes alignment training as using the held-out split of InsAVE-80K, while Section 4.1 describes evaluation on its held-out split. This wording does not prove overlap, but leaves train/test separation unclear. Until the results and split definitions are reconciled, the reported advantages are difficult to audit.

Questions for authors
  1. Can you compare asymmetric scheduling with shared shifted schedules such as (2,2) and (9,9), and with independently sampled modality noise levels, while keeping clean-branch subtasks and the rest of training fixed? Separately, what changes when clean-branch supervision is removed? These controls would determine whether the claimed scheduling contribution is supported.

  2. Please specify the exact InsAVE-80K training/evaluation split identifiers and whether source clips are disjoint. Please also clarify the matched data/update/parameter budgets in the single-stage versus progressive ablation, and reconcile the sampling probabilities with the numbers of samples seen in Table A1. This would determine how much confidence to place in the paired-data-efficiency claim.

  3. Which MiniMax-H3 results in Tables 2 and A2 are correct, and do both tables use the same checkpoint, examples, and inference protocol? For Table 2, please specify whether UniAVC uses self-refinement and how its total denoising budget compares with the baselines.

7 comments
AuthorsOpenReview ↗

Response to Reviewer FtdY (1/2)

We thank the reviewer for the constructive feedbacks and questions. We hereby address each of them below.

[W1 & Q1] Asymmetry versus shift magnitude.

We agree that the original comparison confounded asymmetry with shift magnitude. We therefore conduct additional experiments following your suggested settings to isolate the two factors, holding clean-branch subtasks and everything else fixed. Three findings stand out:

  1. Symmetric pairs (2,2)(2,2) and (9,9)(9,9), which vary shift magnitude while keeping the two schedules symmetric, both underperform our (2,9)(2,9) across all metrics, so asymmetry rather than magnitude drives the gain;
  2. Reversing the ordering to (9,2)(9,2), which keeps our shift magnitude but makes video resolve first, degrades AQ the most (2.31 vs 2.59), confirming that audio requires early temporal anchoring while video latents, which carry higher spatial redundancy, denoise over a longer trajectory;
  3. Even (3,12)(3,12), which uses strictly larger shifts than our (2,9)(2,9), still scores lower across all metrics, so the benefit is not simply from increasing shift strength.

Beside, we argue that independently sampled noise levels form a separate training regime rather than a variant of our schedule: uncoupled draws break the deterministic link between the noise pair and the step counter that our sampler and inference modes rely on (§2, p.4, L171–L189), and let the leading modality flip freely, spending about half of training on video-first states that our (9,2)(9,2) control shows to be detrimental. Additionally, Unison [1] also reports that fully decoupled timesteps destabilise optimisation without a curriculum. All three controls are reported below and have been added to Table 4 of §4.3.

SettingVQ$\uparrow$AQ$\uparrow$AVS$\uparrow$IC$\uparrow$VF$\uparrow$AVQ$\uparrow$
(ka,kv)=(1,1)(k_a,k_v)=(1,1)0.05892.480.25454.344.353.67
(ka,kv)=(2,2)(k_a,k_v)=(2,2)0.05922.500.26074.394.413.80
(ka,kv)=(9,9)(k_a,k_v)=(9,9)0.05922.480.26084.414.403.77
(ka,kv)=(9,2)(k_a,k_v)=(9,2)0.05912.310.25204.364.403.58
(ka,kv)=(3,12)(k_a,k_v)=(3,12)0.05922.510.26184.414.403.80
(ka,kv)=(2,9)(k_a,k_v)=(2,9)0.05992.590.29754.424.483.85

[W1 & Q1] Clean-branch supervision.

We conduct another training for the model without clean-branch subtasks while holding all other settings fixed. Since these boundary-state samples are what place intent-aware editing and self-refinement inside the training distribution, neither mode is available to this established variant (3), so we compare it against our Base variant (2) with clean-branch subtasks included during training but using identical basic inference mode.

Clean-branchSelf-refinementVQ$\uparrow$AQ$\uparrow$AVS$\uparrow$IC$\uparrow$VF$\uparrow$AVQ$\uparrow$
(1)✓✓0.05992.590.29754.424.483.85
(2)✓✗0.05842.370.26434.224.313.68
(3)✗✗0.05792.360.25454.234.273.55

AVS and AVQ show significant drops under Base mode, indicating that clean-branch subtasks implicitly help strengthen cross-modal alignment learning without affecting unimodal qualities (VQ, AQ). The Full variant (1), with both inference modes enabled, marks the ceiling that removing the subtasks places out of reach. The schedule and clean-branch supervision are therefore jointly necessary: the schedule provides the asymmetric trajectory while the subtasks make the boundary states trainable (p.3, L138–L140).

AuthorsOpenReview ↗

Response to Reviewer FtdY (2/2)

[W1 & Q2] Single-stage budget.

The single-stage baseline is initialised from the same LTX-2.3 checkpoint with all parameters trainable and finetuned in one stage over the union of the unimodal and paired corpora, with sampling probabilities re-calibrated to match the numbers of samples seen in Table A1, so both paradigms consume identical data exposures and differ only in curriculum (§4.3, p.8, L425–L430). A unimodal sample contributes loss on its own modality while gradients still update the whole model. With the budget fixed, we attribute the performance gap to the optimisation conflict between unimodal and cross-modal objectives inside a mixed batch (§4.3, p.8 L431 to p.9 L474), together with the scarcity of paired supervision (§1 C1, p.2, L77–L87). Given the target to tackle pair-wise data scarcity, we demonstrate that our progressive paradigm is particularly effective for this adverse data situation, and leave paired-data scaling studies for future work (§A2.4, p.18, L968–L970).

[W2 & Q2 & Q3] InsAVE-80K and Table 2 versus Table A2.

Both are human errors on our side, and we thank the reviewer for catching them. The official InsAVE-80K splitting is 79K training and 1K evaluation pairs, with source clips disjoint between the two. Alignment finetuning uses only the 79K training pairs, and the 1K evaluation pairs are used solely for assessment (§A1.1, p.16, L850–L852). On the tables, Table A2's per-task columns had been carried over from an earlier preliminary run instead of the run behind Table 2. Table A2 is now corrected and agrees with Table 2 throughout (p.21), and the corrected rows can also be seen below:

MetricMethodOverallSub. EditBG. EditSpeechSub. AddSub. Rm
VQ$\uparrow$MiniMax-H30.06050.06090.06180.06140.06180.0557
VQ$\uparrow$UniAVC0.05990.06370.06340.06310.06190.0423
AQ$\uparrow$MiniMax-H32.422.322.312.692.572.25
AQ$\uparrow$UniAVC2.592.592.522.822.572.40
AVS$\uparrow$MiniMax-H30.23070.23000.20460.29980.21110.1643
AVS$\uparrow$UniAVC0.29750.32030.28290.32290.26720.0579
IC$\uparrow$MiniMax-H34.114.084.374.404.123.48
IC$\uparrow$UniAVC4.424.354.474.534.404.36
VF$\uparrow$MiniMax-H34.174.104.334.233.884.29
VF$\uparrow$UniAVC4.484.654.234.534.364.56
AVQ$\uparrow$MiniMax-H33.613.753.673.833.583.08
AVQ$\uparrow$UniAVC3.854.153.974.004.162.76

All comparisons and observations were always based on Table 2, so they remain valid and consistent.

[Q3] Table 2 protocol and denoising budget.

In Table 2, UniAVC uses Full mode with self-refinement enabled. We leverage human inspection to subjectively determine which stream is satisfactory and which necessitates refinement. When needed, a single refinement round is applied to only the target stream while the accepted one is preserved, adding a cost of one extra denoising trajectory on that branch, equivalently a 50% increase in total NFE at the default λ=0.5\lambda=0.5 (§A2.3, p.18, L956–L958). We present self-refinement as an optional trade-off, and note that the incurred latency is already acknowledged as a limitation of our model (§A2.4, p.18, L963–L965). However, we highlight that the Base row in Table 4, with neither inference mode enabled, already surpasses the SOTA baselines such as MiniMax-H3 on instruction compliance (4.22 vs 4.11), video fidelity (4.31 vs 4.17), synchrony (0.2643 vs 0.2307) and overall audio-video quality (3.68 vs 3.61) while evaluating at a similar denoising budget.

We thank the reviewer once again for their careful reading and constructive feedback, which has significantly strengthened the paper.

Reference

[1] Unison: Harmonizing Motion, Speech, and Sound for Human-Centric Audio-Video Generation - ECCV 2026

Thank you for the additional experiments and clarifications. The shifted symmetric schedules and clean-branch ablation meaningfully strengthen the evidence, and the split clarification and corrected Table A2 address my original concerns about those reporting inconsistencies. I also appreciate the Base results and the Qwen3-Omni selector evaluation.

However, the interpretation of the headline comparison remains insufficiently established. Table 2 uses human-selected refinement with additional denoising, whereas the Base and automatic-selector results describe different operating conditions. Their promising scores do not fully resolve how selection and computation affect the reported advantage. Please specify which results use human or automatic selection, what information the selector receives, and the total inference cost, including selection and refinement. Identifying which baseline comparisons use comparable budgets would help establish the practical advantage of the method.

The training attribution also remains unresolved: matching data exposures does not isolate curriculum order when the single-stage baseline updates all parameters but the progressive recipe freezes unimodal branches during alignment. This supports a comparison of complete recipes, rather than a curriculum-only conclusion. These remaining issues affect the interpretation of the main comparative gains and the specific methodological contribution. I therefore maintain my rating of 2, while remaining open to reconsideration after clarification.

Reviewer_SWeEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents UniAVC, a unified framework for text-conditioned audio-video generation, bidirectional cross-modal generation, and instruction-guided editing using a single set of model weights. Built on LTX-2.3, it adopts progressive training: first developing generation and editing capabilities on unimodal data, then freezing the modality-specific branches and learning cross-modal alignment using paired data. Modality-specific noise schedules allow audio to become clean earlier than video, while clean-branch training supports modality-preserving editing and selective self-refinement.

Critical strengths

A coherent framework covering a broad range of audio-video creation tasks. UniAVC supports joint generation, bidirectional cross-modal generation, and instruction-guided editing using one set of weights without task-specific adapters. The progressive training strategy offers a sensible separation between learning unimodal capabilities and learning cross-modal alignment.

Critical weaknesses

Table 1 reports 89.2 million unimodal sample exposures before alignment on 0.67 million paired samples. However, the single-stage baseline in Table 4 is not described sufficiently to establish whether it receives comparable data, optimization steps, and computational resources. Consequently, the observed improvements cannot be confidently attributed to separating unimodal learning from cross-modal alignment, rather than differences in training exposure. A matched-budget comparison and a paired-data scaling study would more directly substantiate the claimed data efficiency.

Table 4 compares (ka,kv)=(1,1)(k_a,k_v)=(1,1), (3,12)(3,12), and (2,9)(2,9). These configurations change both the relationship between modalities and the noise-level distribution seen by each branch. Therefore, the results do not establish whether the gains specifically arise from audio becoming clean before video. Additional symmetric shifted schedules, a reversed ordering, and independently sampled modality noise levels would help distinguish the proposed mechanism from more general benefits of timestep reweighting.

Section 3 introduces partner-noise-conditioned cross-modal gates, an asymmetric reference-attention mask, and explicit clean-branch subtasks. Yet Table 4 only ablates the overall training paradigm, inference modes, and scheduler settings. In particular, clean-branch subtasks directly expose the model to the boundary conditions used for cross-modal generation and modality-preserving editing, but their contribution is not separated from that of asymmetric scheduling. Component-level ablations are needed to establish which mechanisms are essential to the reported capabilities.

Self-refinement performs additional denoising after an initial output has been generated. Although Table 4 reports improvements, the main paper do not provide corresponding latency, total model evaluations, or comparisons against alternatives using equivalent computation, such as longer sampling or repeated generation. The protocol for selecting the accepted modality during evaluation is also unclear. Without these details, it is difficult to assess the practical advantage of selective regeneration over simpler inference-time strategies.

Questions for authors
  1. How exactly is the single-stage baseline in Table 4 trained, including its initialization, access to unimodal data, trainable parameters, sample exposures, and total compute? Does progressive training retain its advantage when these factors are matched?

  2. Can the authors separate the effects of asymmetric scheduling and clean-branch supervision through a factorial ablation, including shifted symmetric schedules and removal of clean-branch subtasks? How do these changes affect cross-modal generation and modality-preserving editing, beyond the aggregate joint-editing scores?

4 comments
AuthorsOpenReview ↗

Response to Reviewer SWeE (1/2)

We thank the reviewer for the constructive feedbacks and questions. We hereby address each of them below.

[W1 & Q1] Single-stage baseline and matched budgets.

We clarify that the matched-budget comparison is already what the main paper runs. The single-stage baseline is initialised from the same LTX-2.3 checkpoint with all parameters trainable and finetuned in one stage over the union of the unimodal and paired corpora, with sampling probabilities re-calibrated to match the numbers of samples seen in Table A1, so both paradigms consume identical data exposures and differ only in curriculum (§4.3, p.8, L425–L430). A unimodal sample contributes loss only on its own modality while gradients still update the whole model. We agree a paired-data scaling study would substantiate the training efficiency claim more directly. However, given the target to tackle the contemporary challenge of pair-wise data scarcity, we demonstrate that our progressive paradigm is particularly effective for this adverse data situation, and leave the paired-data scaling study for future work (§A2.4, p.18, L968–L970).

[W2 & Q2] Schedule controls.

We agree that the original comparison confounded asymmetry with shift magnitude. We therefore conduct additional experiments following your suggested settings to isolate the two factors, holding clean-branch subtasks and everything else fixed. Three findings stand out:

  1. Symmetric pairs (2,2)(2,2) and (9,9)(9,9), which vary the shift magnitude while keeping the two schedules symmetric, both underperform our (2,9)(2,9) across metrics, so asymmetry rather than magnitude drives the gain;
  2. Reversing the ordering to (9,2)(9,2), which keeps our shift magnitudes but makes video resolve first, degrades AQ the most, confirming the physical intuition that audio-first directional asymmetry matters because video latents carry higher spatial redundancy and require slower denoising trajectories, whereas audio requires early temporal anchoring;
  3. Even (3,12)(3,12), which uses strictly larger shifts than our (2,9)(2,9), still scores lower, so the benefit is not simply from increasing shift strength.

Besides, we argue that independently sampled noise levels form a separate training regime rather than a variant of our schedule: uncoupled draws break the deterministic link between the noise pair and the step counter that our sampler and inference modes rely on (§2, p.4, L171–L189), and let the leading modality flip freely, spending about half of training on video-first states that our (9,2)(9,2) control shows to be detrimental. Unison [1] also reports that fully decoupled timesteps destabilise optimisation without a curriculum. All results are shown below and have been added to Table 4 of §4.3 in the revised version, together with the accompanying discussion.

SettingVQ$\uparrow$AQ$\uparrow$AVS$\uparrow$IC$\uparrow$VF$\uparrow$AVQ$\uparrow$
(ka,kv)=(1,1)(k_a,k_v)=(1,1)0.05892.480.25454.344.353.67
(ka,kv)=(2,2)(k_a,k_v)=(2,2)0.05922.500.26074.394.413.80
(ka,kv)=(9,9)(k_a,k_v)=(9,9)0.05922.480.26084.414.403.77
(ka,kv)=(9,2)(k_a,k_v)=(9,2)0.05912.310.25204.364.403.58
(ka,kv)=(3,12)(k_a,k_v)=(3,12)0.05922.510.26184.414.403.80
(ka,kv)=(2,9)(k_a,k_v)=(2,9)0.05992.590.29754.424.483.85
AuthorsOpenReview ↗

Response to Reviewer SWeE (2/2)

[W3 & Q2] Component attribution.

We conduct another training for the model without clean-branch subtasks while holding all other settings fixed, reported as Tab. A3 (p.21). Since these boundary-state samples are what place intent-aware editing and self-refinement inside the training distribution, neither mode is available to this established variant (3), so we compare it against our Base variant (2) with clean-branch subtasks included during training but using identical basic inference mode.

Clean-branchSelf-refinementVQ$\uparrow$AQ$\uparrow$AVS$\uparrow$IC$\uparrow$VF$\uparrow$AVQ$\uparrow$
(1)✓✓0.05992.590.29754.424.483.85
(2)✓✗0.05842.370.26434.224.313.68
(3)✗✗0.05792.360.25454.234.273.55

AVS and AVQ show significant drops under Base mode, indicating that clean-branch subtasks implicitly help strengthen cross-modal alignment learning without affecting unimodal qualities (VQ, AQ). The Full variant (1), with both inference modes enabled, marks the ceiling that removing the subtasks places out of reach.

[W4] Self-refinement cost and alternatives.

We clarify that when a stream is deemed unsatisfactory, a single refinement round is applied to that stream only while the accepted one is preserved, adding a cost of one extra denoising trajectory on that branch, equivalently a 50% increase in total NFE at the default λ=0.5\lambda=0.5 (§A2.3, p.18, L956–L958). The selection for refinement is made by human subjective inspection (§A2.3, p.18, L949–L951), yet we empirically find that Qwen3-Omni serves as a reliable automatic judge for the same decision, as shown below and in Tab. A3 (p.21).

InspectorVQ$\uparrow$AQ$\uparrow$AVS$\uparrow$IC$\uparrow$VF$\uparrow$AVQ$\uparrow$
Human0.05992.590.29754.424.483.85
Qwen3-Omni0.05952.500.28544.244.463.82

We note that longer sampling runs both branches and repeated generation discards the accepted stream and starts afresh, so neither provides the intended controllability. We present self-refinement as an optional trade-off, and note that the incurred latency is already acknowledged as a limitation of our model (§A2.4, p.18, L963–L965). Crucially, we highlight that the Base row in Table 4, with neither inference mode enabled, already surpasses the SOTA baselines such as MiniMax-H3 on instruction compliance (4.22 vs 4.11), video fidelity (4.31 vs 4.17), synchrony (0.2643 vs 0.2307) and overall audio-video quality (3.68 vs 3.61) while evaluating at a similar denoising budget.

We thank the reviewer once again for their careful reading and constructive feedback, which has significantly strengthened the paper.

Reference

[1] Unison: Harmonizing Motion, Speech, and Sound for Human-Centric Audio-Video Generation - ECCV 2026

Thanks for authors' response. Some of my concerns are resolved. I have a question about the generality of the method. The authors claim the method is better than Minimax-H3. It seems that H3 is treated as the best baseline. Why not train the model based on H3? Does the method work on H3?

AuthorsOpenReview ↗

Follow-up Response to Reviewer SWeE

We thank the reviewer for the follow-up and are glad that some of the concerns have been resolved. We address the question on generality below.

Why not train the model based on MiniMax-H3.

Our work started before MiniMax-H3 was released, which is why we treat it as concurrent work (§2, p.4) and include it as a baseline rather than a backbone. We selected LTX-2.3 because at that time it was arguably the strongest open-source joint audio-video generation model, and designed our method for its dual-stream architecture. Training on MiniMax-H3 would also be considerably more expensive, given MiniMax-H3 has 33B parameters against the 22B of LTX-2.3.

Does the method work on MiniMax-H3.

Our method is not tied to LTX-2.3, but it does rely on a dual-stream design. Progressive training first trains each branch on its own unimodal corpus and then learns only the cross-modal bridges between them (§3.3, p.5). MiniMax-H3 instead processes both modalities within a single dense omni-transformer. We therefore do not expect the method to transfer to it directly. However, we do expect it to carry over to other dual-stream DiTs in the future. Finally, since MiniMax-H3 demonstrates more robust performance than LTX-2.3, we aim to design a suitable approach for it and other dense omni-transformers in future work too.

We thank the reviewer again and welcome any further questions.

Reviewer_tWBKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper uses LTX-2.3 to develop a unified model for audio-video generation, including V2A, A2V, joint generation and instruction-based editing. It adopts a two-stage training strategy: each branch first learns unimodal generation and editing capabilities from large-scale unimodal data, and then the cross-modal modules are trained using a smaller paired audio-video dataset. The method also introduces asymmetric noise scheduling to account for differences in representation density and denoising dynamics between video and audio. The reported results show competitive performance across multiple generation and editing tasks.

Critical strengths
  1. The paper is well-written and easy to follow.
  2. The paper addresses a meaningful problem. Unifying multiple generation and editing tasks within a single model could simplify practical content creation workflows. The proposed design is generally reasonable and consistent with its motivation.
  3. The reported results demonstrate competitive performance across joint audio-video generation, bidirectional cross-modal generation, and editing tasks.
Critical weaknesses
  1. The technical novelty of the asymmetric scheduling is limited. MOVA[1] already employs modality-specific sigma shifts in the first phase of joint training, using different settings for video and audio. Thus, the general principle of assigning asymmetric noise configurations to the two modalities has already been explored. Unison[2] also investigates cross-modal guidance through decoupled denoising schedules, allowing the cleaner modality to guide the noisier one. The technical contribution beyond these existing designs needs to be clarified.
  2. There are several inconsistencies in the manuscript that require clarification. For example, the overall MiniMax-H3 scores differ between Table 2 and Table A2. In addition, Line 851 lists the “held-out split of InsAVE-80K” as part of the alignment training data, while the main text states that evaluation uses the held-out evaluation split of this dataset.

[1] MOVA: Towards Scalable and Synchronized Video–Audio Generation

[2] Unison: Harmonizing Motion, Speech, and Sound for Human-Centric Audio-Video Generation

Questions for authors
  1. The paper describes self-refinement as regenerating an unsatisfactory modality while retaining the satisfactory one. How is an unsatisfactory output identified in practice? Is the decision made through human inspection, an automatic metric, or a predefined rule? In particular, how are the refinement decision and the target modality determined in the reported experiments?
  2. For the single-stage baseline in the training-paradigm ablation in Table 4, is training performed only on paired audio-video data?
  3. For the metrics in Table A2, what is the relationship between the overall scores and the per-category scores? I noticed that, for some metrics, the overall score is higher than all the corresponding per-category scores, which seems somewhat counterintuitive.
4 comments
AuthorsOpenReview ↗

Response to Reviewer tWBK (1/2)

We thank the reviewer for the constructive feedbacks and questions. We hereby address each of them below.

[W1] Novelty relative to MOVA and Unison.

Our novelty relies on two core pillars: (1) Deterministic Asymmetric Coupling during training and sampling, and (2) Boundary Supervision enabled by this coupling. We do not claim the general philosophy of asymmetric scheduling concept as our novelty, and we already discuss related works that follow it (§2, p.4, L171–L189). There is as yet no universally optimal design for it, and our difference lies in how the noise levels of the two branches are coupled:

  • MOVA explores timestep decoupling for joint audio-video generation by independently sampling tvt_v and tat_a from U(0,1)\mathcal{U}(0,1) throughout training. Phase 1 uses asymmetric shifts (shiftv=5.0,shifta=1.0\mathrm{shift}_v=5.0, \mathrm{shift}_a=1.0) but Phase 2 aligns shifta\mathrm{shift}_a to 5.05.0, making the operating configuration effectively symmetric. The two draws remain uncoupled across both phases so the noise pair (σa,σv)(\sigma_a,\sigma_v) varies at random from sample to sample with no fixed direction. Unison also decouples the timesteps with a curriculum that starts from synchronous warmup (tv=tat_v=t_a) in Phase I, introduces bounded decoupling (∣tv−ta∣≤0.25|t_v-t_a|\le0.25) with loss reweighting on the noisier branch in Phase II, and removes the cap entirely in Phase III with unconstrained updates, so the direction of guidance changes freely between samples. Neither model therefore maintains a fixed asymmetric direction throughout the training distribution. We instead fix the shift pair at (ka,kv)=(2,9)(k_a,k_v)=(2,9) throughout: one shared step index always maps to a prescribed asymmetric pair with audio consistently resolving ahead of video, and the same pair is used at sampling (Fig. 2). The comparison with MOVA and Unison has been expanded in the Non-uniform Noise Schedules paragraph of §2 (p.4, L171–L189).

  • Beyond scheduling, the critical contribution is boundary supervision. Neither MOVA nor Unison trains on the states where one stream is fully clean while the other is heavily noised, because their randomly drawn or curriculum-scheduled pairs never specifically reach those extremes. Clean-branch subtasks explicitly train those boundary states, which allows the model to preserve one stream at full fidelity while altering or generating the other (p.3, L138–L140). Prior works hardly attempt modality-preserving editing and are hence susceptible to unwanted changes in the preserved stream. In addition, we are one of, if not the first, capable of joint generation, bidirectional cross-modal generation and editing of either or both modalities within a single set of weights (p.3, L132–L134).

[W2(a) & Q3] Table 2 vs Table A2, and the Overall column.

This was a human error on our side and we thank the reviewer for catching it. The per-task columns of Table A2 had been carried over from an earlier run under different settings instead of being taken from the run behind Table 2, which also explains your counterintuitive observation of an Overall value outside the range of its own per-task values. Table A2 is now corrected, agrees with Table 2 in every cell both report (p.21), and the corrected rows can be seen below:

MetricMethodOverallSub. EditBG. EditSpeechSub. AddSub. Rm
VQ$\uparrow$MiniMax-H30.06050.06090.06180.06140.06180.0557
VQ$\uparrow$UniAVC0.05990.06370.06340.06310.06190.0423
AQ$\uparrow$MiniMax-H32.422.322.312.692.572.25
AQ$\uparrow$UniAVC2.592.592.522.822.572.40
AVS$\uparrow$MiniMax-H30.23070.23000.20460.29980.21110.1643
AVS$\uparrow$UniAVC0.29750.32030.28290.32290.26720.0579
IC$\uparrow$MiniMax-H34.114.084.374.404.123.48
IC$\uparrow$UniAVC4.424.354.474.534.404.36
VF$\uparrow$MiniMax-H34.174.104.334.233.884.29
VF$\uparrow$UniAVC4.484.654.234.534.364.56
AVQ$\uparrow$MiniMax-H33.613.753.673.833.583.08
AVQ$\uparrow$UniAVC3.854.153.974.004.162.76

Regarding the Overall column, we follow the JAVEditBench protocol. It is the mean over all test videos with a valid score, given by Overall=∑tntmt/∑tnt\mathrm{Overall} = \sum_t n_t m_t / \sum_t n_t, where ntn_t is the number of valid videos in task tt and mtm_t is that task's mean. Each per-task column applies the same function within its task, so Overall always falls inside the per-task range, which every row of Table A2 now satisfies. As all comparisons and observations were based on Table 2, they remain valid and consistent.

AuthorsOpenReview ↗

Response to Reviewer tWBK (2/2)

[W2(b)] InsAVE-80K.

We also thank the reviewer for catching this. We clarify that the official splitting is 79K training and 1K evaluation pairs, with source clips disjoint between the two. Alignment finetuning uses only the 79K training pairs, and the 1K evaluation pairs are used solely for assessment. It has been corrected in the revised version accordingly (§A1.1, p.16, L850–L852).

[Q1] How the refinement target is determined.

We leverage human inspection to subjectively determine which stream is satisfactory and which necessitates refinement. When needed, a single refinement round is applied to only the target stream while the accepted one is preserved, adding a cost of one extra denoising trajectory on that branch, equivalently a 50% increase in total NFE at the default λ=0.5\lambda=0.5 (§A2.3, p.18, L956–L958). Besides, we conduct an additional experiment and empirically find that Qwen3-Omni can be adopted for reliable automatic judgement here (evaluating full holistic audio-video sequences against prompt conditions), aligning with JAVEdit's finding of strong positive Spearman correlations with human preference across the judge-based metrics. The two routes agree closely:

InspectorVQ$\uparrow$AQ$\uparrow$AVS$\uparrow$IC$\uparrow$VF$\uparrow$AVQ$\uparrow$
Human0.05992.590.29754.424.483.85
Qwen3-Omni0.05952.500.28544.244.463.82

[Q2] Single-stage baseline.

Both training paradigms are conducted with identical data exposures, and the single-stage baseline simply interleaves all of it in one stage. During training, a unimodal sample contributes loss only on its own modality while gradients still update the whole model, so the only difference from progressive training is the curriculum. With the budget fixed, we attribute the performance gap to the optimisation conflict between the unimodal and cross-modal objectives inside a mixed batch (§4.3, p.8 L431 to p.9 L474), together with the scarcity of paired supervision that interleaving spends on competence the unimodal corpora already supply (§1 C1, p.2, L77–L87). This demonstrates that our progressive paradigm is particularly effective for this adverse data situation. We leave recipes that scale further as paired data grows to future work (§A2.4, p.18, L968–L970).

We thank the reviewer once again for their careful reading and constructive feedback, which has significantly strengthened the paper.

Thank you for the detailed response and clarifications. The authors have addressed most of my concerns, and I will raise my score accordingly.

AuthorsOpenReview ↗

Reply to Reviewer tWBK

Thank you for taking the time to revisit our submission and for raising your score. We are glad the revisions addressed your concerns. Should any further questions arise during the discussion, we remain happy to respond. We appreciate your thorough and constructive reviewing.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.