acceptodds
Under review as a conference paper at ICLR 2027

OmniNFT: Modality-wise Omni Diffusion Reinforcement for Joint Audio-Video Generation

Abstract

Recent advances in joint audio-video generation have been remarkable, yet real-world applications demand strong per-modality fidelity, cross-modal alignment, and fine-grained synchronization. Reinforcement Learning (RL) offers a promising paradigm, but its extension to and joint audio-video generation remains unexplored. Notably, our in-depth analysis first reveals that the primary obstacles to applying RL in this stem from: multi-objective advantages inconsistency, where the advantages of multimodal outputs are not always consistent within a group; multi-modal gradients imbalance, where video-branch gradients leak into shallow audio layers responsible for intra-modal generation; uniform credit assignment, where fine-grained cross-modal alignment regions fail to get efficient exploration. These shortcomings suggest that vanilla RL fine-tuning strategy with a single global advantage often leads to suboptimal results. To address these challenges, we propose OmniNFT, a novel modality-aware online diffusion RL framework with three key innovations: Modality-wise advantage routing, which routes independent per-reward advantages to their respective modality generation branches. Layer-wise gradient surgery, which selectively detaches video-branch gradients on shallow audio layers while retaining those for cross-modal interaction layers. Region-wise loss reweighting, which modulates policy optimization toward critical regions related to audio-video synchronization and fine-grained alignment. Extensive experiments on JavisBench and VBench with the LTX-2 backbone demonstrate that OmniNFT achieve comprehensive improvements in audio and video perceptual quality, cross-modal alignment, and audio-video synchronization.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Reinforcement Learning, Joint Audio-Video Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_w3e9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies RL fine-tuning for joint audio-video generation. The authors identify three obstacles. Video and audio rewards often disagree for the same sample. Video gradients leak into shallow audio layers through cross-attention. A uniform loss also ignores sound-emitting regions. Building on DiffusionNFT, the proposed OmniNFT addresses these issues with three components. Modality-wise advantage routing computes one advantage per reward and sends each to the branch it evaluates, while the synchronization advantage goes to both branches. Layer-wise gradient surgery scales down the video-to-audio gradients in the shallow A2V cross-attention blocks. Region-wise loss reweighting uses V2A attention maps to up-weight sound-related video tokens. Experiments on LTX-2 and MOVA show gains over the base model and GDPO on JavisBench and VBench. Lip-sync, ASR, and human preference evaluations further support these results.

Critical strengths
  1. Modality-wise advantage routing is the main contribution: Sending each advantage to the branch that the corresponding reward evaluates is well suited to RL for joint audio-video generation. Table 3 isolates this component, where DeSync improves from 0.412 to 0.322. This is the main reason for my positive rating.

  2. The human study also prefers OmniNFT over GDPO: This agrees with the automatic metrics and provides a second, independent form of evidence for the reported improvement.

Critical weaknesses

Major weaknesses

W1. The synchronization gain from region-wise reweighting is not explained: This component reweights the video loss only, yet in Table 3 it improves DeSync from 0.334 to 0.269. The paper does not explain how a change confined to the video loss produces a gain in temporal alignment. The mechanism behind a reported synchronization gain is therefore left unaccounted for.

W2. The video evaluation is not comprehensive: Only three VBench dimensions are reported, namely aesthetic quality, imaging quality, and subject consistency. The remaining dimensions are not shown, so the video evaluation is incomplete.

Minor weaknesses

W3. The ablation is not monotonic and the trade-offs are not discussed: In Table 3, gradient surgery slightly worsens DeSync from 0.322 to 0.334, and region-wise reweighting lowers AQ from 5.917 to 5.715. The paper presents the three components as complementary without addressing either regression.

W4. The main baseline is not clearly described: The LTX-2 + GDPO row in Table 1 and the LTX-2 + vanilla RL row in Table 3 report the same numbers. The paper should state which advantage formulation and which reward set this baseline uses. Without that, the size of the reported improvement over the baseline cannot be interpreted.

W5. Related work on multi-reward RL for visual generation is missing: Work on multi-reward RL for image generation and video generation is not discussed. Relevant examples include:

  • He et al., Learning profitable NFT image diffusions via multiple visual-policy guided reinforcement learning, ACMMM 2023.
  • Lee et al., Parrot: Pareto-optimal Multi-Reward Reinforcement Learning Framework for Text-to-Image Generation, ECCV 2024.
  • Li et al., T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model with Mixed Reward Feedback, NeurIPS 2024.
  • Lee et al., Calibrated Multi-Preference Optimization for Aligning Diffusion Models, CVPR 2025.
  • Liu et al., VideoDPO: Omni-Preference Alignment for Video Diffusion Generation, CVPR 2025.
Questions for authors

See Weaknesses above

7 comments
AuthorsOpenReview ↗

Response to reviewer w3e9 - part 1

We sincerely thank you for your thorough review. We appreciate your recognition of modality-wise advantage routing as a core contribution of our paper and your endorsement that routing advantages to their corresponding modality branches is well suited for joint audio-video reinforcement learning.

Below, we respond point-by-point to your comments on mechanism explanation, evaluation comprehensiveness, and related work, detailing the computational pathway of region-wise loss reweighting, additional benchmark evaluations, and how our work relates to prior literature.

W1: The synchronization gain from region-wise reweighting is not explained: This component reweights the video loss only, yet in Table 3 it improves DeSync from 0.334 to 0.269. The paper does not explain how a change confined to the video loss produces a gain in temporal alignment. The mechanism behind a reported synchronization gain is therefore left unaccounted for.

Region-wise loss reweighting improves temporal synchronization because it redistributes shared synchronization feedback across video tokens. Specifically, the shared synchronization advantage AavA_{av} enters the video policy loss through Eq. (7), while the spatial weights wiw_i preserve frame indices and emphasize visual tokens in sound-emitting regions. Consequently, even though the weights are applied solely to the video loss, they directly reshape how the synchronization objective is optimized within the video branch. We further verify spatial correspondence using localization statistics and test whether accurate spatial placement provides additional gains via spatial shuffling. We construct an automated reference benchmark using the union of bounding boxes generated by frozen Gemma3 and GroundingDINO (where Gemma3 extracts sound-source phrases and GroundingDINO detects the corresponding objects), and compare against equal-area random pixel selection:

Region MethodEqual-Area OverlapIoU
Attention-based68.640%54.820%
Random Selection46.150%34.250%

These localization results and visualizations appear in Appendix D.5 (Table 15 and Figure 8) of the revised manuscript.

Our critical-region selection based on V2A cross-attention maps substantially outperforms equal-area random selection, confirming its strong spatial correspondence with visible sound-emitting sources.

We further validate OmniNFT's region-wise reweighting design via an attention-map spatial shuffle control: randomly permuting spatial weight positions within each latent frame while preserving the exact weight distribution, sum, training budget, and all other components, as shown below:

Region SettingVQ ↑AQ ↑JavisScore ↑DeSync ↓
OmniNFT3.332 ± 0.0965.715 ± 0.0700.220 ± 0.0080.270 ± 0.031
OmniNFT − region3.246 ± 0.0835.917 ± 0.0730.209 ± 0.0070.334 ± 0.034
OmniNFT + region shuffle3.008 ± 0.0895.487 ± 0.0780.183 ± 0.0080.417 ± 0.038

These spatial-shuffle results are reported in Section 6.3 (Table 3), with the protocol in Appendix D.3 of the revised manuscript.

These experimental results demonstrate that: after spatial shuffling, VQ(3.008), AQ (5.487) and DeSync (0.417) degrade substantially compared to full OmniNFT, supporting the importance of accurate spatial localization in this reweighting design.

W2: The video evaluation is not comprehensive: Only three VBench dimensions are reported, namely aesthetic quality, imaging quality, and subject consistency. The remaining dimensions are not shown, so the video evaluation is incomplete.

We provide results across all 16 VBench dimensions.

ModelSCBCTFMSDDAesQImgQOCMOHACSRSASTSOC′
LTX-20.8920.9430.9900.9820.7420.6440.6460.8720.6500.9740.7620.7640.5510.2210.2490.263
GDPO0.9280.9410.9810.9790.7440.6810.7270.9260.7540.9780.7740.8120.5330.2190.2450.265
OmniNFT0.9380.9440.9790.9850.6670.6830.7310.9300.7680.9790.7800.8230.5210.2230.2420.268

The complete 16-dimension VBench results are reported in Appendix E.1, Table 16 of the revised manuscript.

Abbreviations: SC = Subject Consistency; BC = Background Consistency; TF = Temporal Flickering; MS = Motion Smoothness; DD = Dynamic Degree; AesQ = Aesthetic Quality; ImgQ = Imaging Quality; OC = Object Class; MO = Multiple Objects; HA = Human Action; C = Color; SR = Spatial Relationship; S = Scene; AS = Appearance Style; TS = Temporal Style; OC′ = Overall Consistency.

AuthorsOpenReview ↗

Response to reviewer w3e9 - part 2

W3: The ablation is not monotonic and the trade-offs are not discussed: In Table 3, gradient surgery slightly worsens DeSync from 0.322 to 0.334, and region-wise reweighting lowers AQ from 5.917 to 5.715. The paper presents the three components as complementary without addressing either regression.

To examine the independent contribution of each component in the full framework, we conduct controlled ablations by removing modality-wise advantage routing (−routing), layer-wise gradient surgery (−surgery), and region-wise loss reweighting (−region) one at a time. The table below summarizes the results of OmniNFT and its ablated configurations across multiple random seeds (reported as mean±std\text{mean} \pm \text{std}).

ConfigurationRoutingSurgeryRegionVQ ↑AQ ↑JavisScore ↑DeSync ↓
OmniNFT✓✓✓3.332 ± 0.0965.715 ± 0.0700.220 ± 0.0080.270 ± 0.031
OmniNFT−routing✗✓✓2.631 ± 0.0815.470 ± 0.0730.209 ± 0.0080.357 ± 0.037
OmniNFT−surgery✓✗✓2.948 ± 0.0895.426 ± 0.0710.200 ± 0.0080.398 ± 0.036
OmniNFT−region✓✓✗3.246 ± 0.0835.917 ± 0.0730.209 ± 0.0070.334 ± 0.034

These ablations and trade-offs are discussed in Section 6.3 (Table 2) and Appendix D.1 (Table 13) of the revised manuscript.

These results support the contributions of the three components within the full framework, while also revealing trade-offs between audio quality and audiovisual synchronization::

  • Modality-wise advantage routing effectively coordinates joint audio-video optimization by directing each branch toward feedback aligned with its own objective;
  • Layer-wise gradient surgery improves the reported VQ, AQ, JavisScore, and DeSync when routing and region reweighting are retained, supporting its contribution within the complete framework.;
  • Region-wise loss reweighting reflects a reasonable trade-off, accepting a localized drop in AQ to secure substantial gains in overall synchronization and multi-dimensional alignment.

W4: The main baseline is not clearly described: The LTX-2 + GDPO row in Table 1 and the LTX-2 + vanilla RL row in Table 3 report the same numbers. The paper should state which advantage formulation and which reward set this baseline uses. Without that, the size of the reported improvement over the baseline cannot be interpreted.

Thank you for pointing out this naming ambiguity. GDPO in Table 1 and vanilla RL in Table 3 refer to the exact same baseline experiment, whose reward weights and normalization formulas are detailed below. We will unify the terminology as GDPO in the revision.

To address the substantial scale differences among multiple training rewards, we adopt a "normalize-first, route-by-modality" strategy: each of the 5 rewards is first standardized independently and then aggregated into three advantage channels—video, audio, and shared synchronization—eliminating scale bias in reward weighting. The exact definitions and coefficients are detailed below:

ScorerScore DefinitionWeightChannel
VideoAlign(VQ+TA)/2(VQ+TA)/2 calibrated with fixed mean/std1Video
HPSv3Frame-wise scoring; mean of top 30% frames as video quality score1.5Video
Audiobox(CE+CU+PQ−PC)/40(CE+CU+PQ-PC)/400.5Audio
CLAPCosine similarity between generated audio and prompt_a embeddings1Audio
DeSync1/(1+d)1/(1+d), where dd is the segment-wise mean of predicted absolute temporal offset1Shared Sync

These reward definitions are specified in Section 5.1 and Appendix C.3 (Table 12) of the revised manuscript.

Following GRPO, the per-reward advantage for each prompt is computed as:

ak,ci=sk,ci−mean⁡({sk,c1,sk,c2,…,sk,cG})std⁡({sk1,sk2,…,skG})+ϵ.a_{k,c}^{i}=\frac{s_{k,c}^{i}-\operatorname{mean}(\{s_{k,c}^{1},s_{k,c}^{2},\ldots,s_{k,c}^{G}\})}{\operatorname{std}(\{s_k^{1},s_k^{2},\ldots,s_k^{G}\})+\epsilon}.

where GG is the rollout group size and ϵ=10−4\epsilon=10^{-4}. We then construct the three advantage channels:

Av=aVA+1.5aHPS,Aa=0.5aAB+aCLAP,Aav=aDeSync.A_v=a_{VA}+1.5a_{HPS},\quad A_a=0.5a_{AB}+a_{CLAP},\quad A_{av}=a_{DeSync}.

For advantage assignment, Av+AavA_v+A_{av} and Aa+AavA_a+A_{av} are routed to the video and audio branches respectively (whereas GDPO assigns the shared sum Av+Aa+AavA_v+A_a+A_{av} to both branches). Both are subsequently clipped to [−5,5][-5,5] and mapped to optimality probabilities via r=0.5+A/10r=0.5+A/10.

W5: Related work on multi-reward RL for visual generation is missing

Thank you for recommending these relevant works. Multi-reward aggregation and preference optimization provide an important foundation for visual generation. Unlike prior methods that focus on unimodal image or video generation, OmniNFT specifically targets the cross-modal coupling structure of joint audio-video generative models. We will clearly distinguish and discuss these works in the revised manuscript.

AuthorsOpenReview ↗

Gentle Reminder for the Discussion

Dear Reviewer w3e9,

We sincerely thank you for taking the time to review our paper. In our rebuttal, we respond to your comments and summarize the additional evidence and revisions.

If you have time, we would greatly appreciate hearing which concerns remain unresolved or where our explanations are still insufficient. Your perspective would help us understand what we may have missed, and we would welcome the opportunity to clarify these points.

Best regards,

Paper 17080 Authors

Reviewer_PiRROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies reward-based post-training of an existing joint audio-video model to improve video, audio, and synchronization together. Building on DiffusionNFT, it makes three changes: video and audio losses receive different combinations of relevant reward feedback; gradients from the video side are attenuated along selected paths into shallow audio layers; and cross-modal attention maps give some visual regions greater weight during training. The main experiments use LTX-2 and evaluate generation quality, content alignment, and synchronization on JavisBench and VBench. The appendices also report results on MOVA, lip synchronization, speech completeness, and human preference.

Critical strengths
  1. The paper addresses a practical issue in joint generation: a sample can have good video but poor audio, and training both outputs with one aggregate score may lose useful feedback. The proposed changes address the scores assigned to each output, the gradients passed through the network, and the visual locations emphasized during training. The approach is clear and seems reasonably easy to apply to existing models.

  2. Several results support the complete method. Compared with the RL-tuned GDPO baseline, Table 1 reduces DeSync from 0.412 to 0.269 and increases JavisScore from 0.185 to 0.220. The second backbone, lip-synchronization tests, and human evaluation in Appendices A and B also support some benefit. Appendix C.2 reports a small increase in training time, from 23.9 to 24.1 hours.

Critical weaknesses
  1. I could not fully determine which training settings are matched between OmniNFT and the comparison method. GDPO in Table 1 and vanilla RL in Table 3 have identical results on all ten metrics. These may simply be two names for the same baseline, but this needs to be stated. Reward composition is another detail that matters for this comparison: Section 6.1 uses five scoring models, while Section 5.1 groups feedback into video, audio, and synchronization rewards without fully specifying the within-group weights or the normalization actually used. Some key training parameters are also missing from Appendix C. This makes it difficult to tell how much of the gain comes from modality-wise routing and how much might be related to the reward combination or training settings. A comparison that holds the other choices fixed would make the effect of routing more convincing.

  2. Table 3 supports the overall combination, but the explanation of why each change works would benefit from more direct checks. Figure 2 mainly shows gradient magnitudes; a larger gradient does not necessarily point against the audio objective. The attention maps in Figure 4 focus on relevant subjects, but do not yet establish that the gains mainly come from identifying the regions most important for synchronization. Table 3 also shows some tradeoffs worth explaining: gradient surgery raises AQ from 5.399 to 5.917, while DeSync increases slightly from 0.322 to 0.334; adding region weighting then improves synchronization but lowers AQ to 5.715. These changes may reflect an ordinary multi-objective tradeoff or some experimental variation. Simple control comparisons and repeated runs would help clarify the role of each component and whether these tradeoffs are stable.

Questions for authors
  1. [Clarifying the comparison] Are GDPO in Table 1 and vanilla RL in Table 3 the same baseline? How are the VideoAlign, HPSv3, Audiobox, CLAP, and DeSync scores combined, weighted, and normalized (Sections 5.1 and 6.1)? Please provide the key training settings for the baseline and OmniNFT, and explain what happens when only the modality-wise routing in Equation (7) is changed while the other choices are held fixed.

  2. [Checking which feedback is harmful] Figure 2 shows that shallow audio layers receive some relatively strong gradients. Is there further evidence that these conflict with the update direction preferred by the audio objective itself? Could a simpler change, such as reducing the audio-branch learning rate or uniformly attenuating incoming video-side gradients across all layers, produce similar gains? This would help assess the added value of selecting particular layers and paths, and interpret the higher audio quality but slightly worse synchronization error in Table 3.

  3. [Checking the importance of location] What happens if the spatial positions or sample assignments of the attention maps are shuffled, while preserving the weight distribution and training budget (Section 5.3 and Table 3)? This could help distinguish the benefit of locating synchronization-critical regions from that of loss reweighting more generally. Does region weighting also help when the sound source is off screen or the audio is mainly background music?

  4. [Checking reliability] Which evaluators produce VQ and AQ, and are they also used during training (Section 6.1)? Table 7 reports a 57.46% overall preference rate against GDPO. Could you give the number of evaluators, judgments per pair, whether ties were allowed, and confidence intervals? Is the Sync-C increase from 5.901 to 5.964 in Table 5 stable across repeated generations? If the results are available, adding GDPO to the speech-completeness comparison in Table 6 would also help isolate the gain over an existing RL method.

9 comments
AuthorsOpenReview ↗

Response to reviewer PiRR - part 1

We sincerely thank you for your careful review and for recognizing that our design across feedback assignment, gradient propagation, and spatial weighting is clean, easily transferable, and incurs minimal overhead. Below, we address your questions regarding baseline definitions, reward formulations, and ablation experiments.

Q1: [Clarifying the comparison] Are GDPO in Table 1 and vanilla RL in Table 3 the same baseline? How are the VideoAlign, HPSv3, Audiobox, CLAP, and DeSync scores combined, weighted, and normalized (Sections 5.1 and 6.1)? Please provide the key training settings for the baseline and OmniNFT, and explain what happens when only the modality-wise routing in Equation (7) is changed while the other choices are held fixed.

(a) Baseline Identity

Thank you for pointing out the naming ambiguity. GDPO in Table 1 and vanilla RL in Table 3 refer to the exact same baseline experiment, with results shared across both tables. We will unify the terminology as GDPO in the revision and explicitly specify its advantage formulation and reward setup.

(b) Reward Formulation and Normalization

To address the substantial scale differences among multiple training rewards, we adopt a "normalize-first, route-by-modality" strategy: each of the 5 rewards is first standardized independently and then aggregated into three advantage channels—video, audio, and shared synchronization—eliminating scale bias in reward weighting. The exact definitions and coefficients are detailed below:

ScorerScore DefinitionWeightChannel
VideoAlign(VQ+TA)/2(VQ+TA)/2 calibrated with fixed mean/std1Video
HPSv3Frame-wise scoring; mean of top 30% frames as video quality score1.5Video
Audiobox(CE+CU+PQ−PC)/40(CE+CU+PQ-PC)/400.5Audio
CLAPCosine similarity between generated audio and prompt_a embeddings1Audio
DeSync1/(1+d)1/(1+d), where dd is the segment-wise mean of predicted absolute temporal offset1Shared Sync

These reward definitions are specified in Section 5.1 and Appendix C.3 (Table 12) of the revised manuscript.

Following GRPO, the per-reward advantage for each prompt is computed as:

ak,ci=sk,ci−mean⁡({sk,c1,sk,c2,…,sk,cG})std⁡({sk1,sk2,…,skG})+ϵ.a_{k,c}^{i}=\frac{s_{k,c}^{i}-\operatorname{mean}(\{s_{k,c}^{1},s_{k,c}^{2},\ldots,s_{k,c}^{G}\})}{\operatorname{std}(\{s_k^{1},s_k^{2},\ldots,s_k^{G}\})+\epsilon}.

where GG is the rollout group size and ϵ=10−4\epsilon=10^{-4}. We then construct the three advantage channels:

Av=aVA+1.5aHPS,Aa=0.5aAB+aCLAP,Aav=aDeSync.A_v=a_{VA}+1.5a_{HPS},\quad A_a=0.5a_{AB}+a_{CLAP},\quad A_{av}=a_{DeSync}.

For advantage assignment, Av+AavA_v+A_{av} and Aa+AavA_a+A_{av} are routed to the video and audio branches respectively (whereas GDPO assigns the shared sum Av+Aa+AavA_v+A_a+A_{av} to both branches). Both are subsequently clipped to [−5,5][-5,5] and mapped to optimality probabilities via r=0.5+A/10r=0.5+A/10.

(c) Matched Training Settings

We summarize the complete training configuration below to facilitate reproducibility. GDPO and OmniNFT share the exact same optimizer, LoRA setup, sampling parameters, and update budget, while OmniNFT additionally enables layer-wise gradient surgery and region-wise loss reweighting.

ItemSetting
OptimizerAdamW; LR 3×10−53\times10^{-5}; betas=(0.9, 0.999); eps 10−810^{-8}; weight decay 10−410^{-4}; no LR scheduler
Gradient / Precision / ParallelismGlobal gradient clipping 1.0; BF16, FSDP, activation checkpointing
LoRArank/alpha = 32/64, Gaussian initialization
LoRA Target ModulesQ/K/V/output projections of self-attention, text cross-attention, and bidirectional AV cross-attention in both branches; net.0.proj and net.2 in FFN
Training Budget400 updates; global batch size 32, rollout group size 8 (16 GPUs ×\times batch size 2 per GPU)
Video Specification512×768512\times768, 121 frames, 24 FPS
Sampling Settings20 steps with video/audio CFG = 1.5/3 during training; 40 denoising steps with CFG = 3/7 during evaluation
NFT Mixing / Reference RegularizationMixing parameter β=1\beta = 1; reference regularization coefficient 10−410^{-4}
Old Policy EMAθold←ηgθold+(1−ηg)θ\theta_{old}\leftarrow\eta_g\theta_{old}+(1-\eta_g)\theta, ηg=min⁡(0.001g,0.5)\eta_g=\min(0.001g,0.5); EMA disabled during extra evaluation
OmniNFT Region Warmupλg=0.5min⁡(g/400,1)\lambda_g=0.5\min(g/400,1); wi=1+λgsigmoid⁡(λgzi)w_i=1+\lambda_g\operatorname{sigmoid}(\lambda_g z_i); weight upper bound increases from 1 to 1.5
Public OmniNFT A2V Context BackwardBlock 0: 0; Blocks 1–10: 0.1; Blocks 40–47: 0.3; Remaining blocks: 1

These training settings are provided in Appendix C.2, Table 11 of the revised manuscript.

AuthorsOpenReview ↗

Response to reviewer PiRR - part 2

(d) Effect of Changing Only Modality-Wise Routing

To examine the role of modality-wise advantage routing both on top of the baseline and within the full framework, we compare four configurations: Routing only enables solely modality-wise advantage routing; OmniNFT−Routing retains layer-wise gradient surgery and region-wise loss reweighting while removing only advantage routing; and OmniNFT enables all three components.

ConfigurationRoutingSurgeryRegionVQ ↑AQ ↑JavisScore ↑DeSync ↓
Base (LTX-2)———2.0385.1970.1920.569
Routing only✓✗✗3.264 ± 0.0975.399 ± 0.0770.199 ± 0.0070.322 ± 0.031
OmniNFT−Routing✗✓✓2.631 ± 0.0815.470 ± 0.0730.209 ± 0.0080.357 ± 0.037
OmniNFT✓✓✓3.332 ± 0.0965.715 ± 0.0700.220 ± 0.0080.270 ± 0.031

These results are reported in Section 6.3, Table 2 of the revised manuscript.

The comparison shows that adding routing alone to the base setup substantially improves VQ, JavisScore, and DeSync, though audio quality (AQ) also improves over the base model but remains below that of the full method.. Conversely, removing routing from the full method (OmniNFT−Routing) causes noticeable degradation across all metrics (notably a 0.701 drop in VQ and a 0.087 worsening in DeSync), demonstrating that without reward-level decoupling, both branches remain disrupted by irrelevant modality signals. Among the configurations compared here, the full method achieves the highest VQ, AQ, and JavisScore and the lowest DeSync.

Q2: [Checking which feedback is harmful] Figure 2 shows that shallow audio layers receive some relatively strong gradients. Is there further evidence that these conflict with the update direction preferred by the audio objective itself? Could a simpler change, such as reducing the audio-branch learning rate or uniformly attenuating incoming video-side gradients across all layers, produce similar gains? This would help assess the added value of selecting particular layers and paths, and interpret the higher audio quality but slightly worse synchronization error in Table 3.

(a) Gradient Direction vs. Audio Objective

Layer-wise gradient surgery is motivated by the simultaneous occurrence of magnitude imbalance and objective directional conflict in shallow layers. To verify this directly, we conduct measurements on the fixed LTX-2 base model (512 prompts ×\times 8 rollout samples ×\times 3 timesteps). The backward gradient magnitude ratio (A2V audio KV vs. V2A audio Q) and gradient conflict proportion (cosine similarity <−0.08< -0.08) across layers are summarized below:

Block RangeA2V Audio KV / V2A Audio Q Gradient Magnitude RatioProportion of Objective Gradient Cosine < −0.08
0–94.20330.040%
10–191.48030.650%
20–322.56517.990%
33–390.3448.920%
40–470.3897.240%

These gradient diagnostics are summarized in Section 4.2 and detailed in Appendix D.4 (Table 14) of the revised manuscript.

Both shallow backward magnitude imbalance and objective directional conflict jointly support layer-wise gradient surgery. In the first 10 blocks, the A2V audio KV backward gradient norm is 4.203 ×\times that of V2A audio Q, and roughly 30% of objective gradient pairs in the first 20 blocks exhibit notable negative correlation, significantly higher than in deeper layers. This quantitative evidence validates that cross-modal gradients heavily disrupt shallow audio states, providing direct justification for layer-wise gradient surgery.

We also provide an ablation comparing OmniNFT with and without gradient surgery (OmniNFT − surgery), keeping routing and region reweighting active in both settings. Enabling gradient surgery yields markedly higher audio quality, confirming its positive impact on audio generation fidelity.

ConfigurationAQ ↑
OmniNFT5.715 ± 0.070
OmniNFT − surgery5.426 ± 0.071

These results are reported in Section 6.3, Table 2 of the revised manuscript.

AuthorsOpenReview ↗

Response to reviewer PiRR - part 3

(b) Simple Alternatives

To further validate the effectiveness of our layer-wise gradient surgery design, we introduce two simple controls: (1) reducing the audio-branch learning rate (scaling down the overall update magnitude of the audio branch); and (2) uniformly attenuating A2V context backward gradients across all layers (globally scaling the cross-modal pathway). Both controls retain routing and region reweighting and use identical initialization, reward functions, global batch size, rollout group size, and update budget.

ConfigurationVideo / Audio LRA2V Context BackwardVQ ↑AQ ↑JavisScore ↑DeSync ↓
Base (LTX-2)——2.0385.1970.1920.569
OmniNFT3e-5 / 3e-5Layer-wise customized3.332 ± 0.0965.715 ± 0.0700.220 ± 0.0080.270 ± 0.031
Low Audio LR3e-5 / 3e-61.0 across all layers2.772 ± 0.0845.423 ± 0.0770.199 ± 0.0080.413 ± 0.036
Uniform Attenuation3e-5 / 3e-50.3 across all layers2.995 ± 0.0935.472 ± 0.0730.208 ± 0.0080.442 ± 0.039

These controls are reported in Section 6.3 (Table 3), with settings in Appendix D.3 of the revised manuscript.

These experimental results demonstrate that:

  • Both simple controls improve the reported quality metrics over the base model, but their DeSync values of 0.413 and 0.442 remain higher than OmniNFT’s 0.270. They also obtain lower VQ, AQ, and JavisScore than OmniNFT.
  • Reducing the audio learning rate scales updates throughout the audio branch, while uniform attenuation applies the same scaling factor across all A2V context backward layers. These controls therefore differ from selecting particular layers and gradient paths.
  • OmniNFT's layer-wise, path-selective gradient control adapts precisely to the functional roles of different Transformer blocks, enabling joint improvements in both generation quality and audio-video synchronization.

Q3: [Checking the importance of location] What happens if the spatial positions or sample assignments of the attention maps are shuffled, while preserving the weight distribution and training budget (Section 5.3 and Table 3)? This could help distinguish the benefit of locating synchronization-critical regions from that of loss reweighting more generally. Does region weighting also help when the sound source is off screen or the audio is mainly background music?

(a) Spatial Correspondence and Shuffled Controls

To verify the effectiveness of our spatial reweighting design, we construct an attention-map spatial shuffle control: while preserving the exact weight distribution, sum, and training budget, we randomly permute the spatial positions of the weights within each latent frame.

Region SettingVQ ↑AQ ↑JavisScore ↑DeSync ↓
OmniNFT3.332 ± 0.0965.715 ± 0.0700.220 ± 0.0080.270 ± 0.031
OmniNFT − region3.246 ± 0.0835.917 ± 0.0730.209 ± 0.0070.334 ± 0.034
OmniNFT + region shuffle3.008 ± 0.0895.487 ± 0.0780.183 ± 0.0080.417 ± 0.038

These spatial-shuffle results are reported in Section 6.3 (Table 3), with the protocol in Appendix D.3 of the revised manuscript.

These experimental results demonstrate that: after spatial shuffling, VQ(3.008), AQ (5.487) and DeSync (0.417) degrade substantially compared to full OmniNFT, supporting the importance of accurate spatial localization in this reweighting design.

Furthermore, to evaluate the reliability of V2A cross-attention maps as an intrinsic proxy for critical regions, we construct an automated reference benchmark using the union of bounding boxes generated by frozen Gemma3 and GroundingDINO (where Gemma3 extracts sound-source phrases and GroundingDINO detects the corresponding objects), and compare against equal-area random pixel selection:

Region MethodEqual-Area OverlapIoU
Attention-based68.640%54.820%
Random Selection46.150%34.250%

These localization results and visualizations appear in Appendix D.5 (Table 15 and Figure 8) of the revised manuscript.

Our critical-region selection based on V2A cross-attention maps substantially outperforms equal-area random selection, confirming its strong spatial correspondence with visible sound-emitting sources.

(b) Off-Screen Sound Sources and Background Music

Region-wise loss reweighting is specifically designed for visual regions associated with visible sound-emitting sources. When the audio consists of off-screen sound sources, voice-over narration, or background music, the attention weights naturally flatten toward uniform weighting (or the module effectively deactivates), while modality-wise advantage routing and layer-wise gradient surgery continue to decouple cross-modal optimization normally.

Reviewer_ffivOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

OmniNFT proposes a post-training framework for joint audio–video diffusion models, building on DiffusionNFT. It identifies three optimization challenges: disagreement between modality-specific advantages, imbalanced cross-modal gradient propagation, and spatially uniform credit assignment. The method combines three components. Modality-wise advantage routing assigns video and audio advantages to their respective output losses while sharing the synchronization advantage. Layer-wise gradient surgery attenuates gradients through shallow audio key/value paths used by audio-to-video attention. Region-wise loss reweighting uses video-to-audio attention maps to emphasize video tokens associated with sound-producing regions. Experiments primarily fine-tune LTX-2 using a 20K-sample prompt pool and evaluate on JavisBench and VBench. Additional experiments cover MOVA, lip synchronization, speech completeness, and human preference. The results show improved perceptual quality and audiovisual synchronization, with some tradeoffs in text–video alignment and audio quality across component configurations.

Critical strengths

1.Substantial synchronization improvements with evidence beyond the optimized metric. On LTX-2, OmniNFT reduces DeSync from 0.569 to 0.269, compared with 0.412 for GDPO, while improving AV-IB from 0.232 to 0.262 and JavisScore from 0.192 to 0.220. The supplementary lip-sync and human-preference evaluations provide additional evidence: overall preference against GDPO is reported as 57.46%. These complementary results make the gains more convincing than improvement in a training reward alone.

2.A lightweight intervention with preliminary cross-backbone support. OmniNFT modifies reward assignment, backward gradient flow, and loss weighting without introducing a new generation architecture. The reported training time increases only from 23.9 to 24.1 hours. On MOVA, it also improves six of seven reported metrics over GDPO, including reducing DeSync from 0.3380 to 0.2697. This suggests practical applicability beyond one backbone.

Critical weaknesses

The component analysis does not fully support the proposed mechanisms or the breadth of the improvement claims. The ablations add components cumulatively rather than testing the full method with each component removed. Gradient surgery improves AQ but worsens DeSync from 0.322 to 0.334; adding region weighting improves synchronization but reduces AQ from 5.917 to 5.715. Thus, the components exhibit tradeoffs rather than uniformly complementary gains. The gradient-norm observations do not establish harmful gradient conflict, and no matched gradient-scaling control or quantitative localization evaluation establishes why the proposed surgery and attention masks are preferable. Evaluation also reuses CLAP and DeSync as both training rewards and reported metrics. Independent results partly address this concern, but the human study omits the number of raters and judgments, uncertainty estimates are absent, and text–video ImageBind declines from 0.272 to 0.261. These limitations weaken the claim of comprehensive improvement.

Questions for authors
  1. Can you provide full-method-minus-one-component ablations, together with simple controls such as uniformly attenuating cross-modal gradients and using shuffled or alternative saliency masks? Do the gains remain reliable across seeds?

  2. Gradient interpretation: Do video-derived gradients actually conflict with audio-objective gradients, as measured by gradient cosine similarity or their effect on held-out audio quality? Large gradient norms alone do not establish that those gradients are harmful.

  3. Reproducibility and independent validation: Please specify how VideoAlign/HPSv3 and Audiobox/CLAP are combined into the three reward channels, along with the normalizers, optimizer settings, LoRA configuration, EMA schedule, and region-weight warmup. For the human study, how many raters and judgments were collected, how were ties handled, and what are the confidence intervals?

6 comments
AuthorsOpenReview ↗

Response to reviewer ffiv - part 1

We sincerely thank you for your thorough review and constructive feedback. In response to your suggestions to further clarify component contributions, gradient mechanisms, and result reliability, we provide point-by-point responses below supported by additional experiments and implementation details.

Q1: Can you provide full-method-minus-one-component ablations, together with simple controls such as uniformly attenuating cross-modal gradients and using shuffled or alternative saliency masks? Do the gains remain reliable across seeds?

(a) Full-Method-Minus-One-Component Ablations

To examine the independent contribution of each component in the full framework, we conduct ablations by removing modality-wise advantage routing (−routing), layer-wise gradient surgery (−surgery), and region-wise loss reweighting (−region) one at a time. The table below summarizes the results of OmniNFT and its ablated configurations across multiple random seeds (reported as mean±std\text{mean} \pm \text{std}).

ConfigurationRoutingSurgeryRegionVQ ↑AQ ↑JavisScore ↑DeSync ↓
OmniNFT✓✓✓3.332 ± 0.0965.715 ± 0.0700.220 ± 0.0080.270 ± 0.031
OmniNFT−routing✗✓✓2.631 ± 0.0815.470 ± 0.0730.209 ± 0.0080.357 ± 0.037
OmniNFT−surgery✓✗✓2.948 ± 0.0895.426 ± 0.0710.200 ± 0.0080.398 ± 0.036
OmniNFT−region✓✓✗3.246 ± 0.0835.917 ± 0.0730.209 ± 0.0070.334 ± 0.034

These results are reported in Section 6.3, Table 2 of the revised manuscript.

These results clearly demonstrate the complementary gains among the three components:

  • Modality-wise advantage routing effectively coordinates joint audio-video optimization by directing each branch toward feedback aligned with its own objective;
  • Layer-wise gradient surgery improves the reported VQ, AQ, JavisScore, and DeSync when routing and region reweighting are retained, supporting its contribution within the complete framework.;
  • Region-wise loss reweighting reflects a reasonable trade-off, accepting a localized drop in AQ to secure substantial gains in overall synchronization and multi-dimensional alignment.

Multi-Seed Reliability Verification: Across multiple training seeds, OmniNFT achieves robust and consistent performance gains over the GDPO baseline in visual quality, audio quality, and audio-video alignment.

MethodVQ ↑AQ ↑JavisScore ↑DeSync ↓
GDPO3.217 ± 0.0975.524 ± 0.0730.186 ± 0.0070.413 ± 0.038
OmniNFT3.332 ± 0.0965.715 ± 0.0700.220 ± 0.0080.270 ± 0.031

These results are reported in Section 6.3, Table 2 of the revised manuscript.

(b) Gradient Surgery Ablations

To further validate the effectiveness of our layer-wise gradient surgery design, we introduce two simple controls: (1) reducing the audio-branch learning rate (scaling down the overall update magnitude of the audio branch); and (2) uniformly attenuating A2V context backward gradients across all layers (globally scaling the cross-modal pathway). Both controls retain routing and region reweighting and use identical initialization, reward functions, global batch size, rollout group size, and update budget.

ConfigurationVideo / Audio LRA2V Context BackwardVQ ↑AQ ↑JavisScore ↑DeSync ↓
Base (LTX-2)——2.0385.1970.1920.569
OmniNFT3e-5 / 3e-5Layer-wise customized3.332 ± 0.0965.715 ± 0.0700.220 ± 0.0080.270 ± 0.031
Low Audio LR3e-5 / 3e-61.0 across all layers2.772 ± 0.0845.423 ± 0.0770.199 ± 0.0080.413 ± 0.036
Uniform Attenuation3e-5 / 3e-50.3 across all layers2.995 ± 0.0935.472 ± 0.0730.208 ± 0.0080.442 ± 0.039

These controls are reported in Section 6.3 (Table 3), with settings in Appendix D.3 of the revised manuscript.

These experimental results demonstrate that:

  • Both simple controls improve the reported quality metrics over the base model, but their DeSync values of 0.413 and 0.442 remain higher than OmniNFT’s 0.270. They also obtain lower VQ, AQ, and JavisScore than OmniNFT.
  • Reducing the audio learning rate scales updates throughout the audio branch, while uniform attenuation applies the same scaling factor across all A2V context backward layers. These controls therefore differ from selecting particular layers and gradient paths.
  • OmniNFT's layer-wise, path-selective gradient control adapts precisely to the functional roles of different Transformer blocks, enabling joint improvements in both generation quality and audio-video synchronization.
AuthorsOpenReview ↗

Response to reviewer ffiv - part 2

(c) Region-Wise Reweighting Ablations

To verify the effectiveness of our spatial reweighting design, we construct an attention-map spatial shuffle control: while preserving the exact weight distribution, sum, and training budget, we randomly permute the spatial positions of the weights within each latent frame.

Region SettingVQ ↑AQ ↑JavisScore ↑DeSync ↓
OmniNFT3.332 ± 0.0965.715 ± 0.0700.220 ± 0.0080.270 ± 0.031
OmniNFT − region3.246 ± 0.0835.917 ± 0.0730.209 ± 0.0070.334 ± 0.034
OmniNFT + region shuffle3.008 ± 0.0895.487 ± 0.0780.183 ± 0.0080.417 ± 0.038

These spatial-shuffle results are reported in Section 6.3 (Table 3), with the protocol in Appendix D.3 of the revised manuscript.

These experimental results demonstrate that: after spatial shuffling, VQ(3.008), AQ (5.487) and DeSync (0.417) degrade substantially compared to full OmniNFT, supporting the importance of accurate spatial localization in this reweighting design.

Furthermore, to evaluate the reliability of V2A cross-attention maps as an intrinsic proxy for critical regions, we construct an automated reference benchmark using the union of bounding boxes generated by frozen Gemma3 and GroundingDINO (where Gemma3 extracts sound-source phrases and GroundingDINO detects the corresponding objects), and compare against equal-area random pixel selection:

Region MethodEqual-Area OverlapIoU
Attention-based68.640%54.820%
Random Selection46.150%34.250%

These localization results and visualizations appear in Appendix D.5 (Table 15 and Figure 8) of the revised manuscript.

Our critical-region selection based on V2A cross-attention maps substantially outperforms equal-area random selection, confirming its strong spatial correspondence with visible sound-emitting sources.

Q2: Gradient interpretation: Do video-derived gradients actually conflict with audio-objective gradients, as measured by gradient cosine similarity or their effect on held-out audio quality? Large gradient norms alone do not establish that those gradients are harmful.

Layer-wise gradient surgery is motivated by the simultaneous occurrence of magnitude imbalance and objective directional conflict in shallow layers. To verify this directly, we conduct measurements on the fixed LTX-2 base model (512 prompts ×\times 8 rollout samples ×\times 3 timesteps). The backward gradient magnitude ratio (A2V audio KV vs. V2A audio Q) and gradient conflict proportion (cosine similarity <−0.08< -0.08) across layers are summarized below:

Block RangeGradient Magnitude RatioProportion of Gradient Conflict
0–94.20330.040%
10–191.48030.650%
20–322.56517.990%
33–390.3448.920%
40–470.3897.240%

These gradient diagnostics are summarized in Section 4.2 and detailed in Appendix D.4 (Table 14) of the revised manuscript.

Both shallow backward magnitude imbalance and objective directional conflict jointly support layer-wise gradient surgery. In the first 10 blocks, the A2V audio KV backward gradient norm is 4.203 ×\times that of V2A audio Q, and roughly 30% of objective gradient pairs in the first 20 blocks exhibit notable negative correlation, significantly higher than in deeper layers. This quantitative evidence validates that cross-modal gradients heavily disrupt shallow audio states, providing direct justification for layer-wise gradient surgery.

We also provide an ablation comparing OmniNFT with and without gradient surgery (OmniNFT − surgery), keeping routing and region reweighting active in both settings. Enabling gradient surgery yields markedly higher audio quality, confirming its positive impact on audio generation fidelity.

ConfigurationAQ ↑
OmniNFT5.715 ± 0.070
OmniNFT − surgery5.426 ± 0.071

These results are reported in Section 6.3, Table 2 of the revised manuscript.

AuthorsOpenReview ↗

Response to reviewer ffiv - part 3

Q3: Reproducibility and independent validation: Please specify how VideoAlign/HPSv3 and Audiobox/CLAP are combined into the three reward channels, along with the normalizers, optimizer settings, LoRA configuration, EMA schedule, and region-weight warmup. For the human study, how many raters and judgments were collected, how were ties handled, and what are the confidence intervals?

(a) Reward Formulation and Normalization

To address the substantial scale differences among multiple training rewards, we adopt a "normalize-first, route-by-modality" strategy: each of the 5 rewards is first standardized independently and then aggregated into three advantage channels: video, audio, and shared synchronization, eliminating scale bias in reward weighting. The exact definitions and coefficients are detailed below:

ScorerScore DefinitionWeightChannel
VideoAlign(VQ+TA)/2(VQ+TA)/2 calibrated with fixed mean/std1Video
HPSv3Frame-wise scoring; mean of top 30% frames as video quality score1.5Video
Audiobox(CE+CU+PQ−PC)/40(CE+CU+PQ-PC)/400.5Audio
CLAPCosine similarity between generated audio and prompt_a embeddings1Audio
DeSync1/(1+d)1/(1+d), where dd is the segment-wise mean of predicted absolute temporal offset1Shared Sync

These reward definitions are specified in Section 5.1 and Appendix C.3 (Table 12) of the revised manuscript.

Following GRPO, the per-reward advantage for each prompt is computed as:

ak,ci=sk,ci−mean⁡({sk,c1,sk,c2,…,sk,cG})std⁡({sk1,sk2,…,skG})+ϵ.a_{k,c}^{i}=\frac{s_{k,c}^{i}-\operatorname{mean}(\{s_{k,c}^{1},s_{k,c}^{2},\ldots,s_{k,c}^{G}\})}{\operatorname{std}(\{s_k^{1},s_k^{2},\ldots,s_k^{G}\})+\epsilon}.

where GG is the rollout group size and ϵ=10−4\epsilon=10^{-4}. We then construct the three advantage channels:

Av=aVA+1.5aHPS,Aa=0.5aAB+aCLAP,Aav=aDeSync.A_v=a_{VA}+1.5a_{HPS},\quad A_a=0.5a_{AB}+a_{CLAP},\quad A_{av}=a_{DeSync}.

For advantage assignment, Av+AavA_v+A_{av} and Aa+AavA_a+A_{av} are routed to the video and audio branches respectively (whereas GDPO assigns the shared sum Av+Aa+AavA_v+A_a+A_{av} to both branches). Both are subsequently clipped to [−5,5][-5,5] and mapped to optimality probabilities via r=0.5+A/10r=0.5+A/10.

(b) Training, EMA, and Region-Weight Warmup Details

We summarize the complete training configuration below to facilitate reproducibility. GDPO and OmniNFT share the exact same optimizer, LoRA setup, sampling parameters, and update budget, while OmniNFT additionally enables layer-wise gradient surgery and region-wise loss reweighting.

ItemSetting
OptimizerAdamW; LR 3×10−53\times10^{-5}; betas=(0.9, 0.999); eps 10−810^{-8}; weight decay 10−410^{-4}; no LR scheduler
Gradient / Precision / ParallelismGlobal gradient clipping 1.0; BF16, FSDP, activation checkpointing
LoRArank/alpha = 32/64, Gaussian initialization
LoRA Target ModulesQ/K/V/output projections of self-attention, text cross-attention, and bidirectional AV cross-attention in both branches; net.0.proj and net.2 in FFN
Training Budget400 updates; global batch size 32, rollout group size 8 (16 GPUs ×\times batch size 2 per GPU)
Video Specification512×768512\times768, 121 frames, 24 FPS
Sampling Settings20 steps with video/audio CFG = 1.5/3 during training; 40 denoising steps with CFG = 3/7 during evaluation
NFT Mixing / Reference RegularizationMixing parameter β=1\beta = 1; reference regularization coefficient 10−410^{-4}
Old Policy EMAθold←ηgθold+(1−ηg)θ\theta_{old}\leftarrow\eta_g\theta_{old}+(1-\eta_g)\theta, ηg=min⁡(0.001g,0.5)\eta_g=\min(0.001g,0.5); EMA disabled during extra evaluation
OmniNFT Region Warmupλg=0.5min⁡(g/400,1)\lambda_g=0.5\min(g/400,1); wi=1+λgsigmoid⁡(λgzi)w_i=1+\lambda_g\operatorname{sigmoid}(\lambda_g z_i); weight upper bound increases from 1 to 1.5
Public OmniNFT A2V Context BackwardBlock 0: 0; Blocks 1–10: 0.1; Blocks 40–47: 0.3; Remaining blocks: 1

These training settings are provided in Appendix C.2, Table 11 of the revised manuscript.

Comments on the paper

3 comments

Dear Reviewers,

The authors have responded to your questions. If you haven’t already done so, please review their responses and let them know whether your concerns have been addressed. Please use the discussion to raise any remaining concerns or follow-up questions.

Thank you for your time!

Best,

AC

AuthorsOpenReview ↗

Revision Clarification

Dear Reviewers and ACs,

Thank you for your careful reviews and constructive feedback. We have revised the OmniNFT manuscript and provided additional experiments and analyses in our point-by-point responses. The main updates are summarized below.

In the main paper:

  1. Added a layer-wise analysis of gradient magnitudes and directional conflicts to motivate gradient surgery in the shallow audio layers (Section 4.2).
  2. Clarified how the five rewards are normalized and combined into video, audio, and shared synchronization advantages, and identified which evaluation metrics overlap with training rewards (Sections 5.1 and 6.1).
  3. Added multi-seed comparisons with GDPO, a routing-only ablation, and ablations that remove each component in turn (Section 6.3, Table 2).
  4. Added matched-budget controls using a lower audio-branch learning rate, uniform cross-modal gradient attenuation, and spatially shuffled attention weights (Section 6.3, Table 3).

In the appendix:

  1. Added multi-seed lip-sync results, a GDPO comparison for speech completeness, and the blind A/B study protocol and confidence intervals (Appendices B.1–B.3; Tables 7–9).
  2. Documented the training and reward configurations, including optimizer and LoRA settings, the EMA schedule, and region-weight warmup (Appendices C.2–C.3; Tables 11–12).
  3. Added detailed ablations, control protocols, gradient diagnostics, and an evaluation of how well cross-attention maps localize visible sound sources (Appendices D.1 and D.3–D.5; Tables 13–15 and Figure 8).
  4. Reported all 16 VBench dimensions and added results on TA-IB and JavisScore, neither of which was used as a training reward (Appendices E.1–E.3; Tables 16–18).

In our replies, we also explained why video-only reweighting can affect synchronization: the video loss already incorporates the shared synchronization advantage. We clarified that “vanilla RL” and “GDPO” refer to the same baseline. We expand the discussion of prior work on multi-reward RL for visual generation.

Thank you again for your time and feedback.

Best regards,
Paper 17080 Authors

AuthorsOpenReview ↗

Rebuttal Summary

Dear AC,

We sincerely appreciate the time and effort you have devoted to reviewing Paper 17080. We would like to summarize the reviewers’ feedback and our response.

We are encouraged that Reviewer w3e9 retained a Weak Acceptance. During rebuttal, we also addressed the main technical concerns raised by Reviewers ffiv and PiRR through a comprehensive set of controlled experiments and analyses.

We appreciate that the reviewers highlighted the following strengths:

  • OmniNFT is well-motivated and lightweight. It preserves audio- and video-specific feedback that a single aggregate reward can obscure and can be applied to existing models without changing their architecture, with only a small increase in training time.
  • OmniNFT shows substantial synchronization gains over GDPO, supported by evaluations beyond the training rewards and preliminary results on MOVA in addition to LTX-2.

During rebuttal, we addressed the main technical questions:

  • Component contributions: We added three-seed ablations that remove one component at a time.
  • Mechanisms and controls: We added gradient-conflict analyses and matched-budget controls for gradient attenuation and spatial reweighting.
  • Fair comparison and broader evaluation: We documented the matched GDPO setup, reported the full VBench evaluation, and added evidence beyond the training rewards.

We thank you and the reviewers again for your time and consideration.

Best regards,
Paper 17080 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.