acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Reward Routing: Dynamic Multi-Reward Optimization for Joint Audio-Video Diffusion via Forward-Process RL

Abstract

Multi-reward guided reinforcement learning (i.e., RL) offers a promising way to improve joint audio-video diffusion models along several complementary objectives, including modality-specific quality, cross-modal semantic alignment, and temporal synchronization. Its effectiveness, however, depends on two quantities that change during training: where reward-driven updates should act, and how competing rewards should be combined. Existing methods tend to rely on fixed routing and reward weights, failing to track evolving model functions. To address these limitations, we propose Adaptive Reward Routing to jointly adapt update locations and reward coordination during forward-process RL (i.e., DiffusionNFT) of joint audio-video diffusion models. Our method consists of two components. (i) Cross-Modal Influence-Guided Routing (Localizing Updates): We use bidirectional cross-attention responses as an efficient proxy for evolving cross-modal influence, dynamically reweighting token-aware losses and scaling gradients across cross-modal layers without additional model interventions. (ii) Preference-Preserving Modality-Aware Reweighting (Coordinating Rewards): We preserve predefined weights as preference priors and use branch-specific reward-gradient interactions as residual corrections after warm-up. This resolves evolving conflicts without letting dominant rewards suppress weak but essential objectives. Extensive experiments demonstrate consistent improvements in modality quality, semantic consistency, and audio-video synchronization over strong RL baselines. Ablations and mechanism analyses further validate the complementary benefits of adaptive update routing and reward coordination.

Primary area:
generative models
Keywords:
Joint Audio-Video Generation, Diffusion Reinforcement Learning, Multi-Reward Optimization, Forward-Process RL

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_Fc2zOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Adaptive Reward Routing, a multi-reward optimization method for joint audio-video diffusion models based on forward-process reinforcement learning. The method consists of two components: (1) Cross-Modal Influence-Guided Routing, which dynamically adjusts token-level loss weights and layer-wise gradient routing based on cross-modal attention responses, and (2) Preference-Preserving Modality-Aware Reweighting, which adjusts reward weights according to modality-specific gradient interactions while preserving predefined reward preferences. The proposed approach is evaluated on LTX-2 and LTX-2.3 using JavisBench, demonstrating quantitative improvements in generation quality, cross-modal consistency, and audio-video synchronization over existing baselines.

Critical strengths
  1. Practical formulation of multi-reward optimization for joint audio-video diffusion. The paper addresses the challenge of optimizing multiple potentially conflicting reward signals across audio and video modalities. This is an important problem for joint audio-video generation, where modality-specific quality and cross-modal consistency must be optimized simultaneously.

  2. Simple and practical techniques. The proposed routing and reward reweighting mechanisms are relatively straightforward and appear easy to implement within existing diffusion RL frameworks. Their practical simplicity, together with the reported improvements on LTX-2 and LTX-2.3, is a strength of the paper.

Critical weaknesses
  1. Limited methodological novelty and insufficient justification of the interaction between the proposed components. The proposed method retains the underlying DiffusionNFT framework and primarily introduces heuristic mechanisms for token weighting, layer-wise gradient scaling, and reward reweighting. Although these techniques are empirically effective, the methodological contribution remains relatively incremental. Furthermore, Cross-Modal Influence-Guided Routing and Preference-Preserving Modality-Aware Reweighting are developed largely independently. In particular, gradient-conflict estimation explicitly disables token routing, even though token routing changes the gradient geometry of the actual training objective. The paper does not sufficiently justify this design choice or establish whether jointly applying these mechanisms provides benefits or causes harm.

  2. Insufficient justification of the influence criterion for reward-driven optimization. The proposed routing mechanism uses cross-attention response magnitudes as a proxy for cross-modal influence. Although Section 5.4 provides intervention-based validation showing that the proxy correlates with changes in predicted velocity, this does not necessarily establish that layers with greater influence on model predictions are the most effective locations for reward-driven optimization. Similarly, the layer-wise gradient norms illustrated in Figure 1(b) are noisy and not necessarily reliable indicators of functional importance. The paper would benefit from further analysis connecting the proposed influence criterion to actual improvements in reward objectives. Additionally, comparisons with recent alternative layer-wise influence or adaptability estimation approaches, such as the constructive influence function proposed in MusicInfuser (Hong et al., CVPR 2026), would help clarify the advantages and limitations of the proposed criterion.

Questions for authors
  1. Interaction between routing and reward reweighting: Section 4.3 disables token routing during gradient-conflict estimation to avoid bias from the current token weights. However, token routing changes the gradient geometry of the actual optimization objective. Is there empirical evidence that estimating reward conflicts with uniform token weights is preferable to estimating them under the actual routed objective? How sensitive is performance to this design choice?

  2. Influence estimation and optimization effectiveness: Section 5.4 demonstrates that the proposed cross-attention response proxy correlates strongly with changes in predicted velocity under direct interventions. However, it remains unclear whether layers with greater prediction influence are necessarily more beneficial for reward-driven optimization. Can the authors provide evidence connecting the influence scores to actual reward improvements achieved by updating the corresponding layers? How does the proposed criterion compare conceptually or empirically with other layer-wise influence or adaptability criteria?

7 comments
AuthorsOpenReview ↗

Response to Reviewer Fc2z (Part 1)

Thank you for your thoughtful and constructive feedback. We sincerely appreciate the time and care you devoted to reviewing our work. Your comments have helped us sharpen our claims and strengthen the supporting evidence.

(W1) 1. Methodological Novelty

Thank you for recognizing the problem’s importance and our method’s practicality. Our novelty lies in unified optimization, model-adaptive mechanisms, and cross-architecture applicability. To our knowledge, within joint audio-video diffusion RL, ours is the first framework to:

1-a. Unify update allocation and reward coordination. We jointly adapt where reward-driven updates act and how competing rewards are combined, addressing both dimensions within one optimization framework.

1-b. Use model evolution to guide optimization. Evolving cross-modal functionality provides online routing signals for token weighting and layer-wise gradient scaling, while changing reward conflicts demand adaptive coordination anchored to user preferences. Both mechanisms respond to the evolving model rather than follow fixed optimization rules.

1-c. Support both dual-stream and single-stream architectures. We instantiate and validate the framework on dual-stream LTX and single-stream JavisDiT++, extending adaptive reward routing beyond architecture-specific layer structures.

(W1,Q1) 2. Interaction Between Reward Coordination and Influence-Guided Routing

The two components jointly define the training objective and update, rather than operate independently:

2-a. Reward coordination determines the supervision. It combines rewards into branch-specific advantages, which determine the weights of DiffusionNFT’s positive/negative-policy regression losses.

2-b. Influence-guided routing allocates that supervision. It applies token weights to these losses and controls gradient propagation through cross-modal pathways.

Empirically, the components are complementary. On LTX-2, routing alone achieves a JavisScore of 0.190, coordination alone achieves 0.183, and their combination reaches 0.206 (Table 2), supporting the benefit of applying both.

(W1,Q1) 3. Why Disable Token Routing During Conflict Estimation?

3-a. Rationale Clarification. We choose uniform probing to separate branch-level reward coordination from local update allocation for two reasons:

  • Avoiding route-dependent reward priorities. Token weighting can change gradient directions, making branch-wide reward coefficients depend disproportionately on currently emphasized regions. Gradient normalization does not remove this directional effect.
  • Separating update timescales. Routing changes every round, whereas reward coefficients are refreshed every 20 rounds and EMA-smoothed. Uniform probing avoids carrying a particular round’s token-weighting effects into coefficients reused across subsequent rounds.

3-b. Controlled comparison. To test this choice, we change only the token weights used during conflict estimation, keeping priors, warm-up, refresh frequency, training budget, and actual training routing unchanged. Uniform probing performs better across all six metrics, supporting our default design.

Table R8. Uniform versus routed token weights for conflict estimation on LTX-2.

Conflict estimationVQ↑AQ↑CLAP↑AV-IB↑JavisScore↑DeSync↓
Routed-token probe3.2985.7390.4190.2300.2010.349
Uniform-token probe (Ours)3.3365.8680.4250.2350.2060.341
AuthorsOpenReview ↗

Response to Reviewer Fc2z (Part 2)

(W2,Q2) 4. Prediction Influence vs. Reward-Update Utility

Thank you for highlighting the distinction between prediction influence and update utility. We clarify why prioritizing influential locations does not simply reinforce their current behavior (4-a), then test update utility directly: response-based routing outperforms alternative criteria (4-b, Table R1), and stronger routing improves optimization within the tested range (4-c, Table R2).

4-a. Rationale clarification. As validated in Sec. 5.4, our proxy measures how strongly a location influences the model’s output. This identifies where to focus updates, while DiffusionNFT’s reward-conditioned positive/negative-policy regression losses determine how to update. Allocating more updates to influential locations does not unconditionally reinforce their current behavior. Instead, it enables DiffusionNFT’s reward supervision to reinforce beneficial behavior and correct unfavorable behavior at those locations.

4-b. Experiment 1: Comparison with alternative routing criteria. (Table R1) To test whether gains depend on the response-based criterion rather than nonuniform weighting alone, we replace only the routing criterion while keeping other training settings fixed. Gradient-magnitude routing uses gradients of the unweighted DiffusionNFT loss with respect to cross-modal responses, while random routing shuffles token weights and layer scales while preserving their distributions. Our method outperforms both alternatives across all six metrics. These results support the effectiveness of cross-modal responses for allocating reward-driven updates.

Table R1. Comparison of routing criteria on LTX-2.

MethodVQ↑AQ↑CLAP↑AV-IB↑JavisScore↑DeSync↓
OmniNFT3.1365.6140.4160.2220.1950.390
Random routing3.1025.5340.3950.2100.1810.405
Gradient-magnitude routing3.2015.7120.4110.2210.2000.367
Response-based routing (Ours)3.3365.8680.4250.2350.2060.341

4-c. Experiment 2: Effect of routing strength. (Table R2) To verify whether prioritizing influential locations benefits reward optimization, we evaluate both OmniNFT (using its own routing strategy) and our method at routing strengths η∈{0.2,0.6,1.0}\eta\in\{0.2,0.6,1.0\}, keeping other settings and post-training steps fixed on LTX-2. Here, λη=1+η(λ−1)\lambda_\eta=1+\eta(\lambda-1) and αη=ηα\alpha_\eta=\eta\alpha scale token reweighting and layer detachment, respectively, from disabled (η=0\eta=0) to each method’s default (η=1\eta=1). Both methods improve as routing strengthens, while ours performs better overall at each matched strength

Table R2. Effect of routing strength on LTX-2 under the same number of post-training steps.

Methodη\etaVQ↑AQ↑TV-IB↑TA-IB↑CLIP↑CLAP↑AV-IB↑AVHScore↑JavisScore↑DeSync↓
GDPO (reference)—2.7225.4500.2610.1380.3120.3470.1740.1750.1550.671
OmniNFT0.22.8915.5030.2620.1400.3100.3730.1930.1920.1710.566
OmniNFT0.63.0485.5710.2640.1430.3110.4010.2120.2090.1870.448
OmniNFT1.03.1365.6140.2650.1450.3120.4160.2220.2190.1950.390
Ours0.23.0985.5920.2640.1490.3120.3860.2020.2010.1780.419
Ours0.63.2145.7540.2670.1600.3130.4120.2240.2230.1970.392
Ours1.03.3365.8680.2680.1670.3140.4250.2350.2340.2060.341

4-d. Clarifying Fig. 1(b). The gradient norms illustrate how gradient propagation changes during training; they are not used to compute our routing scores. Prediction influence is validated through interventions in §5.4, while the training experiments above establish the practical utility of response-based update allocation.

AuthorsOpenReview ↗

Response to Reviewer Fc2z (Part 3)

(W2,Q2) 5. Comparison with Other Layer-Wise Method (MusicInfuser[3])

Thank you for highlighting MusicInfuser[3]. Both methods use internal model signals to identify useful adaptation locations, but differ in three respects:

5-a. Estimation criterion. MusicInfuser constructs guidance from full-model and layer-skipped predictions, then ranks layers using the resulting videos’ VBench imaging quality. Our criterion uses bidirectional cross-modal response magnitudes to identify interactions influencing audio and video outputs.

5-b. Fixed selection vs. dynamic allocation. MusicInfuser selects 16 of 48 layers for music-conditioning modules before training and keeps this selection fixed. We refresh layer-wise gradient scales throughout training, while deriving token weights from the same responses, thereby tracking evolving cross-modal functionality rather than selecting adaptation locations once.

5-c. Suitability for online adaptation. MusicInfuser’s selection requires guided generation and quality evaluation. Our routing signals are collected during normal rollout, without additional intervention passes or gradient probes for routing estimation, making frequent refreshes practical.

Our criterion therefore targets lightweight, continuous token/layer update allocation, whereas MusicInfuser targets quality-guided selection of fixed adaptation layers. We will incorporate this comparison into the revision.

[3] Hong S, Kemelmacher-Shlizerman I, Curless B, et al. Musicinfuser: Making video diffusion listen and dance[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026: 43751-43761.

Reviewer_1itrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Adaptive Reward Routing for multi-reward reinforcement learning in joint audio-video diffusion models. The method consists of two main components: dynamically adjusting token- and layer-level gradient updates based on cross-attention responses, and balancing different rewards by combining predefined weights with gradient conflict information. The authors evaluate their approach on LTX-2, LTX-2.3, and JavisDiT++, demonstrating improvements in generation quality, cross-modal consistency, and audio-video synchronization through extensive experiments and ablation studies.

Critical strengths
  1. The problem is practically relevant. The authors identify that fixed reward routing may become less effective as cross-modal interactions evolve during training. They provide convincing evidence through gradient analysis and intervention experiments.

  2. The experimental evaluation is comprehensive. The method consistently improves performance across multiple backbones and evaluation metrics. The ablation studies also provide good evidence for the effectiveness of individual components, particularly regarding audio-video synchronization.

Critical weaknesses
  1. The methodological novelty is somewhat limited. The overall framework mainly combines existing ideas from reward balancing and gradient routing. While the adaptive design is reasonable, the individual components are relatively straightforward, and the work offers limited new theoretical insights.

  2. Some conclusions could benefit from further validation. Cross-attention responses can indicate how sensitive model outputs are to different tokens or layers, but this does not necessarily mean these locations are optimal for reward-driven updates. Additionally, some evaluation metrics overlap with the training rewards, and the human evaluation covers a relatively small number of prompts.

Questions for authors

Is the magnitude of cross-attention responses directly correlated with the effectiveness of reward optimization? Could the authors further compare response-based routing with gradient-magnitude-based and random routing strategies?

Could the authors expand the human evaluation to cover more diverse prompts and further validate whether the improvements remain significant under evaluation criteria independent of the training rewards?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 1itr (Part 1)

Thank you for your thoughtful and constructive feedback. We sincerely appreciate the time and care you devoted to reviewing our work. Your comments have helped us sharpen our claims and strengthen the supporting evidence.

(W1) 1. Methodological Novelty

Thank you for recognizing the problem’s relevance and our experimental evidence. Our novelty lies in unified optimization, model-adaptive mechanisms, and cross-architecture applicability. To our knowledge, within joint audio-video diffusion RL, ours is the first framework to:

1-a. Unify update allocation and reward coordination. We jointly adapt where reward-driven updates act and how competing rewards are combined, addressing both dimensions within one optimization framework.

1-b. Use model evolution to guide optimization. Evolving cross-modal functionality provides online routing signals for token weighting and layer-wise gradient scaling, while changing reward conflicts demand adaptive coordination anchored to user preferences. Both mechanisms respond to the evolving model rather than follow fixed optimization rules.

1-c. Support both dual-stream and single-stream architectures. We instantiate and validate the framework on dual-stream LTX and single-stream JavisDiT++, extending adaptive reward routing beyond architecture-specific layer structures.

(W2,Q1) 2. Response-Based Routing

Thank you for suggesting these comparisons. We distinguish prediction influence from optimization utility and test the latter by replacing only the routing criterion with gradient-magnitude and distribution-matched random alternatives.

2-a. Rationale clarification. As validated in Sec. 5.4, our proxy identifies locations that strongly influence cross-modal outputs. It determines where to emphasize updates, while DiffusionNFT’s reward-conditioned positive/negative-policy regression losses determine how to update. Prioritizing these locations enables reward supervision to reinforce beneficial behavior and correct unfavorable behavior—not unconditionally amplify their existing contribution.

2-b. Controlled comparison of routing criteria. (Table R1) To test whether response-based allocation improves optimization, we replace only the routing criterion while keeping other training settings fixed:

  • Gradient-magnitude routing: use gradient norms of the DiffusionNFT loss without token routing with respect to cross-modal responses, aggregated into token/layer scores.
  • Random routing: shuffle token weights and layer scales while preserving their distributions and refresh frequency.

Response-based routing performs best across all six metrics, reducing DeSync by 7.1% relative to gradient-magnitude routing. These comparisons show that response-guided location selection improves optimization beyond nonuniform weighting alone.

Table R1. Comparison of routing criteria on LTX-2.

MethodVQ↑AQ↑CLAP↑AV-IB↑JavisScore↑DeSync↓
OmniNFT3.1365.6140.4160.2220.1950.390
Random routing3.1025.5340.3950.2100.1810.405
Gradient-magnitude routing3.2015.7120.4110.2210.2000.367
Response-based routing (Ours)3.3365.8680.4250.2350.2060.341
AuthorsOpenReview ↗

Response to Reviewer 1itr (Part 2)

(Q2) 3. Expanded Human Evaluation

Thank you for suggesting broader human evaluation. We expanded human evaluation from 10 to 30 distinct prompts, with the additional 20 assessed by 40 new film-school experts. The expanded set covers diverse events, visual styles, and sound types. Our method ranks first across all five criteria on both backbones in this additional evaluation (Table R6).

Table R6. Additional expert evaluation on 20 prompts.

BackboneMethodVisual Quality↑Audio Quality↑Text–AV Alignment↑AV Alignment↑Overall Preference↑
LTX-2Base Model23.530.134.131.528.5
LTX-2GDPO42.543.035.529.339.2
LTX-2MARBLE36.731.540.933.436.5
LTX-2OmniNFT61.065.156.070.866.5
LTX-2Ours86.380.383.585.079.3
LTX-2.3Base Model23.531.532.533.026.5
LTX-2.3GDPO43.738.533.030.538.6
LTX-2.3MARBLE38.139.045.538.336.8
LTX-2.3OmniNFT68.561.557.064.768.3
LTX-2.3Ours76.279.582.083.579.8

(Q2) 4. Evaluation Beyond Training Rewards

4-a. Existing independent metrics. Table 1 already reports ImageBind, CLIP, and JavisScore, none of which is directly optimized as a training reward.

4-b. Additional independent evaluators. We further evaluate with VA-Judger[1] using Prompt Alignment, Video-Audio Consistency, Audio Quality, and Video Quality, and with EvalVerse[2] using Vocal and Soundscape. Neither evaluator is used as a training reward. Our method achieves the best results across all six metrics on both backbones, supporting improvements beyond the directly optimized reward functions.

Table R7. Evaluation with additional non-training metrics. VA-Judger reports 1–10 scores; EvalVerse reports pairwise win rates.

BackboneMethodVA: Prompt Alignment↑VA: Video-Audio Consistency↑VA: Audio Quality↑VA: Video Quality↑EV: Vocal↑EV: Soundscape↑
LTX-2Base Model6.105.765.845.970.310.30
LTX-2GDPO6.015.705.926.760.440.39
LTX-2MARBLE6.175.895.786.420.380.40
LTX-2OmniNFT6.736.696.647.010.640.66
LTX-2Ours7.127.207.027.230.730.75
LTX-2.3Base Model6.346.136.026.180.320.29
LTX-2.3GDPO6.256.076.046.930.410.39
LTX-2.3MARBLE6.436.255.976.600.400.40
LTX-2.3OmniNFT7.007.026.787.310.630.65
LTX-2.3Ours7.357.507.207.510.740.77

[1] Huang Y, Tu S, Yan X, et al. VA-Judger: Reward Modeling from Human Preference Feedback for Joint Video-Audio Generation[J]. arXiv preprint arXiv:2608.18607, 2026.

[2] Yang S, Zhong H, Zhang R, et al. EvalVerse: Pipeline-Aware and Expert-Calibrated Benchmarking for Professional Cinematic Video Generation[J]. arXiv preprint arXiv:2605.23271, 2026.

Thanks for conducting all the additional experiments. All my concerns have been addressed. I will maintain my score.

Dear Reviewer 1itr,

We are very pleased to know that our rebuttal has successfully addressed your concerns. We sincerely appreciate your thoughtful evaluation and your consistent support for our paper throughout the review process. Thank you again for your time, careful consideration, and constructive feedback.

We wish you all the best and hope you are doing well.

The Authors

Reviewer_ic1uOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a reinforcement learning approach to improve joint audio-video diffusion models. By reweighting rewards and selectively routing updates, the method achieves promising results on relevant benchmarks.

Critical strengths

1. The authors provide a relatively rich set of examples and discussions, along with comparisons against existing baselines.

2. The individual design choices are discussed in reasonable detail, and the rationale behind the proposed improvements is relatively clear.

3. The experimental details are comprehensive, which should facilitate reproducibility.

Critical weaknesses

1. The paper presents its main contribution as adaptive reward routing. However, OmniNFT has already explored modality-aware credit assignment and routing-related techniques, including layer-wise diagnosis and targeted interventions. A side-by-side comparison of Table 3 in OmniNFT and Table 2 in this paper suggests that the contribution of the proposed method in terms of routing is relatively limited. This component appears more like a set of hyperparameter optimizations built upon existing approaches than a substantially new routing mechanism. The authors should clarify their specific contributions in this aspect and provide a more detailed comparison with OmniNFT.

2. The reward reweighting component also draws heavily on design choices from existing reinforcement learning algorithms, such as group-wise reward normalization. Although these techniques appear effective, their combination does not seem sufficient to establish a clear methodological contribution in terms of originality. The warm-up strategy is also a common technique, so I am somewhat surprised that it warrants a separate discussion as a key design choice. Have the authors conducted experiments to investigate the effect of the additional hyperparameter (ewarme_{\text{warm}}) introduced by this strategy?

3. Although Fig. 1 presents reward conflicts as a key motivation and the paper proposes a reweighting mechanism to address them, the paper does not explicitly analyze how the proposed method mitigates such conflicts. It is unrealistic to expect reviewers to carefully compare individual reward values in the figures to infer this effect. Can the authors provide a more convincing way to isolate and demonstrate the contribution of the reweighting mechanism? In the current Fig. 4, the effect of warm-up is so pronounced that it is difficult to assess qualitatively or quantitatively whether reweighting leads to a better balance among the reward objectives. In addition, simply rescaling the individual rewards could potentially lead to different conclusions, making it difficult to attribute the observed improvements specifically to the proposed reweighting mechanism.

Overall, I find the proposed improvements somewhat engineering-oriented, with the paper appearing to accumulate a large number of engineering refinements. This makes the paper relatively difficult to read. More importantly, as discussed above, several components overlap with existing work. After accounting for these overlaps, the remaining contributions may not be sufficient to support the paper's claims of methodological novelty. I welcome the authors' response should I have misunderstood any aspects of the proposed method or its relationship to prior work.

Minor Comments

The figures in Figs. 4 and 5 are of relatively poor quality, with issues such as blurriness, incorrect scaling, and overlapping elements. These issues impair readability and should be addressed.

Questions for authors

Please refer to the weakness part.

8 comments
AuthorsOpenReview ↗

Response to Reviewer ic1u (Part 1)

Thank you for your thoughtful and constructive feedback. We sincerely appreciate the time and care you devoted to reviewing our work. Your comments have helped us sharpen our claims and strengthen the supporting evidence.

(W1) 1. Novelty Clarification

Our novelty lies in unified optimization, model-adaptive mechanisms, and cross-architecture applicability. To our knowledge, within joint audio-video diffusion RL, ours is the first framework to:

1-a. Unify update allocation and reward coordination. We jointly adapt where reward-driven updates act and how competing rewards are combined, addressing both dimensions within one optimization framework.

1-b. Use model evolution to guide optimization. Evolving cross-modal functionality provides online routing signals for both token weighting and layer-wise gradient scaling, while changing reward conflicts demand adaptive coordination anchored to user preferences.

1-c. Support both dual-stream and single-stream architectures. We instantiate and validate the framework on dual-stream LTX and single-stream JavisDiT++, extending adaptive reward routing beyond architecture-specific layer structures.

(W1) 2. Technical Differences from OmniNFT

We recognize OmniNFT’s contributions to modality-wise credit assignment, layer-wise gradient control, and region reweighting. Our work advances these directions in four respects:

2-a. Beyond fixed layer rules and video-only token weighting. OmniNFT uses only V2A attention to reweight video tokens, but fixes its layer boundaries and gradient scales. We use both V2A and A2V attention to adapt both audio/video token weights and layer scales.

2-b. Beyond reward assignment. OmniNFT assigns rewards to their respective modalities but does not dynamically coordinate them using within-branch gradient conflicts. We introduce branch-conditioned, preference-anchored reward coordination, adapting how rewards are combined. This is a capability that layer-routing hyperparameter tuning cannot provide.

2-c. Beyond LTX. Appendix J extends our framework to single-stream JavisDiT++, using cross-modal attention submatrices to construct directional responses and route gradients. Three-seed experiments demonstrate applicability beyond predefined layer partitions in dual-stream LTX.

2-d. Stronger performance under matched conditions. Across three independent runs per method, ours reduces DeSync relative to reproduced OmniNFT by 12.6% on LTX-2 and 18.2% on LTX-2.3, while improving audio quality and cross-modal consistency. Table 1 also evaluates the official OmniNFT checkpoints under the same evaluation protocol, confirming that the overall advantage is not limited to our reproduction.

AuthorsOpenReview ↗

Response to Reviewer ic1u (Part 2)

(W2) 3. Reward Coordination Novelty

Thank you for requesting clarification. Our novelty is a dynamic reward-coordination principle that jointly accounts for modality responsibility, evolving gradient conflicts, and user-defined objective importance.

3-a. Beyond modality assignment. Assigning rewards to the correct modality does not resolve conflicts within that modality. We introduce branch-conditioned coordination to determine how competing rewards jointly guide each branch’s updates.

3-b. Separating compatibility from importance. Purely gradient-based weighting can suppress essential synchronization objectives. We separate user priorities from gradient geometry: priors specify importance, while gradient relationships provide adaptive corrections. This differs from both fixed weighting and directly adopting geometry-derived coefficients.

(W2) 4. Warm-Up: Contributions and Sensitivity

4-a. Gains independent of warm-up. In the weighting-only ablation, branch conditioning and prior anchoring reduce MARBLE’s DeSync from 0.618 to 0.455 without warm-up (Table 2). Warm-up is a supporting stabilization measure, not an independent novelty claim.

4-b. Warm-up sensitivity. To assess dependence on warm-up length, we compared ewarm∈{0,25,50,100}e_{\mathrm{warm}}\in\{0,25,50,100\}, keeping other settings fixed. Fifty rounds capture most of the benefit, while extending to 100 yields only marginal gains, supporting our default choice.

Table R4. Sensitivity to warm-up length on LTX-2.

Methodewarme_{\mathrm{warm}}VQ↑AQ↑CLAP↑AV-IB↑JavisScore↑DeSync↓
OmniNFT—3.1365.6140.4160.2220.1950.390
Ours03.1745.7680.4210.2290.1970.372
Ours253.2785.8130.4230.2310.2030.350
Ours50 (default)3.3365.8680.4250.2350.2060.341
Ours1003.3425.8760.4250.2350.2070.338
AuthorsOpenReview ↗

Response to Reviewer ic1u (Part 3)

(W3) 5. Reward Conflict Mitigation: Direct Evidence

To isolate whether adaptive reweighting mitigates conflicts, we compare fixed and adaptive reward weights at steps 0, 300, 600, and 900, keeping routing, priors, and training budget unchanged. Following Fig. 1, a conflict occurs when two rewards assign opposite signs to a sample relative to their respective prompt-group means. Adaptive coordination reduces both within-modality and cross-modal disagreement more substantially than fixed weighting (Table R5).

Table R5. Sample-level reward conflict rates during training (%, lower indicates less disagreement).

CoordinationTraining stepWithin-video conflictWithin-audio conflictCross-modal conflict
Shared initialization035.050.051.25
Fixed weights30033.147.549.1
Fixed weights60032.446.946.2
Fixed weights90032.245.645.9
Adaptive coordination (Ours)30029.643.140.4
Adaptive coordination (Ours)60025.437.835.1
Adaptive coordination (Ours)90022.834.530.6

(Within-video conflict compares HPSv3 and VideoAlign; within-audio conflict compares AudioBox and CLAP; cross-modal conflict averages the four video–audio reward pairs.)

(W3) 6. Reward Rescaling: Our Adaptive Strategy vs. Manual Rescaling

We agree that reward coordination amounts to rescaling rewards. However, our contribution is how these scales are determined: evolving branch-wise gradient conflicts drive adaptive corrections, while preference anchors prevent important objectives from being discarded. Once priors are specified, weights adapt automatically throughout training.

Manual scaling faces a changing optimization problem. Fixed weights cannot respond to evolving conflicts, while repeatedly adjusting them by hand requires costly trial and error during expensive post-training. Our mechanism automates this adaptation using conflict information rather than repeated human intervention.

7. Figure Readability

Thank you for noting these issues. We will revise Figs. 4 and 5 to improve clarity, correct scaling, and eliminate overlapping elements.

Reviewer_R5EXOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Adaptive Reward Routing for multi-reward post-training of joint audio-video diffusion models. It combines response-based token and layer routing with branch-conditioned reward reweighting anchored to predefined preferences. Experiments report improvements over fixed-weight, gradient-based, and static-routing baselines, with additional ablations, intervention studies, and an extension to a unified backbone.

Critical strengths
  1. The method addresses a practical problem: balancing modality quality, semantic alignment, and synchronization during joint generation.
  2. The experiments include strong baselines, component ablations, and controlled route-refresh comparisons.
Critical weaknesses
  1. The proxy validation measures prediction sensitivity, rather than reward-specific update utility. A highly influential path may be beneficial or harmful, so the justification for allocating it more gradient remains incomplete.
  2. The reweighting rule preserves a nonzero contribution from positive priors, but does not necessarily preserve their relative priorities or resulting user preferences. The claim should be stated more narrowly.
Questions for authors

What exactly does “preference-preserving” guarantee? How sensitive are the trade-offs to the residual mixing strength and prior weights?

2 comments
AuthorsOpenReview ↗

Response to Reviewer R5EX (Part 1)

Thank you for your thoughtful and constructive feedback. We sincerely appreciate the time and care you devoted to reviewing our work. Your comments have helped us sharpen our claims and strengthen the supporting evidence.

(W1) 1. Prediction Influence vs. Reward-Update Utility

Thank you for highlighting the distinction between prediction influence and update utility. We clarify why prioritizing influential locations does not simply reinforce their current behavior (1-a), then test update utility directly: response-based routing outperforms alternative criteria (1-b, Table R1), and stronger routing improves optimization within the tested range (1-c, Table R2).

1-a. Rationale clarification. As validated in Sec. 5.4, our proxy measures how strongly a location influences the model’s output. This identifies where to focus updates, while DiffusionNFT’s reward-conditioned positive/negative-policy regression losses determine how to update. Allocating more updates to influential locations does not unconditionally reinforce their current behavior. Instead, it enables DiffusionNFT’s reward supervision to reinforce beneficial behavior and correct unfavorable behavior at those locations.

1-b. Experiment 1: Comparison with alternative routing criteria. (Table R1) To test whether gains depend on the response-based criterion rather than nonuniform weighting alone, we replace only the routing criterion while keeping other training settings fixed. Gradient-magnitude routing uses gradients of the unweighted DiffusionNFT loss with respect to cross-modal responses, while random routing shuffles token weights and layer scales while preserving their distributions. Our method outperforms both alternatives across all six metrics. These results support the effectiveness of cross-modal responses for allocating reward-driven updates.

Table R1. Comparison of routing criteria on LTX-2.

MethodVQ↑AQ↑CLAP↑AV-IB↑JavisScore↑DeSync↓
OmniNFT3.1365.6140.4160.2220.1950.390
Random routing3.1025.5340.3950.2100.1810.405
Gradient-magnitude routing3.2015.7120.4110.2210.2000.367
Response-based routing (Ours)3.3365.8680.4250.2350.2060.341

1-c. Experiment 2: Effect of routing strength. (Table R2) To verify whether prioritizing influential locations benefits reward optimization, we evaluate both OmniNFT (using its own routing strategy) and our method at routing strengths η∈{0.2,0.6,1.0}\eta\in\{0.2,0.6,1.0\}, keeping other settings and post-training steps fixed on LTX-2. Here, λη=1+η(λ−1)\lambda_\eta=1+\eta(\lambda-1) and αη=ηα\alpha_\eta=\eta\alpha scale token reweighting and layer detachment, respectively, from disabled (η=0\eta=0) to each method’s default (η=1\eta=1). Both methods improve as routing strengthens, while ours performs better overall at each matched strength

Table R2. Effect of routing strength on LTX-2 under the same number of post-training steps.

Methodη\etaVQ↑AQ↑TV-IB↑TA-IB↑CLIP↑CLAP↑AV-IB↑AVHScore↑JavisScore↑DeSync↓
GDPO (reference)—2.7225.4500.2610.1380.3120.3470.1740.1750.1550.671
OmniNFT0.22.8915.5030.2620.1400.3100.3730.1930.1920.1710.566
OmniNFT0.63.0485.5710.2640.1430.3110.4010.2120.2090.1870.448
OmniNFT1.03.1365.6140.2650.1450.3120.4160.2220.2190.1950.390
Ours0.23.0985.5920.2640.1490.3120.3860.2020.2010.1780.419
Ours0.63.2145.7540.2670.1600.3130.4120.2240.2230.1970.392
Ours1.03.3365.8680.2680.1670.3140.4250.2350.2340.2060.341
AuthorsOpenReview ↗

Response to Reviewer R5EX (Part 2)

(W2) 2. Preference Preservation: Guarantees and Sensitivity to Mixing Strength and Prior Weights

Thank you for pointing out this distinction. We agree that “preference-preserving” overstates the guarantee and clarify its scope below, together with sensitivity results.

2-a. Guarantee clarification. For positive priors and 0≤κ<10\leq\kappa<1, our rule guarantees ωm,k≥(1−κ)ωm,kprior\omega_{m,k}\geq(1-\kappa)\omega^{\mathrm{prior}}_{m,k} and preserves the total weight within each branch. It does not guarantee unchanged weight rankings or resulting output preferences. We will use “preference-anchored” to accurately describe the retained prior contribution.

2-b. Sensitivity to mixing strength. To assess the effect of the residual correction, we vary only κ\kappa, keeping priors, routing, warm-up, and training time fixed. Both larger mixing strengths underperform the default κ=0.3\kappa=0.3 across all six metrics, supporting our default choice within the tested range (Table R3).

Table R3. Sensitivity to residual mixing strength on LTX-2.

κ\kappaVQ↑AQ↑CLAP↑AV-IB↑JavisScore↑DeSync↓
0.33.3365.8680.4250.2350.2060.341
0.53.2475.8250.4240.2310.2020.356
0.73.2115.7830.4210.2320.2000.372

2-c. Sensitivity to prior weights. Appendix H, Table 8 compares visual-, audio-, and cross-modal-oriented priors with equal total weight at fixed κ=0.3\kappa=0.3. Each profile achieves the best results on its corresponding metrics among the three settings. This demonstrates that priors meaningfully steer performance trade-offs, without implying a universal guarantee of preference preservation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.