- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies reward fine-tuning for few-step autoregressive joint audio and video generation. In this setting every generated chunk is simultaneously an output to be optimized and part of the context on which later chunks depend.
The proposed method, Fork-NFT, is a local NFT update. It branches multiple joint continuations from a shared generated prefix and compares them under the same generated history. Modality-specific relative advantages then split the credit: video quality credit is assigned to the full continuation after the sink boundary fork, and audio quality credit is assigned to short windows after each fork. The update trains on paired sampler states while re-materializing the causal KV context under the current policy, and it regularizes how cross-modal attention reads the committed history.
Validation uses a 22B eight-step autoregressive model (LTX-AR). The paper reports improved quality, consistency, and synchronization over matched autoregressive baselines on JavisBench and on long-horizon generation, together with ablations for each design component.
Critical strengths
- Sensible problem decomposition: The paper decomposes the problem in a way that makes sense for this setting, where every generated chunk is both an output to be optimized and part of the context on which later chunks depend.
- Sampler-state-aware training: Sampler-state-aware training for joint audio and video autoregressive generation is somewhat novel.
Critical weaknesses
Major weaknesses
W1. Missing baselines and comparisons: Four directly relevant methods are not compared against experimentally.
- DiffusionNFT. DiffusionNFT is the direct foundation of OmniNFT and of Fork-NFT, but it is never used as a baseline. Without it, the gain attributable to the proposed update cannot be separated from the gain already provided by its foundation.
- OmniForcing. The paper discusses OmniForcing but reports no experimental comparison with it.
- GDPO. GDPO is a relevant multi-reward optimization baseline and is not compared against.
- AR-CoPO. AR-CoPO targets video generation only, and my understanding is that it also updates the chunk. It nevertheless appears adaptable to joint audio and video generation, and should therefore be considered as a baseline.
References for the methods named above:
- Kaiwen Zheng et al., DiffusionNFT: Online diffusion reinforcement with forward process, ICLR, 2026.
- Guohui Zhang et al., OmniNFT: Modality-wise omni diffusion reinforcement for joint audio-video generation, arXiv preprint, 2026.
- Yaofeng Su et al., OmniForcing: Unleashing real-time joint audio-visual generation, ECCV, 2026.
- Dailan He et al., AR-CoPO: Align autoregressive video generation with contrastive policy optimization, ECCV, 2026.
W2. No user study: The central quality and synchronization claims rest entirely on automatic metrics, with no human evaluation to corroborate them.
W3. No videos in the supplementary material: No generated videos are provided, so the visual quality of the outputs cannot be assessed.
W4. Possibly unfair comparison with OmniNFT: The paper does not describe its reimplementation of OmniNFT in sufficient detail, and the improvement that OmniNFT achieves here is far lower than the improvement reported in the original OmniNFT paper. A further concern is that this paper uses no audio-video synchronization reward. Such a reward is important for joint audio and video generation, and it is used in OmniNFT.
Minor weaknesses
W5. No discussion of limitations or failure cases: The paper contains no limitations section and no failure case analysis.
Questions for authors
Please see Weaknesses above. I would be happy to raise my score if the author can resolve my concerns.
6 comments
W2 Human evaluation of quality and synchronization:
Fork-NFT receives more preferences than OmniNFT for video quality, audio quality, and audiovisual synchronization in our blinded pairwise evaluation. Ten participants evaluated 50 cases with matched prompts and generation seeds, rating each criterion separately with a tie option.
| Criterion | Fork-NFT preferred (%) | Tie (%) | OmniNFT preferred (%) |
|---|---|---|---|
| Video quality | 56 | 29 | 15 |
| Audio quality | 57 | 16 | 27 |
| Audiovisual synchronization | 47 | 21 | 32 |
The human-evaluation website demo source is included in ForkNFT_Materials.zip under website_source/; open website_source/index.html to run the demo. During evaluation, participants see the prompt and two comparison videos and choose A, Tie, or B for each of the three criteria.
Revision note. Sec. 5.3 and Appendix C.3, Table 10, now report the human study and its protocol.
W3 Playable audiovisual supplementary material:
We apologize for omitting playable comparisons from the original submission. We now provide three playable comparisons in ForkNFT_Materials.zip: Case_1.mp4–Case_5.mp4 in short_videos/ForkNFT/ and short_videos/OmniNFT/, with prompts in short_videos/prompts.txt. Both OmniNFT and Fork-NFT were trained on the same LTX-AR backbone with the same training budget.
Revision note. Appendix C.3 describes the playable audiovisual supplement and the human-evaluation web demo.
W4 OmniNFT reimplementation, performance gains, and synchronization reward:
We retain all three OmniNFT designs on the same LTX-AR initialization and eight-step sampler, with the same training-update budget as Fork-NFT. Fork-NFT also uses a synchronization reward on the intersection of its video and audio credit windows; the submitted description omitted this term and its scope.
OmniNFT reimplementation. The three designs are implemented as follows:
- Modality-wise advantage routing. We compute advantages separately for each reward, route video and audio rewards to their respective NFT losses, and route the synchronization advantage to both modalities. Whole-rollout advantages supervise the corresponding losses across chunks. Following the official code, NFT operates on clean predictions, with training inputs obtained by forward-noising generated endpoints.
- Layer-wise gradient surgery. We retain the official layer-dependent gradient scaling on the audio context supplying K/V to audio-to-video attention. This attenuates gradients from the video loss into the audio branch while preserving the forward computation. With detached AR history caches, it acts on the current chunk’s cross-modal gradient paths.
- Region-wise reweighting. We derive video-token weights from audio-query attention to video keys and apply them to the video NFT loss. For AR training, we recompute these weights on the current chunk at the first sampled training timestep; the official implementation collects them during rollout sampling.
Scope in the AR setting. Gradient surgery and region reweighting address within-chunk gradient interference and spatial loss allocation, respectively. Cached history is detached, so gradient surgery acts on current-chunk cross-modal paths without directly constraining how the current chunk attends to history. Region reweighting redistributes loss within the current chunk while retaining the same whole-rollout advantage; it does not distinguish reward variation associated with the shared prefix from that of the sampled continuation. Fork-NFT explicitly addresses these temporal-credit and historical-attention questions through modality-specific fork windows and history-marginal regularization.
Causal caches are rebuilt from generated prefixes using the current policy. Our reported gains are relative to LTX-AR, whereas the original OmniNFT results use a bidirectional LTX-2 baseline, so the gain magnitudes are measured from different starting points.
Synchronization reward correction. We checked the training code and confirmed that Fork-NFT uses an audiovisual synchronization reward. We apologize for omitting this term and its temporal scope from the submitted reward specification. Synchronization is evaluated only within the intersection of the video and audio credit windows, and its advantage is assigned to both modalities within that intersection. This is the initial short window at the sink-boundary fork. The remaining video continuation receives only video-quality credit, and later audio-only credit windows receive no synchronization term. We have corrected the reward specification and its temporal support in the revised manuscript. This corrects the description of the training used for the reported results.
Revision note. Synchronization credit is explicit in Sec. 4.4, Eq. 7, and Sec. 5.1; Appendix A.1, Eq. 12 and Tables 6–7, now specify its definition, weight, and temporal support. Appendix A.2 details all three OmniNFT components and the AR adaptation.
W5 Limitations and failure cases:
We acknowledge that the original manuscript omitted a discussion of limitations and failure cases. Long video generation can still exhibit temporal drift, and validation across different model backbones remains limited. The revised manuscript adds a focused discussion of long-video failure modes, supported by the segment-wise comparisons, and analyzes applicability to other backbones.
Revision note. The scope paragraph in Sec. 5.4, Appendix D.3 (Table 14), and Appendix E now discuss long-horizon drift and cross-backbone limitations.
We hope these results and clarifications help address your concerns, and we would be happy to discuss any remaining questions.
We thank Reviewer dSks for recognizing our problem decomposition and the novelty of sampler-state-aware training. We sincerely hope you will reconsider your assessment in light of the results and clarifications below.
W1 Missing baselines and comparisons:
We have added comparisons with DiffusionNFT, GDPO, AR-CoPO, and the released OmniForcing model. The retrained RL baselines use the same autoregressive initialization and number of training updates as the Fork-NFT comparison run, with matched training prompts, rollout collection budget, optimizer settings, and LoRA configuration.
We evaluate on the complete 1,000-prompt JavisBench-mini suite, using the same generation seed for each prompt across methods and five-second outputs.
Fork-NFT achieves the highest VQ, AQ, and JavisScore and the lowest DeSync. Its AV-IB is close to GDPO (0.2391 versus 0.2395).
AV adaptation and training losses. DiffusionNFT and GDPO use the same modality-wise clean-prediction NFT loss construction as OmniNFT: compute the implicit positive/negative prediction loss separately in video and audio latent space, average within each modality, and combine the two modality losses with equal weight:
=\tfrac12\left[\ell^v_{\mathrm{NFT}}(A)+\ell^a_{\mathrm{NFT}}(A)\right] +\lambda_{\mathrm{ref}}\mathcal L_{\mathrm{ref}}.$$ Here, A is the joint-reward advantage. The reference term averages the video/audio squared prediction deviations from the reference predictor, whose parameters are frozen. Both predictors use the same input state and detached current-policy cache. - **DiffusionNFT:** independently sample complete joint audio–video rollouts and forward-noise both clean endpoints. Combine quality, alignment, and synchronization rewards into a joint group-relative advantage A, which weights both modality losses above. - **GDPO:** adapt its reward-decoupled advantage normalization to joint audio–video generation using the AV-NFT objective above. Construct A by normalizing each reward separately within its group, aggregating the weighted advantages, and normalizing the aggregate over the collected pool. - **AR-CoPO:** extend the on-policy contrastive objective to paired video/audio candidates. Perturb both modalities at the pivot while sharing non-pivot noise, and score each completed joint sequence. Average the video and audio mean-squared clean-prediction distances to define the joint candidate distribution, then apply the clipped contrastive GRPO loss only at the pivot, together with the same reference-prediction penalty. The three added RL baselines follow OmniNFT’s reward components, coefficients, and whole-rollout scoring convention. DiffusionNFT and GDPO apply the resulting joint advantage across chunks; AR-CoPO retains its pivot-only update. Fork-NFT uses its modality-specific scoring and credit windows, with synchronization credit confined to their intersection. **OmniForcing.** OmniForcing uses DMD, causal ODE regression, and joint Self-Forcing distillation rather than reward-based RL. We evaluate its released causal checkpoint with the native inference configuration as a system-level reference. [OmniForcing, Secs. 3.3–3.4](https://arxiv.org/html/2603.11647#S3.SS3). **Revision note.** Sec. 5.3 (Table 4) now presents the consolidated comparison; Appendices A.2 and C.4 specify baseline implementations, matched RL budgets, and evaluation settings.