acceptodds
Under review as a conference paper at ICLR 2027

Fork-NFT: Forked-Credit Negative-Aware Fine-Tuning for Autoregressive Joint Audio–Video Generation

Abstract

Autoregressive joint audio–video generation poses a reinforcement learning problem beyond optimizing complete clips. Each continuation inherits an autoregressive history, while its audio and video streams are tightly coupled: an update may improve one modality while degrading the other or disrupting their synchronization. Consequently, clip-level rewards do not identify which chunks under a shared history should receive video or audio credit. We introduce Fork-NFT, a local reinforcement learning update that branches multiple joint audio–video continuations from a shared prefix and compares them under the same generated history. Modality-specific relative advantages assign video-quality credit to the full continuation after the retained prefix and audio-quality credit to short windows after each fork, while both streams remain jointly generated. To align this local credit with few-step generation, Fork-NFT trains on paired intermediate states recorded from the sampler while re-materializing the causal KV context from each generated prefix under the current policy. We further constrain how cross-modal attention reads the committed history while leaving attention within the active chunk free to adapt. On a 22B eight-step autoregressive model, Fork-NFT improves quality and audio–video consistency over matched baselines on both short- and long-form evaluations, while retaining the same sampler and backbone architecture.

Primary area:
generative models
Keywords:
Audio–Video Generation, Reinforcement Fine-Tuning, Diffusion and Flow Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_dSksOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies reward fine-tuning for few-step autoregressive joint audio and video generation. In this setting every generated chunk is simultaneously an output to be optimized and part of the context on which later chunks depend.

The proposed method, Fork-NFT, is a local NFT update. It branches multiple joint continuations from a shared generated prefix and compares them under the same generated history. Modality-specific relative advantages then split the credit: video quality credit is assigned to the full continuation after the sink boundary fork, and audio quality credit is assigned to short windows after each fork. The update trains on paired sampler states while re-materializing the causal KV context under the current policy, and it regularizes how cross-modal attention reads the committed history.

Validation uses a 22B eight-step autoregressive model (LTX-AR). The paper reports improved quality, consistency, and synchronization over matched autoregressive baselines on JavisBench and on long-horizon generation, together with ablations for each design component.

Critical strengths
  1. Sensible problem decomposition: The paper decomposes the problem in a way that makes sense for this setting, where every generated chunk is both an output to be optimized and part of the context on which later chunks depend.
  2. Sampler-state-aware training: Sampler-state-aware training for joint audio and video autoregressive generation is somewhat novel.
Critical weaknesses

Major weaknesses

W1. Missing baselines and comparisons: Four directly relevant methods are not compared against experimentally.

  • DiffusionNFT. DiffusionNFT is the direct foundation of OmniNFT and of Fork-NFT, but it is never used as a baseline. Without it, the gain attributable to the proposed update cannot be separated from the gain already provided by its foundation.
  • OmniForcing. The paper discusses OmniForcing but reports no experimental comparison with it.
  • GDPO. GDPO is a relevant multi-reward optimization baseline and is not compared against.
  • AR-CoPO. AR-CoPO targets video generation only, and my understanding is that it also updates the chunk. It nevertheless appears adaptable to joint audio and video generation, and should therefore be considered as a baseline.

References for the methods named above:

  • Kaiwen Zheng et al., DiffusionNFT: Online diffusion reinforcement with forward process, ICLR, 2026.
  • Guohui Zhang et al., OmniNFT: Modality-wise omni diffusion reinforcement for joint audio-video generation, arXiv preprint, 2026.
  • Yaofeng Su et al., OmniForcing: Unleashing real-time joint audio-visual generation, ECCV, 2026.
  • Dailan He et al., AR-CoPO: Align autoregressive video generation with contrastive policy optimization, ECCV, 2026.

W2. No user study: The central quality and synchronization claims rest entirely on automatic metrics, with no human evaluation to corroborate them.

W3. No videos in the supplementary material: No generated videos are provided, so the visual quality of the outputs cannot be assessed.

W4. Possibly unfair comparison with OmniNFT: The paper does not describe its reimplementation of OmniNFT in sufficient detail, and the improvement that OmniNFT achieves here is far lower than the improvement reported in the original OmniNFT paper. A further concern is that this paper uses no audio-video synchronization reward. Such a reward is important for joint audio and video generation, and it is used in OmniNFT.

Minor weaknesses

W5. No discussion of limitations or failure cases: The paper contains no limitations section and no failure case analysis.

Questions for authors

Please see Weaknesses above. I would be happy to raise my score if the author can resolve my concerns.

6 comments

We thank Reviewer dSks for recognizing our problem decomposition and the novelty of sampler-state-aware training. We sincerely hope you will reconsider your assessment in light of the results and clarifications below.

W1 Missing baselines and comparisons:

We have added comparisons with DiffusionNFT, GDPO, AR-CoPO, and the released OmniForcing model. The retrained RL baselines use the same autoregressive initialization and number of training updates as the Fork-NFT comparison run, with matched training prompts, rollout collection budget, optimizer settings, and LoRA configuration.

We evaluate on the complete 1,000-prompt JavisBench-mini suite, using the same generation seed for each prompt across methods and five-second outputs.

MethodVQ ↑AQ ↑AV-IB ↑DeSync ↓JavisScore ↑
DiffusionNFT2.87774.95320.23700.74600.1929
GDPO2.89534.95670.23950.70100.1953
AR-CoPO2.91094.96270.22550.69900.1878
Fork-NFT3.0134.97820.23910.54900.2003
OmniForcing1.98584.81010.15921.05700.1270

Fork-NFT achieves the highest VQ, AQ, and JavisScore and the lowest DeSync. Its AV-IB is close to GDPO (0.2391 versus 0.2395).

AV adaptation and training losses. DiffusionNFT and GDPO use the same modality-wise clean-prediction NFT loss construction as OmniNFT: compute the implicit positive/negative prediction loss separately in video and audio latent space, average within each modality, and combine the two modality losses with equal weight:

=\tfrac12\left[\ell^v_{\mathrm{NFT}}(A)+\ell^a_{\mathrm{NFT}}(A)\right] +\lambda_{\mathrm{ref}}\mathcal L_{\mathrm{ref}}.$$ Here, A is the joint-reward advantage. The reference term averages the video/audio squared prediction deviations from the reference predictor, whose parameters are frozen. Both predictors use the same input state and detached current-policy cache. - **DiffusionNFT:** independently sample complete joint audio–video rollouts and forward-noise both clean endpoints. Combine quality, alignment, and synchronization rewards into a joint group-relative advantage A, which weights both modality losses above. - **GDPO:** adapt its reward-decoupled advantage normalization to joint audio–video generation using the AV-NFT objective above. Construct A by normalizing each reward separately within its group, aggregating the weighted advantages, and normalizing the aggregate over the collected pool. - **AR-CoPO:** extend the on-policy contrastive objective to paired video/audio candidates. Perturb both modalities at the pivot while sharing non-pivot noise, and score each completed joint sequence. Average the video and audio mean-squared clean-prediction distances to define the joint candidate distribution, then apply the clipped contrastive GRPO loss only at the pivot, together with the same reference-prediction penalty. The three added RL baselines follow OmniNFT’s reward components, coefficients, and whole-rollout scoring convention. DiffusionNFT and GDPO apply the resulting joint advantage across chunks; AR-CoPO retains its pivot-only update. Fork-NFT uses its modality-specific scoring and credit windows, with synchronization credit confined to their intersection. **OmniForcing.** OmniForcing uses DMD, causal ODE regression, and joint Self-Forcing distillation rather than reward-based RL. We evaluate its released causal checkpoint with the native inference configuration as a system-level reference. [OmniForcing, Secs. 3.3–3.4](https://arxiv.org/html/2603.11647#S3.SS3). **Revision note.** Sec. 5.3 (Table 4) now presents the consolidated comparison; Appendices A.2 and C.4 specify baseline implementations, matched RL budgets, and evaluation settings.

W2 Human evaluation of quality and synchronization:

Fork-NFT receives more preferences than OmniNFT for video quality, audio quality, and audiovisual synchronization in our blinded pairwise evaluation. Ten participants evaluated 50 cases with matched prompts and generation seeds, rating each criterion separately with a tie option.

CriterionFork-NFT preferred (%)Tie (%)OmniNFT preferred (%)
Video quality562915
Audio quality571627
Audiovisual synchronization472132

The human-evaluation website demo source is included in ForkNFT_Materials.zip under website_source/; open website_source/index.html to run the demo. During evaluation, participants see the prompt and two comparison videos and choose A, Tie, or B for each of the three criteria.

Revision note. Sec. 5.3 and Appendix C.3, Table 10, now report the human study and its protocol.

W3 Playable audiovisual supplementary material:

We apologize for omitting playable comparisons from the original submission. We now provide three playable comparisons in ForkNFT_Materials.zip: Case_1.mp4–Case_5.mp4 in short_videos/ForkNFT/ and short_videos/OmniNFT/, with prompts in short_videos/prompts.txt. Both OmniNFT and Fork-NFT were trained on the same LTX-AR backbone with the same training budget.

Revision note. Appendix C.3 describes the playable audiovisual supplement and the human-evaluation web demo.

W4 OmniNFT reimplementation, performance gains, and synchronization reward:

We retain all three OmniNFT designs on the same LTX-AR initialization and eight-step sampler, with the same training-update budget as Fork-NFT. Fork-NFT also uses a synchronization reward on the intersection of its video and audio credit windows; the submitted description omitted this term and its scope.

OmniNFT reimplementation. The three designs are implemented as follows:

  1. Modality-wise advantage routing. We compute advantages separately for each reward, route video and audio rewards to their respective NFT losses, and route the synchronization advantage to both modalities. Whole-rollout advantages supervise the corresponding losses across chunks. Following the official code, NFT operates on clean predictions, with training inputs obtained by forward-noising generated endpoints.
  2. Layer-wise gradient surgery. We retain the official layer-dependent gradient scaling on the audio context supplying K/V to audio-to-video attention. This attenuates gradients from the video loss into the audio branch while preserving the forward computation. With detached AR history caches, it acts on the current chunk’s cross-modal gradient paths.
  3. Region-wise reweighting. We derive video-token weights from audio-query attention to video keys and apply them to the video NFT loss. For AR training, we recompute these weights on the current chunk at the first sampled training timestep; the official implementation collects them during rollout sampling.

Scope in the AR setting. Gradient surgery and region reweighting address within-chunk gradient interference and spatial loss allocation, respectively. Cached history is detached, so gradient surgery acts on current-chunk cross-modal paths without directly constraining how the current chunk attends to history. Region reweighting redistributes loss within the current chunk while retaining the same whole-rollout advantage; it does not distinguish reward variation associated with the shared prefix from that of the sampled continuation. Fork-NFT explicitly addresses these temporal-credit and historical-attention questions through modality-specific fork windows and history-marginal regularization.

Causal caches are rebuilt from generated prefixes using the current policy. Our reported gains are relative to LTX-AR, whereas the original OmniNFT results use a bidirectional LTX-2 baseline, so the gain magnitudes are measured from different starting points.

Synchronization reward correction. We checked the training code and confirmed that Fork-NFT uses an audiovisual synchronization reward. We apologize for omitting this term and its temporal scope from the submitted reward specification. Synchronization is evaluated only within the intersection of the video and audio credit windows, and its advantage is assigned to both modalities within that intersection. This is the initial short window at the sink-boundary fork. The remaining video continuation receives only video-quality credit, and later audio-only credit windows receive no synchronization term. We have corrected the reward specification and its temporal support in the revised manuscript. This corrects the description of the training used for the reported results.

Revision note. Synchronization credit is explicit in Sec. 4.4, Eq. 7, and Sec. 5.1; Appendix A.1, Eq. 12 and Tables 6–7, now specify its definition, weight, and temporal support. Appendix A.2 details all three OmniNFT components and the AR adaptation.

W5 Limitations and failure cases:

We acknowledge that the original manuscript omitted a discussion of limitations and failure cases. Long video generation can still exhibit temporal drift, and validation across different model backbones remains limited. The revised manuscript adds a focused discussion of long-video failure modes, supported by the segment-wise comparisons, and analyzes applicability to other backbones.

Revision note. The scope paragraph in Sec. 5.4, Appendix D.3 (Table 14), and Appendix E now discuss long-horizon drift and cross-backbone limitations.

We hope these results and clarifications help address your concerns, and we would be happy to discuss any remaining questions.

Reviewer_8RDgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

Fork-NFT studies reward-based post-training for autoregressive audio–video generation. The method compares continuations from a shared prefix and uses different temporal credit windows for audio and video. It also introduces sampler-state replay, current-policy cache reconstruction, and history-marginal attention regularization. Experiments on an LTX-2.3-derived autoregressive model show improvements in perceptual quality, audiovisual consistency, and synchronization over matched baselines. The authors also provide ablations of the main components and temporal-window settings.

Critical strengths
  • Relevant problem and reasonable design. The paper addresses an important challenge in reward optimization for autoregressive audio–video generation, where each generated chunk also affects future predictions. The combination of shared-prefix comparisons, sampler-state replay, and history regularization is well motivated.

  • Promising experimental results. Fork-NFT improves several quality and audiovisual consistency metrics over the matched autoregressive baselines under the same inference setup. The component and temporal-window ablations also help illustrate the effects of different design choices.

Critical weaknesses
  • Unclear contribution beyond prior work. The differences from OmniNFT and existing shared-prefix optimization methods are not sufficiently clear. In particular, I would like to understand how the LTX-AR + OmniNFT baseline handles sampler-state replay, current-policy cache reconstruction, and temporal credit assignment. Without a clearer comparison under similar training budgets, it is difficult to judge how much of the improvement comes from the proposed changes.

  • Limited evidence for the stability of improvements. The gains over OmniNFT are encouraging, but some margins are relatively small, and the results are mainly reported as point estimates. Paired confidence intervals or per-prompt comparisons would help determine whether the improvements are consistent and whether the component rankings are reliable.

  • Temporal credit assignment is not fully isolated. The ablations in Table 6 are useful, but the different configurations appear to change both the scoring horizon and the chunks receiving credit. This makes it difficult to identify which factor accounts for the improvements. A more controlled comparison that varies these factors separately would better support the proposed temporal credit design.

Questions for authors
  • Long-form generation: The method is partly motivated by long-form and streaming generation, but the training rollouts are approximately 20.5 seconds, and the longer evaluation covers only 20.2 seconds. Could the authors provide longer generation examples and analyze quality and synchronization across successive segments? This would help assess whether Fork-NFT reduces accumulated drift over time.

  • Generalization across architectures: All experiments use an LTX-derived backbone. Could the authors consider testing Fork-NFT on a different architecture, such as a suitable causal MiniMax-H3 variant? If this is not feasible, it would be helpful to discuss which components, particularly the history regularizer, depend on the LTX architecture and how they might transfer to other models.

7 comments

We thank Reviewer 8RDg for recognizing the importance of our problem, our design rationale, and promising results. We sincerely hope you will reconsider your assessment in light of the results and clarifications below.

W1 Contribution beyond prior work and baseline comparability:

Compared with OmniNFT. Fork-NFT replaces endpoint re-noising and whole-rollout reward credit with recorded sampler states and modality-specific shared-prefix credit, and adds history-marginal attention regularization. Our OmniNFT baseline uses the same LTX-AR initialization and eight-step sampler as Fork-NFT, with matched training updates, rollout budgets, optimizer settings, and LoRA configurations. It retains OmniNFT’s three designs: separate reward advantages routed to modality losses, layer-dependent gradient scaling on the audio K/V path in audio-to-video attention, and attention-derived video-token loss weights. For the AR adaptation, region weights are recomputed on the current chunk during training. The key implementation choices are:

  • Training states: OmniNFT forward-noises generated endpoints; Fork-NFT reuses paired audio/video states recorded at the same step of the actual few-step sampler.
  • Causal context: both methods rebuild detached KV caches from generated prefixes using the current policy. Cache reconstruction is therefore shared by the controlled baseline.
  • Reward credit and history: OmniNFT applies whole-rollout advantages across chunks. Fork-NFT uses shared-prefix comparisons with modality-specific credit windows and explicitly regularizes attention to committed history.

In the AR implementation, these mechanisms operate within each chunk. With detached history caches, gradient surgery acts on current-chunk cross-modal paths, while region reweighting redistributes current-chunk losses under the same whole-rollout advantage. These mechanisms leave temporal reward attribution and attention to committed history implicit, motivating Fork-NFT’s modality-specific credit windows and history-marginal regularization.

Compared with AR-CoPO. AR-CoPO perturbs the pivot chunk’s initial noise, shares other noise sources across candidates, scores completed sequences, and applies a contrastive GRPO update only at the pivot. AR-CoPO, §3.2 Fork-NFT independently samples joint continuations after a shared prefix, excludes that prefix from reward credit, and applies NFT updates over different temporal spans: full-continuation video-quality credit at the sink boundary, and short-window audio-quality credit at early and later forks. This follows our observation that video and audio rewards exhibit different variation across shared-prefix continuations. Synchronization credit is restricted to the intersection of the two credit windows.

Our AR-CoPO baseline adapts its on-policy formulation to joint audio–video generation by perturbing both modalities at the pivot and averaging their modality-normalized clean-prediction distances to construct the contrastive objective, while retaining pivot-only updates.

Revision note. Secs. 4.3–4.5 and Appendix A.2 specify the state inputs, shared cache conditioning, temporal credit, and AV baseline implementations.

W2 Reliability of improvements and component rankings:

We reevaluated the OmniNFT and Fork-NFT checkpoints reported in the main table on 200 JavisBench-mini prompts using three matched generation seeds, generating 600 five-second audio–video samples per method. Fork-NFT achieves statistically significant improvements in VQ, AQ, AV-IB, and JavisScore.

These gains are consistent across all three generation seeds and extend to a majority of prompts. DeSync also improves on average, although the improvement is not statistically significant.

MetricFork-NFTOmniNFTPaired difference95% CIPrompt win / tie / loss (%)
VQ ↑3.03792.9515+0.0864[+0.0456, +0.1262]63.5 / 1.0 / 35.5
AQ ↑4.92894.9037+0.0252[+0.0122, +0.0386]59.0 / 0.0 / 41.0
AV-IB ↑0.23460.2210+0.0136[+0.0095, +0.0179]70.0 / 0.0 / 30.0
DeSync ↓0.56600.5830-0.0170[-0.0823, +0.0367]46.0 / 10.5 / 43.5
JavisScore ↑0.20010.1909+0.0092[+0.0056, +0.0128]64.0 / 0.0 / 36.0

Differences are Fork-NFT minus OmniNFT. Prompt win/tie/loss rates compare each method's mean score across the three seeds for that prompt; confidence intervals use paired bootstrap resampling over prompts, retaining all three seeds within each sampled prompt. The revised manuscript includes these paired estimates and the seed-specific results.

For component effects, Table 5 of the revised manuscript and the controlled comparisons in W3 provide ablation estimates. The confidence intervals above quantify the complete method's gain over OmniNFT; they do not establish statistical significance of individual component rankings.

Revision note. Sec. 5.3 and Appendix C.2, Tables 8–9, include paired uncertainty and seed-specific results.

W3 Separating scoring horizons from credit horizons:

With full-continuation scoring held fixed, extending video credit to the full continuation further improves VQ, AQ, AV-IB, and JavisScore while reducing DeSync. We isolated this effect through finer-grained ablations on JavisBench-mini at the same sink-boundary fork, varying video scoring and credit spans while keeping the audio configuration and other training settings fixed. We thank the reviewer for highlighting the need to separate these factors.

SettingVideo scoring spanVideo credit spanVQ ↑AQ ↑AV-IB ↑DeSync ↓JavisScore ↑
AFirst four chunksFirst four chunks2.80554.95730.22210.53600.1916
BFull continuationFirst four chunks2.88694.95690.23140.58600.1910
C: FullFull continuationFull continuation3.0134.97820.23910.54900.2003

Comparing A with B, full-continuation scoring improves VQ and AV-IB, but does not improve JavisScore. Comparing B with C, with the same full-continuation scoring, extending credit to the complete continuation further improves VQ, AQ, AV-IB, and JavisScore, while reducing DeSync. Setting C achieves the highest VQ, AQ, AV-IB, and JavisScore among the three configurations. These comparisons distinguish the benefit of evaluating a longer continuation from that of assigning reward credit over that continuation.

Our temporal design is grounded in the analysis of modality-dependent reward variation among continuations sharing the same prefix (Sec. 4.2, Table 1, and Fig. 2b). This analysis motivates full-continuation video credit from an early fork and short-window audio credit at multiple fork positions. Our contribution is to characterize these different temporal requirements and design the fork locations and reward-credit windows accordingly.

Revision note. Sec. 5.4 and Appendix D.1, Table 11, now separate scoring span from credit span.

Q1 Generation beyond the training horizon and accumulated drift:

We provide two 30-second comparisons and segment-wise analyses showing better preservation of video and motion quality, while synchronization can still deteriorate in the final segment. Our core contribution is reinforcement learning for autoregressive audio–video generation. We retain the backbone’s existing extrapolation strategy without additional optimization for generation beyond the training horizon, so longer outputs can still exhibit accumulated errors.

We provide two playable 30-second comparisons with LTX-AR in ForkNFT_Materials.zip (long_videos/case_A_30s_comparison.mp4 and long_videos/case_B_30s_comparison.mp4; prompts in long_videos/prompts.txt), extending beyond the approximately 20.5-second training horizon. The side-by-side videos show Fork-NFT on the left and LTX-AR on the right. For quantitative evaluation, we evaluate 200 30-second videos per model, using each video’s corresponding audio track, over three consecutive windows: W₁ = 0–10 s, W₂ = 10–20 s, and W₃ = 20–30 s. The table reports the score changes between adjacent windows (W₂ − W₁ and W₃ − W₂), averaged over the 200 videos for each model. Positive changes indicate improvement for VQ, MQ, and AQ; negative changes indicate improvement for DeSync.

MetricLTX-AR: W₂−W₁Fork-NFT: W₂−W₁LTX-AR: W₃−W₂Fork-NFT: W₃−W₂
VQ ↑-1.406-0.344-1.109+0.484
MQ ↑-2.207-0.945-0.129-0.108
AQ ↑+0.139+0.178-0.089-0.098
DeSync ↓+0.4000.000-0.267+0.167

In these examples, Fork-NFT better preserves video and motion quality over time. VQ declines less from W₁ to W₂ and recovers from W₂ to W₃, whereas LTX-AR continues to decline. MQ also declines less in both transitions. AQ changes are similar between the two models; Fork-NFT's DeSync remains stable between the first two windows but worsens in the final window. These comparisons indicate partial mitigation of long-horizon degradation, most clearly in video quality, even without extrapolation-specific optimization. The revised manuscript includes these comparisons and segment-wise analyses, together with a discussion of the remaining long-video failure modes.

Revision note. The scope discussion in Sec. 5.4 and Appendix D.3, Table 14, now include the 30-second diagnostic and remaining drift.

Reviewer_Cd7qOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies reward-based fine-tuning for few-step autoregressive joint audio–video generation and proposes Fork-NFT. The method replays paired states recorded from the sampler, reconstructs historical KV caches under the current policy, assigns long-horizon video credit and short-window audio credit, and regularizes cross-modal attention to historical context through a history-marginal attention constraint. Using the same 22B LTX-AR backbone, Fork-NFT outperforms both the original backbone and an adapted OmniNFT baseline, with ablation studies supporting its main design choices.

Critical strengths
  1. The method addresses practical challenges in autoregressive joint generation by combining sampler states, historical context, and modality-specific reward credit assignment. The history-marginal KL regularizer constrains attention to historical context while allowing attention within the current chunk to adapt. This design is clearly motivated and mathematically characterized.
  2. The comparisons using the same backbone and sampling settings are valuable, and the ablations cover state replay, credit windows, and attention regularization. Compared with OmniNFT, JavisScore improves from 0.192 to 0.205 for short-form generation and from 0.2782 to 0.2969 for approximately 20-second outputs.
Critical weaknesses
  1. The statistical reliability of the improvements remains unclear. The paper does not report repeated evaluations across multiple generation seeds, confidence intervals, or per-prompt paired differences. Although the long-horizon evaluation uses matched prompts and seeds, it remains unclear whether the gains over OmniNFT are consistent or driven by a small subset of samples. Some differences are small; for example, short-form VQ increases only from 3.101 to 3.127.
  2. Evidence of perceptual improvements is insufficient. I could not find an anonymous demo with playable audiovisual samples, and no human evaluation is reported. The video frames in Figure 3 show limited visible differences, while changes in audio waveforms indicate differences in sound activity but do not establish improvements in audio quality or audiovisual synchronization. Whether the automatic metric gains translate into perceptible improvements and consistent human preferences therefore remains unverified.
Questions for authors
  1. Could you repeat the evaluation on the same prompts using multiple generation seeds and report paired differences, win rates, and confidence intervals relative to OmniNFT, particularly for JavisScore, AV-IB, and DeSync? Results from multiple independent training runs would provide additional evidence of training stability, but I regard these as a valuable addition rather than a requirement.
  2. Could you provide anonymous, playable comparisons using matched prompts and seeds, together with a blinded human preference evaluation assessing video quality, audio quality, and audiovisual synchronization separately?
  3. What is the duration of the short-form outputs evaluated in Table 2?
6 comments
AuthorsOpenReview ↗

Response to Reviewer Cd7q

We thank Reviewer Cd7q for recognizing our method design, mathematical formulation, and controlled experiments. Below, we address your questions and concerns point by point.

W1/Q1 Statistical reliability across prompts and generation seeds:

We reevaluated the OmniNFT and Fork-NFT checkpoints reported in the main table on 200 JavisBench-mini prompts using three matched generation seeds, generating 600 five-second audio–video samples per method. Fork-NFT improves VQ, AQ, AV-IB, and JavisScore across all three seeds; DeSync improves on average, but its confidence interval includes zero.

Fork-NFT improves VQ, AQ, AV-IB, and JavisScore by 0.0864, 0.0252, 0.0136, and 0.0092, respectively, with paired 95% confidence intervals excluding zero for all four metrics. Their prompt-level win rates are 63.5%, 59.0%, 70.0%, and 64.0%, showing improvements across a majority of prompts. DeSync also decreases from 0.5830 to 0.5660 on average, although its confidence interval includes zero.

MetricFork-NFTOmniNFTPaired difference95% CIPrompt win / tie / loss (%)
VQ ↑3.03792.9515+0.0864[+0.0456, +0.1262]63.5 / 1.0 / 35.5
AQ ↑4.92894.9037+0.0252[+0.0122, +0.0386]59.0 / 0.0 / 41.0
AV-IB ↑0.23460.2210+0.0136[+0.0095, +0.0179]70.0 / 0.0 / 30.0
DeSync ↓0.56600.5830-0.0170[-0.0823, +0.0367]46.0 / 10.5 / 43.5
JavisScore ↑0.20010.1909+0.0092[+0.0056, +0.0128]64.0 / 0.0 / 36.0

Differences are Fork-NFT minus OmniNFT. Prompt win/tie/loss rates compare each method's mean score across the three seeds for that prompt; confidence intervals use paired bootstrap resampling over prompts, retaining all three seeds within each sampled prompt. The seed-specific differences are:

MetricSeed 0 differenceSeed 1 differenceSeed 2 difference
VQ ↑+0.1180+0.0374+0.1036
AQ ↑+0.0373+0.0207+0.0175
AV-IB ↑+0.0170+0.0116+0.0123
DeSync ↓-0.0510+0.0560-0.0560
JavisScore ↑+0.0087+0.0091+0.0097

Revision note. Sec. 5.3 and Appendix C.2, Tables 8–9, now report the paired estimates and generation-seed results.

W2/Q2 Perceptual improvements, human evaluation, and playable samples:

We apologize for omitting playable comparisons from the original submission. We now provide three playable comparisons in ForkNFT_Materials.zip: Case_1.mp4–Case_5.mp4 in short_videos/ForkNFT/ and short_videos/OmniNFT/, with prompts in short_videos/prompts.txt. Both OmniNFT and Fork-NFT were trained on the same LTX-AR backbone with the same training budget.

Fork-NFT receives more preferences than OmniNFT for video quality, audio quality, and audiovisual synchronization in our blinded pairwise evaluation. Ten participants evaluated 50 cases with matched prompts and generation seeds, rating each criterion separately with a tie option.

CriterionFork-NFT preferred (%)Tie (%)OmniNFT preferred (%)
Video quality562915
Audio quality571627
Audiovisual synchronization472132

The human-evaluation website demo source is included in ForkNFT_Materials.zip under website_source/; open website_source/index.html to run the demo. During evaluation, participants see the prompt and two comparison videos and choose A, Tie, or B for each of the three criteria.

Revision note. Sec. 5.3 and Appendix C.3, Table 10, now include the human study and describe the playable supplement and web-demo interface.

Q3 Duration of the short-form outputs in Table 2:

The short-form outputs in Table 2 are 5 seconds long, as are the additional short-form evaluations in this response. This duration is now explicit in the revised manuscript.

Revision note. The five-second short-form duration is explicit in Sec. 5.1 and Appendix C.1.

We hope these results and clarifications help address your concerns, and we would be happy to discuss any remaining questions.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Revisions

Thank you again for your careful review and helpful suggestions. We have added paired statistical analyses across multiple generation seeds, a blinded human evaluation, and playable audiovisual comparisons, and clarified the five-second short-form evaluation protocol. As the discussion period is nearing its end, we wanted to follow up and would appreciate your feedback on whether these additions address your concerns. We would be happy to discuss any remaining concerns or provide further clarification.

Thank you to the authors for the response. The additional paired comparison results and human evaluation convince me that the method is statistically work, although the improvements appear relatively modest. I have the following questions:

  1. In the provided short_videos materials, for cases 2–5, there is almost no discernible difference between ForkNFT and OmniNFT in terms of audio-visual content; the audio and video are highly similar and, from my perspective, nearly imperceptible. I would appreciate an explanation from the authors.

  2. In the provided long_videos materials, differences among the generation methods are now visible. However, it is not possible to distinguish which audio segment is the LTX-AR output and which is the ForkNFT output.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Revisions

We thank the Area Chair and reviewers for their careful evaluation and constructive feedback. We appreciate the reviewers’ recognition of our problem formulation, design rationale, and sampler-state-aware training.

Our responses and revised manuscript provide the following additional evidence and clarifications:

  1. Reliable improvements across prompts and seeds. Using the main-table checkpoints, paired evaluation on 200 JavisBench-mini prompts with three matched generation seeds shows statistically significant improvements in VQ, AQ, AV-IB, and JavisScore. Improvements cover a majority of prompts. DeSync also improves on average, although the improvement is not statistically significant (Tables 8–9).

  2. Additional baselines and isolated credit analysis. On the complete JavisBench-mini suite, Fork-NFT outperforms AV-adapted DiffusionNFT, GDPO, and AR-CoPO in VQ, AQ, JavisScore, and DeSync under matched RL training budgets (Table 4). Further ablations show that extending video credit improves results even when the scoring span is fixed, supporting our modality-specific temporal design (Table 11).

  3. Perceptual evidence. In a blinded study involving ten participants and 50 matched cases, Fork-NFT receives more preferences than OmniNFT for video quality, audio quality, and synchronization (Table 10). The supplementary archive ForkNFT_Materials.zip includes short-video examples (short_videos/), 30-second comparisons (long_videos/), and the human-evaluation web-demo source (website_source/).

  4. Implementation and scope clarified. We specify all three OmniNFT components in our AR adaptation and correct the omitted synchronization-reward term: its advantage reaches both modalities only within the intersection of their credit windows. We also add 30-second diagnostics and discuss remaining temporal drift and cross-backbone limitations.

Revision note. These changes are incorporated into Secs. 4.4, 5.1, and 5.3–5.4, with implementation details in Appendix A, additional evidence in Appendices C–D, and cross-backbone discussion in Appendix E. Our individual responses identify the corresponding sections and tables.

We respectfully ask the reviewers to reassess our work in light of these results and clarifications, and welcome any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.