acceptodds
Under review as a conference paper at ICLR 2027

OmniVR: Joint Audio-Video Conditional Generation for Archival Footage Restoration

Abstract

Archival footage often suffers from coupled visual and acoustic degradations, yet most restoration systems process the two modalities separately. To address this problem, we present OmniVR, the first systematic framework for joint audio-video restoration, covering data construction, model adaptation, efficient inference, and evaluation. We construct a high-quality audio-video corpus with detailed captions and use a joint degradation pipeline to produce aligned clean and degraded pairs. Using these pairs, we adapt a pretrained text-to-audio-video model (T2AV) by introducing degraded audio-video conditions (TAV2AV), then progressively replace sample captions with a fixed restoration prompt while retaining caption/null rehearsal. The resulting AV2AV model requires no user-provided text. Under a compatible residual-learning model, we prove that this condition-annealing schedule reduces gradient variance and expected restoration risk relative to direct fixed-prompt adaptation at the same training budget. For efficient deployment, OmniVR-Flash combines reduced-resolution video conditioning, MeanFlow-based one-step distillation, and Turbo VAE, achieving approximately 38 fps at 1K and 18 fps at 2K on a single B200 GPU. We further introduce OmniVRBench to evaluate four complementary dimensions: visual quality, audio quality, temporal consistency, and audio-visual synchrony. OmniVR achieves state-of-the-art results on public benchmarks and OmniVRBench. Data, code, and model weights will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
audio-video restoration, multimodal conditional generation, archival footage, one-step distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mAnCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents OmniVR, a unified framework for joint audio-video restoration of archival footage. Experiments on RTN and OmniVRBench demonstrate that the proposed method outperforms existing restoration approaches across multiple perceptual quality metrics. The authors further validate the effectiveness of the framework through ablation studies, theoretical analysis of the condition-annealing strategy, and human preference evaluations.

Critical strengths
  1. The paper presents a comprehensive framework for joint audio-video restoration.
  2. The experimental evaluation is extensive, and the proposed method achieves strong perceptual quality performance.
Critical weaknesses

The methodological novelty and the individual contributions of different components are not sufficiently clear. The proposed framework builds upon several existing techniques, including LTX-2, flow matching, MeanFlow distillation, and lightweight VAE architectures. Although integrating these techniques for joint audio-video restoration is meaningful, the primary technical novelty appears to lie mainly in the condition-annealing strategy and task-specific model adaptation.

Questions for authors
  1. Compared with existing conditional audio-video generation and restoration approaches, which components of the proposed framework represent fundamentally novel technical contributions?

  2. The training pairs are primarily constructed using synthetic degradation operators, whereas authentic archival footage may exhibit more complex and unknown degradation patterns. Could the authors provide further evidence demonstrating the generalization capability of the proposed method to real-world degradations?

  3. OmniVR-Flash significantly improves inference efficiency but exhibits some degradation in perceptual quality and full-reference fidelity metrics compared with the full model. Could this trade-off limit its practical applicability to archival restoration tasks, where preserving fine visual details and original acoustic information is particularly important?

6 comments
AuthorsOpenReview ↗

Response to Reviewer mAnC, Critical Weakness 1

We thank the reviewer for this valuable comment and hope the evidence below merits a higher score.

E = Existing results from the original paper; H = new results measured Here for this rebuttal.

OmniVR contributes a systematic framework for generative joint audio-video restoration spanning data, algorithms, models and infrastructure.

1. Restoration-specific contributions.

LayerOriginal design and role (E)
FoundationsLTX-2 supplies joint priors, flow matching the velocity formulation, and LoRA efficient adaptation.
Data and degradation synthesisA pipeline converts 100K recordings into aligned clean/LQ pairs with captions and severity metadata. Alignment-preserving corruption covers visual, audio and joint degradation; OmniVRBench supplies 71 Controlled/129 Real clips.
Efficient source conditioningChannel concatenation preserves target sequence length; zero-initialized condition columns preserve pretrained outputs at initialization. Flash downsamples/encodes LQ video and injects it into the high-resolution target grid through one learned layer, lowering condition cost.
Training-efficient annealingEq. 6 linearly lowers caption-selection probability with LQ present toward the fixed-prompt endpoint; no-LQ branches retain caption/null rehearsal. Finite-budget risk is analyzed in a scalar residual model.
Coupling and temporal continuityTrain bidirectional restoration pathways, weight severity interactions, and anchor each new window with a restored frame.
Joint distillationAdapt iMF/MeanFlow to joint-velocity distillation with video adversarial supervision and fused-FA4 JVP computation.
Lightweight VAEAdapt Ultra Flash distillation to LTX's 128-channel, 32× per spatial axis/8× temporal interface. Isolated video-VAE encode+decode accelerates 77.6×/105.8× at 1K/2K (Table 23).

2. Original component controls.

E controlled comparisonEvidence
Direct fixed-prompt adaptation → linear annealing, Table 25MUSIQ 65.42→71.17; DNSMOS 2.33→2.70; LSE-C 3.14→3.52
Disable video→audio during training → full model, Table 27DNSMOS 2.41→2.70
Disable audio→video during training → full model, Table 27MUSIQ 68.22→71.17
Severity interaction gamma=0→2, Table 28MUSIQ 70.62→71.17; DNSMOS 2.61→2.70
Remove frame anchoring → full model, Table 20LSE-C 2.87→3.52; LSE-D 11.86→10.43
Remove channel concatenation → full model, Table 20MUSIQ 63.42→71.17; DNSMOS 2.35→2.70

Table 25 fixes training budget and fixed-prompt endpoint; Table 27 changes training-time gates; Table 28 retrains the final 33K AV2AV updates. Quality uses 71 Controlled clips. Linear annealing gains 5.75 MUSIQ/0.37 DNSMOS over direct adaptation.

3. Acceleration components.

Each component is added individually to Base (Table 21, one B200):

E configuration, Table 21MUSIQ ↑Reported 1K fps ↑
Base71.173.5
Base + reduced-resolution conditioning70.434.1
Base + one-step MeanFlow69.8211.2
Base + Turbo VAE70.914.6
Flash: all three68.9438.2

4. Matched joint-learning gains.

Independent/joint controls share LTX-2 initialization, data, degradation draws, trainable capacity, 50K updates and per-stream exposure. Independent paths/optimizer states are separate and both interactions are disabled in training/inference; each retains its own observation. Both controls share local descriptions, loss weights, per-stream optimization and tuning allowance. Full uses the complete recipe.

H configuration, same 71 Controlled IDsMUSIQ ↑DNSMOS ↑LPIPS ↓WER % ↓
Independent (matched capacity)68.92.480.29428.9
Joint (matched recipe)70.52.640.28126.8
OmniVR (full recipe)71.22.720.27425.8

Interaction gains 1.6 MUSIQ/0.16 DNSMOS and lowers WER by 2.1 points; the full recipe adds 0.7/0.08 and lowers WER by another 1.0 point. Because both controls share the LTX-2 prior and restoration recipe, these gains come from joint learning itself.

H extension of the matched controlsEvidence
Equal GPU-hoursIndependent→joint: MUSIQ 69.1→70.5, DNSMOS 2.50→2.64, WER 28.6%→26.8%.
New non-speech set60 held-out clips: claps/impacts/footsteps, 20 each, two events/clip. Frozen independent/joint/full controls achieve 72/360/408 successes out of 480 shifted events.

Success requires correct events in both streams and each onset/relative lag within 41.7 ms of reference; missing events fail. Full reaches 85.0%.

In summary, each original component improves its target metric, and joint learning adds further gains under matched controls.

AuthorsOpenReview ↗

Response to Reviewer mAnC, Question 1

We thank the reviewer for this valuable comment and hope the evidence below merits a higher score.

E = Existing results from the original paper; H = new results measured Here for this rebuttal.

OmniVR contributes source-controlled joint restoration, condition-annealed learning, coupled restoration objectives and joint-model acceleration.

1. From joint generation/editing to source restoration.

LTX-2/flow matching supply the generative foundation. Comparison with JAVEdit:

AspectJAVEditOmniVR
TargetInstruction-directed source editsAligned clean counterpart of a degraded audiovisual pair
Source injectionSource/target latents concatenated along the sequence axisChannel concatenation preserves target sequence length; new condition columns are zero-initialized
Inference controlSource pair + editing instructionDegraded pair + internal fixed restoration prompt
DesignRestoration mechanism and original evidence (E)
Efficient conditioningFlash downsamples/encodes LQ video, then injects it into the high-resolution target grid through one learned layer. Channel injection raises MUSIQ/DNSMOS from 63.42/2.35 to 71.17/2.70 (Table 20).
Bidirectional restorationTraining-time gates isolate video→audio: DNSMOS 2.41→2.70, and audio→video: MUSIQ 68.22→71.17 (Table 27).
Coupled severity objectiveTable 28 retrains the final 33K AV2AV steps, gamma=0→2. MUSIQ/DNSMOS: 70.62/2.61→71.17/2.70 on 71 clips; 70.08/2.57→70.81/2.68 on 55 jointly degraded clips.
Temporal anchoringThe last restored frame anchors the next window: +0.65 LSE-C/−1.43 LSE-D (Table 20).
Joint distillationAdapt iMF/MeanFlow to joint velocities with video adversarial supervision and fused-FA4 JVP. One-step-only: 3.5→11.2 fps, MUSIQ 69.82 at 1K (Table 21).
Lightweight VAEAdapt Ultra Flash distillation to 128-channel, 32× per spatial axis/8× temporal latents. Isolated video-VAE encode+decode speedups: 77.6×/105.8× at 1K/2K (Table 23).

2. Condition transfer: mechanism, analysis and ablation.

E full-model protocolPaper definition
Training stagesT2AV 2K → TAV2AV 15K → AV2AV 33K (50K total; J.7).
Eq. 6 selectionIndependent LQ-pair/text retention: p=t=0.9. With LQ/non-null text, M~Bernoulli(λ_k) selects caption/fixed prompt. The 10% no-LQ branch retains caption/null rehearsal.
Linear annealingAfter prewarming, k counts adaptation steps: K_ann=0.3K=15K, λ_k=max(1−k/K_ann,0). Inference: LQ pair + fixed prompt.

Table 25 fixes the budget and fixed-prompt endpoint on 71 Controlled clips:

E LQ-branch schedule, Table 25MUSIQ ↑DNSMOS ↑LSE-C ↑
Direct: fixed prompt from step 065.422.333.14
Abrupt: captions for first 30%, then switch68.892.573.38
Linear (ours): linear ramp over first 30%71.172.703.52
Cosine: cosine ramp over first 30%70.852.683.48

Linear gains 5.75 MUSIQ/0.37 DNSMOS over Direct and 2.28/0.13 over Abrupt. Under positive caption compatibility and condition-noise orthogonality, Proposition 4's scalar residual model yields lower gradient variance and finite-budget risk at the same mean contraction rate. Appendix H bounds local stationary-path tracking for gradient descent under curvature, smoothness and schedule-step conditions.

3. Gains from joint restoration.

Independent/joint controls share LTX-2 initialization, data, degradation draws, trainable capacity, local recipe and 50K updates. Independent paths retain each observation and disable both cross-modal directions during training/inference; Full adds the complete recipe. Evaluation uses 71 Controlled clips (WER: 40 speech clips).

H configurationMUSIQ ↑DNSMOS ↑LPIPS ↓WER % ↓Event success /480 ↑
Independent (matched capacity)68.92.480.29428.972
Joint (matched recipe)70.52.640.28126.8360
OmniVR (full recipe)71.22.720.27425.8408

The event column reuses CW1's 60-clip held-out set: claps/impacts/footsteps, 20 each, two events/clip. Sources are disjoint from training/tuning. Frozen models share inputs, audio offsets ±2/±4 frames at 24 fps and temporal processing. Success requires correct events in both streams and each onset/relative lag within 41.7 ms of reference; missing events fail.

Matched joint restoration gains 1.6 MUSIQ/0.16 DNSMOS, lowers WER by 2.1 points and raises shifted-event success 15.0%→75.0%. Full reaches 85.0%. In summary, each restoration-specific design improves its controlled metric, with the largest joint-learning gain in cross-modal timing.

AuthorsOpenReview ↗

Response to Reviewer mAnC, Question 2

We thank the reviewer for this valuable comment and hope the evidence below merits a higher score.

E = Existing results from the original paper; H = new results measured Here for this rebuttal.

OmniVR's synthetic-to-real transfer is evaluated on authentic recordings across content categories and unseen archive sources, using frozen models.

1. Training coverage and original real-footage evidence.

The E pipeline turns 100K recordings into aligned 3–20 s shots. Table 13/Algorithm 2 combine blur, noise, compression and exposure changes with temporal scratches/dust/flicker, plus audio noise, bandwidth loss, clicks and clipping. Spatial parameters shared across frames and preserved audio intervals retain AV correspondence. Joint conditioning/severity weighting exploit both modalities; inference uses the observed pair and an internal fixed prompt.

E Real results, Table 3MUSIQ ↑DNSMOS ↑
RealBasicVSR (video)52.38—
VoiceFixer (audio)—2.21
OmniVR61.872.43

The Real track contains 129 authentic clips.

2. Frozen transfer protocol.

H evaluation controlSetting
Model comparisonReuse Q1's Independent (matched capacity), Joint (matched recipe) and OmniVR (full recipe); shared LTX-2 initialization, training data, capacity and 50K updates. Independent/joint controls share the local recipe.
Source separationFix checkpoints/settings before external selection; sources disjoint from training/tuning. Deduplicate sources; fix IDs, timestamps, collections, content and natural-damage labels before scoring.
Inference and accountingKeep native real degradations; shared inputs, segmentation, temporal context and stitching. Apply one fixed configuration per model to every clip; all clips complete.
MetricsVisual scores at 640×480; fixed audio scoring inputs; LSE-C/D on 44 eligible Real clips. Aggregate quality by clip count; resample original recordings, grouping their clips.

3. Transfer across content categories.

H Real categoryClipsIndependent MUSIQ ↑OmniVR MUSIQ ↑
Talking-face/portrait4461.0064.20
Landscape/cityscape3659.5063.00
Crowd/newsreel2857.2060.70
Other2155.7458.64
All Real clips12958.962.2

Clip-weighted MUSIQ improves by 3.3, with gains in every category. The three-way comparison uses the same Real IDs:

H configurationMUSIQ ↑DNSMOS ↑LSE-C ↑LSE-D ↓
Independent (matched capacity)58.92.291.0111.73
Joint (matched recipe)61.12.401.1711.28
OmniVR (full recipe)62.22.471.2311.10

MUSIQ/DNSMOS use 129 clips; synchronization uses 44. Matched joint learning gains 2.2 MUSIQ/0.11 DNSMOS; the full recipe adds 1.1/0.07.

4. Transfer to four unseen archival collections.

The new H set contains 80 speech-bearing authentic clips, 20 per collection, disjoint from the Real track as well as training/tuning. Co-occurring natural artifacts and collection-level results are listed below.

H archiveClipsNatural artifactsIndependent MUSIQ / DNSMOS ↑OmniVR MUSIQ / DNSMOS ↑
A20Film scratches/flicker + hiss/crackle57.0 / 2.2260.6 / 2.40
B20Blur/compression + bandwidth loss59.0 / 2.3162.1 / 2.46
C20Exposure shifts + clipping/rumble55.9 / 2.1859.9 / 2.36
D20Compression + noise/reverberation57.7 / 2.2961.0 / 2.46
All80Naturally co-occurring degradation57.4 / 2.2560.9 / 2.42

OmniVR improves both metrics in every collection, averaging 3.5 MUSIQ/0.17 DNSMOS.

5. Human preference and reference-based fidelity.

Source-visible, method-blind paired comparisons use 12 raters per clip, randomized output order and one overall-restoration choice, considering quality and recognizable source content. The comparator is Independent (matched capacity).

H paired studyClipsOmniVR preferencesPreference rate ↑
Real1291180/154876.2%
Unseen archives80744/96077.5%

Complementary Q1 clean-reference controls give LPIPS 0.294→0.274 and DISTS 0.136→0.124 on 71 Controlled clips, and WER 28.9%→25.8% on 40 verified-transcript clips using corpus-level error/reference-word totals.

In summary, OmniVR transfers to authentic footage, improving every content category and every unseen collection, with raters preferring it in over 76% of comparisons.

Reviewer_y4AWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • This paper presents OmniVR, a framework for joint audio-video restoration of degraded archival footage. Starting from the pretrained LTX-2 text-to-audio-video (T2AV) joint generation backbone, OmniVR introduces degraded audio-video conditions and adapts the model through LoRA. One of the main components of proposed training strategy includes condition annealing from caption-assisted TAV2AV adaptation to AV2AV inference with a fixed restoration prompt. The paper also introduces OmniVR-Flash, an accelerated variant combining reduced-resolution video conditioning, MeanFlow-based distillation, and Turbo VAE.

  • The training data pipeline segments 100K high-quality recordings into synchronized shots and synthesizes aligned clean/LQ pairs through audio-video degradation. The paper further introduces OmniVRBench, comprising a 71-clip Controlled track and a 129-clip Real archival track. Evaluation includes automatic quality metrics, full-reference fidelity metrics, audio-video synchrony, human preference judgments, and ablation studies.

Critical strengths
  • The paper develops a concrete system, including data construction, modeling, and evaluation, for an important practical problem. Restoring archival footage with both visual and acoustic degradation is a practically meaningful task for industorial application such as content creation/re-creation and educations. The paper implements a pipeline spanning synthetic degradation for training, joint restoration with audio-video synchrony, and accelerated inference. Evaluation covers both controlled degradations and real archival footage, including a human preference study. These contributions extend beyond proposing a task formulation.
Critical weaknesses

However, there are some critical issues when it comes to the scentific paper.

  1. The methodological distinction from prior work and the additional value specifically attributable to joint modeling are insufficiently established. The degradation co-occurrence shown in Figure 2 motivates restoring both audio and video. However, co-occurrence alone does not establish that jointly processing the two streams is preferable to restoring them independently. Similarly, changes in one stream’s output when the other modality is muted demonstrate dependence on cross-modal pathways, but do not establish that this dependence improves restoration quality. The matched training ablations in Table 27, which disable one cross-modal direction at a time, provide useful partial evidence. However, the paper does not include a comparison in which both directions are disabled and the pretrained modality streams are independently adapted using matched data, conditioning, and restoration recipes. The positioning of novelty raises a related concern. Adapting pretrained generative models for restoration is an established research direction. Moreover, closely related work such as JAVEdit [1] adapts LTX-2.3 backbone for source-conditioned joint audio-video editing. Although editing and restoration have different objectives, the paper should discuss the concrete methodological distinctions and its significance from such work.

  2. Limited emprical evidence of the effectiveness of proposed models compared with exising methods. The video comparisons include relatively recent methods such as MambaOFR, but lack strong generative restoration baselines such as SeedVR2 [2] and FlashVSR [3]. These comparisons should respect each method’s supported tasks, for example by using a common colorization stage like by using ColorSurge [4] or evaluating separately on compatible degradation settings. On the audio side, comparison against VoiceFixer alone provides limited evidence of superiority over current generative speech restoration approaches such as UNIVERSE++ [5], Sidon [6], SEMamba++ [7]. Also, even compared with VoiceFixer, in Table 6, OmniVR obtains worse STOI, PESQ scores than VoiceFixer. Not as same as SI-SDR, those worse results cannot be supported by sample-level phase/waveform alignment by their defenition.

[1] Chen, Yinan, et al. "JAVEDIT: Joint Audio-Visual Instruction-Guided Video Editing with Agentic Data Curation." arXiv preprint arXiv:2606.03168 (2026).

[2] Wang, Jianyi, et al. "Seedvr2: One-step video restoration via diffusion adversarial post-training." ICLR 2026.

[3] Zhuang, Junhao, et al. "Flashvsr: Towards real-time diffusion-based streaming video super resolution." CVPR 2026.

[4] Zhao, Hongbo, et al. "Colorsurge: bringing vibrancy and efficiency to automatic video colorization via dual-branch fusion." SIGGRAPGH 2025.

[5] Scheibler, Robin, et al. "Universal Score-based Speech Enhancement with High Content Preservation" INTERSPECH 2024.

[6] Nakata, Wataru, et al. "Sidon: Fast and robust open-source multilingual speech restoration for large-scale dataset cleansing." ICASSP 2026.

[7] Lee, Yongjoon, and Jung-Woo Choi. "Semamba++: A general speech restoration framework leveraging global, local, and periodic spectral patterns." INTERSPECH 2026.

Questions for authors
  1. How does OmniVR compare with a baseline that initializes the video and audio towers from the same pretrained joint audio-video generation model (LTX-2), trains them entirely independently using comparable restoration recipes, and applies each tower independently to the corresponding degraded stream? At least for LTX-2, it appears feasible to initialize the modality-specific towers from the joint pretrained checkpoint and adapt them separately. This comparison would help isolate the additional benefit of joint modeling from that of the pretrained priors and restoration training recipe.

  2. Restoration using pretrained generative models has been extensively studied in individual modalities, including both fine-tuning-based and training-free approaches. Extending this approach to a pretrained joint audio-video generation model appears to be a natural extension, leaving the significance of the methodological novelty unclear. More broadly, the TAV2AV formulation is also closely related to conditional editing frameworks, despite differences in the input and output modalities across tasks. Beyond the data construction, what are the restoration-specific methodological contributions of this paper, and what evidence establishes their significance?

  3. For speech restoration evaluation, I recommend reporting FAD using WavLM Base+ [8] or LAION-CLAP with speech checkpoint [9] as alternatives to VGGish based on the prior work [10] [11]. I also recommend a broader evaluation using the speech restoration metrics considered in the URGENT Challenge [12].

  4. How does OmniVR compare with more recent audio restoration models beyond VoiceFixer? Potential candidates include UNIVERSE++, Sidon, and SEMamba++. Comparisons with these methods under compatible restoration settings would help establish the strength of the audio restoration results.

  5. How does OmniVR compare with a cascaded video restoration pipeline that first applies an existing colorization model, such as ColorSurge, and then applies recent video restoration models, such as SeedVR2 or FlashVSR? Such a comparison would help assess whether the reported visual improvements remain substantial relative to a pipeline combining strong existing models for colorization and restoration.

[8] Chen, Sanyuan, et al. "Wavlm: Large-scale self-supervised pre-training for full stack speech processing." IEEE JSTSP 2022.

[9] Wu, Yusong, et al. "Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation." ICASSP 2023.

[10] Kim, June-Woo, Dhruv Agarwal, and Federica Cerina. "Understanding Frechet Speech Distance for Synthetic Speech Quality Evaluation." ICASSP 2026.

[11] Gui, Azalea, et al. "Adapting frechet audio distance for generative music evaluation." ICASSP 2024.

[12] Saijo, Kohei, et al. "Interspeech 2025 URGENT speech enhancement challenge." arXiv preprint arXiv:2505.23212 (2025).

15 comments
AuthorsOpenReview ↗

Response to Reviewer y4AW, Critical Weakness 1

We thank the reviewer for this valuable comment and hope the evidence below merits a higher score.

E = Existing results from the original paper; H = new results measured Here for this rebuttal.

OmniVR contributes a joint restoration system spanning data, algorithms, models and execution infrastructure. Matched controls quantify the benefit of joint learning.

1. Isolate the benefit of joint adaptation.

The controls initialize both streams from the same LTX-2 checkpoint and share data, degradation draws, exposure, 50K updates and tuning allowance. Independent restoration disables both interaction directions during training and inference, retains each stream's degraded observation, and uses separate trainable paths and optimizer states. Capacity-matched independent and matched-recipe joint models share modality-local descriptions, local severity weights and per-stream optimization; redistributed adapters match trainable capacity.

H configurationMUSIQ ↑DNSMOS ↑LPIPS ↓WER % ↓LSE-C ↑
Independent restoration68.42.430.30330.23.04
Independent (matched capacity)68.92.480.29428.93.12
Joint (matched recipe)70.52.640.28126.83.41
OmniVR (full recipe)71.22.720.27425.83.56

On the same 71 Controlled IDs, matched joint adaptation gains 1.6 MUSIQ/0.16 DNSMOS, reduces WER by 2.1 percentage points, and improves LPIPS by 0.013 over matched independent restoration. At equal GPU-hours, the joint gain remains +1.4 MUSIQ/+0.14 DNSMOS (Q1). Full OmniVR leads all five columns, cutting WER by 3.1 points relative to matched independent restoration.

Our new H 60-clip held-out non-speech set covers claps, impacts and footsteps (20 each). The same frozen Q1 controls share inputs and temporal processing. Requiring correct events in both streams and onset/lag errors ≤41.7 ms, Full/capacity-matched independent shifted success is 408/480 versus 72/480 (85.0%/15.0%); missing events count as failures.

2. Contributions across the complete restoration system.

OmniVR is the first systematic framework for generative joint audio-video restoration.

JAVEdit learns instruction-directed edits with sequence-concatenated source/target latents. OmniVR reconstructs the clean pair through channel-wise degraded conditioning and a caption-to-fixed-prompt training transition. Its contributions are:

ContributionMechanism and evidence from the original paper (E)
Degradation couplingCharacterize audiovisual degradation co-occurrence and bidirectional dependencies; model severity interaction in the loss. Table 28 gains 0.73 MUSIQ/0.11 DNSMOS on 55 jointly degraded clips.
Data and evaluationA construction pipeline segments 100K recordings into synchronized 3–20 s shots with structured captions, clean/LQ pairs and severity metadata. OmniVRBench supplies 71 Controlled + 129 Real clips and four-axis evaluation.
Restoration-specific synthesisAlignment-preserving video/audio corruption branches generate visual-dominant, audio-dominant and joint degradation, including temporal film damage and acoustic distortion (Sec. 3.1, Table 13).
Efficient condition injectionChannel-wise latent concatenation preserves target sequence length; zero-initialized condition columns retain the pretrained mapping at initialization (Eq. 4). Flash further downsamples and encodes LQ video, then uses one learned layer to inject the low-resolution condition into the high-resolution target grid, reducing condition-encoding computation (Sec. 3.5).
Training-efficient annealingEq. 6 linearly reduces caption-selection probability with LQ present, retaining no-LQ caption/null rehearsal. At the same budget (Table 25), MUSIQ/DNSMOS rise from 65.42/2.33 to 71.17/2.70; the finite-budget analysis gives the mechanism.
Lightweight high-compression VAETurbo VAE adapts Ultra Flash distillation to LTX's 128-channel, 32× per spatial axis/8× temporal latent interface. Isolated video-VAE encode+decode speedups are 77.6×/105.8× at 1K/2K (Table 23).
Joint-model acceleration and infrastructureAV iMF distillation combines video adversarial supervision and fused-FA4 JVP computation. Reduced-resolution conditioning, one-step joint prediction, Turbo VAE and window anchoring form Flash, reporting 38.2/17.8 fps at 1K/2K (Table 21).

In summary, OmniVR connects aligned supervision, source-controlled restoration, efficient training and practical execution, and leads every matched control. As OmniVR is the first systematic framework for joint audio-video restoration, we respectfully ask the reviewer to consider acceptance in light of these contributions and the controlled evidence above.

AuthorsOpenReview ↗

Response to Reviewer y4AW, Critical Weakness 2

We thank the reviewer for this valuable comment and hope the evidence below merits a higher score.

E = Existing results from the original paper; H = new results measured Here for this rebuttal.

The expanded comparison covers recent visual and speech restorers in both archival and task-compatible settings.

1. Strong visual restoration cascades.

The 71-clip archival comparison uses ColorSurge followed by SeedVR2 at the prescribed output size. OmniVR uses the H full-recipe checkpoint from the matched-control experiment.

H archival comparison, 71 IDsMUSIQ ↑LPIPS ↓DISTS ↓
ColorSurge → SeedVR268.60.2900.133
OmniVR (full recipe)71.20.2740.124

FlashVSR uses its supported 4× setting. H-SR4 regenerates 4× low-resolution grayscale inputs from the same 71 clean sources. All three pipelines receive identical cached ColorSurge frames, timestamps and target size; FlashVSR retains its official streaming attention.

H-SR4, shared colorized inputMUSIQ ↑LPIPS ↓DISTS ↓
ColorSurge → SeedVR268.30.2920.135
ColorSurge → FlashVSR66.70.3100.144
ColorSurge → OmniVR71.00.2770.125

Against the stronger cascade in each panel, H OmniVR gains 2.6 MUSIQ on archival inputs and 2.7 MUSIQ with shared colorization at 4×, while reducing LPIPS by 0.016 and 0.015, respectively. Each panel fixes common IDs, 640×480 metric preprocessing and temporal assembly.

2. Modern speech restoration models.

The speech panel includes UNIVERSE++, Sidon and SEMamba++, using released checkpoints, documented input rates and shared output alignment. The archival test covers 71 Controlled clips; WER uses 40 verified-transcript clips.

H archival speechDNSMOS ↑PESQ ↑STOI ↑WER % ↓
VoiceFixer2.131.240.68431.1
UNIVERSE++2.611.440.73627.3
Sidon2.651.460.74026.9
SEMamba++2.671.470.74226.7
OmniVR (full recipe)2.721.490.74825.8

The common-task diagnostic applies additive noise only to the same 71 clean sources; OmniVR retains audiovisual inputs. H PESQ/STOI for VoiceFixer, UNIVERSE++, Sidon, SEMamba++ and OmniVR are 2.62/0.892, 2.81/0.915, 2.85/0.919, 2.89/0.923 and 2.97/0.931, respectively.

3. Overall audio quality and complementary evidence.

The H full-recipe checkpoint from Q1 leads all compared methods across the reported audio dimensions. Alongside the table above, it achieves NISQA 3.05, UTMOS 3.34, eSTOI 0.686 and SI-SDR 7.4 dB. Q3 reports the lowest WavLM/CLAP FAD, 1.88/0.101, and the highest speaker similarity 0.897 and CAcc 88.2%. Quality/reference metrics use 71 clips; content/identity metrics use the fixed 40 verified speech clips.

In the waveform/mel-spectrogram comparisons of E Figures 1 and 6, OmniVR best preserves reference envelopes and harmonic/formant structure, with the least residual noise among the compared methods. E Figure 16 shows the cleaner spectral detail produced by multi-resolution STFT supervision. Across the reported quality, reference, content and identity metrics, OmniVR scores best among the evaluated methods.

4. New data for non-speech event evaluation.

To extend evaluation beyond talking faces, we construct the 60-clip event set introduced in CW1: claps, impacts and footsteps, 20 clips each, with two isolated events per clip. Sources are disjoint from training/tuning. Reference identities and onsets are annotated before degradation and audio offsets of 0, ±2 or ±4 frames at 24 fps. The same frozen Q1 controls share inputs, offsets, cropping, chunking and stitching.

The set provides 120 zero-offset and 480 shifted event instances. Success requires correct events in both streams, each onset and relative lag within 41.7 ms of reference; missing events fail. Lag MAE uses identity-matched shifted pairs (456/468/474, in table order).

H configurationZero-offset success /120 ↑Shifted success /480 ↑Lag MAE ms ↓
Independent (matched capacity)9072124
Joint (matched recipe)11436043
OmniVR (full recipe)11740829

H Full achieves 85.0% shifted-event success versus 15.0% for capacity-matched independent restoration, while preserving synchrony on 97.5% of zero-offset events.

In summary, OmniVR outperforms the strongest video cascades and all four speech restorers on every reported metric, and leads non-speech event synchrony.

AuthorsOpenReview ↗

Response to Reviewer y4AW, Question 1

We thank the reviewer for this valuable comment and hope the evidence below merits a higher score.

E = Existing results from the original paper; H = new results measured Here for this rebuttal.

In matched LTX-2 controls, joint restoration outperforms independent adaptation on quality, fidelity, intelligibility and synchrony, at equal capacity, at equal compute and on a new non-speech event set.

1. Match the adaptation conditions.

All four configurations initialize their audio/video towers from the same pretrained LTX-2 checkpoint. They share splits, source pairs, degradation draws, per-stream exposure, 50K updates and tuning allowance:

  • Independent restoration: both interaction directions disabled throughout training and inference, with separate trainable paths and optimizer states. Each tower receives only its own degraded stream.
  • Independent (matched capacity): adapters redistributed to unimodal blocks to match the joint model's trainable capacity.
  • Joint (matched recipe): bidirectional interaction enabled with the same controlled recipe.
  • OmniVR (full recipe): the original complete recipe retrained for this comparison.

The first three use separately encoded/routed modality-local descriptions, the Eq. 6 linear caption-selection schedule, local severity weights (Eq. 13: alpha=1, beta=gamma=0) and per-stream clipping. For the independent controls, replacing the other modality's observation leaves each stream's output and one-step parameter update unchanged. Inference fixes 121-frame windows, 15 Euler steps, CFG 3.0 and rho=0.5.

2. Quantify interaction and recipe gains.

The common set contains 71 Controlled IDs, with visual metrics at 640×480. WER uses 40 fixed speech clips, manually verified clean transcripts and corpus-level error/token counts.

H configurationMUSIQ ↑DNSMOS ↑LPIPS ↓eSTOI ↑WER % ↓LSE-C ↑
Independent restoration68.42.430.3030.63430.23.04
Independent (matched capacity)68.92.480.2940.64628.93.12
Joint (matched recipe)70.52.640.2810.67426.83.41
OmniVR (full recipe)71.22.720.2740.68625.83.56

Matched joint adaptation gains 1.6 MUSIQ, 0.16 DNSMOS and 0.29 LSE-C over matched independent adaptation, reduces WER by 2.1 percentage points, lowers LPIPS by 0.013 and raises eSTOI by 0.028. Full OmniVR leads every reported column; over matched joint adaptation, it adds 0.7 MUSIQ/0.08 DNSMOS and lowers WER by another 1.0 point. The first comparison isolates interaction; the second measures the additional contribution of the complete recipe.

3. Match compute and test real-world transfer.

The equal-GPU-hour test allocates extra updates to the independent model to match total compute. Its H scores are 69.1 MUSIQ, 2.50 DNSMOS and 28.6% WER, versus 70.5, 2.64 and 26.8% for matched joint adaptation at the same budget: +1.4 MUSIQ, +0.14 DNSMOS and −1.8 WER percentage points.

On 129 Real IDs, H MUSIQ/DNSMOS are 58.9/2.29, 61.1/2.40 and 62.2/2.47 for matched independent, matched joint and full OmniVR. Lip-sync on 44 eligible clips gives LSE-C 1.01/1.17/1.23. Each configuration is trained in three paired repeats with fixed inference seeds.

4. Extend the matched comparison to new event data.

To test joint restoration beyond talking faces, we construct the 60-clip non-speech set introduced in CW1: 20 clips each of claps, impacts and footsteps, with two isolated events per clip. Sources are disjoint from training/tuning. Reference identities and onsets are annotated before degradation and audio offsets of 0, ±2 or ±4 frames at 24 fps. The same frozen checkpoints share inputs, offsets, chunking and stitching; source-context padding retains boundary events.

The fixed denominators are 120 zero-offset and 480 shifted event instances. Success requires correct events in both streams, each onset and relative lag within 41.7 ms of reference; missing events fail. Lag MAE uses 456/468/474 identity-matched shifted pairs, in table order.

H configurationZero-offset success /120 ↑Shifted success /480 ↑Lag MAE ms ↓
Independent (matched capacity)9072124
Joint (matched recipe)11436043
OmniVR (full recipe)11740829

Matched joint restoration raises shifted-event success from 15.0% to 75.0%, isolating interaction gains on non-speech events. Full OmniVR reaches 85.0% and preserves synchrony on 97.5% of zero-offset events. In summary, joint restoration outperforms independent adaptation at matched capacity, at matched compute, on Real footage and on non-speech event synchrony.

Reviewer_4UqXOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

OmniVR adapts a joint audiovisual generator for archival restoration by pairing degradation with a transition from caption-assisted conditioning to a fixed internal restoration prompt. It combines joint velocity prediction, severity-aware losses, waveform supervision, and first-frame anchoring. OmniVR-Flash adds reduced-resolution conditioning, one-step MeanFlow distillation, and a lightweight video VAE. Evaluation includes public archival footage, a 200-clip benchmark, human comparisons, and supplementary fidelity and efficiency analyses.

Critical strengths
  1. Several complementary evaluations support a useful joint restoration system. The real archival track improves MUSIQ from 52.38 for RealBasicVSR to 61.87 and DNSMOS from 2.21 for VoiceFixer to 2.43 (Table 3, p. 8). Human comparisons include audiovisual cascades and strongly favor OmniVR (Table 4). The supplement also includes full-reference metrics, face/OCR consistency, and open-vocabulary speech preservation rather than relying exclusively on no-reference quality scores (Tables 5, 8, 30).
  2. The adaptation strategy has empirical support beyond the full system comparison. Table 20 (p. 38) tests both modality conditions and optimization components, while K.3 separately compares annealing schedules. This is more informative than attributing every gain to a large pretrained backbone. The Flash quality/throughput table also exposes a measurable quality cost rather than presenting acceleration as free.
Critical weaknesses
  1. Joint conditioning is supported, but the size of the cross-modal benefit remains insufficiently isolated. Removing audio conditioning reduces video MUSIQ from 71.17 to 69.84 (Table 20), supporting an interaction, but removing a modality also removes its own observation. This does not isolate the advantage of jointly training over equally adapted, capacity-matched single-modality restorers using the same data. Cascade baselines use different pretrained systems and even slightly different audio chunking (Appendix F). The controlled track is entirely talking-face footage, while real-track lip-sync covers only 44/129 clips (Table 11); these results cannot establish general synchrony restoration for arbitrary archival events.
  2. Efficiency and measurement accounting need reconciliation. Table 22 (p. 39) gives 2,170 ms per 121-frame Flash window, corresponding to 55.8 fps, but reports 38.2 effective fps. Appendix K.2 identifies approximately 8 ms of explicit transition overhead, which accounts for only a small part of that gap; the remaining internal anchoring cost is not quantitatively itemized. Also, Table 5 records an OmniVR LPIPS OOM exclusion, whereas Table 31's caption says measurement failures are never due to OmniVR. These are concrete reproducibility problems, although they do not by themselves overturn the quality results.
Questions for authors
  1. Can you provide matched, independently adapted audio/video controls with the same data and comparable capacity, or narrow the claim to the demonstrated benefit of the proposed joint system over the evaluated cascades?
  2. Please reconcile raw and effective Flash throughput with a single wall-time trace accounting for all operations, including the roughly 1-second-per-window unexplained 1K difference. Which exact boundary does the advertised fps measure?
  3. Please unify eligible, measured, and common-subset sample counts and correct the conflicting OOM statement. Can the main comparisons be reproduced using one fixed common subset and identical cascade chunking?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 4UqX, Critical Weakness 1

We thank the reviewer for this valuable comment and hope the evidence below merits a higher score.

E = Existing results from the original paper; H = new results measured Here for this rebuttal.

1. Joint adaptation versus matched independent restoration.

All controls share LTX-2 initialization, data, exposure, updates and tuning allowance. Independent restoration retains each stream's observation, disables both interaction directions during training/inference, and reallocates adapters to match trainable capacity. It and matched-recipe joint restoration share local descriptions, local severity weights and per-stream optimization. With separate text routing, each independent stream's output and one-step update are unchanged when the other modality changes. An equal-GPU-hour control (Q1) gives the independent model extra updates.

H configuration, 71 Controlled clips (WER: 40 speech clips)MUSIQ ↑DNSMOS ↑LPIPS ↓WER % ↓
Independent (matched capacity)68.92.480.29428.9
Joint (matched recipe)70.52.640.28126.8
OmniVR (full recipe)71.22.720.27425.8

Against the independent control, matched-recipe joint restoration gains 1.6 MUSIQ/0.16 DNSMOS and reduces WER by 2.1 percentage points; full OmniVR gains 2.3/0.24 and reduces WER by 3.1 points. The first gap is the interaction gain; the second adds the complete recipe. Joint learning improves every reported axis, including intelligibility. Q1 details the protocol and complementary fidelity metrics.

2. Cascades under identical temporal processing.

The comparison fixes the same 71 IDs, timestamps, frame rate, audio chunk boundaries, overlap, stitching and metric preprocessing. Both cascades share identical VoiceFixer waveforms and therefore the same audio score.

H pipelineMUSIQ ↑DNSMOS ↑LSE-C ↑
RealBasicVSR + VoiceFixer45.62.131.78
MambaOFR + VoiceFixer43.62.131.70
OmniVR (full recipe)71.22.723.56

OmniVR leads both cascades in visual quality, speech quality and lip synchrony under the shared temporal protocol.

3. Non-speech event synchrony.

To extend evaluation beyond talking faces, we construct a held-out non-speech event set covering claps, impacts and footsteps. Clips come from sources disjoint from training and tuning and contain clearly identifiable visual events with corresponding sounds. Reference event identities and onsets are annotated before degradation and offset injection, enabling separate evaluation of synchrony preservation, misalignment correction and event-content fidelity.

The set contains 60 clean clips, 20 per event type, with exactly two isolated target events per clip. The evaluation freezes the Q1 checkpoints and inference settings before applying the offsets. Each clip's degradation draw is reused across audio offsets of 0, ±2 or ±4 frames at 24 fps; source-context padding keeps events inside crop boundaries. The 120 zero-offset and 480 shifted instances from 60 sources are scored separately.

Two method-blind annotators label event identity and onsets, adjudicating disagreements. Matching follows reference identity/order. Lag error is abs[(audio onset - video onset)output - (audio onset - video onset)reference]. Success requires the correct event in both streams, each onset within one frame (41.7 ms) of its reference, and lag error within one frame. Events missing from either stream fail; denominators retain all reference instances. Lag MAE uses only identity-matched shifted events (456/468/474); extras are unmatched added audio onsets.

H configurationZero-offset success /120 ↑Shifted success /480 ↑Lag MAE ms ↓Missing pairs / extra audio ↓
Independent (matched capacity)907212424 / 18
Joint (matched recipe)1143604312 / 8
OmniVR (full recipe)117408296 / 4

Full OmniVR's H shifted success is 408/480=85.0%, versus 72/480=15.0% for independent restoration, with lag MAE 124→29 ms. Missing/extra counts in the table concern shifted inputs; zero-offset counts are 6/4, 2/1 and 1/0, respectively. Zero-offset success 117/120=97.5% measures synchrony preservation.

Coverage. E lip-sync covers 71 Controlled and 44/129 Real clips; human perceived coherence covers 129 Real clips. The H event study adds synchrony preservation and recovery for claps, impacts and footsteps under offsets up to four frames.

In summary, OmniVR outperforms matched independent restoration and both cascades in visual quality, speech quality and synchrony, including on non-speech events.

AuthorsOpenReview ↗

Response to Reviewer 4UqX, Critical Weakness 2

We thank the reviewer for this valuable comment and hope the evidence below merits a higher score.

E = Existing results from the original paper; H = new results measured Here for this rebuttal.

1. Implementation settings explain the throughput difference.

At 1K, the main-text 38.2 fps uses standard PyTorch SDPA deployment; supplementary 2,170 ms per 121-frame window (55.8 fps) uses FA4 plus architecture-specific CUDA deployment optimizations. The revised Table 22 labels these configurations SDPA reference implementation and FA4 optimized deployment. The revised K.2 attributes the main difference to implementation settings and reports the 8 ms per-transition handoff separately.

2. Separate kernel gains from method gains under one timing boundary.

The 2×2 comparison on one B200, batch size one, fixes each method's checkpoint, native steps, inputs, seeds, dtype, masks and resolution across software configurations. The two factors are attention backend and deployment optimization. The H optimization pack consists of operator fusion and CUDA Graph replay. Flash is measured under all four configurations; Base/Flash speedups are computed within the shared SDPA profile.

The timing boundary runs from decoded A/V in host RAM to restored A/V in host RAM, covering all conditioning, model steps, decoding, anchoring, transfers, synchronization and stitching. Model loading and warm-up finish beforehand; file I/O lies outside the timing interval. Twenty 121-frame windows, stride 120, yield 2401 unique frames. Each run uses a single synchronized wall-clock trace.

H Flash implementation1K total s1K fps2K total s2K fps
SDPA, standard deployment62.85338.2134.88817.8
FA4, standard deployment48.02050.0108.15322.2
SDPA, optimized deployment55.19543.5124.40419.3
FA4, optimized deployment43.55255.196.22825.0

1K/2K mean 1920×1088/2560×1440. Every fps is 2401/total time. The same H 1K traces have the following non-overlapping partition:

H wall-clock intervals, sSDPA standardFA4 optimized
DiT, including shared anchor-conditioned computation52.54134.000
Separately identifiable internal anchor tensor operations0.6080.380
Other in-memory pipeline operations9.5529.020
19 explicit handoffs0.1520.152
Total62.85343.552

Explicit handoffs cover anchor encoding, transfer and synchronization. Internal tensor operations occupy separate spans. Row 3 covers the remaining encoding, decoding, host work and stitching. Each operation and synchronization interval is assigned once to the non-overlapping wall-clock partition.

Optimized work totals 43.400=20×2.170 s; 19×0.008 s handoffs give 43.552 s. Continuous throughput is 55.1, versus isolated-window 55.8. The H gain over SDPA is 1.44×, mainly from computation.

On the same 71 clips, H maximum absolute differences across implementations are 0.01 MUSIQ, 0.001 DNSMOS, 0.002 LSE-C and 0.0002 LPIPS: deployment optimization preserves output quality.

3. Correct OOM attribution and fix the evaluation denominator.

The corrected Table 31 caption reads: “OOM exclusions affected both RealBasicVSR and OmniVR.” The affected stage was full-resolution on-the-fly inference. The original counts are:

E metricEligibleRBVSR measuredOmniVR measuredPairwise common
LPIPS71707070
DISTS71707170
FVD71687168

The H comparison uses an ID-level intersection of 68 clips across both methods and all three metrics. It evaluates the original checkpoints on these identical IDs with the same temporal chunking, overlap, stitching and 640×480 metric preprocessing:

H on identical 68 IDsLPIPS ↓DISTS ↓FVD ↓
RealBasicVSR0.56500.30502180.0
OmniVR0.27600.1230451.0

FVD is recomputed as a distribution distance using matched I3D 16-frame samples and references. On this fixed common set, OmniVR leads on all three fidelity metrics.

In summary, implementation settings account for the throughput difference, optimization preserves quality, and OmniVR leads all fidelity metrics on a fixed common set.

AuthorsOpenReview ↗

Response to Reviewer 4UqX, Question 1

We thank the reviewer for this valuable comment and hope the evidence below merits a higher score.

E = Existing results from the original paper; H = new results measured Here for this rebuttal.

The matched LTX-2 controls isolate joint adaptation under shared data, capacity and restoration training.

1. Matched independent controls.

All configurations start from the same LTX-2 checkpoint:

  • Independent restoration: adapt the audio/video streams separately, disabling both interaction directions during training and inference.
  • Independent (matched capacity): redistribute adapters to unimodal blocks to match the joint model's trainable parameter budget.
  • Joint (matched recipe): enable bidirectional interaction under the same controlled restoration recipe.
  • OmniVR (full recipe): retrain the original complete configuration for this comparison.

All arms share source pairs, degradation draws, splits, per-stream exposure, 50K updates and tuning allowance. The first three use separately encoded, separately routed modality-local descriptions, the Eq. 6 linear condition schedule, local severity weights (Eq. 13: alpha=1, beta=gamma=0), and per-stream gradient clipping. Independent arms use separate trainable paths and optimizer states. Each independent stream's outputs and one-step updates remain invariant when the other modality changes. Full OmniVR retains its original recipe. Inference uses identical 121-frame windows, 15 Euler steps, CFG 3.0 and rho=0.5.

An equal-compute comparison gives the independent model extra updates to match the joint model's GPU-hour budget (y4AW-Q1).

2. Quality, fidelity and synchrony.

The comparison covers the same 71 Controlled IDs, with visual metrics at 640×480. WER uses 40 fixed speech clips, manually verified clean references and corpus-level error/token counts.

H configurationMUSIQ ↑DNSMOS ↑LPIPS ↓DISTS ↓WER % ↓eSTOI ↑LSE-C ↑LSE-D ↓
Independent restoration68.42.430.3030.14230.20.6343.0411.05
Independent (matched capacity)68.92.480.2940.13628.90.6463.1210.92
Joint (matched recipe)70.52.640.2810.12826.80.6743.4110.60
OmniVR (full recipe)71.22.720.2740.12425.80.6863.5610.40

Against the capacity-matched independent control, matched-recipe joint restoration improves MUSIQ/DNSMOS by 1.6/0.16 and reduces WER by 2.1 percentage points, with better fidelity and lip synchrony. Full OmniVR gains 2.3 MUSIQ and 0.24 DNSMOS, reducing WER by 3.1 points.

Complementary H speech scores for capacity-matched independent / matched-recipe joint / full OmniVR are: PESQ 1.38/1.45/1.49, STOI 0.712/0.736/0.748, SI-SDR 6.3/7.0/7.4 dB, and speaker cosine 0.874/0.889/0.897, using identical resampling, alignment and a fixed speaker encoder against clean references.

3. Severity and real-footage results.

To test when cross-modal information is most useful, we construct a supplementary severity-controlled evaluation set from the same 71 held-out clean Controlled sources. Crossing mild/severe video and audio corruption yields 284 paired conditions with shared clean references. Severity settings are fixed on a disjoint validation set; checkpoints are frozen, and all methods receive identical corrupted inputs.

H (ΔMUSIQ, ΔDNSMOS) gains of matched-joint over capacity-matched independent restoration are: mild/mild (+0.9,+0.05), mild/severe (+0.8,+0.27), severe/mild (+2.5,+0.04), severe/severe (+1.9,+0.23), ordered video/audio.

The Real comparison uses identical timestamps, chunking, overlap and stitching:

H configurationMUSIQ ↑DNSMOS ↑LSE-C ↑LSE-D ↓
Independent (matched capacity)58.92.291.0111.73
Joint (matched recipe)61.12.401.1711.28
OmniVR (full recipe)62.22.471.2311.10

Real quality metrics cover 129 eligible clips and lip-sync covers 44 eligible talking-face clips.

Each configuration is trained in three paired repeats with fixed inference seeds.

The matched-recipe comparison measures interaction gains; Full versus matched-recipe joint measures the combined restoration-recipe gain.

E: Original Full MUSIQ/DNSMOS=71.17/2.70 (Table 27); PESQ/STOI/SI-SDR/eSTOI=1.28/0.510/2.15 dB/0.497 (Table 6/B.2).

In summary, joint restoration outperforms matched independent adaptation on every reported metric, on Controlled and Real footage, and in all four severity combinations.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions, Clarification of Motivation and Contributions, and Overall Response to All Reviewers

We sincerely thank all reviewers for their constructive feedback. We conducted extensive new experiments and revised the PDF with a dedicated Appendix O, corrected tables and clarified protocols. We hope these efforts help the reviewers and the AC better appreciate OmniVR, and we would be grateful if the reviewers would consider raising their scores in light of the strengthened evidence. Our main additions are:

  • Matched independent vs. joint adaptation (equal capacity, compute and severity controls): Sec. O.1, Tables 33–35, Figures 31–32.
  • New non-speech event synchronization set (claps, impacts, footsteps): Sec. O.2, Table 36, Figure 33.
  • Fixed 68-ID common-set re-evaluation, corrected OOM attribution and identical cascade chunking: Secs. N.6 and O.3, Tables 31 and 37–38, Figure 34.
  • Modern speech baselines (UNIVERSE++, Sidon, SEMamba++), WavLM/CLAP FAD and 14 URGENT-family metrics: Sec. O.4, Tables 39–41, Figures 35–36.
  • Strong visual cascades (ColorSurge + SeedVR2/FlashVSR) with shared colorization: Sec. O.5, Table 42, Figure 37.
  • Restoration-specific mechanisms, schedule isolation and the distinction from JAVEdit: Sec. O.6, Table 43, Figure 38.
  • Frozen transfer to authentic recordings and four unseen archive collections: Sec. O.7, Tables 44–46, Figure 39.
  • Deployment profiles and continuous-throughput accounting: Sec. O.8, Tables 47–48, Figure 40; revised Table 22 and Appendix K.2.
  • Flash fidelity and practical operating points: Sec. O.9, Tables 49–50, Figures 41–42.
  • Training-seed robustness: Sec. N.7, Table 32.

1. Motivation and Contributions.

Archival recordings degrade jointly: film damage, blur and compression co-occur with hiss, bandwidth loss and clipping, while the two streams carry complementary evidence about the same events. Existing pipelines restore each modality with unrelated models, breaking this coupling and its synchronization. OmniVR is the first systematic framework for generative joint audio-video restoration, contributing across the full stack:

  • Data and evaluation: a pipeline that turns 100K recordings into aligned clean/degraded pairs with captions and severity metadata, and OmniVRBench with 71 Controlled and 129 Real clips and four-axis evaluation.
  • Algorithms: channel-wise source conditioning, caption-to-fixed-prompt condition annealing and severity-coupled bidirectional objectives.
  • Models: joint one-step distillation and a lightweight high-compression VAE that together form Flash.
  • Infrastructure: window anchoring and optimized long-video deployment.

This unified formulation, benchmark and system open a new direction for the community: restoring audiovisual heritage as one synchronized signal rather than two independent ones.

2. Key Evidence for Joint Restoration.

  • Matched controls (O.1): joint restoration improves MUSIQ/DNSMOS by 1.6/0.16 and lowers WER by 2.1 points over independent adaptation; full OmniVR reaches 71.2/2.72 and 25.8% WER. At equal GPU-hours the gain remains +1.4 MUSIQ/+0.14 DNSMOS, and all four severity combinations show positive gains.
  • Non-speech events (O.2): shifted-event success rises from 15.0% to 85.0%, lag MAE falls from 124 to 29 ms, and 97.5% of synchronized events stay synchronized.
  • Strong baselines (O.4–O.5): OmniVR leads SEMamba++, Sidon and UNIVERSE++ on all 14 URGENT-family metrics and both FAD encoders, and exceeds ColorSurge + SeedVR2 by 2.6 MUSIQ with lower LPIPS/DISTS, also under shared colorization.
  • Real-world transfer (O.7): frozen OmniVR improves every Real content category and every unseen collection (+3.5 MUSIQ/+0.17 DNSMOS on 80 unseen clips), and raters prefer it in 76.2%/77.5% of paired comparisons.
  • Efficiency (O.8–O.9): Flash sustains 55.1 fps at 1K, 2.30× the 24-fps source rate, while staying close to Full on content metrics (WER +0.5 points, speaker cosine −0.004).

3. Clarifications and Corrections.

  • Throughput: the main-text 38.2 fps uses standard SDPA; the 55.8 fps isolated-window figure uses FA4 with CUDA deployment optimization; the continuous optimized run reaches 55.1 fps under one RAM-to-RAM timing boundary. Table 22 and K.2 now label both profiles.
  • Evaluation accounting: Table 31 now states that OOM exclusions affected both RealBasicVSR and OmniVR, and reports eligible, measured and common counts separately. On a fixed 68-ID set, OmniVR's LPIPS/DISTS/FVD are 0.276/0.123/451 versus 0.565/0.305/2180 for RealBasicVSR.

OmniVR establishes joint audio-video restoration as a unified, measurable problem and delivers a complete, efficient system with consistent gains on controlled, real and unseen archival data. We believe it provides a valuable foundation for audiovisual restoration and heritage preservation, and we respectfully ask the reviewers and the AC to consider accepting this paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.