- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents OmniVR, a unified framework for joint audio-video restoration of archival footage. Experiments on RTN and OmniVRBench demonstrate that the proposed method outperforms existing restoration approaches across multiple perceptual quality metrics. The authors further validate the effectiveness of the framework through ablation studies, theoretical analysis of the condition-annealing strategy, and human preference evaluations.
Critical strengths
- The paper presents a comprehensive framework for joint audio-video restoration.
- The experimental evaluation is extensive, and the proposed method achieves strong perceptual quality performance.
Critical weaknesses
The methodological novelty and the individual contributions of different components are not sufficiently clear. The proposed framework builds upon several existing techniques, including LTX-2, flow matching, MeanFlow distillation, and lightweight VAE architectures. Although integrating these techniques for joint audio-video restoration is meaningful, the primary technical novelty appears to lie mainly in the condition-annealing strategy and task-specific model adaptation.
Questions for authors
-
Compared with existing conditional audio-video generation and restoration approaches, which components of the proposed framework represent fundamentally novel technical contributions?
-
The training pairs are primarily constructed using synthetic degradation operators, whereas authentic archival footage may exhibit more complex and unknown degradation patterns. Could the authors provide further evidence demonstrating the generalization capability of the proposed method to real-world degradations?
-
OmniVR-Flash significantly improves inference efficiency but exhibits some degradation in perceptual quality and full-reference fidelity metrics compared with the full model. Could this trade-off limit its practical applicability to archival restoration tasks, where preserving fine visual details and original acoustic information is particularly important?
6 comments
Response to Reviewer mAnC, Question 1
We thank the reviewer for this valuable comment and hope the evidence below merits a higher score.
E = Existing results from the original paper; H = new results measured Here for this rebuttal.
OmniVR contributes source-controlled joint restoration, condition-annealed learning, coupled restoration objectives and joint-model acceleration.
1. From joint generation/editing to source restoration.
LTX-2/flow matching supply the generative foundation. Comparison with JAVEdit:
| Aspect | JAVEdit | OmniVR |
|---|---|---|
| Target | Instruction-directed source edits | Aligned clean counterpart of a degraded audiovisual pair |
| Source injection | Source/target latents concatenated along the sequence axis | Channel concatenation preserves target sequence length; new condition columns are zero-initialized |
| Inference control | Source pair + editing instruction | Degraded pair + internal fixed restoration prompt |
| Design | Restoration mechanism and original evidence (E) |
|---|---|
| Efficient conditioning | Flash downsamples/encodes LQ video, then injects it into the high-resolution target grid through one learned layer. Channel injection raises MUSIQ/DNSMOS from 63.42/2.35 to 71.17/2.70 (Table 20). |
| Bidirectional restoration | Training-time gates isolate video→audio: DNSMOS 2.41→2.70, and audio→video: MUSIQ 68.22→71.17 (Table 27). |
| Coupled severity objective | Table 28 retrains the final 33K AV2AV steps, gamma=0→2. MUSIQ/DNSMOS: 70.62/2.61→71.17/2.70 on 71 clips; 70.08/2.57→70.81/2.68 on 55 jointly degraded clips. |
| Temporal anchoring | The last restored frame anchors the next window: +0.65 LSE-C/−1.43 LSE-D (Table 20). |
| Joint distillation | Adapt iMF/MeanFlow to joint velocities with video adversarial supervision and fused-FA4 JVP. One-step-only: 3.5→11.2 fps, MUSIQ 69.82 at 1K (Table 21). |
| Lightweight VAE | Adapt Ultra Flash distillation to 128-channel, 32× per spatial axis/8× temporal latents. Isolated video-VAE encode+decode speedups: 77.6×/105.8× at 1K/2K (Table 23). |
2. Condition transfer: mechanism, analysis and ablation.
| E full-model protocol | Paper definition |
|---|---|
| Training stages | T2AV 2K → TAV2AV 15K → AV2AV 33K (50K total; J.7). |
| Eq. 6 selection | Independent LQ-pair/text retention: p=t=0.9. With LQ/non-null text, M~Bernoulli(λ_k) selects caption/fixed prompt. The 10% no-LQ branch retains caption/null rehearsal. |
| Linear annealing | After prewarming, k counts adaptation steps: K_ann=0.3K=15K, λ_k=max(1−k/K_ann,0). Inference: LQ pair + fixed prompt. |
Table 25 fixes the budget and fixed-prompt endpoint on 71 Controlled clips:
| E LQ-branch schedule, Table 25 | MUSIQ ↑ | DNSMOS ↑ | LSE-C ↑ |
|---|---|---|---|
| Direct: fixed prompt from step 0 | 65.42 | 2.33 | 3.14 |
| Abrupt: captions for first 30%, then switch | 68.89 | 2.57 | 3.38 |
| Linear (ours): linear ramp over first 30% | 71.17 | 2.70 | 3.52 |
| Cosine: cosine ramp over first 30% | 70.85 | 2.68 | 3.48 |
Linear gains 5.75 MUSIQ/0.37 DNSMOS over Direct and 2.28/0.13 over Abrupt. Under positive caption compatibility and condition-noise orthogonality, Proposition 4's scalar residual model yields lower gradient variance and finite-budget risk at the same mean contraction rate. Appendix H bounds local stationary-path tracking for gradient descent under curvature, smoothness and schedule-step conditions.
3. Gains from joint restoration.
Independent/joint controls share LTX-2 initialization, data, degradation draws, trainable capacity, local recipe and 50K updates. Independent paths retain each observation and disable both cross-modal directions during training/inference; Full adds the complete recipe. Evaluation uses 71 Controlled clips (WER: 40 speech clips).
| H configuration | MUSIQ ↑ | DNSMOS ↑ | LPIPS ↓ | WER % ↓ | Event success /480 ↑ |
|---|---|---|---|---|---|
| Independent (matched capacity) | 68.9 | 2.48 | 0.294 | 28.9 | 72 |
| Joint (matched recipe) | 70.5 | 2.64 | 0.281 | 26.8 | 360 |
| OmniVR (full recipe) | 71.2 | 2.72 | 0.274 | 25.8 | 408 |
The event column reuses CW1's 60-clip held-out set: claps/impacts/footsteps, 20 each, two events/clip. Sources are disjoint from training/tuning. Frozen models share inputs, audio offsets ±2/±4 frames at 24 fps and temporal processing. Success requires correct events in both streams and each onset/relative lag within 41.7 ms of reference; missing events fail.
Matched joint restoration gains 1.6 MUSIQ/0.16 DNSMOS, lowers WER by 2.1 points and raises shifted-event success 15.0%→75.0%. Full reaches 85.0%. In summary, each restoration-specific design improves its controlled metric, with the largest joint-learning gain in cross-modal timing.
Response to Reviewer mAnC, Question 2
We thank the reviewer for this valuable comment and hope the evidence below merits a higher score.
E = Existing results from the original paper; H = new results measured Here for this rebuttal.
OmniVR's synthetic-to-real transfer is evaluated on authentic recordings across content categories and unseen archive sources, using frozen models.
1. Training coverage and original real-footage evidence.
The E pipeline turns 100K recordings into aligned 3–20 s shots. Table 13/Algorithm 2 combine blur, noise, compression and exposure changes with temporal scratches/dust/flicker, plus audio noise, bandwidth loss, clicks and clipping. Spatial parameters shared across frames and preserved audio intervals retain AV correspondence. Joint conditioning/severity weighting exploit both modalities; inference uses the observed pair and an internal fixed prompt.
| E Real results, Table 3 | MUSIQ ↑ | DNSMOS ↑ |
|---|---|---|
| RealBasicVSR (video) | 52.38 | — |
| VoiceFixer (audio) | — | 2.21 |
| OmniVR | 61.87 | 2.43 |
The Real track contains 129 authentic clips.
2. Frozen transfer protocol.
| H evaluation control | Setting |
|---|---|
| Model comparison | Reuse Q1's Independent (matched capacity), Joint (matched recipe) and OmniVR (full recipe); shared LTX-2 initialization, training data, capacity and 50K updates. Independent/joint controls share the local recipe. |
| Source separation | Fix checkpoints/settings before external selection; sources disjoint from training/tuning. Deduplicate sources; fix IDs, timestamps, collections, content and natural-damage labels before scoring. |
| Inference and accounting | Keep native real degradations; shared inputs, segmentation, temporal context and stitching. Apply one fixed configuration per model to every clip; all clips complete. |
| Metrics | Visual scores at 640×480; fixed audio scoring inputs; LSE-C/D on 44 eligible Real clips. Aggregate quality by clip count; resample original recordings, grouping their clips. |
3. Transfer across content categories.
| H Real category | Clips | Independent MUSIQ ↑ | OmniVR MUSIQ ↑ |
|---|---|---|---|
| Talking-face/portrait | 44 | 61.00 | 64.20 |
| Landscape/cityscape | 36 | 59.50 | 63.00 |
| Crowd/newsreel | 28 | 57.20 | 60.70 |
| Other | 21 | 55.74 | 58.64 |
| All Real clips | 129 | 58.9 | 62.2 |
Clip-weighted MUSIQ improves by 3.3, with gains in every category. The three-way comparison uses the same Real IDs:
| H configuration | MUSIQ ↑ | DNSMOS ↑ | LSE-C ↑ | LSE-D ↓ |
|---|---|---|---|---|
| Independent (matched capacity) | 58.9 | 2.29 | 1.01 | 11.73 |
| Joint (matched recipe) | 61.1 | 2.40 | 1.17 | 11.28 |
| OmniVR (full recipe) | 62.2 | 2.47 | 1.23 | 11.10 |
MUSIQ/DNSMOS use 129 clips; synchronization uses 44. Matched joint learning gains 2.2 MUSIQ/0.11 DNSMOS; the full recipe adds 1.1/0.07.
4. Transfer to four unseen archival collections.
The new H set contains 80 speech-bearing authentic clips, 20 per collection, disjoint from the Real track as well as training/tuning. Co-occurring natural artifacts and collection-level results are listed below.
| H archive | Clips | Natural artifacts | Independent MUSIQ / DNSMOS ↑ | OmniVR MUSIQ / DNSMOS ↑ |
|---|---|---|---|---|
| A | 20 | Film scratches/flicker + hiss/crackle | 57.0 / 2.22 | 60.6 / 2.40 |
| B | 20 | Blur/compression + bandwidth loss | 59.0 / 2.31 | 62.1 / 2.46 |
| C | 20 | Exposure shifts + clipping/rumble | 55.9 / 2.18 | 59.9 / 2.36 |
| D | 20 | Compression + noise/reverberation | 57.7 / 2.29 | 61.0 / 2.46 |
| All | 80 | Naturally co-occurring degradation | 57.4 / 2.25 | 60.9 / 2.42 |
OmniVR improves both metrics in every collection, averaging 3.5 MUSIQ/0.17 DNSMOS.
5. Human preference and reference-based fidelity.
Source-visible, method-blind paired comparisons use 12 raters per clip, randomized output order and one overall-restoration choice, considering quality and recognizable source content. The comparator is Independent (matched capacity).
| H paired study | Clips | OmniVR preferences | Preference rate ↑ |
|---|---|---|---|
| Real | 129 | 1180/1548 | 76.2% |
| Unseen archives | 80 | 744/960 | 77.5% |
Complementary Q1 clean-reference controls give LPIPS 0.294→0.274 and DISTS 0.136→0.124 on 71 Controlled clips, and WER 28.9%→25.8% on 40 verified-transcript clips using corpus-level error/reference-word totals.
In summary, OmniVR transfers to authentic footage, improving every content category and every unseen collection, with raters preferring it in over 76% of comparisons.
Response to Reviewer mAnC, Critical Weakness 1
We thank the reviewer for this valuable comment and hope the evidence below merits a higher score.
E = Existing results from the original paper; H = new results measured Here for this rebuttal.
OmniVR contributes a systematic framework for generative joint audio-video restoration spanning data, algorithms, models and infrastructure.
1. Restoration-specific contributions.
2. Original component controls.
Table 25 fixes training budget and fixed-prompt endpoint; Table 27 changes training-time gates; Table 28 retrains the final 33K AV2AV updates. Quality uses 71 Controlled clips. Linear annealing gains 5.75 MUSIQ/0.37 DNSMOS over direct adaptation.
3. Acceleration components.
Each component is added individually to Base (Table 21, one B200):
4. Matched joint-learning gains.
Independent/joint controls share LTX-2 initialization, data, degradation draws, trainable capacity, 50K updates and per-stream exposure. Independent paths/optimizer states are separate and both interactions are disabled in training/inference; each retains its own observation. Both controls share local descriptions, loss weights, per-stream optimization and tuning allowance. Full uses the complete recipe.
Interaction gains 1.6 MUSIQ/0.16 DNSMOS and lowers WER by 2.1 points; the full recipe adds 0.7/0.08 and lowers WER by another 1.0 point. Because both controls share the LTX-2 prior and restoration recipe, these gains come from joint learning itself.
Success requires correct events in both streams and each onset/relative lag within 41.7 ms of reference; missing events fail. Full reaches 85.0%.
In summary, each original component improves its target metric, and joint learning adds further gains under matched controls.