acceptodds
Under review as a conference paper at ICLR 2027

CAVC: Convergence-Adaptive Audio-Visual Cache for Modality-Asymmetric Diffusion Generation

Abstract

Diffusion-based audio-visual generation methods typically perform joint denoising of audio and video latents throughout the entire sampling trajectory, repeatedly executing modality-specific denoising branches and cross-modal attention at every timestep. However, we observe that such uniform joint computation is not always necessary. The key is a modality-asymmetric recoverability: audio-side drift introduced by mid-stage caching can be substantially reduced by only a few terminal joint denoising steps, whereas video-side degradation caused by caching leads to persistent loss of motion and temporal details. This asymmetry suggests that inference acceleration should selectively cache recoverable audio-side computation while preserving full video refinement. Motivated by this observation, we propose Convergence-Adaptive Audio-Visual Cache (CAVC), a training-free modality-asymmetric caching framework. CAVC adopts a sandwich-structured computation allocation: after detecting that audio denoiser outputs enter a cache-tolerant regime, it caches and reuses the audio denoiser output to bypass audio denoising and cross-modal attention in the middle stage, while the video branch continues independent spatio-temporal refinement; a few joint steps are then restored at the end to reduce accumulated cross-modal drift and improve synchronization. Experiments across multiple public benchmarks demonstrate that CAVC achieves 1.10–1.35× wall-clock acceleration, improves several visual dynamics and temporal statistics, and maintains competitive audio fidelity and audio-visual alignment.

Primary area:
generative models
Keywords:
Audio-Visual Generation, Diffusion Model, Inference Acceleration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UXbZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes CAVC (Convergence-Adaptive Audio-Visual Cache), a training-free inference acceleration method for audio-video diffusion models that selectively caches the audio branch during denoising because audio errors appear easier to recover than video errors. Existing acceleration methods generally assume that all components can be cached or skipped in a similar manner. The hyphothesis is audio and video modalities exhibit asymmetric recoverability during diffusion denoising. The proposed method (CAVC) divides the denoising process into three phases: 1) normal joint denoising, 2) audio caching phase, and 3) recovery phase. Though these approaches, the proposed method could recover from modality-asymmetric problems. The method is evaluated on an audio-video diffusion model derived from the LTX-Video ecosystem with approximately 10%–35% inference acceleration and improvement on the benchmarks.

Critical strengths
  • Well-motivated empirical observation. The paper identifies a potentially interesting phenomenon: errors introduced by caching the audio branch appear substantially more recoverable than errors introduced by caching the video branch. This observation is supported by diagnostic branch-freezing and branch-caching experiments and provides a clear motivation for the proposed method.

  • Practical training-free acceleration. CAVC can be applied directly to pretrained audio-visual diffusion models without retraining, additional fine-tuning, or architectural modifications. This makes deployment straightforward and increases practical applicability in real-world inference pipelines.

  • Simple and easy-to-implement proposed method. The method consists of a relatively lightweight modification to the sampling process involving audio caching and a terminal recovery phase. The implementation complexity is low compared with methods requiring new training objectives or specialized architectures.

  • Comprehensive ablation studies. The paper includes several ablations investigating cache intervals, convergence thresholds, and recovery schedules. These experiments help clarify the contribution of different design choices and improve the reproducibility of the method.

Critical weaknesses
  • Limited novelty relative to existing diffusion caching methods (Major Concern). The core idea appears closely related to prior diffusion acceleration methods such as DeepCache, TeaCache, AdaCache, ToCa, and PAB. These methods already exploit temporal redundancy and selectively reuse computations during diffusion sampling. CAVC mainly applies this principle at the modality level (audio branch) rather than introducing a fundamentally new caching mechanism.

  • Missing important baseline. The most important missing experiment is an audio-only version of TeaCache (or another adaptive caching method) combined with the same terminal recovery stage. Without this comparison, it is difficult to determine whether the improvement stems from the claimed modality-asymmetric recoverability or simply from applying existing caching techniques selectively to the audio branch.

  • Insufficient evidence for the claimed scientific phenomenon. The central claim of "modality-asymmetric recoverability" is demonstrated only on a single model family. The paper does not establish whether this phenomenon generalizes to other audio-visual diffusion architectures, datasets, or modality balancing strategies. As a result, the proposed principle may be architecture-specific rather than fundamental.

  • Modest acceleration gains. The reported speedups are relatively limited compared with other diffusion acceleration approaches such as distillation, step reduction, consistency models, or aggressive caching methods. Since video computation remains largely intact, the achievable acceleration appears inherently constrained.

  • Quality trade-offs are not fully resolved. Although some metrics improve, others degrade noticeably. In particular, quality and naturalness metrics show non-negligible drops in certain settings. The paper therefore does not fully demonstrate a clear Pareto improvement between quality and efficiency.

  • Lacking of deeper analysis of cross-modal interactions. The paper argues that video information must remain active while audio can be cached, but the underlying reasons for this asymmetry remain unclear. A stronger analysis of cross-modal information flow would make the claims more convincing.

Questions for authors
  • Is the contribution fundamentally different from TeaCache/AdaCache?
  • Can we compare with the baseline in the weaknesses?
  • How general is the "modality-asymmetric recoverability" phenomenon? For instance, does this observation hold for other audio-video diffusion architectures?
  • What is the underlying cause of the asymmetry? The underlying problem is not explained. Is it due to stronger video-to-audio conditioning?
  • Why are the speedups relatively modest?
1 comment
Reviewer_UXbZOpenReview ↗

Rebuttal Acknowledgement

We thank the authors for providing a rebuttal addressing the concerns raised above. Although the rebuttal partially resolves some of these concerns, several issues remain, as below:

  • Beyond the reported performance improvements, the technical and methodological novelty appears limited. As noted in the original review, the authors should better justify the novelty of the proposed method and clarify how it is distinct from prior approaches, rather than appearing primarily as an integration of existing techniques. In particular, we encourage to elaborate on what is fundamentally new about the proposed caching strategy.
  • The comparison with TeaCache also remains unconvincing. The gains in performance and speedup are relatively marginal, while TeaCache achieves a higher CLAP score. These results make the advantage over TeaCache unclear, and we encourage the authors to provide further justification or analysis.
  • From the reported results, CAVC’s speedup appears to depend substantially on both the architecture and the benchmark. Could the authors elaborate on the factors underlying this variation? In particular, why do the speedup gains vary considerably across the benchmarks in Table 5?

Some other points for consideration:

  • On a separate point, could the authors explain why MiniMax-H3-CAVC achieves a significant lower CLAP score?
  • Lastly, despite the concerns raised during the review, the manuscript remains unchanged and does not appear to incorporate the reviewers’ comments or suggestions.

Given the remaining concerns, we maintain the current rating.

Reviewer_LC3qOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

In this paper, the authors propose CAVC, a training-free caching method for joint audio-video diffusion generation. Motivated by the different recoverability of audio and video during denoising, CAVC reuses audio predictions and skips cross-modal attention during intermediate sampling steps, while preserving video refinement and restoring joint denoising at the end. An adaptive trigger determines when audio caching begins. Experiments on three benchmarks using LTX-2.3 demonstrate 1.10–1.35× inference speedups, supported by ablation studies and a conditional error analysis.

Critical strengths
  1. Modality-asymmetric caching. The proposed CAVC selectively caches the audio branch while preserving video denoising. The branch ablations in Table 3a support this design, showing that caching the video branch substantially reduces motion dynamics.

  2. Training-free acceleration. The method requires no additional training and retains the original denoising schedule, making it practical for accelerating pretrained audio-video diffusion models.

  3. Effective terminal refinement. Table 4 shows that restoring a few joint denoising steps substantially recovers audio-text alignment after caching, supporting the proposed sandwich structure.

  4. Quality-efficiency trade-off. CAVC achieves 1.10–1.35× speedups across three benchmarks, with improvements in several visual metrics and AV-F1. The ablations also examine different caching strategies and trigger settings.

Critical weaknesses
  1. Insufficient baseline comparison. TeaCache and Cache-DiT are considerably slower than full inference in Tables 1–2. More importantly, the paper lacks a controlled comparison with audio-only TeaCache under the same terminal refinement schedule, making it difficult to isolate the benefit of the proposed adaptive caching strategy.
  2. Lack of human evaluation. Based on the experimental tables, the naturalness drops from 0.6349 to 0.490 on General-80, and AV Sync decreases on two benchmarks. No human evaluation is provided to determine whether these degradations are perceptually acceptable.
  3. Limited generalization. All experiments use LTX-2.3, remaining unclear whether the proposed caching strategy and speedups can be generalized to other architectures or deployment settings.
  4. Limited justification of the convergence trigger. The trigger relies on adjacent audio-prediction changes, which do not guarantee small future caching errors. The paper does not quantitatively establish how reliably the trigger predicts cache tolerance across different generation trajectories.
Questions for authors
  1. Could the authors compare CAVC with an audio-only TeaCache baseline under matched caching schedules and terminal refinement settings?

  2. Could the authors provide human perceptual evaluations of video quality, audio quality, and synchronization to better assess the reported quality trade-offs?

  3. How does CAVC perform on other joint audio-video diffusion models or hardware configurations without CPU offloading?

  4. How well does the convergence trigger correlate with the actual accumulated audio-prediction error during caching?

  5. It would be better to provide a video demo for watching and listening.

1 comment
Reviewer_LC3qOpenReview ↗

Rebuttal Acknowledgement

I appreciate authors' efforts on the rebuttal, which solves most of my concerns. However, after carefully going through the rebuttal and revisiting the paper, I still have the following concerns which are not fully responsed:

  1. Why the performance of naturalness and synchronization evaluation on General-80 present a drop?
  2. Whether the asymmetric denoising observation on the LTX2.3 is the same to the one on other models? Because it will give more evidence about generealization capability of the proposed method.
  3. Whether and how the proposed caching method can be applied long-duration audio-video generation?

Based on the remaining concerns, I'm willing to maintain my current rating.

Reviewer_GybAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Convergence-Adaptive Audio-Visual Cache (CAVC), a training-free modality-asymmetric caching framework for joint audio-visual diffusion generation. It finds that inference acceleration can be achieved by selectively cache recoverable audio-side computation while preserving full video refinement. Therefore, CAVC employs a three-stage schedule: full joint denoising, adaptive audio-prediction caching with continued video refinement, and terminal joint refinement. It reports 1.10–1.35× speedups while maintaining competitive audio fidelity and audio-visual alignment on three benchmarks.

Critical strengths
  1. The motivation is well established. Modality-asymmetric recoverability motivates selectively caching audio-side computations while preserving full video refinement.
  2. Extensive experiments convincingly demonstrate the quality–efficiency trade-offs of the approach.
Critical weaknesses
  1. The paper acknowledges that TeaCache on audio-branch with the same terminal refinement would be a close-related baseline. But the paper lacks the necessary comparison.
  2. The speedup is modest and comes at a cost of quality trade-offs. On General-80, Naturalness drops from 0.6349 to 0.490, while AV Sync also decreases on AVGen-Bench and VABench T2AV. These results raise concerns of quality-preservation claim.
  3. The theoretical analysis does not sufficiently justify the caching trigger. The observed small change monitored by the trigger between adjacent audio-predictions does not guarantee the accuracy of future cached predictions. The assumptions of bounded prediction residuals and local Lipschitz conditions are not definitely established by the trigger. Consequently, the connection between the online triggering criterion and the actual cache-error behavior remains unsupported.
  4. Some related work is missing, such as:
  • AV-DiT: Taming Image Diffusion Transformers for Efficient Joint Audio and Video Generation
  • Javisdit: Joint audio-video diffusion transformer with hierarchical spatio-temporal prior synchronization
Questions for authors
  1. Could the authors provide a comparison with audio-only TeaCache under the same terminal-refinement schedule?
  2. Considering the substantial decrease in Naturalness score on General-80 (Table 1), further evidence is needed to clarify that the increased Dynamic Degree reflects meaningful motion rather than distortions or artifacts.
  3. Does the adjacent-prediction-change criterion accurately predict the actual residual over subsequent caching intervals? Furthermore, how robust is this predictive performance across varying prompts under early or late trigger activation?

Comments on the paper

1 comment
Reviewer_GybAOpenReview ↗

Rebuttal Acknowledgement by Reviewer GybA

I appreciate the authors' efforts in preparing the rebuttal, which addresses several of my previous concerns. However, after carefully revisiting the paper and the rebuttal, I still have several concerns regarding the inherent limitations of the proposed method that remain insufficiently addressed.

  1. Limited architectural generality. CAVC does not appear to be a general acceleration framework for arbitrary joint audio-visual diffusion models. Instead, its effectiveness seems highly dependent on the underlying architecture and the proportion of audio-side computation that can be skipped. The newly added JavisDiT and MiniMax-H3 experiments achieve only 1.86% and 1.2% end-to-end speedups, respectively.

  2. Limitation of caching strategy. Once caching is activated, CAVC continuously reuses the same audio denoiser output until the final joint refinement stage, without monitoring subsequent prediction changes or allowing dynamic cache invalidation. If a substantial change in audio prediction occurs after cache activation, the stale prediction may introduce substantial errors. Moreover, the bounded residual assumptions on ϵa\epsilon_a and ϵv\epsilon_v in Section 3.3 are not guaranteed by the current trigger mechanism.

  3. Insufficient evidence for error recovery through terminal joint refinement. The extent to which terminal joint denoising actually recovers caching-induced errors is not directly validated. In Table 4, some metrics even exceed the Full AV baseline under certain refinement settings, but these metric changes do not directly quantify the recovery of prediction or latent errors. A more direct error analysis or targeted human evaluation would provide stronger evidence. The theoretical analysis in Section 3.3 also does not establish this recovery capability. In particular, Eq. (14) depends on KrefK_{\mathrm{ref}}, which may amplify errors if it is large. The authors do not establish Kref<1K_{\mathrm{ref}} < 1, and thus cannot guarantee that terminal joint refinement contracts or recovers the accumulated errors.

  4. Lack of long-horizon evaluation. The evaluations are largely based on relatively short audio-visual clips of 5–10 seconds. The robustness of the caching strategy under longer and more complex audio-visual sequences remains unclear, particularly when multiple abrupt audio events occur during caching. Further evaluation on longer videos is needed to establish its reliability and generalizability.

  5. Unclear presentation. The paper is somewhat hard to follow. Several ablation settings are insufficiently explained, making the results difficult to interpret. In Table 3(a), the proposed Audio Only strategy does not demonstrate clear advantages in the audio and alignment metrics. In Table 3(b), the configurations of "Cache Only", "Fixed 20%", and "Fixed 80%" are not adequately defined. Table 4 is also confusing: why does "None (Full AV)" have a Stage 2 setting of 3 steps, given that Full AV performs joint denoising throughout the entire sampling trajectory? Also, Fig. 3 uses TT for the trigger thresholds, which should be corrected to θ\theta for consistency with the text.

  6. Unexplained degradation in visual naturalness. The substantial decrease in Naturalness on General-80 (0.6349 to 0.490) remains unexplained. Since the video branch is still evaluated at every denoising step, it is unclear why bypassing audio computation and AV cross-attention leads to such a substantial degradation in visual naturalness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.