acceptodds
Withdrawn from ICLR 2027

AsymRefresh: Modality-Asymmetric Refresh for Efficient Joint Audio-Video Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Feature caching accelerates joint audio–video generation, but joint refreshes tie additional audio computation to video recomputation. We introduce AsymRefresh, a training-free method that makes modality choice part of the refresh decision. Between full joint refreshes, selected steps recompute the native audio pathway against reconstructed video contexts while predicting the video output. A shallow joint prefix and periodically renewed contexts preserve audio access to the evolving video trajectory, with unchanged model weights and samplers. We evaluate JavisDiT++ on all 10,140 JavisBench prompts and both JavisDiT++ and JavisDiT on all 500 T2AV-Compass prompts, covering joint-attention and dual-stream backbones. On JavisBench, AsymRefresh achieves a speedup in a separate timing study with competitive full-benchmark quality against four caching baselines. On JavisDiT, the selected T2AV operating point leads the compared accelerators on four of six primary quality means. Controlled JavisDiT++ studies on 512 prompts and three seeds show improved audio–text alignment over additional joint refreshes at comparable latency under two budgets, with positive multiplicity-adjusted intervals. The six-update gain is also observed at a second prefix depth. Stage and context controls characterize the alignment benefit and accompanying quality and memory trade-offs. These results support modality selection as a useful dimension of inference-budget allocation in joint generation.

Primary area:
generative models
Keywords:
joint audio-video generation, diffusion transformers, training-free inference acceleration, feature caching, modality-asymmetric refresh

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_kYZ6OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes AsymRefresh, a training-free method for speeding up joint audio-video generation. Existing caching methods skip computation on most sampling steps and periodically run full “joint” refreshes that recompute both modalities. AsymRefresh adds a third option: selective steps that recompute only the audio while predicting the video output.

To keep audio conditioned on the evolving video, a shallow joint prefix (1–2 blocks) runs at every step. Deeper audio blocks attend to video context reconstructed from the two most recent full refreshes. Output prediction uses the existing DiCache predictor.

The method is evaluated on JavisDiT++ (JavisBench, 10,140 prompts; T2AV-Compass, 500 prompts) and JavisDiT (T2AV-Compass), against TeaCache, PAB, TaylorSeer, and DiCache. On JavisBench, it achieves a 1.93× speedup over the unaccelerated model with quality comparable to the baselines. Controlled studies on 512 prompts and three seeds show that, at matched latency, audio-only updates improve audio–text alignment (CLAP) slightly more than extra joint refreshes. Late-stage updates outperform middle-stage ones. These gains come with small drops in some video and audio quality scores and about 2.5 GiB of additional memory. All evaluation uses automatic metrics.

Critical strengths
  • Clear, simple idea. Treating “which modality to recompute” as a budget choice is easy to grasp and offers a new option for speeding up joint audio-video generation. It’s also training-free, so it works with existing models without retraining.
  • Transparent reporting. The authors report all metrics, including unfavorable ones, plus per-seed results and inconclusive findings. They clearly label which results are exploratory or tuned on observed data. This openness is what makes the weaknesses easy to identify.
  • Matched-budget comparison. The key experiment compares audio-only updates against extra full refreshes at nearly the same speed, with the predictor held fixed. This directly tests the paper’s central question, even though the measured gain is small.
Critical weaknesses

No human subjective evaluation

The main gains are very small, around +0.001 to +0.004 on CLAP. Without a listening test, we don’t know whether anyone could actually hear a difference.


Novelty

The paper’s main idea is to recompute only the audio on some caching steps, while predicting the video, instead of always refreshing both modalities together. Framing modality choice as part of inference-budget allocation is clear and reasonable. However, the individual components are largely borrowed:

  • The output and context predictor is taken directly from DiCache (Bu et al., 2026).
  • Modality-asymmetric computation, where video is reused and audio recomputed, appears in SyncCache (Ma et al., 2026).
  • Audio attending to cached video context appears in H3 AudioRefine. The distinct contribution is applying this asymmetry during joint sampling, where both modalities are still evolving. This requires the shallow joint prefix and the reconstruction of video context from anchors. That is a genuine technical adaptation, but an incremental one in my opinion.

Small Gains Obscure the Contribution of the New Components

The novelty concern is amplified by the size of the reported benefit. The gains over extra joint refreshes are small (+0.0007 to +0.0035 CLAP). The main-benchmark results are largely indistinguishable from DiCache, and the paper’s own ablation could not show a benefit from context extrapolation, the component most specific to the new setting. As a result, it is unclear whether the new elements, beyond DiCache plus an audio-only schedule, contribute meaningfully. I would encourage the authors to:

  1. state more precisely what is new relative to SyncCache and H3 AudioRefine,
  2. show a case where the joint-sampling-specific design (prefix plus reconstructed context) clearly matters.

No synchronization measurement

The method’s goal is keeping sound tied to the video, but the sync metric (DeSync) was dropped because the clips are too short. There’s no direct evidence that timing between sound and picture is preserved.


Settings tuned on the test data

The T2AV operating points were picked after looking at benchmark results, and the authors call them exploratory. The follow-up test on new prompts found no clear advantage, and those prompts were written by AI with no human review.


Weak timing evidence.

Speed is measured on only 4 prompts and 3 repeats. For JavisDiT, the Native timing comes from a different setup (concurrent runs), so its speedup isn’t a fair comparison. The speed advantage over DiCache is also small (17.68 s vs 18.00 s on T2AV).


Quality and memory costs.

Some video and audio quality scores drop (VQ, MQ, PQ), and memory use rises by about 2.5 GiB, roughly 15%.


Narrow scope

  • Only short clips (~4 seconds) are used.
  • Only audio-only refreshes are explored. The reverse, video-only refreshes, is never tried.
  • The method is compared only to other caching tricks, not to fewer-step samplers or distilled models, and it isn’t tested in combination with them.

I am willing to increase the score if the weaknesses can be addressed.

Questions for authors

See critical weaknesses.

Reviewer_Y9xeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AsymRefresh, a training-free feature caching method for DiTs that jointly generate audio and video. At some steps between full joint refreshes, the video output is replaced with an estimate from a predictor, and only the remaining audio blocks are recomputed. The intermediate video states that audio attends to are obtained from two sources: the output of a shallow joint prefix computed at every step, and an extrapolation from the states stored at the two most recent full refreshes. The authors claim that, at comparable latency, spending computation on audio-only updates improves audio–text alignment more than adding joint refreshes. In the experiments, the proposed method is compared with four existing caching methods on JavisDiT++ (JavisBench and T2AV-Compass) and JavisDiT (T2AV-Compass).

Critical strengths
  • The problem setting tackled in this paper is important, and the motivation is clear.
    • In joint audio-video generation, audio and video are commonly represented and processed as separate token sequences. The goal of optimizing how a limited computational budget is allocated across modalities is therefore well motivated and convincing.
    • The proposed method would be broadly applicable to models built on a DiT architecture.
Critical weaknesses
  • Most components of the proposed method build on prior work, which limits its novelty.
    • The predictor (Eqs. (6) and (7)) and the shallow prefix are taken directly from DiCache. The idea of reusing video token computation while recomputing only audio tokens can also be found in prior work such as SyncCache and H3 AudioRefine. The authors position their contribution as applying this idea to joint audio–video generation, where both modalities evolve. However, this appears to be mainly a difference in the application setting, and the methodological difference seems small.
    • The component unique to this method, extrapolation of intermediate video context (Eq. (3)), is not shown to be effective in comparison with held context (Table 15). Similarly, no prefix depth is shown to be preferable (Table 4). As a result, the contribution with a confirmed effect appears to be limited to the scheduling choice of placing audio-only recomputation in late steps.
  • The experimental results do not sufficiently support the effectiveness of the proposed method.
    • The speedup over DiCache is very small on JavisBench (Table 1), and given the small number of measurements, it is unclear whether this difference is significant. On T2AV-Compass (Table 2), there is almost no speedup.
    • The authors’ main claim is that, at the same latency, spending additional computation on audio-only updates yields better alignment than spending it on joint computation. However, the differences in the comparison with the joint-only controls (Tables 3 and 15) are very small. Considering that the required peak allocated memory increases by more than 10% (Table 20), the practical significance of this gain is unclear.
    • The generated examples in Figure 2 do not show a noticeable difference between the proposed method and joint-only. In addition, no subjective evaluation is reported, and no supplementary material or demo page is provided. This makes it difficult to assess the perceptual validity of the results.
    • The evaluation is limited to two models from the same family, JavisDiT and JavisDiT++, so generalization to other architectures remains unclear.
Questions for authors
  • Compared with simply combining the DiCache predictor with a fixed schedule, what is the novelty of the proposed method?
  • Given that the benefit of extrapolation is not demonstrated against held context, what is the rationale for including Eq. (3) as a component of the proposed method?
  • Could the MP4/WAV playback files mentioned in Appendix K be made available?
  • Would a similar allocation benefit at comparable latency be obtained on joint generation models with different architectures, such as Ovi?
Reviewer_sdJpOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AsymRefresh is a training-free acceleration method for joint audio-video diffusion models. Instead of refreshing both modalities whenever additional computation is allocated, it selectively recomputes the audio pathway while predicting the video output. A shallow joint prefix and periodically refreshed video contexts maintain cross-modal conditioning, and intermediate video inputs are reconstructed using a DiCache-based predictor. Experiments on JavisDiT++ and JavisDiT evaluate the resulting quality–latency trade-offs on JavisBench and T2AV-Compass, including controlled comparisons between selective audio updates and additional joint refreshes.

Critical strengths
  1. Selective audio refreshes improve computation allocation without retraining the backbone.
  2. The separate JavisBench timing study reports a 1.93× speedup with competitive full-benchmark quality.
  3. Matched-predictor comparisons and multi-seed studies support the allocation benefit, and the paper clearly reports trade-offs and inconclusive results.
Critical weaknesses
  1. Limited backbone coverage. Both evaluated models belong to the JavisDiT family, despite representing different attention layouts. The cost and effectiveness of selective refreshes may change with modality interactions, token ratios, and sampling schedules. Results on production-scale models such as LTX-2 and MiniMax-H3 would substantially strengthen the practical relevance.
  2. Resource and quality trade-offs. Peak allocated memory rises from 17.400 to 19.964 GiB in the depth-two audio configurations, and some video- and audio-quality measures decrease in the allocation studies. These are meaningful costs for an inference-acceleration method, even though the paper reports them transparently. A compact comparison of latency, memory, and quality would make the useful operating range clearer.
  3. Unresolved benefit of context extrapolation. The controlled comparison does not establish an additional alignment benefit over held context. This leaves open whether extrapolation is essential to the method's practical advantage. Please clarify whether a simpler held-context configuration retains the main benefit and what is gained by the extra reconstruction mechanism.
Questions for authors

please see weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.