- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper presents JoyAI-Video-Edit, a 16B-parameter autoregressive model for streaming video editing. Its main algorithmic component, Resilient-DMD, monitors source-versus-history attention and adaptively increases teacher-side source guidance when source attention falls below matched moving baselines. Long-Horizon Autoregressive Distillation extends training to deeper generated histories through forward-reverse source traversal and segmented optimization. The paper also develops a bounded-state asynchronous inference engine and introduces LongV2VBench, containing 229 one-minute editing instances. Evaluations report improvements over streaming baselines, competitive editing quality relative to offline models, and 34.87 FPS steady-state end-to-end throughput at 720p on a single B200.
Critical strengths
-
The paper addresses a concrete failure mode specific to streaming editing: generated history can override changes in the current source. The proposed teacher-side guidance mechanism is practical because it introduces no monitoring or additional guidance branches at inference. The component ablations and comparison against the best tested static-guidance baseline provide meaningful evidence that adaptive source guidance improves editing quality.
-
The work combines algorithmic improvements with a substantial deployment implementation. It reports full-pipeline throughput, chunk-latency distributions, component-level optimizations, and quality changes under deployment precision settings. The evaluation spans an existing benchmark, a new minute-long benchmark, qualitative comparisons, and human preferences, supporting the practical value of the resulting editor.
Critical weaknesses
-
It is unclear whether low source attention reliably identifies editing errors. The appropriate reliance on the source can differ between, for example, recoloring an object and replacing an entire background, but the EMA baseline does not account for the editing task. Figure 9 shows that dynamic guidance works better than the tested fixed weight; it does not show whether the attention signal assigns stronger guidance to the right chunks. The authors could address this by measuring how well the signal predicts source-following errors across editing categories, or by shuffling guidance weights among samples within matched noise/depth bins. The latter would test whether matching the weight to the sample matters beyond the weight distribution itself.
-
The minute-long benchmark shows better average editing quality, but does not directly measure recovery from history errors. The human study also uses clips shorter than ten seconds. A focused comparison against static guidance on events such as subject disappearance, measuring how long phantom content persists, would better support the resilience claim.
-
The main streaming baselines with reported model sizes use 1.3B–2B parameters, whereas JoyAI-Video-Edit uses 16B parameters and a substantial foundation-training pipeline. This makes it difficult to attribute the cross-model quality gains to the proposed methods rather than model capacity and training data. The same-backbone ablations are helpful, but a smaller-scale implementation or a capacity-matched streaming baseline would make the comparison more convincing.
Questions for authors
How reliably does the attention score identify chunks that benefit from stronger source guidance? Does the same controller configuration work across editing categories and scene types, or does it require substantial manual tuning of the monitored layers, EMA settings, and guidance parameters?
3 comments
I thank the authors for the detailed response and for conducting the suggested controls.
The AUROC analysis and the guidance weight shuffling experiment satisfactorily demonstrate the predictive utility of the attention signal. Additionally, the response delay measurements provide concrete quantitative evidence regarding error recovery and phantom content suppression. While the capacity difference relative to external streaming baselines remains a contextual caveat, the same-backbone ablations and the new validation results adequately address my primary concerns.
Overall, the response clarifies the open questions, and I maintain my rating of 3 (Weak Acceptance). Please ensure that all newly presented experimental results—particularly the attention reliability tests and the response delay metrics—are fully integrated into the final manuscript.
Response to Reviewer UyuW
Dear UyuW,
Thank you very much for your positive feedback. We are glad that our responses have addressed your concerns, and we will incorporate all these revisions into the final manuscript.
Best regards,
Paper 6937 Authors
Response to Reviewer UyuW (1/1)
We thank you for the valuable feedback. Our responses below address each concern through controlled experiments and configuration details.
q1/w1: How reliably does the attention score identify chunks that benefit from stronger source guidance?
w2: Recovery from history errors and phantom persistence
History error recovery: We define response delay as the interval from the subject's disappearance in the source video to the disappearance of the corresponding phantom in the edited output. As shown in Table 16 in our response to Reviewer NFd5, the maximum delay is 0.3 s with Resilient-DMD, while phantom content persists for at least 4 s without it. These measurements support rapid source following and suppression of persistent history artifacts.
Dynamic versus static recovery: Both static and dynamic guidance provide source-conditioned correction. On the same source-change cases, dynamic guidance reduces the maximum response delay from 1 s (three chunks) to 0.3 s (within one chunk) (Table 19), supporting the benefit of adapting correction strength to each chunk.
Table 19: Response delays under dynamic and static guidance after substantial source changes.
Full-video human evaluation: As shown in Table 15 in our response to Reviewer NFd5, human evaluation of complete one-minute and five-minute videos yields win rates of 88–94% and 98–99%, respectively, for our method against the three streaming baselines. These results support sustained editing quality over the evaluated durations.
w3: Attributing cross-model quality gains to methods rather than model capacity and training data
q2: Does the same controller configuration work across editing categories and scene types without substantial manual tuning?
Shared controller configuration: We use one controller configuration during distillation and the same distilled checkpoint across all editing categories and tested scenes. Since monitoring and guidance adaptation operate only during training, inference requires no task- or scene-specific controller tuning. Table 2 reports per-task performance, while Figure 20 and Tables 15 and 16 in our response to Reviewer NFd5 support sustained editing quality and source responsiveness.
Monitoring layer selection: No specific choice of monitoring layers is required. Our layer-selection ablations showed 0.8% variation in final scores. We therefore use all layers for monitoring in the final configuration. Monitoring reuses the generator's existing queries and keys and adds no forward or backward model passes.
EMA configuration: We use fixed EMA settings across tasks, without task-specific tuning: β=0.99, two exit-step noise bins, and three rollout-depth bins. Each EMA entry is initialized from its first available attention statistic and updated after each rollout. An entry is used for adaptive normalization once it has accumulated 24 attention measurements.
Guidance configuration: We selected the guidance parameters through a standard hyperparameter search and reuse the same settings across tasks and in both short-rollout and LHAD training, without task-specific retuning. Table 5 and Fig. 10 report sensitivity results and guidance trajectories under alternative settings.