- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents MatAnyone-E, an efficient framework for high-resolution video matting that reduces redundant computation in both temporal memory retrieval and decoding. Sparse Memory Propagation selectively performs native-resolution memory matching only where precise retrieval is needed, while a lightweight, distillation-based decoder further lowers high-resolution computation. The framework preserves the strong matting quality of MatAnyone 2, demonstrating that selective high-resolution computation is a practical approach for scaling memory-based video matting.
Critical strengths
- This paper focuses on an important and timely problem of high-resolution video matting in terms of its computation and memory efficiency on modern GPU platforms
- This paper gives a clear and systematic design for the overall algorithm and implementation.
- Evaluation also demonstrates its advantages over those existing baselines and ablation studies for breakdown analysis.
Critical weaknesses
- More detailed description of the high-resolution subsets of "VideoMatte" and "YoutubeMatte" should be provided, including its major scene and the scope of its content.
- The cost of training the revised model should be disclosed for better understanding of its effectiveness.
- The figures (such as Figure 2B) are too tight and highly compacted, making it hard to read and at least the author should give highlight to some part of the figure instead treat all components equally. Or maybe a better way to present it is to present them separately instead of as a whole.
- Another concerns based on the provided FPS seems to shows that the output video at 2160p may not be smooth given its low FPS compared to regular 30FPS for better viewing experience.
Questions for authors
- What is the cost of such training this revised model? also compare its training cost with its original MatAnyone 2.
- Considering those largely saved memory effects of the proposed method and its reported reduced TFLOPs, Just feel that the proposed solution are more likely to under utilize the GPUs computing and memory resources, considering the costs of maintaining those GPUs is relatively fixed (such as renting GPU servers from the cloud services), what is the potential application settings of the proposed solution? Are you considering multiplexing a GPUs for different requests or larger throughputs to fully utilize the more idle GPUs if with this revised solution?
- Consider the efficiency of SMP relies on the effectiveness of the reuse those low-resolution drafts, one thing we are wondering is that how does the drastically changed scenes (e.g., a soccer player running from the top-right corner to the bottom-left corner of a scene in several continuous frames) would affect its efficiency, where more static scene or foreground objects demonstrate higher speed up while drastically moving foreground objects would be hard to benefit from SMP.
3 comments
Thanks for providing more evidence and it helps address my concerns, and I will keep my current score.
Thank you for taking the time to review our additional results. We appreciate your consideration.
Dear Reviewer WKWP,
We sincerely thank you for the positive feedback and recognition of our systematic design and experimental evaluation. We address your concerns below.
[W1] Benchmark Composition and Characteristics
We thank the reviewer for the comment. Both VideoMatte and YouTubeMatte provide low-resolution (512 × 288) and high-resolution (1920 × 1080) test sets. We use their complete high-resolution test sets for evaluation. Both benchmarks focus on human video matting.
VideoMatte HR contains 50 videos (5,000 frames), generated by compositing 5 human foreground clips onto 25 video and 25 image backgrounds. It covers upper-body and full-body subjects performing simple actions, with backgrounds spanning natural landscapes, urban environments, and indoor scenes.
YouTubeMatte HR contains 64 videos (6,400 frames), generated from 32 human foreground clips with video and image backgrounds. It includes diverse human appearances, framing, motions, and object interactions, covering children, elderly people, and multiple subjects. Its backgrounds span similar scene categories, with additional harmonization for improved foreground-background consistency. We have added this description to Appendix N.
[W2, Q1] Training Cost
Training our revised matting model requires approximately 258 GPU-hours (around 32 hours on 8 NVIDIA A800-80GB GPUs), with a total batch size of 16. In contrast, MatAnyone 2 trains both the matting model and a separate Matting Quality Evaluator. Since MatAnyone 2 does not report its training cost, a direct comparison in GPU-hours is not possible.
[W3] Figure Presentation
We thank the reviewer for the suggestion. We have adjusted the layout and some details of Figure 2 to improve readability.
[W4] 4K Inference Speed
We have made further implementation-level optimizations, including replacing PyTorch's adaptive average pooling with an equivalent strided average pooling operation and compiling additional modules. These optimizations improve the 4K inference speed from 28.6 to 33.9 FPS on an NVIDIA H200 GPU without compromising matting quality. We have updated Tables 2 and 9 and documented the optimizations in Appendix N.
[Q2] GPU Utilization
Thank you for the question. We evaluated concurrent 4K video inference on an NVIDIA H200 GPU with model compilation.
With four concurrent streams, the total throughput increases from 33.7 to 38.7 FPS (14.8% improvement). Moreover, MatAnyone-E supports up to 28 concurrent 4K streams on a single H200 GPU, whereas MatAnyone 2 can only support one. These results demonstrate the potential of MatAnyone-E for memory-efficient multi-request deployment. We have added the concurrent-stream results to Table 19 in Appendix M.
[Q3] Efficiency under Fast Motion
We thank the reviewer for the question. To evaluate SMP under fast motion, we conducted additional experiments at 4K resolution on an NVIDIA H200 GPU with model compilation, using our test set and 16 motion-augmented sequences.
We grouped frames by foreground motion magnitude, measured using optical flow. The results are summarized below.
As foreground motion increases, the full-resolution query selection ratio rises from 3.2% to 6.7%, while FPS decreases slightly from 34.1 to 32.1. To further evaluate extremely large displacements, we applied additional horizontal translations to foreground videos. The results are summarized below.
At the highest motion level, the foreground moves nearly the full width of a 4K frame in one second at 30 FPS. Even under this extreme motion, SMP maintains 30.1 FPS while selecting only 10.9% of full-resolution queries. These results demonstrate that SMP remains computationally efficient even under extreme foreground motion. We have added the motion-group analysis and extreme-motion experiments to Tables 17 and 18 in Appendix L, respectively.
We hope our responses have addressed the reviewer's concerns. If anything remains unclear or if you have further questions, we would be happy to clarify and discuss them in more detail.