acceptodds
Under review as a conference paper at ICLR 2027

MatAnyone-E: Sparse Memory Propagation for Efficient High-Resolution Video Matting

Abstract

High-resolution video matting requires preserving fine details, such as hair strands and soft transparency, while maintaining consistency across frames. Memory-based methods achieve strong matting quality by matching current-frame features with information stored from earlier frames, but this process becomes increasingly costly as resolution grows. We present MatAnyone-E, an efficient extension of MatAnyone 2 that focuses computation where it is most needed. Our approach first performs low-resolution propagation to identify uncertain regions that require closer inspection and the memory candidates most relevant to them. It then performs high-resolution matching only for these regions using the selected memory, while reusing low-resolution matching results elsewhere. This strategy reduces both the number of locations requiring detailed matching and the amount of memory searched, while preserving the information needed to produce a complete matte. We further reduce the cost of generating high-resolution mattes with a lightweight decoder designed around measured runtime bottlenecks and trained to reproduce the original decoder’s representations. Experiments show that MatAnyone-E maintains matting quality comparable to MatAnyone 2 while achieving an estimated 7.4× inference speedup and reducing peak GPU memory usage to 7.6% of MatAnyone 2 at 4K. With compilation-enabled inference on a single NVIDIA H200 GPU, it reaches 73.07 FPS at 1080p and 28.62 FPS at 4K.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Efficient Video Matting, Sparse Memory Propagation, Model Distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_WKWPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents MatAnyone-E, an efficient framework for high-resolution video matting that reduces redundant computation in both temporal memory retrieval and decoding. Sparse Memory Propagation selectively performs native-resolution memory matching only where precise retrieval is needed, while a lightweight, distillation-based decoder further lowers high-resolution computation. The framework preserves the strong matting quality of MatAnyone 2, demonstrating that selective high-resolution computation is a practical approach for scaling memory-based video matting.

Critical strengths
  • This paper focuses on an important and timely problem of high-resolution video matting in terms of its computation and memory efficiency on modern GPU platforms
  • This paper gives a clear and systematic design for the overall algorithm and implementation.
  • Evaluation also demonstrates its advantages over those existing baselines and ablation studies for breakdown analysis.
Critical weaknesses
  • More detailed description of the high-resolution subsets of "VideoMatte" and "YoutubeMatte" should be provided, including its major scene and the scope of its content.
  • The cost of training the revised model should be disclosed for better understanding of its effectiveness.
  • The figures (such as Figure 2B) are too tight and highly compacted, making it hard to read and at least the author should give highlight to some part of the figure instead treat all components equally. Or maybe a better way to present it is to present them separately instead of as a whole.
  • Another concerns based on the provided FPS seems to shows that the output video at 2160p may not be smooth given its low FPS compared to regular 30FPS for better viewing experience.
Questions for authors
  • What is the cost of such training this revised model? also compare its training cost with its original MatAnyone 2.
  • Considering those largely saved memory effects of the proposed method and its reported reduced TFLOPs, Just feel that the proposed solution are more likely to under utilize the GPUs computing and memory resources, considering the costs of maintaining those GPUs is relatively fixed (such as renting GPU servers from the cloud services), what is the potential application settings of the proposed solution? Are you considering multiplexing a GPUs for different requests or larger throughputs to fully utilize the more idle GPUs if with this revised solution?
  • Consider the efficiency of SMP relies on the effectiveness of the reuse those low-resolution drafts, one thing we are wondering is that how does the drastically changed scenes (e.g., a soccer player running from the top-right corner to the bottom-left corner of a scene in several continuous frames) would affect its efficiency, where more static scene or foreground objects demonstrate higher speed up while drastically moving foreground objects would be hard to benefit from SMP.
3 comments

Dear Reviewer WKWP,

We sincerely thank you for the positive feedback and recognition of our systematic design and experimental evaluation. We address your concerns below.


[W1] Benchmark Composition and Characteristics

We thank the reviewer for the comment. Both VideoMatte and YouTubeMatte provide low-resolution (512 × 288) and high-resolution (1920 × 1080) test sets. We use their complete high-resolution test sets for evaluation. Both benchmarks focus on human video matting.

VideoMatte HR contains 50 videos (5,000 frames), generated by compositing 5 human foreground clips onto 25 video and 25 image backgrounds. It covers upper-body and full-body subjects performing simple actions, with backgrounds spanning natural landscapes, urban environments, and indoor scenes.

YouTubeMatte HR contains 64 videos (6,400 frames), generated from 32 human foreground clips with video and image backgrounds. It includes diverse human appearances, framing, motions, and object interactions, covering children, elderly people, and multiple subjects. Its backgrounds span similar scene categories, with additional harmonization for improved foreground-background consistency. We have added this description to Appendix N.


[W2, Q1] Training Cost

Training our revised matting model requires approximately 258 GPU-hours (around 32 hours on 8 NVIDIA A800-80GB GPUs), with a total batch size of 16. In contrast, MatAnyone 2 trains both the matting model and a separate Matting Quality Evaluator. Since MatAnyone 2 does not report its training cost, a direct comparison in GPU-hours is not possible.


[W3] Figure Presentation

We thank the reviewer for the suggestion. We have adjusted the layout and some details of Figure 2 to improve readability.


[W4] 4K Inference Speed

We have made further implementation-level optimizations, including replacing PyTorch's adaptive average pooling with an equivalent strided average pooling operation and compiling additional modules. These optimizations improve the 4K inference speed from 28.6 to 33.9 FPS on an NVIDIA H200 GPU without compromising matting quality. We have updated Tables 2 and 9 and documented the optimizations in Appendix N.


[Q2] GPU Utilization

Thank you for the question. We evaluated concurrent 4K video inference on an NVIDIA H200 GPU with model compilation.

MethodConcurrent StreamsTotal FPS ↑FPS / StreamGPU Memory (GB) ↓
MatAnyone 212.42.479.6
MatAnyone 22––OOM
MatAnyone-E133.733.76.0
MatAnyone-E237.218.612.0
MatAnyone-E438.79.723.5
MatAnyone-E838.74.839.5
MatAnyone-E2838.71.4138

With four concurrent streams, the total throughput increases from 33.7 to 38.7 FPS (14.8% improvement). Moreover, MatAnyone-E supports up to 28 concurrent 4K streams on a single H200 GPU, whereas MatAnyone 2 can only support one. These results demonstrate the potential of MatAnyone-E for memory-efficient multi-request deployment. We have added the concurrent-stream results to Table 19 in Appendix M.


[Q3] Efficiency under Fast Motion

We thank the reviewer for the question. To evaluate SMP under fast motion, we conducted additional experiments at 4K resolution on an NVIDIA H200 GPU with model compilation, using our test set and 16 motion-augmented sequences.

We grouped frames by foreground motion magnitude, measured using optical flow. The results are summarized below.

Motion LevelMotion (px/frame)Half-res. Query RatioFull-res. Query RatioFPS
Slow1.94.8%3.2%34.1
Moderate9.25.9%4.0%33.5
Fast32.59.2%6.7%32.1

As foreground motion increases, the full-resolution query selection ratio rises from 3.2% to 6.7%, while FPS decreases slightly from 34.1 to 32.1. To further evaluate extremely large displacements, we applied additional horizontal translations to foreground videos. The results are summarized below.

Added Motion (px/frame)Motion (px/frame)Half-res. Query RatioFull-res. Query RatioFPS
+06.66.6%4.5%33.2
+1618.17.6%5.3%32.7
+6461.310.4%7.8%31.6
+128116.813.7%10.9%30.1

At the highest motion level, the foreground moves nearly the full width of a 4K frame in one second at 30 FPS. Even under this extreme motion, SMP maintains 30.1 FPS while selecting only 10.9% of full-resolution queries. These results demonstrate that SMP remains computationally efficient even under extreme foreground motion. We have added the motion-group analysis and extreme-motion experiments to Tables 17 and 18 in Appendix L, respectively.


We hope our responses have addressed the reviewer's concerns. If anything remains unclear or if you have further questions, we would be happy to clarify and discuss them in more detail.

Thanks for providing more evidence and it helps address my concerns, and I will keep my current score.

Thank you for taking the time to review our additional results. We appreciate your consideration.

Reviewer_Nub7OpenReview ↗
Rating
3(+1)
Confidence
3(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper accelerates MatAnyone 2, a memory-based video matting model, so that it can run at native 2K, 4K and 8K resolutions. Its Sparse Memory Propagation first performs memory matching at low resolution, uses the inherited uncertainty predictor to select query locations and the low-resolution affinities to select relevant memory entries, and then repeats the matching at full resolution only between the selected pairs, while a profiling-guided lightweight decoder trained by feature distillation reduces decoding cost. The authors report quality comparable to MatAnyone 2 on three 1080p benchmarks at roughly twice the speed, and native 4K inference with far lower peak memory in a setting where MatAnyone 2 runs out of memory.

Critical strengths
  1. The work removes the memory barrier that keeps strong memory-based matting models away from native high resolution. Dense query-memory matching grows so quickly with resolution that the base model cannot run at 4K even on an 80 GB GPU. Making a model of this quality run at 4K and 8K, including on consumer GPUs, is the main source of practical value I see in the submission.

  2. The design follows a measured bottleneck and lets one cheap pass make two decisions. The profiling in the appendix shows memory propagation dominating the cost as resolution grows, and the method targets it directly: a single low-resolution pass decides both where full-resolution matching is needed and which memory entries it should read, without adding a new learned module. The efficiency study covers resolutions, GPU types and compilation settings, which makes the speed and memory claims credible.

Critical weaknesses
  1. The central claim of native-4K detail preservation lacks quantitative support. All quality numbers are at 1920x1080 (Tables 1, 3, 4), whereas the speed, memory and detail claims center on 4K, where a different 4-to-2-to-1 multi-level SMP stacks two approximations. The 4K evidence is qualitative only (Figures 1, 3, 5, 7), and the comparison partners in Figures 1, 3 and 5, VideoMaMa and SAM2Matting, run at a fixed internal resolution (their FLOPs stay at 16.48 and 7.22 TFLOPs from 1024x1024 to 4K in Table 2). These figures therefore show the benefit of processing native resolution, not the benefit of SMP. The most relevant baseline, MatAnyone 2 at 2K followed by upsampling, is shown only qualitatively in Figure 6(b). The 1080p ablation does not close the gap either: in Table 4, uncertainty-guided HR matching recovers 0.09 of the 0.11 MAD gap between LR only and dense matching, but only 0.21 of the 1.49 Grad gap (13.23, 13.02, 11.74), so it fixes coarse errors while recovering little fine detail. A quantitative 4K evaluation with ground truth, for example on αMatte4K, is needed against MatAnyone 2 at 2K with upsampling, MatAnyone 2 with a reduced memory bank, and LR only.

  2. Comparisons with high-resolution matting methods are missing, and the selection mechanism adds limited novelty. Efficient high-resolution matting has been addressed by RVM, BGMv2 [2], SparseMat and µMatting, and BGMv2 already reports 4K at 30 FPS on an RTX 2080 Ti (with a captured background). Speed alone does not set this work apart, so its contribution has to rest on quality. Yet SparseMat and µMatting appear only in Sec. 2 and are absent from Tables 1 and 2, and RVM is missing from the speed comparison in Table 2, although Sec. 1 claims "a stronger balance between matting quality and inference efficiency at high resolutions". On the method side, query selection reuses the existing uncertainty predictor of MatAnyone 2, and recomputing only uncertain locations at high resolution is an established idea [1, 2]. Its gain over uniform and alpha-boundary selection is at most 0.08 MAD in a single run (Table 4). A speed and quality comparison with these methods at 2K and 4K is needed to place the contribution.

  3. Typos: "spatisal" (Appendix B.3), "NeurIPs" (references).

[1] A. Kirillov, Y. Wu, K. He, R. Girshick, PointRend: Image Segmentation as Rendering, CVPR 2020
[2] S. Lin, A. Ryabtsev, S. Sengupta, B. Curless, S. Seitz, I. Kemelmacher-Shlizerman, Real-Time High-Resolution Background Matting, CVPR 2021

Questions for authors
  1. (Affects my assessment) Can the authors report quantitative 4K results with ground truth, for example on αMatte4K or 4K composites, comparing MatAnyone-E with MatAnyone 2 at 2K plus upsampling, MatAnyone 2 with a reduced memory bank, LR only, and µMatting, using MAD, Grad and dtSSD?

  2. (Clarification) What fraction of query locations is selected for HR matching at 4K, and how do this fraction and the resulting FPS change on sequences with fast motion?

  3. (Clarification) Is SMP active during training, and which modules are fine-tuned? Table 4 uses the MatAnyone 2 checkpoint, so does SMP work without retraining, and how much of the change in Table 1 comes from retraining together with the lightweight decoder?

  4. (Clarification) On which data and at which resolution is the memory recall in Table 7 measured?

  5. (Suggestion, not decisive) A few results on non-human targets such as animals or semi-transparent objects, reporting the quality gap to MatAnyone 2, would show whether the uncertainty-based selection transfers beyond the training domain.

My main concern is question 1. If the 4K results show a clear advantage in detail-sensitive metrics such as Grad over MatAnyone 2 at 2K with upsampling and over high-resolution matting methods, I am willing to raise my rating.

7 comments

Dear Reviewer Nub7,

We sincerely thank you for the constructive feedback and recognition of our method's practical value and efficient design. We address the concerns regarding 4K quantitative evaluation, high-resolution comparisons, methodological differences, and additional clarification questions below.


[W1, Q1] Quantitative 4K Evaluation and Detail Preservation

We thank the reviewer for the comment. To address these concerns, we provide additional quantitative evaluations and ablation studies at 4K resolution, with runtime and memory measurements conducted on an NVIDIA H200 GPU.

4K Quantitative Evaluation. To enable quantitative evaluation at 4K resolution, we construct a 4K test set following the data preparation pipelines of VideoMatte240K and RVM. It contains 7 foreground videos at 3840 × 2160 resolution, each with 109–336 frames. We extract foregrounds and alpha mattes using chroma keying. Each foreground is composited onto four 4K backgrounds (two videos and two images), resulting in 28 sequences (7,532 frames). The quantitative results are summarized below.

MethodMAD ↓MSE ↓Grad ↓dtSSD ↓Latency ↓GPU Mem. ↓
MatAnyone 2 (2K→4K)3.980.359.681.1745.55.59
MatAnyone 2 (reduced mem.)3.970.3410.601.23267.448.47
MatAnyone 23.920.299.351.21417.879.69
MatAnyone-E LR-only5.701.9113.391.5265.16.07
MatAnyone-E3.880.278.311.1869.06.07

Compared with MatAnyone 2 at native 4K, MatAnyone-E reduces inference latency from 417.8 to 69.0 ms/frame and GPU memory from 79.69 to 6.07 GB, while improving MAD from 3.92 to 3.88 and Grad from 9.35 to 8.31. We cannot include µMatting because its official pretrained checkpoints are not publicly available.

Although MatAnyone 2 (2K→4K) achieves competitive quantitative results, upsampling from 2K to 4K may blur fine structures that are not fully captured by aggregate quantitative metrics. We provide additional 4K visual comparisons in Figure 7 (Appendix F) to further illustrate these differences and highlight the importance of preserving fine details at native resolution. We have added the 4K quantitative evaluation and configuration comparisons to Tables 10 and 11 in Appendix G.

4K SMP Ablation. To isolate the contribution of SMP from native-resolution processing, we conduct additional ablation experiments using the original MatAnyone 2 checkpoint without retraining. All configurations use the same 4K inputs and differ only in their memory matching strategies.

PropagationMAD ↓MSE ↓Grad ↓dtSSD ↓
Dense3.920.299.351.21
LR-only 1/2 (s=2)4.140.4711.461.36
LR-only 1/4 (s=4)7.053.3016.891.69
SMP (2-1)3.920.299.731.22
SMP (4-2-1)3.970.339.961.23

Compared with LR-only matching at 1/4 resolution, SMP (4-2-1) reduces Grad from 16.89 to 9.96, recovering 91.9% of the gap relative to dense matching. Despite the additional approximation in 4-2-1 propagation, it maintains quality close to dense matching. These results demonstrate that SMP effectively preserves fine details through selective high-resolution memory matching. We have added these 4K ablation results to Table 12 in Appendix H.


[W2] High-Resolution Quality–Efficiency Comparison and SMP Design

We thank the reviewer for the constructive comments. We provide additional comparisons at 2K and 4K resolutions and clarify the key design of SMP and its advantages.

High-Resolution Quality–Efficiency Comparison. We extend our comparisons with RVM and SparseMat by reporting matting quality, inference latency, and GPU memory at both 2K and 4K resolutions on an NVIDIA H200 GPU. The quality evaluations are conducted on VideoMatte HR and our 4K test set, respectively. We could not include BGMv2 because captured background images are unavailable in our evaluation setting, and µMatting because its official pretrained checkpoints are unavailable.

Our 4K test set

MethodMAD ↓MSE ↓Grad ↓dtSSD ↓LatencyGPU Mem.
RVM6.401.5723.212.054.50.86
SparseMat8.663.7623.112.5939.41.16
MatAnyone 23.920.299.351.21417.879.70
MatAnyone-E3.880.278.311.1869.06.07

VideoMatte HR

MethodMAD ↓MSE ↓Grad ↓dtSSD ↓LatencyGPU Mem.
RVM6.571.9310.551.903.40.33
SparseMat8.934.309.491.9123.70.51
MatAnyone 24.100.283.451.1543.05.50
MatAnyone-E4.140.253.221.1721.81.65

Although RVM and SparseMat achieve higher inference speeds, MatAnyone-E delivers substantially better matting quality at both resolutions. At 4K, MatAnyone-E achieves a MAD of 3.88, compared with 6.40 for RVM and 8.66 for SparseMat, while reducing Grad from 23.21 and 23.11 to 8.31, respectively. Compared with MatAnyone 2, our method maintains comparable quality at 2K and achieves slightly better quality at 4K, while reducing 4K latency by 6.1× and GPU memory by 13.1×. These results demonstrate that MatAnyone-E achieves a strong balance between matting quality and computational efficiency, particularly at high resolutions. We have added these quality–efficiency comparisons to Tables 13 and 14 in Appendix I.

SMP Design and Selection Effectiveness. Unlike existing spatial selective refinement methods, SMP addresses the computational bottleneck of temporal memory matching. It leverages low-resolution matching results to guide both query pruning and memory pruning at higher resolutions, substantially reducing the cost of high-resolution memory propagation without requiring additional learned modules.

To further evaluate the effectiveness of uncertainty-guided selection, we conduct additional experiments on our 4K test set using the MatAnyone 2 checkpoint, with memory pruning disabled. We compare uniform, alpha-boundary, and uncertainty-guided selection under comparable high-resolution query budgets. The results are summarized below.

StrategyPyramidMAD ↓MSE ↓Grad ↓dtSSD ↓Full-res. Query RatioHalf-res. Query Ratio
Baseline (MA2 Dense)–3.920.299.351.21100%–
LR-only1/24.140.4711.461.360%–
LR-only1/47.053.3016.891.690%0%
Uniform2-14.100.4311.321.353.45%–
Uniform4-2-16.152.4214.911.593.62%4.87%
Alpha-boundary2-13.980.3310.531.293.45%–
Alpha-boundary4-2-14.120.4711.331.363.49%4.84%
Uncertainty-guided2-13.920.299.751.223.45%–
Uncertainty-guided4-2-13.970.339.991.243.43%4.85%

Under the more aggressive 4-2-1 configuration, uncertainty-guided selection reduces MAD by 0.15 and Grad by 1.34 compared with alpha-boundary selection, and by 2.18 and 4.92 compared with uniform selection, respectively. All three strategies use comparable query budgets, selecting approximately 3.4–3.6% of full-resolution queries and 4.8–4.9% of half-resolution queries. Compared with LR-only matching, uncertainty-guided query selection substantially improves matting quality and approaches dense matching performance. These results demonstrate the effectiveness of uncertainty-guided selection in preserving matting quality under sparse temporal memory propagation. We have added the 4K query-selection comparison to Table 15 in Appendix J.


[Q2] High-Resolution Query Selection and Motion Sensitivity

We thank the reviewer for the question. We evaluated the HR query selection ratio and inference speed on our 4K test set using a compiled version of MatAnyone-E on an NVIDIA H200 GPU. We grouped frames by foreground motion magnitude, measured using optical flow. The results are summarized below.

Motion LevelMotion (px/frame)Half-res. Query RatioFull-res. Query RatioFPS
Slow1.94.8%3.2%34.1
Moderate9.25.9%4.0%33.5
Fast32.59.2%6.7%32.1

As foreground motion increases, the full-resolution query selection ratio rises from 3.2% to 6.7%, while FPS decreases slightly from 34.1 to 32.1. These results demonstrate that SMP remains computationally efficient even under fast foreground motion. We have added the motion-sensitivity experiments to Tables 17 and 18 in Appendix L.


[Q3] Training Configuration and the Effect of Retraining

During training, we use the original dense memory matching of MatAnyone 2, while SMP is applied only during inference and does not require retraining. We initialize the model from pretrained MatAnyone 2 weights, except for the lightweight decoder, which is randomly initialized and trained from scratch.

Training consists of two stages. First, we train only the lightweight decoder with feature distillation, keeping all other modules frozen. In the second stage, we keep the memory propagation modules frozen and fine-tune the remaining modules.

As shown in Table 3, applying SMP without retraining slightly increases MAD from 4.24 to 4.26. Introducing the lightweight decoder with two-stage training further increases MAD to 4.28, while improving MSE (1.94 to 1.85), Grad (13.00 to 12.87), and dtSSD (4.69 to 4.61) compared with SMP alone. Overall, these results show that the lightweight decoder and its two-stage training largely preserve matting quality. We have clarified the training configuration and the use of SMP during inference in Appendix N.


[Q4] Evaluation Protocol

The memory recall results in Table 7 are measured on the CRGNN benchmark at 1920 × 1080 resolution. We have clarified the dataset and evaluation resolution in the caption of Table 7 in Appendix C.


[Q5] Generalization Beyond Human Matting

We thank the reviewer for the suggestion. Since our method is primarily trained on human video matting data, its generalization to non-human targets remains an open question. We will further investigate animal and semi-transparent object matting in future work.


[W3] Typos

Thank you for pointing out these typos. We have corrected both typos in Appendix B.3 and the References.


We hope our responses have addressed the reviewer's concerns. If anything remains unclear or if you have further questions, we would be happy to clarify and discuss them in more detail.

Reviewer_8bKLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work addresses the slow inference of existing human video matting models. The authors identify memory propagation as the primary runtime bottleneck and introduce a sparse propagation strategy that substantially improves efficiency, enabling video matting at 20–30 FPS at 1080p on an NVIDIA A800 GPU.

Critical strengths
  1. The authors identify memory matching as a key runtime bottleneck in the MatAnyone architecture, providing a well-motivated target for optimization. The proposed acceleration strategy is intuitive and technically reasonable.

  2. MatAnyone-E achieves a 1.96× speedup at 1080p while largely preserving matting quality.

Critical weaknesses
  1. The approximately 7× speedup highlighted in the introduction appears overstated. First, the baseline runtime is extrapolated rather than directly measured, which weakens the evidence supporting this claim. Second, this speedup is achieved only at 4K, where the paper provides qualitative comparisons but no quantitative evaluation of matting quality. Consequently, the accuracy–efficiency trade-off associated with the claimed speedup remains unclear. At the more commonly used 1080p resolution, the actual speedup is 1.96×, rather than approximately 7×. The headline claim should therefore be qualified to reflect the resolution, the estimated baseline runtime, and the absence of quantitative quality evaluation at 4K.

  2. MatAnyone 2 reports results in an auxiliary-free setting. Evaluating MatAnyone-E under the same setting would provide a more complete comparison.

  3. A direct, side-by-side runtime breakdown of the baseline and the proposed model would be helpful, covering the encoder, memory propagation, and decoder. The authors should also explicitly report the module-level speedups achieved by the memory propagation and decoder optimizations separately.

Questions for authors

See the weaknesses above.

6 comments

Dear Reviewer 8bKL,

We sincerely thank you for the positive assessment. We appreciate the recognition of our efficiency improvements and well-motivated design. We address the concerns regarding the 4K speedup and quantitative evaluation, evaluation setting, and runtime breakdown below.


[W1] Speedup and Quantitative Evaluation at 4K

We thank the reviewer for the suggestion. To address these concerns, we provide additional runtime measurements and quantitative evaluations at 4K resolution.

4K Runtime Measurement. Since MatAnyone 2 runs out of memory at 4K resolution on an NVIDIA A800 GPU, we perform additional experiments on an NVIDIA H200 GPU, where both models can run at 4K resolution.

MetricMatAnyone 2MatAnyone-ESpeedup
Latency (ms/frame) ↓417.868.96.1×

MatAnyone-E achieves a directly measured 6.1× speedup at 4K on an NVIDIA H200 GPU, while achieving a 1.96× speedup at 1080p on an NVIDIA A800 GPU. We have updated the speedup claims throughout the manuscript to reflect the corresponding resolutions.

4K Quantitative Evaluation. To enable quantitative evaluation at 4K resolution, we construct a 4K test set following the data preparation pipelines of VideoMatte240K and RVM. The test set contains 7 foreground videos at 3840 × 2160 resolution, each with 109–336 frames. We extract foregrounds and alpha mattes using chroma keying. Each foreground is composited onto four 4K backgrounds (two videos and two images), resulting in 28 sequences (7,532 frames). The quantitative results are summarized below.

MethodMAD ↓MSE ↓Grad ↓dtSSD ↓
RVM6.401.5723.212.05
FTP-VM4.840.8316.601.98
SparseMat8.663.7623.112.59
MaGGIe4.380.4813.291.56
GVM4.410.4417.441.63
MatAnyone4.570.8515.631.45
VideoMaMa4.260.3214.531.36
SAM2Matting4.720.6121.171.86
MatAnyone 23.920.299.351.21
MatAnyone-E3.880.278.311.18

MatAnyone-E achieves the best results across all four quality metrics, demonstrating that the measured 6.1× speedup at 4K does not compromise matting quality.


[W2] Auxiliary-Free Evaluation Setting

Both MatAnyone 2 and MatAnyone-E require a first-frame segmentation mask rather than operating in an auxiliary-free setting. The first-frame segmentation mask is used to specify the target object for video matting. Therefore, MatAnyone-E follows the same setting as MatAnyone 2.


[W3] Module-Level Runtime and Speedup Breakdown

Thank you for the suggestion. We provide a module-level runtime comparison between MatAnyone 2 and MatAnyone-E at 4K resolution on an NVIDIA H200 GPU.

ModuleMatAnyone 2MatAnyone-ESpeedup
Encoder20.319.61.0×
Memory Propagation340.58.639.6×
Object Transformer6.25.61.1×
Decoder40.424.91.6×
Value Encoder10.410.11.0×
Total latency417.868.96.1×

Latency is measured in ms/frame.

SMP reduces memory propagation latency from 340.5 to 8.6 ms, achieving a 39.6× speedup, while the lightweight decoder reduces decoding latency from 40.4 to 24.9 ms, achieving a 1.6× speedup. The runtimes of the other modules remain nearly unchanged.

  1. 1080p versus native 4K inference. Thank you for the additional experiments. I would appreciate a direct comparison of MatAnyone-E on your mini test set when run at 1080p and at native 4K, including matting quality, runtime, and peak GPU memory. Native 4K inference incurs substantially greater computational cost. If it yields only marginal quality improvements, the added cost may not be justified. This comparison would also clarify whether the method captures details at 4K resolution, rather than merely being able to process 4K inputs.

  2. Auxiliary-free evaluation. I apologize for the typo in my previous comment: it is MatAnyone [1], not MatAnyone 2 [2], that reports auxiliary-free results. As described in MatAnyone, an auxiliary-free model such as RVM [4] can generate the first-frame alpha matte, which is then binarized to provide the initial mask. The same protocol could be applied to MatAnyone 2 and MatAnyone-E. This experiment would help determine whether MatAnyone-E preserves its matting quality after acceleration when the initial mask is automatically generated and potentially imperfect, as is often the case in practice. I would also appreciate including GVM [3] and RVM-Large [4] as auxiliary-free baselines.

References

[1] Yang et al. MatAnyone: Stable Video Matting with Consistent Memory Propagation. CVPR, 2025.

[2] Yang et al. MatAnyone 2: Scaling Video Matting via a Learned Quality Evaluator.

[3] Ge et al. Generative Video Matting. SIGGRAPH, 2025.

[4] Lin et al. Robust High-Resolution Video Matting with Temporal Guidance. WACV, 2022.

We thank the reviewer for the constructive follow-up and additional suggestions. We provide additional experiments to address both questions below.


[Q1] 1080p Upsampling vs. Native 4K Inference

We thank the reviewer for this suggestion. We evaluate MatAnyone-E at both 1080p and native 4K resolutions on our 4K test set, using a single NVIDIA H200 GPU. For the 1080p setting, predictions are upsampled to 4K before evaluation. The results are summarized below.

MatAnyone-EMAD ↓MSE ↓Grad ↓dtSSD ↓Runtime (ms/frame) ↓Peak Memory (GiB) ↓
1080p → 4K3.990.359.371.1624.51.74
Native 4K3.880.278.311.1869.06.07

These results reveal a trade-off between matting quality and computational efficiency. Compared with 1080p upsampling, native 4K inference reduces MSE from 0.35 to 0.27 (22.9% reduction) and Grad from 9.37 to 8.31 (an absolute reduction of 1.06). These improvements suggest that the model benefits from native 4K inputs rather than merely being able to process them, making native 4K inference more suitable for applications prioritizing matting quality. On the other hand, 1080p inference is 2.8× faster and uses 4.33 GiB less GPU memory, offering a more efficient alternative for latency-sensitive applications.


[Q2] Auxiliary-Free Evaluation

We thank the reviewer for the clarification and suggestion. Following the auxiliary-free protocol of MatAnyone, we use RVM to predict the first-frame alpha matte, which is then binarized to obtain the same initial segmentation mask for MatAnyone 2 and MatAnyone-E. We also include RVM-Large and GVM as auxiliary-free baselines. All methods are evaluated on our 4K test set.

MethodMAD ↓MSE ↓Grad ↓dtSSD ↓
RVM6.401.5723.212.05
RVM-Large5.791.2622.802.13
GVM4.410.4417.441.63
MatAnyone 24.250.4812.181.37
MatAnyone-E4.110.4711.561.35

Under automatic initialization, both MatAnyone 2 and MatAnyone-E outperform RVM, RVM-Large, and GVM on MAD, Grad, and dtSSD. The improvements are particularly notable in Grad, where MatAnyone 2 and MatAnyone-E achieve 12.18 and 11.56, respectively, compared with 17.44 for GVM and 22.80 for RVM-Large. These results show that MatAnyone-E maintains the strong matting quality of MatAnyone 2 even with automatically generated initial masks.


We hope our responses have addressed the reviewer's concerns. If anything remains unclear or if you have further questions, we would be happy to clarify and discuss them in more detail.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Contributions and Rebuttal Responses

Dear Reviewers, AC, SAC, and PC,

We sincerely thank you for the time and effort dedicated to reviewing our paper. As the discussion period concludes, we would like to highlight MatAnyone-E's contributions toward efficient, high-quality video matting at native high resolutions and the additional experiments conducted during the rebuttal.

By introducing Sparse Memory Propagation (SMP), we substantially reduce the computational and memory costs of dense temporal memory matching, enabling memory-based video matting to scale efficiently to high resolutions. Together with a lightweight decoder, MatAnyone-E achieves 33.9 FPS at native 4K on a single NVIDIA H200 GPU, with a 6.1× inference speedup and peak GPU memory reduced to 7.6% of MatAnyone 2, while maintaining comparable matting quality.

1. Reviewer Consensus We are encouraged that all three reviewers have confirmed their support for acceptance and recognized the value of our work in efficient high-resolution video matting.

Reviewer 8bKL: Recognized our motivation for optimizing memory matching and described the approach as "intuitive and technically reasonable."

Reviewer Nub7: Recognized the practical value of enabling native high-resolution matting and raised their rating after the rebuttal, stating that our responses "significantly address my concerns."

Reviewer WKWP: Recognized our "clear and systematic design" and the advantages demonstrated in our experiments. Following the rebuttal, the reviewer acknowledged that the additional evidence "helps address my concerns."

2. Addressing Key Concerns with New Evidence During the rebuttal, we conducted experiments to address questions regarding 4K matting quality, computational efficiency, detail preservation, and practical deployment:

Native 4K Quality & Acceleration. We introduced a quantitative 4K benchmark and directly measured both MatAnyone 2 and MatAnyone-E on an NVIDIA H200 GPU. MatAnyone-E achieves a 6.1× speedup (417.8 → 68.9 ms/frame) and reduces peak GPU memory from 79.69 to 6.07 GB, while improving MAD (3.92 → 3.88) and Grad (9.35 → 8.31). These results show that MatAnyone-E achieves higher inference efficiency and improved matting quality compared with MatAnyone 2 at native 4K resolution.

High-Resolution Detail Preservation. We added comparisons with lower-resolution upsampling, reduced-memory baselines, and LR-only propagation. Without retraining, SMP (4-2-1) reduces Grad from 16.89 to 9.96 compared with LR-only matching, recovering 91.9% of the gap to dense matching.

Real-Time Performance and Robustness. With additional implementation optimizations, MatAnyone-E achieves 33.9 FPS at native 4K on a single NVIDIA H200 GPU. We evaluated performance under fast and extreme foreground motion, where the method maintains over 30 FPS. We also conducted concurrent inference experiments, showing that MatAnyone-E can process up to 28 concurrent 4K streams on a single H200 GPU.

We believe our work will be useful to the community in developing more efficient high-resolution video matting methods. We will release our code and models to foster further developments in this direction. We respectfully hope that our contributions merit your favorable consideration, and we sincerely appreciate your time and support.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.