- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper aims to improve streaming video restoration by reducing the effect of degraded inputs on historical memory. The authors focus on how errors in the cache affect the restoration of later video chunks. The proposed method uses the original low-quality input for rendering and a pseudo-clean input for memory writing. The pseudo-clean input comes from downsampling a restoration draft generated by the same frozen model. The method also selects keys from either pass and combines their values with a mixing weight to update the cache. Experiments with FlashVSR show gains in several quality metrics under different degradation settings. These gains come with additional computation, and some other metrics decrease.
Critical strengths
The paper focuses on the visual restoration problem in streaming video. I find the distinction between restoring the current chunk and writing memory for later chunks interesting. Here are two good points.
(1) The method gives a straightforward way to use a restored input for memory writing while keeping the original observation for rendering. It uses a frozen model and requires no additional training. Figure 6 also provides useful evidence that changes to historical memory affect the restored output.
(2) The evaluation covers several degradation settings. Table 1 reports higher MUSIQ, CLIP-IQA, FasterVQA, and DOVER scores than FlashVSR across all reported subsets, although some differences are small. I also appreciate the additional tests in Appendix E. These use a fixed configuration on four standard VSR benchmarks and provide evidence under milder degradations.
Critical weaknesses
Here are my major concerns.
(1) The contribution of the rendering input is not clear to me. Figure 6 examines the effect of historical memory, and Figure 3 studies refinement rounds and KV settings. These experiments provide useful evidence, but they do not directly isolate the benefit of keeping the original input for rendering. A matched variant could use the same fixed pseudo-clean draft for both rendering and memory writing. Comparing the two designs would help distinguish the effect of using different inputs from the benefit of refinement with the draft. This matters because the full pipeline in Appendix A takes about 2.2 to 2.3 times as long as FlashVSR. Some quality metrics in the main evaluation also decrease.
(2) I find the parameter selection procedure difficult to assess from the current description. Appendix B.1 gives the search grid and explains that selection considers fidelity, perceptual quality, and temporal consistency together. The description does not identify the tuning videos or explain the choice when these metrics favor different settings. This leaves some uncertainty about performance beyond the videos used for tuning. The fixed-configuration tests in Appendix E are a useful addition. Their focus is on milder degradations, however, so the same question remains for the main benchmarks with more severe degradations.
Questions for authors
(1) I wonder whether using the same fixed pseudo-clean draft for both rendering and memory writing gives similar results. Using memory keys and lambda = 1 in both variants, with the same noise and initial cache, would make this a useful comparison. This would help me understand the specific benefit of retaining the original rendering input.
(2) I am interested in how the tuning videos are chosen and whether they are separate from the final evaluation videos. This distinction would help me assess how well the selected settings generalize.
(3) I find the runtime analysis in Appendix A useful, and I wonder how the cost varies across the configurations used for the main quality results. Appendix A measures the setting with memory keys and lambda = 1. Runtime and peak memory under the same measurement protocol would give a clearer picture of the balance between quality and cost.
3 comments
Official Comment by Authors (Part 2)
2. W2 / Q2 (Parameter Selection and Tuning Videos)
The two hyperparameters follow from the role each one plays in the method, and we explain this first. We then describe the tuning videos, and finally report applied to all benchmarks.
How is chosen. follows the severity of the corruption: the heavier the corruption, the larger . Writing for the severity, i.e. how much of the content entering the cache is expected to be corrupted, we use , with non-decreasing, and , and the simplest reading is : replace as much of the memory as is corrupted. When the corruption is mild or local, most of the cached content from the original input is reliable, and a small keeps it and preserves fidelity. When the corruption is heavy, the cached content is mostly unreliable, and a large is needed to keep it out of memory. is a coarse property of a benchmark and is not estimated per video, so in practice is taken at the level that matches the benchmark.
- BSD, BSCV and AbsDeg-syn: the corruption affects part of the content, and = 0.15.
- TAPE: dropout, mistracking and scan-line flicker damage whole frames in large numbers, and = 0.5.
- AbsDeg-real and SRWOV/real: real old films with heavy, persistent degradation, and = 1.
thus gives the method flexibility across degradation levels with one mechanism.
How the key source is chosen. The key decides which memory entries a query attends to. Memory keys are the default, since keys from the draft are not misled by the corruption. Real keys are preferred when the draft itself is less reliable in structure, as on TAPE, where the corruption is sampled independently for each frame; keys from the original input then keep the attention correspondence consistent with what the rendering pass sees.
Tuning videos. TAPE, BSD and BSCV provide official splits, and their configurations were selected on videos separate from the test videos: the training split of TAPE, the validation split of BSD and the training split of BSCV. AbsDeg-syn, AbsDeg-real and SRWOV/real have no official split, and no separate tuning set was held out for them.
One configuration for all benchmarks. We agree with the reviewer that the hyperparameters should not be tuned too specifically for each benchmark. We therefore propose a single benchmark-independent configuration, with = 0.15, and evaluated it on all benchmarks without any adjustment. For BSD, BSCV and AbsDeg-syn this is the configuration already used in the main paper, so their results are reported in Table 1 and are not repeated here. The remaining three benchmarks are reported below.
| Benchmark | Method | PSNR↑ | SSIM↑ | LPIPS↓ | NIQE↓ | MUSIQ↑ | CLIP-IQA↑ | DOVER↑ | FasterVQA↑ | Ewarp↓ | tOF↓ |
|---|---|---|---|---|---|---|---|---|---|---|---|
| TAPE | FlashVSR | 20.88 | 0.7175 | 0.2429 | 4.805 | 69.77 | 0.5731 | 8.94 | 0.8020 | 2.801 | 1.529 |
| TAPE | DeCache | 20.82 | 0.7137 | 0.2472 | 4.732 | 69.81 | 0.5794 | 8.98 | 0.8028 | 2.646 | 1.725 |
| SRWOV/real | FlashVSR | – | – | – | 5.000 | 63.15 | 0.6503 | 1.38 | 0.5544 | – | – |
| SRWOV/real | DeCache | – | – | – | 4.929 | 63.69 | 0.6589 | 1.47 | 0.5568 | – | – |
| AbsDeg-real | FlashVSR | – | – | – | 3.275 | 59.83 | 0.5391 | -2.09 | 0.6382 | – | – |
| AbsDeg-real | DeCache | – | – | – | 3.205 | 60.85 | 0.5669 | -1.83 | 0.6457 | – | – |
The single configuration is a conservative, fidelity-preserving operating point. On TAPE it stays close to FlashVSR on every metric. On the two real old-film benchmarks it improves the video quality scores. These gains are smaller than those of the benchmark-specific configurations in Table 1, which is consistent with the analysis above: the heavier corruption of these three benchmarks calls for a larger than 0.15. The benchmark-specific settings remain available as an option when the degradation level is known. A larger , up to 1, is an option when perceptual quality is the priority and some risk of hallucinated detail is acceptable, as in old-film restoration.
Changes in the revision. We have added the selection principle and the tuning videos to Appendix B.1, and the single-configuration results as Table 6.
Official Comment by Authors (Part 3)
3. Q3 (Cost Across the Main Configurations)
We re-measured all four configurations used in Table 5 with the Appendix A.3 protocol in one environment. The table also includes a FlashVSR baseline with the same incremental decoding as DeCache, which is the fairer comparison.
| Configuration | TAPE 512×512: FPS | Wall (s) | Peak alloc. (GiB) | BSCV 1280×640: FPS | Wall (s) | Peak alloc. (GiB) |
|---|---|---|---|---|---|---|
| FlashVSR, incremental decoding | 20.80 | 8.32 | 7.43 | 6.69 | 25.85 | 15.85 |
| DeCache, , =0.5 | 9.06 | 19.09 | 10.05 | 2.90 | 59.67 | 24.03 |
| DeCache, , =0.15 | 9.10 | 19.02 | 9.87 | 2.91 | 59.45 | 23.48 |
| DeCache, , =1 | 9.14 | 18.93 | 9.87 | 2.94 | 58.83 | 23.48 |
| DeCache, , =1 | 9.10 | 19.02 | 9.79 | 2.92 | 59.25 | 23.21 |
Cost is nearly independent of the configuration. Throughput varies by less than 1.5%. All four configurations run three backbone forwards and two decoder invocations per step, and differ only in whether render-pass candidates are kept until the memory pass writes them. Choosing the key source and per dataset is therefore a quality decision that does not change the cost. The cost relative to the baseline is about 2.3× in runtime in every configuration. We present DeCache as a trade-off of additional compute and memory for the quality changes in Table 1.
Changes in the revision. We have added these measurements to Table 3 in Appendix A.3.
We thank the reviewer once again for the valuable comments. We hope our explanations have clarified all the questions. We are happy to address any further concerns or discussions the reviewer may have. We kindly hope the reviewer could consider raising the score if satisfied with our responses. Thank you!
Official Comment by Authors (Part 1)
We thank the reviewer for the constructive feedback. We are pleased that the reviewer finds the distinction between restoring the current chunk and writing memory for later chunks "interesting," considers our method "a straightforward way". Below, we provide detailed responses to the concerns raised.
1. W1 / Q1 (Contribution of the Rendering Input)
We ran the comparison as suggested. Both variants use Kmem and λ=1, the same noise seed, an empty initial cache, the same draft x~ and the default cache length. With these settings the two variants write identical cache contents at every step, so the rendering input is the only difference:
We evaluated the four benchmarks with ground truth. BSCV uses a random sample of 100 of its 526 clips because of the rebuttal time limit.
The two variants do not give similar results. Retaining the original rendering input is better on PSNR, SSIM and LPIPS on all four benchmarks. It also improves temporal consistency: on BSD, AbsDeg-syn and BSCV, it lowers Ewarp by 30%, 11% and 20%, and tOF by 20%, 13% and 17%. On the no-reference metrics NIQE, CLIP-IQA, DOVER and FasterVQA the two variants are close, with neither consistently ahead. Draft rendering therefore produces outputs that a per-frame quality predictor rates slightly higher but that are further from the ground truth, which is the usual signature of added generated detail.
Retaining the original input helps because it keeps first-pass errors from being compounded. The draft x~ is a re-degraded copy of a first restoration. Whatever the first pass removed or hallucinated is already fixed in x~, and no information about the true scene can be added to it afterwards. When x~ is also the rendering input, the second pass treats these errors as observations and restores on top of them, so first-pass errors are carried into the output and compounded. This matches the consistent drop in PSNR, SSIM and LPIPS. DeCache gives the draft a narrower role. The draft only decides what is written into memory, where it keeps corrupted content out of the cache, while every output frame remains conditioned on the actual observation. The output therefore stays anchored to the measured content, and the draft's errors can reach it only indirectly, as temporal context, and not as the signal being restored.
Changes in the revision. We have added this comparison and its explanation as Appendix B.4 (Table 8), referenced from Sec. 3.2.