- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies source-assisted space-time video super-resolution: a sender can spend a shared byte budget on compressed pixel anchors or source-derived structural metadata, while a fixed generative receiver reconstructs the missing frames. MetaST measures the value of candidate actions offline and trains a Metadata Value Predictor to select anchor quality and metadata without running the receiver at the sender. On the HC-Meta benchmark, transmitting three interior edge maps at the lossless-anchor reference budget improves average luma PSNR from 29.63 dB for the strongest pixel-only baseline to 32.65 dB. The paper also reports codec curves, general-video results, and actual mixed-QP stream validation.
Critical strengths
- The study makes the cost of structural side information explicit: anchors, metadata, and signaling share one budget, and the codec settings for each comparison are specified. The 3 dB HC-Meta gain at B0 establishes that source-derived structure can be valuable when intermediate motion is ambiguous.
- The allocation claim is tested separately from the structural metadata gain. On the unpruned 25-action grid, grouped out-of-fold prediction improves mean regret over rates-only ridge.
Critical weaknesses
- The 3 dB headline result demonstrates the benefit of transmitting source-derived edges to a metadata-conditioned receiver, not the incremental benefit of MVP. It should not be read as a gain from learned allocation over an otherwise identical pixel-only system.
- The predictor evaluation is confined to 50 HC-Meta clips and a fixed pair of receiver checkpoints. The 0.072 dB regret gain is credible but modest: it does not show how MVP transfers to new receivers, degradation regimes, or metadata extractors.
Questions for authors
- Can the authors report the realized MVP allocation curve, alongside the best uniform action and direct H.266, over the same budgets and with uncertainty by clip? This would separate the learned decision gain from the value of adding source metadata.
- How well does MVP transfer without refitting to a new receiver or metadata codec, and what are the sender-side extraction and encoding costs per second of input video?
6 comments
Response to Reviewer hoYK (2/4)
A1 (continued).
Decision gain versus value of metadata. To separate the two gains, we ran a new experiment: MVP allocation on the anchors-only part of the menu, an otherwise identical pixel-only system. Receivers, rate model and budgets stay the same.
| Action menu | Best uniform action | MVP allocation | Gain of MVP allocation |
|---|---|---|---|
| Anchors only (pixel-only system, 5 actions) | 29.06 | 29.40 | +0.34 [+0.27, +0.39] |
| Anchors and metadata (25 actions) | 29.86 | 30.70 | +0.84 [+0.62, +1.08] |
| Gain from metadata | +0.80 [+0.55, +1.09] | +1.30 [+0.87, +1.85] |
Average PSNR in dB, mean over the 658 budgets of Table 3.
Metadata alone adds 0.80 dB to the best uniform action. MVP allocation adds 0.84 dB on top of it. Together they place MetaST + MVP 1.64 dB [1.24, 2.15] above the best uniform action of the pixel-only system. Allocation with rates-only ridge already attains 0.77 dB of the 0.84 dB. MVP's own increment is the subject of Q2.
Direct H.266. To answer your request for the direct-codec baseline, we ran a new experiment: a comparison with direct H.266. Direct H.266 codes every frame at full resolution and sends no metadata. MetaST uses H.266 anchors and an MVP retrained on H.266 value labels.
| Rate (KB per interval) | Direct H.266 | MetaST + MVP (H.266 anchors) | MetaST + MVP − direct H.266 |
|---|---|---|---|
| 0.75 | 26.64 | 27.12 | +0.48 [−0.04, +0.80] |
| 1.24 | 28.05 | 28.17 | +0.12 [−0.30, +0.41] |
| 1.61 | 28.94 | 28.48 | −0.45 [−0.79, −0.13] |
Average PSNR in dB at measured direct H.266 rates.
Direct H.266 overtakes MetaST + MVP from 1.36 KB per interval [0.78, 1.57]. The paper reports this crossover near 1.4 KB for the oracle (Sec. 4.5, Fig. 4b). H.264 is still the most widely deployed video codec, and it is the paper's main setting. With stronger anchor codecs such as H.266, metadata needs more efficient metadata codecs to pay off. We leave such codecs to future work.
In the updated PDF, we call the 3.0 dB a system gain and add the realized MVP curves to Figs. 4 and 6, with the oracle as the upper bound. The anchors-only result and the best uniform action are in the appendix.
Q2 (W2): The predictor evaluation is confined to 50 HC-Meta clips, and the 0.072 dB regret gain is modest.
A2: To address your concern that the predictor evaluation is confined to HC-Meta, we ran a new experiment: the same evaluation on two other video datasets and with H.266 anchors. Each value model is retrained for its setting and tested on held-out clips or recordings. Regret is the PSNR gap between a model's allocation and the oracle allocation at the same budget.
| Setting | Item prior | Rates-only ridge | MVP | MVP − rates-only ridge | Oracle gain recovered by MVP |
|---|---|---|---|---|---|
| HC-Meta, H.264 anchors (Table 3) | 0.252 | 0.220 | 0.148 | −0.072 [−0.122, −0.031] | 79.4 % |
| HC-Meta, H.266 anchors | 0.195 | 0.201 | 0.142 | −0.059 [−0.088, −0.039] | 66.8 % |
| Adobe240, 15 recordings, 4,410 intervals | 0.101 | 0.139 | 0.058 | −0.082 | 86.7 % |
| GoPro, 10 recordings, 1,522 intervals | 0.102 | 0.078 | 0.103 | +0.025 | 84.7 % |
Mean regret in dB, lower is better. The last column is MVP's gain over the interpolated envelope of uniform actions (Fig. 5), as a share of the oracle's gain over the same envelope.
MVP has the lowest regret in three of the four settings. Under H.266 it also beats the item prior, the stronger simple model there, by 0.053 dB [0.030, 0.087]. On GoPro, every value model already recovers at least 84 % of the oracle's gain, so little headroom remains.
Response to Reviewer hoYK (3/4)
A2 (continued). Where the increment matters. To show where the 0.072 dB comes from, we split MVP's and rates-only ridge's regret on HC-Meta by rate range.
| Regret on HC-Meta (dB) | Rates-only ridge | MVP | Reduction by MVP |
|---|---|---|---|
| Mean over all 658 budgets (Table 3) | 0.220 | 0.148 | 0.072 [0.031, 0.122] |
| Mean below 2 KB per interval | 0.217 | 0.112 | 0.106 |
| Mean at 2–10 KB | 0.313 | 0.186 | 0.127 |
| Mean at 10–30 KB | 0.207 | 0.180 | 0.027 |
| Mean above 30 KB | 0.043 | 0.064 | −0.021 |
| 90th percentile over the budgets (Table 3) | 0.412 | 0.216 | 0.196 |
| Worst budget | 0.587 | 0.293 | 0.294 |
The gain sits below 10 KB per interval, where rate is scarce. At 2.5 KB, MVP's allocation is 0.27 dB [0.15, 0.43] higher in PSNR than ridge's. Above 30 KB both models are within 0.07 dB of the oracle. We also compare the two rate–PSNR curves with the Bjøntegaard delta rate (BD-rate, Bjøntegaard, 2001), the standard measure in video coding. It is the average rate difference at equal PSNR over the PSNR range both curves cover. We interpolate log-rate piecewise linearly. MVP's BD-rate against rates-only ridge is −5.2 %, so it needs 5.2 % less rate for the same PSNR.
The updated PDF adds the H.266, Adobe240 and GoPro results to Appendix G.
Q3 (W2, Q2 first part): How well does MVP transfer without refitting to a new receiver, metadata codec, degradation regime or metadata extractor?
A3: To answer your question on transfer without refitting, we ran a new experiment: one component changed at a time on HC-Meta, with newly measured values of the changed actions.
- Edge codec: a generic lossless coder (bit-packing + Zstandard) replaces the contour codec. On the same edge maps it changes only the rates. On raw Canny dense maps it also changes the maps and their values.
- Second receiver checkpoint: the edge receiver at training step 22,902 is replaced by its ancestor checkpoint at step 13,701.
- Contour codec tolerance: sparse edge maps are simplified with 3.0 px or 6.0 px instead of 1.5 px.
- Learned edge extractor: PiDiNet (Su et al., 2021) replaces Canny for both edge-map families.
- H.266 anchors, on the twelve-action menu.
- Input degradation: Gaussian blur and ×4 decimation replace bicubic ×4.
"No refit" reuses the models trained on the original setting. "Refit" retrains MVP on the new values. "10 labelled intervals" shifts each changed action's predictions by their mean error on 10 labelled intervals, without retraining.
| Change | MVP, refit | MVP, no refit | Rates-only ridge, no refit | Item prior, no refit | MVP, 10 labelled intervals |
|---|---|---|---|---|---|
| Edge codec, same maps | 0.159 | 0.168 (+0.010) | 0.259 (+0.100) | 0.272 (+0.114) | — |
| Edge codec, raw Canny dense maps | 0.162 | 0.163 (+0.001) | 0.257 (+0.095) | 0.273 (+0.111) | — |
| Second receiver checkpoint | 0.170 | 0.181 (+0.011) | 0.271 (+0.101) | 0.274 (+0.104) | 0.169 (−0.001) |
| Contour codec, 3.0 px | 0.145 | 0.159 (+0.014) | 0.271 (+0.126) | 0.275 (+0.130) | 0.148 (+0.003) |
| Contour codec, 6.0 px | 0.139 | 0.226 (+0.088) | 0.427 (+0.288) | 0.385 (+0.246) | 0.142 (+0.004) |
| Learned edge extractor | 0.165 | 0.233 (+0.068) | 0.575 (+0.410) | 0.403 (+0.238) | 0.200 (+0.035) |
| H.266 anchors | 0.142 | 0.421 (+0.279) | 0.534 (+0.392) | 0.278 (+0.137) | 0.196 (+0.054) |
| Blur + ×4 decimation | 0.098 | 1.064 (+0.966) | 0.884 (+0.786) | 0.981 (+0.883) | 0.141 (+0.043) |
Mean regret in dB, lower is better. In parentheses, the difference to the refitted MVP.
Without refitting, MVP stays within 0.02 dB of a refit after the codec change, and within 0.09 dB after the checkpoint, tolerance and extractor changes. In these rows it also beats both unrefitted simple models. A new anchor codec or input degradation changes every action's value. With 10 labelled intervals per changed action, MVP comes within 0.02 dB of a refit after the checkpoint and tolerance changes. After the other changes a full refit is needed. It takes about 4 CPU-minutes on one thread, or 2.6 GPU-minutes on one RTX 5090, with labels for all 900 intervals. After a refit, MVP beats rates-only ridge by 0.06–0.12 dB in every row.
The updated PDF adds this study to the appendix.
References.
- G. Bjøntegaard. Calculation of average PSNR differences between RD-curves. ITU-T SG16/Q6 VCEG, Doc. VCEG-M33, Austin, TX, April 2001.
- Z. Su, W. Liu, Z. Yu, D. Hu, Q. Liao, Q. Tian, M. Pietikäinen and L. Liu. Pixel difference networks for efficient edge detection. In ICCV, 2021.
Response to Reviewer hoYK (1/4)
Thank you for your valuable feedback and questions. We ran additional experiments and added clarifications to address your central concerns. We updated the PDF and highlighted all new additions in red. We also prepared demo videos of MetaST on three HC-Meta clips, beside the ground truth, in the revised supplementary material. The realized MVP allocation is +0.84 dB [+0.62, +1.08] above the best uniform action over the 658 budgets of Table 3. We now call the 3.0 dB headline a system gain (Q1). On two more datasets and with H.266 anchors, MVP stays within 0.15 dB of the oracle on average, and on HC-Meta it halves the worst-budget regret of rates-only ridge (Q2). Without refitting, MVP stays within 0.09 dB of a refit after a change of metadata codec, receiver checkpoint, contour tolerance or edge extractor. After a change of anchor codec or input degradation, 10 labelled intervals per action close most of the gap to a refit (Q3). The complete sender, with pose extraction and the paper's edge-map codec, runs in real time on one RTX 5090 with 64 CPU cores (Q4). Brackets are 95 % intervals.
Q1 (W1, Q1): The 3 dB headline is a system gain, not MVP's incremental benefit. Please report the realized MVP allocation next to the best uniform action and direct H.266, with uncertainty by clip.
A1: We agree that the 3.0 dB at the lossless-anchor reference budget is a system gain. We now separate the gains and report the realized MVP allocation. As you asked, all uncertainty is by clip. Each bracket is a 95 % interval from a bootstrap over clips. Related clips, such as clips cut from the same source video, are correlated. The bootstrap therefore resamples the 32 clip groups of Appendix G, each with all of its clips.
The headline. To address your concern about what the headline gain measures, we ran a new experiment: the MetaST receiver with the same lossless anchors and no metadata. All three systems below use the same bytes.
Realized MVP allocation. To answer your request for the realized allocation curve, we ran a new experiment: MVP's own choices, scored with their measured reconstructions. MVP is trained on some clip groups and chooses the actions for the held-out groups (five-fold cross-validation). The best uniform action is the single action that, used for every interval, gives the highest PSNR within the budget, chosen in hindsight.
Average PSNR in dB on HC-Meta (50 clips, 900 intervals), H.264 anchors, 25-action grid.
Averaged over the 658 budgets of Table 3, MVP is +0.84 dB [+0.62, +1.08] above the best uniform action, and the oracle +0.99 dB. The interval excludes zero at every budget from 1.03 to 45.9 KB. Averaged over ten budgets from 1 to 50 KB, 46 of the 50 clips gain. Against the interpolated envelope of uniform actions (Fig. 5), a stricter content-blind reference, MVP gains +0.57 dB [+0.32, +0.82].