acceptodds
Under review as a conference paper at ICLR 2027

Learning Adaptive Metadata for Generative Space-Time Video Super-Resolution

Abstract

Space-time video super-resolution (STVSR) reconstructs high-resolution, high-frame-rate video from sparse, degraded pixels, but a generative receiver cannot reliably recover source-specific structure that was never transmitted. We introduce MetaST, a sender–receiver framework that learns what source information to preserve under a shared transmission budget. Pixel anchors and structural metadata compete for the same bytes, valued by their reconstruction benefit to a fixed receiver system. A lightweight Metadata Value Predictor learns these values from offline receiver responses and guides allocation using source and encoder statistics, without sender-side reconstruction. On HC-Meta, a human-centric benchmark with aligned structural representations, reallocating rate to source structure improves Average PSNR by 3.0 dB over the strongest pixel-only baseline at the lossless-anchor reference budget. On the unpruned candidate grid, learned valuation recovers 79.4% of the oracle allocation gain and reduces regret by 33% relative to a rate- only predictor. H.264/H.266 and general-video experiments further show how the useful representation depends on the relative efficiency of pixel and metadata coding. MetaST provides a modular rate–value formulation for learning what to preserve for a generative receiver.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Space-Time Video Super-Resolution, Generative Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hoYKOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies source-assisted space-time video super-resolution: a sender can spend a shared byte budget on compressed pixel anchors or source-derived structural metadata, while a fixed generative receiver reconstructs the missing frames. MetaST measures the value of candidate actions offline and trains a Metadata Value Predictor to select anchor quality and metadata without running the receiver at the sender. On the HC-Meta benchmark, transmitting three interior edge maps at the lossless-anchor reference budget improves average luma PSNR from 29.63 dB for the strongest pixel-only baseline to 32.65 dB. The paper also reports codec curves, general-video results, and actual mixed-QP stream validation.

Critical strengths
  • The study makes the cost of structural side information explicit: anchors, metadata, and signaling share one budget, and the codec settings for each comparison are specified. The 3 dB HC-Meta gain at B0 establishes that source-derived structure can be valuable when intermediate motion is ambiguous.
  • The allocation claim is tested separately from the structural metadata gain. On the unpruned 25-action grid, grouped out-of-fold prediction improves mean regret over rates-only ridge.
Critical weaknesses
  • The 3 dB headline result demonstrates the benefit of transmitting source-derived edges to a metadata-conditioned receiver, not the incremental benefit of MVP. It should not be read as a gain from learned allocation over an otherwise identical pixel-only system.
  • The predictor evaluation is confined to 50 HC-Meta clips and a fixed pair of receiver checkpoints. The 0.072 dB regret gain is credible but modest: it does not show how MVP transfers to new receivers, degradation regimes, or metadata extractors.
Questions for authors
  • Can the authors report the realized MVP allocation curve, alongside the best uniform action and direct H.266, over the same budgets and with uncertainty by clip? This would separate the learned decision gain from the value of adding source metadata.
  • How well does MVP transfer without refitting to a new receiver or metadata codec, and what are the sender-side extraction and encoding costs per second of input video?
6 comments
AuthorsOpenReview ↗

Response to Reviewer hoYK (1/4)

Thank you for your valuable feedback and questions. We ran additional experiments and added clarifications to address your central concerns. We updated the PDF and highlighted all new additions in red. We also prepared demo videos of MetaST on three HC-Meta clips, beside the ground truth, in the revised supplementary material. The realized MVP allocation is +0.84 dB [+0.62, +1.08] above the best uniform action over the 658 budgets of Table 3. We now call the 3.0 dB headline a system gain (Q1). On two more datasets and with H.266 anchors, MVP stays within 0.15 dB of the oracle on average, and on HC-Meta it halves the worst-budget regret of rates-only ridge (Q2). Without refitting, MVP stays within 0.09 dB of a refit after a change of metadata codec, receiver checkpoint, contour tolerance or edge extractor. After a change of anchor codec or input degradation, 10 labelled intervals per action close most of the gap to a refit (Q3). The complete sender, with pose extraction and the paper's edge-map codec, runs in real time on one RTX 5090 with 64 CPU cores (Q4). Brackets are 95 % intervals.

Q1 (W1, Q1): The 3 dB headline is a system gain, not MVP's incremental benefit. Please report the realized MVP allocation next to the best uniform action and direct H.266, with uncertainty by clip.

A1: We agree that the 3.0 dB at the lossless-anchor reference budget is a system gain. We now separate the gains and report the realized MVP allocation. As you asked, all uncertainty is by clip. Each bracket is a 95 % interval from a bootstrap over clips. Related clips, such as clips cut from the same source video, are correlated. The bootstrap therefore resamples the 32 clip groups of Appendix G, each with all of its clips.

The headline. To address your concern about what the headline gain measures, we ran a new experiment: the MetaST receiver with the same lossless anchors and no metadata. All three systems below use the same bytes.

System at the lossless-anchor reference budgetAverage PSNR (dB)Gain (dB)
BF-STVSR + 𝓛_RAFT, the strongest pixel-only baseline29.63reference
MetaST receiver, no metadata30.23+0.61 [+0.30, +0.91] over BF-STVSR + 𝓛_RAFT
MetaST with edge maps (Table 1)32.65+2.42 over the receiver without metadata

Realized MVP allocation. To answer your request for the realized allocation curve, we ran a new experiment: MVP's own choices, scored with their measured reconstructions. MVP is trained on some clip groups and chooses the actions for the held-out groups (five-fold cross-validation). The best uniform action is the single action that, used for every interval, gives the highest PSNR within the budget, chosen in hindsight.

Budget (KB per interval)Best uniform actionMetaST + MVP (realized)Oracle (upper bound)MVP − best uniform action
1.027.1027.6827.72+0.58 [+0.23, +0.88]
2.528.3029.3329.54+1.03 [+0.62, +1.52]
6.030.2230.8431.01+0.62 [+0.30, +1.01]
1431.0631.7931.99+0.73 [+0.39, +1.17]
3331.7132.6432.73+0.93 [+0.71, +1.18]
5031.7132.8632.91+1.15 [−0.01, +1.24]

Average PSNR in dB on HC-Meta (50 clips, 900 intervals), H.264 anchors, 25-action grid.

Averaged over the 658 budgets of Table 3, MVP is +0.84 dB [+0.62, +1.08] above the best uniform action, and the oracle +0.99 dB. The interval excludes zero at every budget from 1.03 to 45.9 KB. Averaged over ten budgets from 1 to 50 KB, 46 of the 50 clips gain. Against the interpolated envelope of uniform actions (Fig. 5), a stricter content-blind reference, MVP gains +0.57 dB [+0.32, +0.82].

AuthorsOpenReview ↗

Response to Reviewer hoYK (2/4)

A1 (continued).

Decision gain versus value of metadata. To separate the two gains, we ran a new experiment: MVP allocation on the anchors-only part of the menu, an otherwise identical pixel-only system. Receivers, rate model and budgets stay the same.

Action menuBest uniform actionMVP allocationGain of MVP allocation
Anchors only (pixel-only system, 5 actions)29.0629.40+0.34 [+0.27, +0.39]
Anchors and metadata (25 actions)29.8630.70+0.84 [+0.62, +1.08]
Gain from metadata+0.80 [+0.55, +1.09]+1.30 [+0.87, +1.85]

Average PSNR in dB, mean over the 658 budgets of Table 3.

Metadata alone adds 0.80 dB to the best uniform action. MVP allocation adds 0.84 dB on top of it. Together they place MetaST + MVP 1.64 dB [1.24, 2.15] above the best uniform action of the pixel-only system. Allocation with rates-only ridge already attains 0.77 dB of the 0.84 dB. MVP's own increment is the subject of Q2.

Direct H.266. To answer your request for the direct-codec baseline, we ran a new experiment: a comparison with direct H.266. Direct H.266 codes every frame at full resolution and sends no metadata. MetaST uses H.266 anchors and an MVP retrained on H.266 value labels.

Rate (KB per interval)Direct H.266MetaST + MVP (H.266 anchors)MetaST + MVP − direct H.266
0.7526.6427.12+0.48 [−0.04, +0.80]
1.2428.0528.17+0.12 [−0.30, +0.41]
1.6128.9428.48−0.45 [−0.79, −0.13]

Average PSNR in dB at measured direct H.266 rates.

Direct H.266 overtakes MetaST + MVP from 1.36 KB per interval [0.78, 1.57]. The paper reports this crossover near 1.4 KB for the oracle (Sec. 4.5, Fig. 4b). H.264 is still the most widely deployed video codec, and it is the paper's main setting. With stronger anchor codecs such as H.266, metadata needs more efficient metadata codecs to pay off. We leave such codecs to future work.

In the updated PDF, we call the 3.0 dB a system gain and add the realized MVP curves to Figs. 4 and 6, with the oracle as the upper bound. The anchors-only result and the best uniform action are in the appendix.

Q2 (W2): The predictor evaluation is confined to 50 HC-Meta clips, and the 0.072 dB regret gain is modest.

A2: To address your concern that the predictor evaluation is confined to HC-Meta, we ran a new experiment: the same evaluation on two other video datasets and with H.266 anchors. Each value model is retrained for its setting and tested on held-out clips or recordings. Regret is the PSNR gap between a model's allocation and the oracle allocation at the same budget.

SettingItem priorRates-only ridgeMVPMVP − rates-only ridgeOracle gain recovered by MVP
HC-Meta, H.264 anchors (Table 3)0.2520.2200.148−0.072 [−0.122, −0.031]79.4 %
HC-Meta, H.266 anchors0.1950.2010.142−0.059 [−0.088, −0.039]66.8 %
Adobe240, 15 recordings, 4,410 intervals0.1010.1390.058−0.08286.7 %
GoPro, 10 recordings, 1,522 intervals0.1020.0780.103+0.02584.7 %

Mean regret in dB, lower is better. The last column is MVP's gain over the interpolated envelope of uniform actions (Fig. 5), as a share of the oracle's gain over the same envelope.

MVP has the lowest regret in three of the four settings. Under H.266 it also beats the item prior, the stronger simple model there, by 0.053 dB [0.030, 0.087]. On GoPro, every value model already recovers at least 84 % of the oracle's gain, so little headroom remains.

AuthorsOpenReview ↗

Response to Reviewer hoYK (3/4)

A2 (continued). Where the increment matters. To show where the 0.072 dB comes from, we split MVP's and rates-only ridge's regret on HC-Meta by rate range.

Regret on HC-Meta (dB)Rates-only ridgeMVPReduction by MVP
Mean over all 658 budgets (Table 3)0.2200.1480.072 [0.031, 0.122]
Mean below 2 KB per interval0.2170.1120.106
Mean at 2–10 KB0.3130.1860.127
Mean at 10–30 KB0.2070.1800.027
Mean above 30 KB0.0430.064−0.021
90th percentile over the budgets (Table 3)0.4120.2160.196
Worst budget0.5870.2930.294

The gain sits below 10 KB per interval, where rate is scarce. At 2.5 KB, MVP's allocation is 0.27 dB [0.15, 0.43] higher in PSNR than ridge's. Above 30 KB both models are within 0.07 dB of the oracle. We also compare the two rate–PSNR curves with the Bjøntegaard delta rate (BD-rate, Bjøntegaard, 2001), the standard measure in video coding. It is the average rate difference at equal PSNR over the PSNR range both curves cover. We interpolate log-rate piecewise linearly. MVP's BD-rate against rates-only ridge is −5.2 %, so it needs 5.2 % less rate for the same PSNR.

The updated PDF adds the H.266, Adobe240 and GoPro results to Appendix G.

Q3 (W2, Q2 first part): How well does MVP transfer without refitting to a new receiver, metadata codec, degradation regime or metadata extractor?

A3: To answer your question on transfer without refitting, we ran a new experiment: one component changed at a time on HC-Meta, with newly measured values of the changed actions.

  • Edge codec: a generic lossless coder (bit-packing + Zstandard) replaces the contour codec. On the same edge maps it changes only the rates. On raw Canny dense maps it also changes the maps and their values.
  • Second receiver checkpoint: the edge receiver at training step 22,902 is replaced by its ancestor checkpoint at step 13,701.
  • Contour codec tolerance: sparse edge maps are simplified with 3.0 px or 6.0 px instead of 1.5 px.
  • Learned edge extractor: PiDiNet (Su et al., 2021) replaces Canny for both edge-map families.
  • H.266 anchors, on the twelve-action menu.
  • Input degradation: Gaussian blur and ×4 decimation replace bicubic ×4.

"No refit" reuses the models trained on the original setting. "Refit" retrains MVP on the new values. "10 labelled intervals" shifts each changed action's predictions by their mean error on 10 labelled intervals, without retraining.

ChangeMVP, refitMVP, no refitRates-only ridge, no refitItem prior, no refitMVP, 10 labelled intervals
Edge codec, same maps0.1590.168 (+0.010)0.259 (+0.100)0.272 (+0.114)—
Edge codec, raw Canny dense maps0.1620.163 (+0.001)0.257 (+0.095)0.273 (+0.111)—
Second receiver checkpoint0.1700.181 (+0.011)0.271 (+0.101)0.274 (+0.104)0.169 (−0.001)
Contour codec, 3.0 px0.1450.159 (+0.014)0.271 (+0.126)0.275 (+0.130)0.148 (+0.003)
Contour codec, 6.0 px0.1390.226 (+0.088)0.427 (+0.288)0.385 (+0.246)0.142 (+0.004)
Learned edge extractor0.1650.233 (+0.068)0.575 (+0.410)0.403 (+0.238)0.200 (+0.035)
H.266 anchors0.1420.421 (+0.279)0.534 (+0.392)0.278 (+0.137)0.196 (+0.054)
Blur + ×4 decimation0.0981.064 (+0.966)0.884 (+0.786)0.981 (+0.883)0.141 (+0.043)

Mean regret in dB, lower is better. In parentheses, the difference to the refitted MVP.

Without refitting, MVP stays within 0.02 dB of a refit after the codec change, and within 0.09 dB after the checkpoint, tolerance and extractor changes. In these rows it also beats both unrefitted simple models. A new anchor codec or input degradation changes every action's value. With 10 labelled intervals per changed action, MVP comes within 0.02 dB of a refit after the checkpoint and tolerance changes. After the other changes a full refit is needed. It takes about 4 CPU-minutes on one thread, or 2.6 GPU-minutes on one RTX 5090, with labels for all 900 intervals. After a refit, MVP beats rates-only ridge by 0.06–0.12 dB in every row.

The updated PDF adds this study to the appendix.

References.

  • G. Bjøntegaard. Calculation of average PSNR differences between RD-curves. ITU-T SG16/Q6 VCEG, Doc. VCEG-M33, Austin, TX, April 2001.
  • Z. Su, W. Liu, Z. Yu, D. Hu, Q. Liao, Q. Tian, M. Pietikäinen and L. Liu. Pixel difference networks for efficient edge detection. In ICCV, 2021.
Reviewer_CKRLOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MetaST, a sender–receiver framework for generative space-time video super-resolution (STVSR) that allocates a shared transmission budget between pixel anchors and structural metadata. The framework considers different anchor qualities and metadata configurations, including skeletons, poses, and edge maps. A lightweight Metadata Value Predictor (MVP) is trained on offline receiver responses to predict the reconstruction value of candidate actions without sender-side reconstruction. These predictions guide sequence-level budget allocation, while a frozen one-step video diffusion receiver reconstructs high-resolution, high-frame-rate videos. Experiments on HC-Meta, GoPro, and Adobe240 demonstrate improvements over pixel-only STVSR baselines and reduced allocation regret.

Critical strengths
  1. The paper is well organized and clearly written. The motivation and overall framework are easy to follow, and the key components are presented in a logical and coherent manner. The supplementary material further provides detailed descriptions of metadata processing, receiver training, the Metadata Value Predictor (MVP), and budgeted allocation, supporting a clear understanding of the proposed approach.
  2. The paper provides extensive experimental evaluations across multiple datasets, metadata configurations, and video codecs. The comparisons with existing STVSR methods, rate–distortion analysis, metadata ablations, and evaluations of different value predictors provide useful evidence for assessing the proposed framework.
Critical weaknesses
  1. Limited methodological novelty. A similar sender–receiver framework has been explored in prior work, such as MetaSR, which investigates adaptive metadata transmission for generative super-resolution. The newly introduced components, including the Metadata Value Predictor (MVP) and sequence-level budgeted allocation, appear to be applications of existing techniques.
  2. The effectiveness of MVP has not been sufficiently validated. First, Figures 4 and 6 report MetaST Oracle results rather than the performance of MVP-based allocation. Second, MVP reduces mean regret by only 0.072 dB relative to the rates-only ridge in Table 3, raising questions about the practical significance of this improvement.
Questions for authors
  1. The main methodological contributions appear to be the Metadata Value Predictor (MVP) and sequence-level budgeted allocation, both of which seem to build on existing techniques. Could the authors further clarify the unique contributions of these designs? Additionally, can the proposed framework generalize to different generative receivers or other reconstruction tasks?
  2. In Figures 4 and 6, MetaST Oracle is compared with other STVSR methods. Since Oracle allocation relies on ground-truth reconstruction values, would it be more appropriate to report the performance of the proposed MVP in these comparisons, while retaining Oracle as an upper bound?
4 comments
AuthorsOpenReview ↗

Response to Reviewer CKRL (1/4)

Thank you for your valuable feedback and questions, and for recognizing the clear motivation and the extensive evaluation. We ran additional experiments and added clarifications to address your central concerns. We updated the PDF and highlighted all new additions in red. We also prepared demo videos of MetaST on three HC-Meta clips, beside the ground truth, in the revised supplementary material. MetaST contributes a generative STVSR receiver conditioned on heterogeneous metadata and the joint choice of anchor quality, metadata type and temporal density (Q1). Each metadata type covers its own rate range, and metadata raises the gain of MVP allocation from 0.34 to 0.84 dB (Q1). MetaST + MVP stays within 0.07–0.22 dB of the oracle and 0.82–2.72 dB above BF-STVSR + 𝓛_RAFT at the rates of Table 11 (Q2). Its 0.072 dB regret reduction halves the worst-budget regret and concentrates where rate is scarce (Q3). MVP keeps its advantage on a second receiver checkpoint without refitting and after five other changes with a refit (Q4). The framework also extends to frame interpolation and video super-resolution (Q5). Brackets are 95 % intervals.

Q1 (W1, Q1 first part): Limited methodological novelty with respect to MetaSR. MVP and sequence-level allocation look like applications of existing techniques. What are their unique contributions?

A1: Our contributions are specific to space-time video super-resolution (STVSR). We propose a generative STVSR receiver that takes heterogeneous metadata as its reconstruction condition. We also propose a learned valuation of each joint choice of anchor quality, metadata type and temporal density, and allocate the budget over the sequence.

MetaSR. MetaSR, the work the reviewer names, is a recent preprint that, to our knowledge, has not appeared at a peer-reviewed venue. It focuses on image super-resolution, and its rate–distortion study uses one metadata type, Canny edge maps. It does not consider interval-level decisions for space–time reconstruction. MetaST studies these decisions: missing frames make the temporal density of metadata a decision variable, and shared anchors couple the costs of neighbouring intervals. The contribution we emphasize is learning the receiver's response to these joint actions and allocating them over a coupled sequence. The updated PDF cites and discusses MetaSR.

(1) Scope. To our knowledge, MetaST is the first framework that conditions a generative space-time video super-resolution receiver on transmitted metadata. Image super-resolution has no temporal dimension, no missing frames, no anchors shared across intervals and no temporal sparsity of metadata.

(2) Heterogeneous metadata under one budget. Fig. 5 shows that each metadata type covers its own rate range. As the budget grows, the oracle moves from anchors and compact skeletons to sparse and then dense edge maps. Anchor quality rises along the way. The table gives the action shares behind Fig. 5 (bottom).

Rate per intervalAnchors only2D skeletonSkeleton + 3D poseSparse edgesDense edgesMost used anchor QP
Below 2 KB88 %7 %0 %5 %0 %37
2–10 KB39 %44 %1 %12 %4 %25
10–30 KB7 %34 %10 %15 %34 %18
Above 30 KB1 %6 %10 %1 %81 %10

Share of the 900 HC-Meta intervals given each metadata type by the oracle, averaged over the budgets of each range.

Every type serves a sizeable share of the intervals in some range, and the frontier uses 22 of the 25 actions. At the rates of the QP-25 sparse-edge, dense-edge and anchors-only actions, the oracle gains 1.32, 1.19 and 1.02 dB over that single action.

One HC-Meta clip of a turning ballet dancer shows what each type adds as the budget grows. With the same QP-10 anchors, four uniform actions cost 6.3, 7.6, 9.4 and 13.1 KB per interval. Without metadata (28.5 dB), limbs blur and fade between anchors during turns. 2D skeletons (31.3 dB) restore the limb trajectories but once paint an arm over the torso. Adding 3D pose (31.7 dB) removes that arm. Dense edge maps (36.3 dB) come closest to the ground truth. On this clip, sparse edges reach 34.2 dB at 7.7 KB, above skeleton + 3D pose at 9.4 KB. The best type thus depends on content and budget, so MVP chooses it per interval. We added this example, with its videos, to the supplementary material.

(3) HC-Meta. The benchmark targets human motion, where these metadata types matter. Its 900 intervals carry aligned 2D skeletons, 3D poses, human contours and edge maps.

AuthorsOpenReview ↗

Response to Reviewer CKRL (2/4)

A1 (continued).

(4) Metadata and adaptive allocation reinforce each other. To answer your question on unique contributions, we ran a new experiment: MVP allocation on the anchors-only part of the menu. Receivers, rate model and budgets stay the same. The best uniform action is the single action that, used for every interval, gives the highest PSNR within the budget.

Action menuBest uniform actionMVP allocationGain of MVP allocation
Anchors only (5 actions)29.0629.40+0.34 [+0.27, +0.39]
Anchors and metadata (25 actions)29.8630.70+0.84 [+0.62, +1.08]
Gain from metadata+0.80 [+0.55, +1.09]+1.30 [+0.87, +1.85]

Average PSNR in dB, mean over the 658 budgets of Table 3.

Metadata adds 0.80 dB to the best uniform action and 1.30 dB to MVP allocation. MVP allocation thus gains 0.50 dB [0.28, 0.77] more when metadata is on the menu. Metadata gives the allocator better options, and the allocator spends metadata bytes only where they pay.

(5) What MVP values. The allocated unit is a joint action of anchor quality and metadata configuration. Its value is a generative receiver's response, measured offline per interval. MVP predicts this value from sender-side statistics, so the sender never runs the receiver.

(6) Why sequence-level allocation. Neighbouring intervals share a boundary anchor frame and boundary maps. The cost of an interval therefore depends on what its neighbour has already sent. MetaST allocates the budget over the whole sequence and charges each shared item once. To show why this matters, we ran a new experiment: two allocators that choose each interval's action independently, with the same MVP values.

  • The first charges each interval its average cost and ignores the sharing. Coded as one stream, its choices exceed the budget at 98 % of the budgets, by 2.5 % on average.
  • The second charges each interval as if it were coded alone, with both of its anchors. It stays within budget but leaves 32 % of the budget unused. Where it finds an allocation at all, its PSNR is 0.59 dB lower than with sequence-level allocation. Below 1.41 KB per interval it finds none.

Sequence-level allocation stays within every budget by construction. Solving it exactly with measured values also gives the oracle allocation, the upper bound against which we measure regret.

Q2 (W2 first part, Q2): Figs. 4 and 6 show the oracle, not MVP-based allocation. Should they report MVP, with the oracle as an upper bound?

A2: Yes, these comparisons should show MVP, with the oracle as the upper bound. To address your concern that the figures show only the oracle, we now report MetaST + MVP next to every curve of Figs. 4 and 6. As in Table 3, MVP chooses actions for clips it was not trained on. We score its choices with their measured reconstructions and the paper's rate model.

Fig. 4 (HC-Meta). With H.264 anchors, MetaST + MVP stays within 0.07–0.22 dB of the oracle at the rates of Table 11 from 1.18 KB per interval. It is above BF-STVSR + 𝓛_RAFT at each of these rates, by +0.82 to +2.72 dB. With H.266 anchors, we retrained MVP on the H.266 value labels. At the six rates of Table 11, it is 0.05–0.29 dB below the oracle and above BF-STVSR + 𝓛_RAFT. The interval of this gain includes zero at 0.21 and 0.65 KB.

Panel, rate (KB per interval)BF-STVSR + 𝓛_RAFTAMT-S + DOVEMetaST + MVP (realized)MetaST oracle (upper bound)
Fig. 4a (H.264 anchors), 1.7627.6927.6328.6428.84
Fig. 4a, 5.3028.7928.2930.5930.79
Fig. 4a, 17.6629.3028.4532.0232.20
Fig. 4b (H.266 anchors), 0.2124.3224.3224.5524.66
Fig. 4b, 1.2327.4727.3028.1028.29
Fig. 4b, 3.7328.6428.0330.1430.28

Average PSNR in dB at three of the rates of Table 11 per panel.

Fig. 6 (GoPro, Adobe240). We retrained MVP on each source and tested it on held-out recordings. MVP crosses BF-STVSR + 𝓛_RAFT slightly later than the oracle and keeps almost all of the oracle's gain at the top rate.

SourceAbove BF-STVSR + 𝓛_RAFT from: MVP / oracleGain over it at the top rate: MVP / oracleMean regret of MVP
GoPro11.3 / 9.6 KB+1.47 / +1.52 dB0.103 dB
Adobe24011.9 / 11.0 KB+0.306 / +0.315 dB0.058 dB

Rates in KB per interval. Top rate: 32.6 KB (GoPro) and 29.1 KB (Adobe240). Crossings and top-rate gains are from the first of five training repeats. On Adobe240, MVP has the lowest regret of the three value models. On GoPro, rates-only ridge is slightly closer to the oracle (0.078 dB).

The updated Figs. 4 and 6 add the MetaST + MVP curves and keep the oracle as the upper bound. The values at the rates of Table 11 are in the appendix.

AuthorsOpenReview ↗

Response to Reviewer CKRL (3/4)

Q3 (W2 second part): MVP reduces mean regret by only 0.072 dB over rates-only ridge. Is this practically significant?

A3: The 0.072 dB [0.031, 0.122] is a mean over all 658 budgets, and it hides where the gain occurs. Regret is the PSNR gap between a model's allocation and the oracle allocation at the same budget. To address your concern about practical significance, we ran a new experiment: MVP and rates-only ridge compared budget by budget, by rate range and with H.266 anchors.

Regret (dB)Rates-only ridgeMVP
Mean over all 658 budgets (Table 3)0.2200.148
90th percentile over the 658 budgets (Table 3)0.4120.216
Mean over the budgets below 10 KB per interval0.2780.159
At the worst budget0.5870.293
Mean with H.266 anchors0.2010.142

MVP removes one third of ridge's mean regret and halves its worst case. Below 10 KB per interval it removes 43 %. At 2.5 KB per interval, MVP's allocation is 0.27 dB [0.15, 0.43] higher in PSNR than ridge's. Over the whole rate range, MVP's rate–PSNR curve reaches the same PSNR as ridge's with 5.2 % less rate on average (Bjøntegaard delta rate, Bjøntegaard, 2001).

For scale, the oracle allocation is 0.99 dB above the best uniform action of Q1 (4) on average. MVP reaches 0.84 dB of this and rates-only ridge 0.77 dB.

Q4 (Q1 second part): Can the framework generalize to different generative receivers?

A4: To address your concern about generalization, we tested whether MVP and the allocation carry over when one part of the system changes. We ran new experiments with six changes, one at a time: the receiver checkpoint, the edge extractor, two edge-codec tolerances, the anchor codec and the input degradation. For each change, we measured the new values of the changed actions with the receiver. "No refit" applies MVP, trained on the original setting, unchanged. "Refit" retrains MVP on the new values. In both cases we measure the regret to the oracle of the new setting and compare MVP with rates-only ridge and the item prior. MVP keeps its advantage without refitting after the checkpoint, extractor and tolerance changes, and it is the best value model after a refit in every case.

  • Second receiver checkpoint: our edge actions are scored by the edge receiver at training step 22,902. We replace it with its ancestor checkpoint at step 13,701, of the same architecture.
  • Learned edge extractor: PiDiNet (Su et al., 2021) replaces Canny for both edge-map families.
  • Contour codec tolerance: sparse edge maps are simplified with 3.0 px or 6.0 px instead of 1.5 px.
  • H.266 anchors: anchors are coded with H.266 instead of H.264, on the twelve-action menu.
  • Input degradation: the low-resolution input comes from Gaussian blur and ×4 decimation instead of bicubic ×4.
ChangeMVP, refitMVP, no refitRates-only ridge, refitItem prior, refit
Second receiver checkpoint0.1700.181 (+0.011)0.234 (+0.064)0.263 (+0.093)
Learned edge extractor0.1650.233 (+0.068)0.288 (+0.124)0.358 (+0.193)
Contour codec, 3.0 px0.1450.159 (+0.014)0.205 (+0.060)0.271 (+0.126)
Contour codec, 6.0 px0.1390.226 (+0.088)0.203 (+0.064)0.263 (+0.125)
H.266 anchors0.1420.421 (+0.279)0.201 (+0.059)0.195 (+0.053)
Blur + ×4 decimation0.0981.064 (+0.966)0.177 (+0.079)0.164 (+0.066)

Mean regret in dB, lower is better. In parentheses, the difference to the refitted MVP. Every difference has a 95 % interval that excludes zero, except no refit on the second checkpoint (+0.011 [−0.001, +0.023]).

On the second checkpoint without any refit, MVP's regret is 0.181 dB, against 0.271 dB and 0.274 dB for the equally unrefitted rates-only ridge and item prior. Without refitting, it stays within 0.09 dB of a refit after the checkpoint, extractor and tolerance changes. A few labels close most of the remaining gap. For each changed action, we run the new receiver on only 10 intervals, drawn from clips that are not used for testing. We compare these 10 measured values with MVP's predictions for the same intervals and take the mean difference. We then add this mean difference to all of MVP's predictions for that action. MVP itself is not retrained. For the checkpoint swap, this needs 100 receiver passes, 10 for each of the ten edge actions. MVP then comes within 0.02 dB of a refit after the checkpoint and tolerance changes. Results average 10 random draws of the labelled intervals. New anchors or a new input degradation change every action's value and need new labels.

AuthorsOpenReview ↗

Response to Reviewer CKRL (4/4)

A4 (continued). Labels and retraining. The value definition, the rate model and the allocation do not depend on the receiver. Only MVP's training labels do. A label is the measured value of one action on one interval, so it takes one receiver pass. Labels are computed offline, once per receiver, and the sender never runs the receiver. A new receiver needs labels only for the actions it scores. For the second checkpoint, these are the ten edge actions on the 900 HC-Meta intervals, that is 9,000 receiver passes. Retraining MVP on them takes about 4 CPU-minutes on one thread, or 2.6 GPU-minutes on one RTX 5090.

Q5 (Q1 second part): Can the framework generalize to other reconstruction tasks?

A5: Yes, the framework can generalize beyond STVSR. To answer your question on other reconstruction tasks, we ran new experiments: frame interpolation and video super-resolution. In frame interpolation, three transmitted edge maps add 3.14 / 2.77 dB over the same receiver design without them (DAVIS-7 / UCF101-7). They lead AMT, the best pixel-only method, by 2.36 / 1.72 dB. In video super-resolution, edge maps of every frame give the lowest LPIPS and DISTS of all methods, at a PSNR close to the CNN baselines. Both tasks use MetaST's edge-map codec and a fixed metadata setting.

Frame interpolation. We trained the same receiver design for interpolation. It interpolates between the two anchors as MetaST does, without spatial upsampling. DAVIS-7 and UCF101-7 served only for evaluation. All methods receive the same two anchors per interval and are scored on the same nine frames with the same metrics. All rows also have the same average byte budget. The pixel-only methods spend it on anchors coded at H.264 QP 0, and so does the row without edge maps. MetaST re-codes its anchors at one rate for the whole benchmark, so that anchors and edge maps together fit the same budget. With three edge maps, PSNR is the highest and FVD the lowest of all compared methods.

MethodEdge maps sent (share of bytes)PSNR (dB) ↑FVD ↓
AMT (best pixel-only PSNR)none25.70 / 30.32260.83 / 358.09
FCVG (best pixel-only FVD)none24.18 / 29.12132.14 / 175.53
MetaST receiver design, no edge mapsnone24.92 / 29.27196.60 / 281.95
MetaST receiver design, three edge maps3.3% / 4.2%28.06 / 32.0491.50 / 136.40

DAVIS-7 / UCF101-7 at 256×256, two anchors and seven missing frames. PSNR is on RGB over all nine frames.

Video super-resolution. The byte budget is the size of the losslessly coded low-resolution frames. MetaST re-codes these frames to fit coded edge maps of every frame into this budget.

MethodBytes per intervalAverage PSNR (dB)LPIPSDISTS
EDVR435,94436.730.10210.0723
BasicVSR435,94436.930.09950.0698
MetaST without metadata435,94435.640.08820.0545
MetaST + coded edge maps of every frame425,15335.990.08350.0501
MetaST + lossless edge maps of every frame576,72036.340.07820.0483

×4 on HC-Meta, 900 intervals, every frame observed at low resolution. Each MetaST receiver is fine-tuned for this input layout. Lower LPIPS and DISTS are better.

In VSR, coded edge maps improve the metadata-free MetaST receiver by 0.35 dB within the reference budget and achieve lower LPIPS and DISTS than the CNN baselines, with lower PSNR. These results demonstrate a perceptual-quality trade-off.

The updated PDF adds both experiments to the appendix.

References.

  • G. Bjøntegaard. Calculation of average PSNR differences between RD-curves. ITU-T SG16/Q6 VCEG, Doc. VCEG-M33, Austin, TX, April 2001.
  • Z. Su, W. Liu, Z. Yu, D. Hu, Q. Liao, Q. Tian, M. Pietikäinen and L. Liu. Pixel difference networks for efficient edge detection. In ICCV, 2021.
Reviewer_MxQNOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies source-assisted space-time video super-resolution. The sender has access to the original high-resolution, high-frame-rate video and jointly selects the encoding quality of pixel anchors and structural information, such as skeletons, poses, and edges, under a total byte budget. A Metadata Value Predictor (MVP) learns the reconstruction value of candidate actions from offline receiver results. At deployment, it predicts gains from the source video and encoding statistics, then allocates the budget using chain-structured dynamic programming. The receiver uses two prespecified, frozen generative model configurations.

Critical strengths
  1. The problem is meaningful, and the cost of additional information is explicitly accounted for. Skeletons, poses, and edges compete with pixels under the same byte budget, avoiding the treatment of structural conditioning as free information. Learning which source information is most valuable to transmit for a fixed receiver is a clear and worthwhile problem.

  2. The learning component receives detailed internal validation. The main results use the full 25-action grid and include an item prior, rates-only ridge regression, ridge regression with the same inputs, and prediction-shuffling controls. The authors use repeated grouped cross-validation, with normalization and model selection performed within the training folds. MVP reduces mean regret from 0.220 to 0.148 dB, an improvement of 0.072 dB, with a paired 95% interval of [0.031, 0.122]. P90 regret decreases from 0.412 to 0.216 dB. These results support a benefit from content-dependent prediction.

  3. The paper demonstrates system-level gains and their dependence on the codec. Under the lossless-anchor reference budget, the fixed k=3k=3 edge configuration achieves 32.65 dB on HC-Meta, compared with 29.63 dB for the best pixel-input baseline. Comparisons involving H.264, H.266, and direct source-video compression also show that the advantageous operating range depends on coding efficiency. The authors openly report that direct H.266 compression becomes better beyond approximately 1.4 KB per interval. The scope of the menu-constrained oracle and the limited Lagrangian sweep is also explicitly qualified.

Critical weaknesses

W1. The actual-bitstream experiments do not yet validate the complete deployed system Appendix E.1 explicitly states that the mixed-QP experiments only re-encode and decode the anchors, while sparse-edge completion retains the cached full maps from the original single-QP setting, including endpoint maps generated from the original decoded anchors. Other metadata and accounting remain unchanged. Table 7 therefore evaluates the effect of changes to the anchor bitstream, but does not yet establish that the receiver can reconstruct all of its inputs solely from the information actually transmitted. Complete bitstream validation is important to the central contribution of allocating a shared budget to pixels and structure. In addition, pose costs are computed as average bytes per frame after compressing an entire clip. The actual size after selecting and repackaging only some windows may differ. In Table 7, replacing estimated anchor costs with actual anchor bytes changes sequence-level accounting by up to +1.92%, and at low rates, up to 54 of 900 intervals exhibit a PSNR change greater than 0.5 dB. The small average differences are encouraging, but budget feasibility and predictor gains still need to be established in the actual system. I recommend generating complete bitstreams containing anchors, metadata, and signaling for the learned policy and the main allocation baselines, regenerating endpoints and completing edges from the anchors actually decoded, and then reporting actual rate and quality. The dense-edge DP implementation already uses actual per-frame byte counts and deduplicates boundaries; it should not be characterized as relying only on average-cost accounting.

W2. External validity and the necessity of the specific learning design remain limited The main MVP results come from 50 clips, 32 groups, and 900 intervals in HC-Meta. Repeated grouped cross-validation is appropriate, but the main curves for GoPro, Adobe240, and H.266 are still oracle results and do not establish that the learned policy is effective in those settings. Two types of gains should be distinguished: the 3.02 dB gain is the improvement of the overall source-assisted system over the pixel-input baseline and cannot be attributed to MVP; MVP’s incremental improvement over the rates-only control is 0.072 dB. The latter is statistically significant, but its absolute magnitude is small, making confirmation of its stability in independent settings particularly important. I recommend reporting learned-policy curves for at least one independent video source or another codec setting, and clarifying whether new value labels and retraining are required. Ablations removing candidate-set pooling or the listwise term, or a comparison with a standard nonlinear regressor using the same inputs, would also more directly establish the necessity of the specific design.

W3. Temporal quality under action changes across windows has not been evaluated Adjacent intervals can change metadata family, anchor QP, and even switch between the two frozen checkpoints. Per-frame PSNR/SSIM and static examples are insufficient to rule out flicker, texture drift, or motion discontinuities at switching boundaries. This risk follows directly from the adaptive allocation mechanism and is directly relevant to high-frame-rate video reconstruction. I recommend providing full videos and temporal metrics or human inspection results grouped by whether a switch occurs, with particular attention to the boundaries of eight-frame intervals. Perceptual metrics could be added, but the paper need not change its explicitly stated PSNR optimization objective.

W4. Reproducibility gaps remain between the supplementary implementation and the main experimental configurations Appendix D describes nine-frame training intervals, whereas both supplied training launch scripts specify 17×720×1280. The MVP entry point recommended in the README invokes chain_lagr.py, which fixes the action count to 12 at line 43 and uses 4-bit signaling at line 62. A corresponding entry point for reproducing the main 25-action protocol is not directly provided. The README states that the supplementary package is a curated subset, which can explain its limited scope, but the actual configurations used for the main results still need to be supplied. The training scripts also reference validation clip lists that are not included. The authors should provide the training, model-selection validation, and final evaluation lists, and confirm that the clips used to select receiver checkpoints are also separate from HC-Meta. The presence of test in a path does not itself demonstrate data leakage; the available material indicates a reproducibility issue requiring clarification.

Questions for authors
  1. Does MVP retain its 0.072 dB advantage over rates-only ridge regression when all receiver inputs are reconstructed from complete actual bitstreams and edge completion is recomputed from the anchors actually decoded?
  2. How are selectively transmitted skeleton and pose records packaged and compressed? Do their actual sizes agree with the whole-clip average-byte model?
  3. Can the authors report learned-MVP results on an independent video source or under H.266, together with the practical costs of collecting new value labels, training, and deployment?
  4. Are there temporal artifacts at action or checkpoint switches? What are the individual contributions of candidate-set pooling and the listwise objective?
  5. Please reconcile the nine-frame and 17-frame training configurations, provide the entry point and rate/value tables for the 25-action experiments, and clarify the separation between validation and final evaluation sets.
9 comments
AuthorsOpenReview ↗

Response to Reviewer MxQN (1/5)

Thank you for your valuable feedback and questions. We ran additional experiments and added clarifications to address your central concerns. We updated the PDF and highlighted all new additions in red. We also prepared demo videos of MetaST on three HC-Meta clips, beside the ground truth, in the revised supplementary material (Q4 a). We now run the complete system end to end, with every receiver input rebuilt from the transmitted bytes and all 40 operating points within budget after re-solving (Q1). MVP's margin over rates-only ridge there is +0.060 dB [−0.027, +0.151], essentially the same as under the paper's rate model. Selected skeleton and pose records travel as run packets, close in size to the average-byte model for long runs and billed at their actual size (Q2). Retrained MVP lowers regret against rates-only ridge by 0.059 dB under H.266 and by 0.082 dB on Adobe240, and the complete sender runs in real time (Q3). For temporal artifacts at switches, frames at interval boundaries are 0.78 luma levels less consistent than inside an interval, and a switch adds a further 0.16–0.25 levels (Q4 a). Q4 (b) explains the design of candidate-set pooling and the listwise term, and Q5 answers each reproducibility point. Brackets are 95 % intervals.

Q1 (W1, Q1): Is the complete system validated on actual bitstreams, and does MVP keep its advantage over rates-only ridge there?

A1: On the complete system, MVP exceeds rates-only ridge by +0.060 dB [−0.027, +0.151], against +0.055 dB under the rate model, a paired change of only +0.004 dB [−0.011, +0.023]. The rate model scores the same final allocations with the paper's rate/value table. This comparison covers ten budgets of one cross-validation repeat, so the 0.072 dB of Table 3 remains the significance test, over 658 budgets and five repeats.

We agree that Table 7 re-coded only the anchors. To address your concern about validating the complete deployed system, we ran a new experiment: the complete sender and receiver, end to end. It covers MVP, rates-only ridge, the item prior and the original constrained oracle policy on the 25-action grid. It uses ten budgets from 1 to 50 KB per interval and all 900 HC-Meta intervals.

  • Anchors are actual H.264 streams with per-picture QPs, decoded at the receiver.
  • Transmitted edge maps are decoded from their coded payloads. Edge endpoints and the completed edge maps that depend on them are recomputed from the decoded anchors.
  • Skeleton and pose records are sent as run packets (Q2). The receiver decodes them and renders its pose inputs from them.
  • Five bits per interval signal the action. Signalling and container framing are counted, and framing adds under 0.6 B per interval at every operating point.
  • The receiver rebuilds every input from the transmitted container and fixed shared settings, such as the action menu, codecs and models. The revised supplement contains two such containers and their decoder.

Budget. On actual container bytes, 22 of the 40 operating points first exceed their budget, by at most 3.75%. The sender then re-solves the allocation at a proportionally reduced target and re-encodes, one to four times. All 40 points then lie within budget.

Quality and regret. Regret is measured against the original constrained oracle policy. This is the paper's 25-action oracle allocation, re-solved and transmitted like the other policies. MVP has the lowest regret of the three value models.

PolicyActual rate, % of budgetAverage PSNR, actual − rate model (dB)Mean regret (dB)
MVP97.5–99.9−0.062 to +0.0010.124 [0.051, 0.209]
Rates-only ridge97.2–100.0−0.081 to −0.0120.183 [0.094, 0.285]
Item prior96.9–100.0−0.095 to −0.0120.220 [0.127, 0.339]
Original constrained oracle policy99.3–100.0−0.116 to −0.013reference

Ranges and means are over the ten budgets. The 95 % intervals come from a bootstrap over the 32 clip groups of Appendix G.

Dense edges. We agree that every dense edge map is coded and charged its measured bytes, with shared boundaries counted once. The updated PDF says so.

The updated PDF adds this validation, with the re-solving step, to Appendix E.

AuthorsOpenReview ↗

Response to Reviewer MxQN (2/5)

Q2 (W1, Q2): How are selectively transmitted skeleton and pose records packaged, and do actual sizes agree with the whole-clip average-byte model?

A2: Each run of consecutive intervals that selects a record stream is sent as one self-contained packet. To answer your question on the actual size of selected records, we ran a new experiment: run-based packaging of the records that the allocations select. Their measured sizes agree with the average-byte model for the long runs that MVP and rates-only ridge select.

Billing in the paper. Appendix C bills each selected interval at its clip's average record size, obtained by compressing the clip's whole record stream. We call this the Appendix C charge.

Packaging and compression.

  • Packets. A run of r consecutive selected intervals forms one packet over its 8r + 1 frames, with shared boundary frames sent once. Each packet decodes on its own. The 5-bit action of each interval already tells the receiver which intervals a packet covers, so a packet needs no extra header.
  • 2D skeleton. Each frame carries 133 whole-body keypoints at integer-pixel precision, one visibility flag per keypoint and one presence flag. Each frame is coded as the difference to the previous frame, and the first frame of a packet is coded on its own. The x and y differences (16-bit integers), the visibility flags and the presence flags form four streams, each compressed with zlib at level 9.
  • 3D pose. Each frame carries the 55 SMPL-X joint rotations and the translation, both quantized to 10⁻³. They are coded as temporal differences in two zlib streams. Each packet also carries the body shape and expression (40 B), the camera parameters (16 B) and the frame count (2 B) once.

Sizes. In an earlier measurement, before the complete system was built, we measured 3,916 packets under the allocations of the four policies. They cover the 2D-skeleton records of all 50 clips and the 3D-pose records of the 48 clips that have them.

Selection2D skeleton, packets vs the Appendix C charge3D pose, sameLargest change of total rate (% of budget)
Whole clip+0.28%+0.11%n/a
MVP and rates-only ridge−0.3% to +6.4%+0.0% to +3.0%1.12%
Oracle+9.9% to +20.3%+14.5% to +28.8%1.97%

The size columns cover allocations in which at least 10 intervals select the stream. One of the 18 such allocations of MVP and rates-only ridge is outside the 2D range. MVP at 1.5 KB per interval sends 20 intervals in 9 runs, at +40.7%. The oracle's selections are the most fragmented. Of its 2D runs, 42.6% are single intervals, against 12.6% for MVP and rates-only ridge.

Effect on the budget. The complete system of Q1 sends 1,767 such packets and bills each at its actual size. With these actual sizes, every operating point ends within budget after re-solving (Q1).

Appendix C of the updated PDF specifies run-based packaging and reports its measured cost.

Q3 (W2, Q3): Learned-MVP results on an independent source or under H.266, the size of MVP's own gain, and the cost of labels, training and deployment

A3: Yes, we now report these learned-MVP results and their practical costs. To answer your question on the learned policy in independent settings, we ran a new experiment: MVP retrained for H.266 anchors and for two independent video sources, Adobe240 and GoPro. In each setting, every value model is trained on some clips or recordings and tested on held-out ones. Regret is the PSNR gap between a model's allocation and the oracle allocation at the same budget. H.266 uses the twelve-action menu behind Fig. 4b. Adobe240 and GoPro use the menus of Fig. 6, with one recording held out at a time (15 and 10 recordings).

SettingItem priorRates-only ridgeMVPMVP − rates-only ridge
HC-Meta, H.264 anchors (Table 3)0.2520.2200.148−0.072 [−0.122, −0.031]
HC-Meta, H.266 anchors0.1950.2010.142−0.059 [−0.088, −0.039]
Adobe2400.1010.1390.058−0.082
GoPro0.1020.0780.103+0.025

Mean regret in dB, lower is better.

On GoPro, every value model is already within 0.11 dB of the oracle.

AuthorsOpenReview ↗

Response to Reviewer MxQN (3/5)

A3 (continued).

Learned-policy curves under H.266. We also compare MetaST + MVP with the state of the art on HC-Meta with H.266 anchors, at the six rates of Table 11. All methods receive the same H.266 anchors.

Rate (KB per interval)BF-STVSR + 𝓛_RAFTAMT-S + DOVEMetaST + MVPMetaST oracle (upper bound)MetaST + MVP − BF-STVSR + 𝓛_RAFT
0.2124.3224.3224.5524.66+0.22 [−0.10, +0.28]
0.3525.4725.5025.8225.87+0.35 [+0.25, +0.46]
0.6526.5326.5126.8027.01+0.27 [−0.02, +0.47]
1.2327.4727.3028.1028.29+0.63 [+0.36, +0.99]
2.0428.0727.7228.8929.18+0.82 [+0.28, +1.28]
3.7328.6428.0330.1430.28+1.50 [+0.91, +2.07]

Average PSNR in dB.

MetaST + MVP is above BF-STVSR + 𝓛_RAFT, the strongest pixel-only method, at every rate, by +0.22 to +1.50 dB. It is also above AMT-S + DOVE at every rate, and within 0.05–0.29 dB of the oracle. The updated Figs. 4 and 6 add the MetaST + MVP curves, with the oracle as the upper bound.

Separating the gains (W2). We agree that the 3.02 dB at the lossless-anchor reference budget is a system gain. It splits into +0.61 dB [+0.30, +0.91] from the MetaST receiver without metadata and +2.42 dB from the edge maps. To address your concern about separating the system gain from MVP's gain, we ran a new experiment: each value model's allocation on HC-Meta, against the best uniform action. The best uniform action is the single action that, used for every interval, gives the highest PSNR within the budget.

Allocation on the 25-action gridAverage PSNR (dB)Gain over the best uniform action (dB)
Best uniform action29.86reference
Item prior30.60+0.74
Rates-only ridge30.63+0.77
MVP30.70+0.84 [+0.62, +1.08]
Oracle (upper bound)30.85+0.99

Means over the 658 budgets of Table 3.

MVP's own increment is the 0.072 dB regret reduction of Table 3. It is concentrated below 10 KB per interval, where MVP cuts mean regret by 43 % (0.278 to 0.159 dB). The updated PDF names the 3.02 dB a system gain and reports the three gains separately.

New labels and retraining (W2). To answer your question on whether new value labels and retraining are required, we ran a new experiment: one component changed at a time on HC-Meta. We measured new values for the changed actions.

  • Edge codec: a generic lossless coder (bit-packing + Zstandard) replaces the contour codec. On the same edge maps it changes only the rates. On raw Canny dense maps it also changes the maps and their values.
  • Second receiver checkpoint: the edge receiver at training step 22,902 is replaced by its ancestor checkpoint at step 13,701.
  • Contour codec tolerance: sparse edge maps are simplified with 3.0 px or 6.0 px instead of 1.5 px.
  • Learned edge extractor: PiDiNet (Su et al., 2021) replaces Canny for both edge-map families.
  • H.266 anchors, on the twelve-action menu.
  • Input degradation: Gaussian blur and ×4 decimation replace bicubic ×4.

"No refit" reuses the models trained on the original setting. "Refit" retrains MVP on the new values. "10 labelled intervals" shifts each changed action's predictions by their mean error on 10 labelled intervals, without retraining.

ChangeMVP, refitMVP, no refitRates-only ridge, no refitItem prior, no refitMVP, 10 labelled intervals
Edge codec, same maps0.1590.168 (+0.010)0.259 (+0.100)0.272 (+0.114)—
Edge codec, raw Canny dense maps0.1620.163 (+0.001)0.257 (+0.095)0.273 (+0.111)—
Second receiver checkpoint0.1700.181 (+0.011)0.271 (+0.101)0.274 (+0.104)0.169 (−0.001)
Contour codec, 3.0 px0.1450.159 (+0.014)0.271 (+0.126)0.275 (+0.130)0.148 (+0.003)
Contour codec, 6.0 px0.1390.226 (+0.088)0.427 (+0.288)0.385 (+0.246)0.142 (+0.004)
Learned edge extractor0.1650.233 (+0.068)0.575 (+0.410)0.403 (+0.238)0.200 (+0.035)
H.266 anchors0.1420.421 (+0.279)0.534 (+0.392)0.278 (+0.137)0.196 (+0.054)
Blur + ×4 decimation0.0981.064 (+0.966)0.884 (+0.786)0.981 (+0.883)0.141 (+0.043)

Mean regret in dB, lower is better. In parentheses, the difference to the refitted MVP.

A small change needs no retraining. After the codec, checkpoint, tolerance and extractor changes, MVP without refitting stays within 0.09 dB of a refit and beats both unrefitted simple models. After the checkpoint and tolerance changes, 10 labelled intervals per changed action bring it within 0.02 dB of a refit. A new source, anchor codec or input degradation needs new value labels and a refit, as used in the H.266, Adobe240 and GoPro results above.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Our Responses for the Area Chair and Reviewers

Dear Area Chair and Reviewers,

We thank the Area Chair for coordinating the review of our paper, and the reviewers for their time and constructive feedback. During the discussion, we did our best to address every concern, and we aimed to answer each question of every reviewer directly. Where a question called for evidence, we ran new experiments throughout the discussion period. The revised PDF adds the realized MVP curves to Figs. 4 and 6 and new Appendices E.2–E.3, G.1–G.4, M.1 and O, with all new text in red. The revised supplementary material adds demo videos of MetaST beside the ground truth, videos at action switches, two complete bitstream containers with their decoder, and the 25-action reproduction package.

Below, we briefly summarize the main points. Where several reviewers raised the same question, we answer it once.

  1. Novelty (CKRL, MxQN). We appreciate the reviewers' questions on novelty. MetaST targets space-time video super-resolution. Missing frames make the temporal density of metadata a decision. Each interval takes a joint action of anchor quality, metadata type and density, and shared anchors couple neighbouring intervals. MVP learns the receiver's response to these actions. MetaSR, which we cite and discuss in Appendix N, is an arXiv preprint on image super-resolution with one metadata type. The NeurIPS 2026 work mentioned in the discussion is contemporaneous with our submission, and at present only its abstract is public.
  2. MVP's own gain versus the 3.0 dB system gain (hoYK, CKRL, MxQN). Following the reviewers' suggestion, we separated the gains. The receiver without metadata adds 0.61 dB over the best pixel-only baseline, and metadata adds 2.42 dB more. Over the 658 budgets, the realized MVP allocation is 0.84 dB [0.62, 1.08] above the best uniform action, against 0.99 dB for the oracle. Figs. 4 and 6 now show MetaST + MVP. With H.264 anchors, it is within 0.07–0.22 dB of the oracle and 0.82–2.72 dB above BF-STVSR + 𝓛_RAFT (Appendix G.1).
  3. Significance of MVP over simple predictors (CKRL, hoYK, MxQN). In a paired bootstrap test, MVP has significantly lower regret than rates-only ridge in eight of nine settings (two-sided p < 0.05). It halves the worst-budget regret, from 0.587 to 0.293 dB, and needs 3.5–8.2 % less rate at equal PSNR.
  4. Independent data and codecs (hoYK, MxQN). Retrained for H.266 anchors, Adobe240 and GoPro, MVP stays within 0.15 dB of the oracle on average (Appendix G.2).
  5. Transfer, labels and retraining (hoYK, CKRL, MxQN). After a change of receiver checkpoint, metadata codec, contour tolerance or edge extractor, MVP without refitting stays within 0.09 dB of a refit. After a change of anchor codec or input degradation, 10 labelled intervals per action close most of the gap (Appendix G.3). We also trained two receivers of other architectures, SeedVR2-3B and Stream-DiffVSR. Without refitting, the paper's MVP gains 1.35 and 1.20 dB over no metadata, above the best uniform action (0.70 and 0.95 dB). A refit takes about 4 CPU-minutes and raises the gains to 1.42 and 1.69 dB (Appendix G.4).
  6. Complete transmitted system (MxQN). As Reviewer MxQN suggested, we ran the complete sender and receiver end to end. Every receiver input is rebuilt from the transmitted bytes, and all 40 operating points lie within budget (Appendices C and E.2).
  7. Temporal consistency at switches (MxQN). Switches have little effect on temporal quality. Over 17,000 boundary frame pairs, a switch of action adds 0.16–0.25 luma levels of warping error, small next to the 0.78 levels that every interval boundary adds (Appendix E.3). The supplement adds three full-length videos with every switch marked, and a qualitative demo of a turning ballet dancer. In this demo, each metadata type plays its own role. 2D skeletons restore the limb trajectories, 3D pose shows which limb is in front, and dense edge maps recover fine texture.
  8. Sender cost (hoYK, MxQN). The complete sender, with pose extraction, edge-map coding, MVP and allocation, runs at 57.8 frames per second on one RTX 5090 with 64 CPU cores. This is 0.52 s per second of 720p, 30 fps video, so it runs in real time (Appendix M.1).
  9. Other tasks (CKRL). In frame interpolation, three transmitted edge maps add 3.14 and 2.77 dB over the same receiver without them (DAVIS-7, UCF101-7). In ×4 video super-resolution within the same byte budget, coded edge maps give the lowest LPIPS and DISTS of all methods (Appendix O).
  10. Reproducibility (MxQN). All receivers use nine-frame intervals. The supplement now contains the 25-action entry point, which reproduces Tables 6 and 8, the 900 × 25 rate/value table and all clip lists.

We hope this summary is helpful. We sincerely thank you all for your time and for your kind consideration of our responses.

Best regards, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.