acceptodds
Under review as a conference paper at ICLR 2027

When Does Local Refinement Help? Diagnosing Routing Targets for Sparse Video Frame Interpolation

Abstract

Sparse video frame interpolation allocates a local Expert to a small subset of image regions. Effective allocation requires distinguishing prediction difficulty from the improvement that the available Expert can deliver. We freeze a full-frame predictor and a local residual Expert, then measure Actual Expert Gain: the signed reduction in reconstruction error from each tile-level intervention. An additive formulation separates the target-aware oracle, the feature-conditioned optimum, and the learned policy. On 500 Vimeo90K development examples, 37.59% of 56,000 interventions increase error. At six selected tiles per image, the gain oracle improves PSNR by 0.243 dB, compared with 0.132 dB for endpoint-disagreement routing. On a pre-frozen locked evaluation of 3,281 valid examples, three Gain-Gate seeds show no statistically resolved improvement over Uncertainty at six or twelve selected tiles (paired image-bootstrap 95% confidence intervals include zero). Separately, vectorizing identical halo crops yields a 1.62× local-stage median speedup, while full sparse and dense systems have near-matched CPU latency. These results distinguish realized intervention value, its predictability from deployable features, and its execution cost, which must be evaluated separately when assessing sparse refinement.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video frame interpolation, conditional computation, dynamic routing, sparse refinement, routing targets

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BchyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Sparse VFI: a lightweight "Light Path" predicts a coarse frame; a small "local Expert" refines only k tiles chosen by a Gate. The paper's central move is Actual Expert Gain, G_t = MSE(coarse_t, gt_t) − MSE(expert_t, gt_t) — the signed reduction from actually executing the frozen Expert. Tile interiors are disjoint and the Expert processes crops independently, so frame MSE is additive in G_t and top-k over signed gains is an exact oracle. This separates V_or (full-information value), V_Z (best from input features), and V_s (what a policy realizes).

Critical strengths
  1. The paper identifies a clear and practically meaningful problem in sparse refinement.
  2. The paper is clearly written and appropriately cautious about its conclusions.
Critical weaknesses
  1. The conceptual novelty appears somewhat limited. The central principle—routing according to the realized benefit of a downstream expert rather than an input-side difficulty/confidence proxy—is closely related to learning-to-defer and outcome-aware routing. Under the deliberately disjoint tile setting, Proposition 1 then follows rather directly from the additivity of MSE.
  2. The paper introduces a useful decomposition between full-information oracle value V_{or}, the feature-conditioned optimum V_Z, and the learned policy V_s, but empirically leaves the most important quantity V_Z essentially unresolved. Consequently, the large oracle-to-Gate gap cannot be attributed to intrinsic unpredictability from deployable inputs versus limitations of the chosen features, the very small Gate, or the surrogate objective.
  3. The empirical conclusions are demonstrated on a rather small and highly specific VFI system.
Questions for authors
  1. Jaccard overlap of tiles selected by Uncertainty vs each Gain-Gate seed (dev and locked)?
  2. Held-out Spearman between Gate score and realized G_t?
  3. Can you train an over-capacity predictor on the same Z and report PSNR at k=6/k=12? (Highest-value addition.)
  4. How often does the coarse value lie between endpoints, and what is ρ(m, u)?
  5. Predicted-gain > 0 threshold policy: PSNR, harmful-call rate, average coverage?
  6. Do the latency conclusions hold on GPU and at 1080p?
1 comment
AuthorsOpenReview ↗

Response to Q1–Q6 — Selection diagnostics, capacity, thresholding, and runtime

Thank you for the detailed experimental suggestions. We have carried out the additional selection, capacity, and thresholding analyses. Below, we report these results and clarify the scope of the existing runtime measurements.

Q1–Q2. “Jaccard overlap of tiles selected by Uncertainty vs each Gain-Gate seed (dev and locked)?” “Held-out Spearman between Gate score and realized GtG_t?”

Selection diagnostics. Jaccard measures selected-set overlap with Uncertainty and is averaged per image. Spearman correlates router scores with realized gains across all 112 tiles within each image, using average ranks for ties, and is then averaged across images.

Gate seedDev Jaccard, k=6/12k=6/12Eval Jaccard, k=6/12k=6/12Eval Spearman
20280.7804 / 0.81980.7928 / 0.82030.41026
20290.7803 / 0.82000.7928 / 0.82080.41033
20300.7787 / 0.81950.7927 / 0.82050.41038

Uncertainty's evaluation score–gain Spearman is 0.43740. The original Gates select substantially overlapping tiles but have lower full-ranking correlation with realized gain.

Q3. “Can you train an over-capacity predictor on the same ZZ and report PSNR at k=6/k=12k=6/k=12?”

Capacity control. We used the same 18-channel features, signed-gain labels, training cache, and development-based checkpoint selection. The table reports mean per-image evaluation PSNR gains (dB) at k=6/12k=6/12, with one common initial seed per configuration.

ParametersSmooth L1MSE
5930.14846 / 0.227660.14886 / 0.22718
5,4410.14849 / 0.227170.14876 / 0.22665
70,9130.14786 / 0.226650.14776 / 0.22650

For the largest Gate, three-seed mean ± sample SD is 0.14785 ± 0.00032 / 0.22681 ± 0.00026 dB (Smooth L1) and 0.14774 ± 0.00019 / 0.22637 ± 0.00013 dB (MSE). Uncertainty gives 0.14849/0.22697 dB; the gain oracle gives 0.2586/0.3562 dB. Increasing tested pointwise Gate capacity about 120-fold does not recover the oracle margin under this protocol. These controls use regression alone; the submitted Gate also used ranking. The unrestricted feature-conditioned optimum VZV_Z is not estimated.

Q4. “How often does the coarse value lie between endpoints, and what is ρ(m,u)\rho(m,u)?”

The coarse RGB value lies within the closed endpoint interval at 69.18%/68.08% of scalar channel positions on development/evaluation. Mean within-image tile-level Spearman between motion mm and disagreement uu is 0.9912/0.9908. The cues therefore induce very similar tile rankings here.

Q5. “Predicted-gain > 0 threshold policy: PSNR, harmful-call rate, average coverage?”

Using the width-64 MSE Gate, we select predicted-positive tiles up to cap kk. Uncertainty is matched to the actual number of selected calls within each image.

Split, capMean callsPSNR gain (dB)Harmful callsMatched U gain (dB)
Dev, k=6k=66.00000.1331720.80%0.13225
Dev, k=12k=1211.99200.2021121.46%0.20132
Eval, k=6k=65.99850.1487818.83%0.14852
Eval, k=12k=1211.98870.2266219.96%0.22696

Gains are relative to the common coarse output. Harmful-call rate is the fraction of all selected calls with Gt<0G_t<0. Evaluation coverage is 5.36%/10.70% of 112 tiles, with no zero-call images. Paired differences from matched Uncertainty are +0.00026/-0.00033 dB, with 95% image-bootstrap CIs [-0.00146, 0.00191]/[-0.00189, 0.00121]. The zero threshold almost fills both caps while about 19–20% of calls remain harmful, so this zero-threshold policy provides little useful abstention.

Q6. “Do the latency conclusions hold on GPU and at 1080p?”

The measured scope is eight-thread CPU at 256×448256\times448: output-matched vectorization reduces local-stage median time from 44.31 to 27.37 ms (1.62-fold). The submitted full-system sparse/dense medians are 241.00/229.70 ms. GPU and 1080p timings have not been measured.

Evaluation protocol. These follow-up analyses are exploratory and reuse the originally held-out split after repeated inspection. The paired image-bootstrap intervals quantify image-sampling variation conditional on the evaluated models; seed variability is reported separately.

Thank you for helping us examine the routing results in greater detail. The larger-Gate experiments directly test the capacity question, and the selection and thresholding diagnostics clarify how the fitted policies allocate and stop refinement calls. We will include these results and their experimental settings in the revision, and make the scope of the CPU latency measurements explicit.

Reviewer_WTwZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies which spatial regions should receive local refinement in video frame interpolation. It measures the error reduction delivered by a fixed Expert, uses this signal to train a Gate, and compares learned selection with an oracle at the same budget. The oracle finds useful improvements, but the trained Gate shows no statistically clear mean-PSNR advantage over a simple disagreement heuristic. The paper also separates local batching speedups from end-to-end runtime.

Critical strengths
  1. The paper makes a useful distinction between where predictions are poor and where the available Expert can help. Measuring actual, signed improvements while keeping the models and budget fixed provides a clear way to evaluate routing decisions.

  2. The empirical reporting is careful. The authors separate development and held-out evaluation, acknowledge the learned Gate's limited PSNR gains, report the positive exploratory MSE result, and distinguish local batching improvements from overall speed.

Critical weaknesses
  1. The central distinction between prediction difficulty and Expert benefit lacks a direct comparison with selection by true coarse-prediction MSE. The Teacher oracle is informative, but it measures another model's improvement. It remains unclear how much Expert-specific gain ranking adds beyond accurately identifying poorly predicted regions.

  2. Given prior work on outcome-based model selection [1] and spatial selective refinement [2], much of the contribution rests on the empirical insight. Yet the main routing result comes from one Light–Expert pair; the Expert perturbations do not retrain the Gate. It is therefore unclear how much the finding extends beyond this configuration.

[1] Wittawat Jitkrittum, Neha Gupta, Aditya Krishna Menon, Harikrishna Narasimhan, Ankit Singh Rawat, and Sanjiv Kumar. “When Does Confidence-Based Cascade Deferral Suffice?” Advances in Neural Information Processing Systems, 36, 2023.

[2] Ping Hu, Simon Niklaus, Lu Zhang, Stan Sclaroff, and Kate Saenko. “Video Frame Interpolation with Many-to-Many Splatting and Spatial Selective Refinement.” IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(2): 823–836, 2024.

Questions for authors
  1. Could the authors compare the gain oracle with selecting the k tiles with the largest coarse-prediction MSE against the ground-truth frame? Please keep the Expert and budget fixed and report reconstruction gains and harmful-call rates.

  2. Would the routing result persist for another representative Expert after regenerating gain labels and retraining the Gate? This additional check would help establish the scope of the diagnosis.

1 comment
AuthorsOpenReview ↗

Response to Q1 and Q2 — True-error comparison and Expert-specific retraining

Thank you for suggesting the true-error comparison and Expert-specific Gate retraining. We have completed both comparisons and report the results below.

Q1. “Could the authors compare the gain oracle with selecting the k tiles with the largest coarse-prediction MSE against the ground-truth frame? Please keep the Expert and budget fixed and report reconstruction gains and harmful-call rates.”

Exact-error selection still allocates many calls to harmful corrections. We fixed the Light Path, Expert, crop geometry, images, and exact budget. Both selectors use the target frame: one ranks true coarse MSE; the other ranks the signed MSE reduction delivered by that same Expert.

BudgetTrue-error PSNR gainGain-oracle PSNR gainHarmful calls: true error / gain oracle
k=6k=60.1607 dB0.2586 dB22.93% / 0.010%
k=12k=120.2425 dB0.3562 dB22.08% / 0.091%

These are mean per-image PSNR gains over the common coarse output on 3,281 evaluation images. The paired gain-oracle advantages are 0.0979 dB (95% image-bootstrap CI [0.0951, 0.1009]) and 0.1138 dB ([0.1105, 0.1172]). Mean frame-MSE reductions, in units of 10−510^{-5}, are 6.3165 versus 10.9201 at k=6k=6 and 9.8003 versus 15.2709 at k=12k=12. Thus, highest-error-first selection leaves about 42%/36% of the oracle's MSE reduction unrealized.

Harmful-call rates pool selected calls with negative realized gain. The gain oracle selects 2 harmful calls out of 19,686 at k=6k=6 and 36 out of 39,372 at k=12k=12. An exact-kk constraint can force negative-gain selections when fewer than kk nonnegative-gain tiles are available. These negative signs persist when tile MSE is recomputed in FP64 from the same FP32 model outputs. The allocation gap remains after coarse-error estimation noise is removed.

Q2. “Would the routing result persist for another representative Expert after regenerating gain labels and retraining the Gate? This additional check would help establish the scope of the diagnosis.”

We regenerated signed-gain labels and retrained a Gate for each of two smaller Expert variants. The Light Path and deployable features remain fixed; each Gate and Uncertainty comparison uses the same variant Expert.

ExpertGate seedsRetrained Gate PSNR gain, k=6/12k=6/12Uncertainty PSNR gain, k=6/12k=6/12
Narrower10.0681 / 0.0970 dB0.0685 / 0.0966 dB
Shallower30.0406 / 0.0559 dB0.0395 / 0.0549 dB

For the narrower Expert, paired Gate-minus-Uncertainty evaluation CIs are [-0.00158, 0.00076] and [-0.00049, 0.00141] dB. For the shallower Expert, three-seed mean differences are +0.00106/+0.00096 dB, with nominal paired CIs [0.00018, 0.00195]/[0.00020, 0.00171] dB; development intervals include zero. The new Gates use Smooth L1 regression, whereas the original Gate also used ranking. The shallow Expert has a three-epoch checkpoint.

Evaluation protocol. These follow-up analyses are exploratory and reuse the originally held-out split after repeated inspection. The paired image-bootstrap intervals quantify image-sampling variation conditional on the evaluated models.

Thank you again for these suggestions. The true-error control gives us a direct test of whether accurate error information is sufficient for highest-error-first allocation, while the retraining experiments extend the routing comparison to two additional Expert variants. We will incorporate the true-error control into the main results and document the Expert-specific retraining protocol in the appendix.

Reviewer_cAJuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates when local refinement is beneficial in sparse video frame interpolation.

The authors introduce Actual Expert Gain to measure the improvement or degradation caused by applying a local Expert to individual image regions. They develop a gain-supervised routing approach and analyze the gap between oracle selection and practical routing decisions.

Critical strengths

The paper raises a meaningful question about sparse video frame interpolation: whether regions that are difficult to predict are necessarily the ones that benefit most from local refinement.

The experimental analysis is carefully conducted. The authors keep the Light Path and Expert fixed across routing comparisons, report statistical uncertainty, and distinguish local execution improvements from end-to-end system performance.

Critical weaknesses

The proposed method mainly replaces conventional routing targets with Actual Expert Gain, followed by regression and Top-k selection. Its methodological contribution beyond existing utility-based routing approaches is limited.

The paper carefully measures when refinement helps or hurts, but provides little explanation of why. It does not establish clear conditions or general principles that could guide routing decisions beyond the studied system.

Questions for authors

What is the main methodological contribution of Actual Expert Gain beyond existing utility-based routing and learning-to-defer approaches?

Beyond measuring intervention gains, can the authors identify general conditions under which local refinement is beneficial or harmful? What insights from this study can guide routing decisions in other systems?

1 comment
AuthorsOpenReview ↗

Response to Q1 and Q2 — Contribution and when refinement helps

Thank you for asking us to clarify the contribution and explain when local refinement helps or hurts. We address these questions with the true-error comparison and an explicit condition on the Expert's correction.

Q1. “What is the main methodological contribution of Actual Expert Gain beyond existing utility-based routing and learning-to-defer approaches?”

Our contribution is a controlled, fixed-intervention diagnosis of routing targets for spatial refinement. Building on outcome-aware routing, we hold the coarse predictor, Expert, crop geometry, and budget fixed. This separates coarse prediction difficulty, the value of the available Expert correction, and the improvement recovered by a deployable Gate.

To test whether exact error information suffices for highest-error-first allocation, we compare two target-aware selectors on the same 3,281 images. Both use the target frame and execute the same frozen Expert.

BudgetTrue-error harmful callsGain-oracle minus true-error PSNR95% image-bootstrap CI
k=6k=622.93%+0.0979 dB[0.0951, 0.1009]
k=12k=1222.08%+0.1138 dB[0.1105, 0.1172]

The gaps in mean frame-MSE reduction are 4.6036×10−54.6036\times10^{-5} and 5.4706×10−55.4706\times10^{-5}. Highest-error-first allocation still wastes Expert calls when coarse error is known exactly. The measured gap therefore cannot be attributed merely to inaccurate coarse-error estimation. The comparison quantifies the reconstruction benefit lost under highest-error-first selection relative to gain ranking for the same Expert.

Q2. “Beyond measuring intervention gains, can the authors identify general conditions under which local refinement is beneficial or harmful? What insights from this study can guide routing decisions in other systems?”

An exact correction-level condition explains the sign of gain under squared error. Let dt=Igt,t−Ic,td_t=I_{gt,t}-I_{c,t} be the coarse error and rt=Ie,t−Ic,tr_t=I_{e,t}-I_{c,t} the actual correction after composition and clipping, vectorized over tile tt. With ntn_t scalar RGB entries,

Gt=∥dt∥22−∥dt−rt∥22nt=2⟨dt,rt⟩−∥rt∥22nt.G_t=\frac{\lVert d_t\rVert_2^2-\lVert d_t-r_t\rVert_2^2}{n_t}=\frac{2\langle d_t,r_t\rangle-\lVert r_t\rVert_2^2}{n_t}.

A call helps exactly when 2⟨dt,rt⟩>∥rt∥222\langle d_t,r_t\rangle>\lVert r_t\rVert_2^2. For nonzero error and correction, set at=∥dt∥2a_t=\lVert d_t\rVert_2, bt=∥rt∥2b_t=\lVert r_t\rVert_2, and qt=⟨dt,rt⟩/(atbt)q_t=\langle d_t,r_t\rangle/(a_tb_t). Then Gt>0G_t>0 if and only if qt>0q_t>0 and bt<2atqtb_t<2a_tq_t. At bt=2atqtb_t=2a_tq_t, the gain is zero. Orthogonal or oppositely directed nonzero corrections are harmful, as is a positively aligned correction whose magnitude exceeds this bound. A zero correction gives zero gain.

Error magnitude describes how much is wrong; correction alignment and magnitude determine whether this Expert repairs it. Consistent with this distinction, an average of 8.03 of ten within-image coarse-error-rank bins contain both positive and negative gains; each bin spans a range of error values.

The transferable evaluation principle is to compare a proposed routing signal with the realized value of the actual downstream intervention at the intended budget, report harmful calls alongside net quality, and regenerate gain labels when the Expert changes. This condition uses the target frame for diagnosis; a deployable Gate must predict utility from available inputs.

Evaluation protocol. These follow-up analyses are exploratory and reuse the originally held-out split after repeated inspection. The paired image-bootstrap intervals quantify image-sampling variation conditional on the evaluated models.

Thank you for encouraging us to make both the contribution and its explanation more explicit. The true-error comparison quantifies the allocation mismatch, and the correction-level condition explains how correction alignment and magnitude determine its benefit. We will incorporate this analysis and use the new experimental evidence to sharpen the contribution statement in the revised manuscript. We would be happy to discuss any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.