acceptodds
Under review as a conference paper at ICLR 2027

PAIRNeRV: Patch-Owned Appearance and Innovation for Neural Video Representation

Abstract

Neural video representations offer compact reconstruction, but fitting them separately to each video incurs substantial encoding cost. Feed-forward token-based methods avoid this optimization by inferring video codes for a shared decoder. However, using a global token memory leaves temporal sharing and spatial relevance implicit in the stored representation, while reconstruction repeatedly retrieves content through attention. We propose PAIRNeRV, a feed-forward framework that organizes video state by temporal role and spatial ownership. A clip-wide encoder gathers global context, from which two writers produce appearance codes shared across frames and innovation codes specific to individual frames, both assigned to spatial patches. A shared decoder reconstructs requested patches through direct lookup of their appearance and innovation codes, without global retrieval attention. Appearance and geometry projections are reused across frames. Eight-bit storage complements the design by retaining more code values within a fixed bit budget. Experiments show improved reconstruction over the compared feed-forward baselines under a common video-state budget and transfer to unseen datasets without fine-tuning. Comparisons with global-attention readout demonstrate faster resident-state decoding, while controlled studies examine the roles of state organization, storage allocation, and cross-frame reuse.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Neural Video Representation, Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VhdWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PAIRNeRV is a feed-forward neural video representation that avoids per-video fitting. A clip-wide encoder writes, for each spatial patch, an appearance code shared across frames and frame-specific innovation codes. A shared coordinate MLP reconstructs queried pixels by directly indexing the relevant codes. Q8 is used to store and evaluate the resident state, while training and reconstruction remain in floating point. The design aims to place more state within a fixed bit budget and support patch-addressable decoding. Under the authors' evaluator, Table 7 reports improvements of 1.28, 2.80, 3.03, and 3.37 dB over an expanded Q8 CoANeRV baseline on Train, K400, SthV2, and UCF101, respectively.

Critical strengths
  1. The paper presents a clear and useful systems design. It combines clip-level encoding, patch-owned appearance and innovation state, and direct local lookup into a simple feed-forward interface. The write, storage, and readout paths are well specified, and the paper is unusually careful about the limits of its claims: the two code families are storage roles rather than supervised semantic factors, patch ownership is an output address rather than an encoder mask, and the speed measurements are resident-state rather than end-to-end codec results.

  2. The evaluation is broad for the claimed setting. It includes matched-bit-budget baselines, cross-dataset tests, ablations, multiple seeds for the expanded comparison, high-resolution tiled experiments, and access-latency measurements. The expanded Q8 comparison and the 3.0-3.8x warm resident-state decoding result show that the organization can be practically useful when many local queries are served from an already encoded clip.

Critical weaknesses
  1. The main contribution is a useful combination of existing ideas - temporal role separation, spatially local state, and coordinate-based reconstruction - rather than a clearly new primitive. In Table 2, A/I local differs from All-I local by only +0.4, +0.5, -0.1, and +0.9 dB on Train, K400, SthV2, and UCF101, and this comparison has no paired-seed statistics. The local-reader gain is also measured against one limited global reader. These results do not yet isolate which part of the design causes the much larger cross-system gap.

  2. The cross-method effect size and practical use case remain uncertain. Appendix A.2 reports local CoANeRV results that are 2.03, 2.53, and 4.00 dB below the public F4/F8/F16 values, while the expanded baselines in Tables 3 and 7 use this local implementation. Most evidence concerns F<=16 short clips at 256x256; high-resolution videos are processed as independent tiles, indexed access loses its latency advantage for larger requests, and the resident-state timing excludes encoding, recovery, and I/O. The L2/PSNR evaluation also leaves visible weaknesses on fine texture and small text, and the traditional codec rate-distortion results are substantially better. These limits make the significance beyond a narrow patch-addressable setting unclear.

Questions for authors
  1. Is the local CoANeRV baseline based on the public code or on a reimplementation? Please state the differences in data, learning-rate schedule, checkpoint selection, numerical precision, and evaluation. If possible, could you evaluate a public checkpoint with the same video list and evaluator, and explain how this calibration changes Tables 3 and 7?

  2. Could you provide paired-seed results for A/I local versus All-I local, and compare with a stronger learned global reader or a CoANeRV-style execution control under the same short-clip protocol? This would help separate the effect of the proposed state organization from other architectural differences.

  3. What video lengths and request sizes define the intended use case? Using existing outputs, could you report per-video results and a focused analysis of the difficult text and fine-texture examples, including how quality and latency change as the requested region grows?

3 comments

Thank you for recognizing the value of patch-level access and for distinguishing resident decoding from end-to-end latency. Your question highlights an important distinction: using the public code is not the same as reproducing the published checkpoint.

Q1. Was the baseline reimplemented? No. We trained CoANeRV using its public model and trainer. N384 is the released 384-token architecture, and N1535 is our expansion to 1,535 tokens. We did not obtain weights verified to produce the published scores, so the published column below reports the paper’s numbers, not a run of its weights under our evaluator.

What differed? Both local N384 runs used seed 42, AdamW at 0.0001, mixed-precision training, and the final 150-epoch checkpoint, not the best validation checkpoint. We used the released K400 video lists (10,000 training and 13,732 validation videos), the first eight frames, 256×256 YUV crops and batch size two. Our earlier runs used attention temperature 0.3 and reduced the learning rate at epoch 135. The current public recipe uses 0.4 and epoch 136. We could not verify the published run’s sample order, precision flags or checkpoint-selection rule. Both local columns use FP32 storage and evaluation with TF32 off. We report YUV PSNR (dB) / SSIM using CoANeRV’s ordered-batch convention. On 32 K400 validation videos, the native evaluator and ours reconstructed identical frames from the same local N384 weights. This checks scoring on those inputs, not the published training run.

SplitLocal public-recipe rerunEarlier local runPublished N384
Train31.81 / 0.92632.24 / 0.93233.1 / 0.939
K40028.55 / 0.90129.57 / 0.91432.1 / 0.927
SthV230.34 / 0.91531.25 / 0.92532.8 / 0.935
UCF10129.48 / 0.91230.73 / 0.92733.3 / 0.938

What changes in Tables 3 and 7? We retain both local comparisons. The N384 rerun did not close the K400 gap: 28.55 dB vs. 29.57 earlier and 32.1 published. Our earlier N384 runs averaged 29.49 ± 0.88 dB over three seeds. With the original seed-42 checkpoints, compact PAIRNeRV outscored local N384 by 2.59 dB but was comparable to published N384 (32.16 vs. 32.1 dB). Substituting the rerun would widen the local gap to 3.61 dB. For expanded Q8 models, Table 3 gives a 2.21 dB local gap at seed 42. Submitted Table 7 (revised Table 8) gives 2.80 dB across three seeds (36.886 ± 0.024 vs. 34.089 ± 1.468, sample SD). We retain that local comparison: the N384 rerun provides no basis for adding or subtracting a correction from N1535.

HD baseline correction. Submitted Table 10 labeled the five-encoder/one-writer-block control as native CoANeRV. Revised Table 11 retains that control and adds native CoANeRV (zero encoder blocks, six writer blocks) on the same first 128 frames of each video. The last two columns report pooled YUV PSNR (dB).

Model / state (8 frames)Values per tile/windowUVG7MCL16
CoA-style 5/1 N384, FP3227,64832.3627.59
Native N384, FP3227,64833.5328.75
Native N1535, Q8110,52038.3233.75
PAIRNeRV, Q8109,05640.9636.75

Native N384 is the earlier seed-42 run (K400 29.57), not the 28.55 rerun. With nearly equal Q8 code counts the local gaps narrow to 2.64/3.00 dB (revised Table 11). The earlier RGB check gave the same ordering (2.71/3.07 dB).

At equal FP32 code count (884.7 kbit, Table 1), compact PAIRNeRV scored above local N384 at all three lengths, but below published N384 at four and sixteen. Both local rows use seed 42. Δ = PAIRNeRV minus published CoANeRV (dB, not seed-paired).

FramesPAIRNeRV K400Local N384 K400Published K400Δ TrainΔ K400Δ SthV2Δ UCF101
434.3032.7734.8−0.89−0.50+0.82−0.04
832.1629.5732.1−0.87+0.06+1.44+0.02
1630.2826.9030.9−1.18−0.62+1.20−0.22

The larger Table 1 gains (4.7/5.1/3.5 dB) compare expanded Q8 PAIRNeRV with published compact FP32 N384 at the same bit ceiling, not the same code count. At the main eight-frame allocation, three independent PAIRNeRV runs excluding the Table 1 checkpoint averaged 37.01 ± 0.06 dB under the same FP32 scoring protocol (+2.92 dB over local expanded CoANeRV).

The within-PAIRNeRV result requires no CoANeRV calibration: F8 expansion added 4.73 dB on K400 for seed42 while the measured step changed by 0.057 ms, within trial variation, and each patch remained directly addressable. The submitted F4/F16 steps showed the same qualitative pattern (CoA +18%/+78%; PAIRNeRV −1.7%/+0.9%). Only F8 was independently retimed after model loading. The small F4 decrease is not evidence that expansion accelerates PAIRNeRV.

Q2. Paired tests of temporal roles and readout. With the same local reader, sharing appearance helped K400 but hurt SthV2 in all three runs. All-I keeps the appearance writer but discards its output. We report A/I minus All-I PSNR (dB) for eight-frame Q8 states (872.5 vs. 868.4 kbit), with SD across seeds.

SplitSeed 42Seed 43Seed 44Mean ± SD
Train+0.452+0.539+0.455+0.482 ± 0.050
K400+0.415+0.503+0.417+0.445 ± 0.050
SthV2−0.220−0.037−0.200−0.152 ± 0.100
UCF101+0.784+0.905+0.769+0.819 ± 0.075

The strongest tested global reader nearly matched direct lookup. At the main allocation, we trained each variant separately using the same encoder, writer and MLP architectures and the same schedule. Against fixed attention over separate global banks, lookup gained +0.226 ± 0.113 dB on K400 (Q8, mean ± SD over three paired seeds) and +0.074 dB with FP32 storage. Learned joint attention scored 33.05 dB vs. 37.00 for lookup (Q8, seed 42). Here lookup retained comparable quality without global retrieval. This comparison does not explain the full 2.80 dB cross-system gap. In Table 2, cold decode took 2.75 vs. 3.34 ms (lookup vs. global, seed 43, reuse off).

Q3. When is selective decoding useful? We target repeated requests to already encoded 4–16-frame clips. Within PAIRNeRV, rendering only the touched tiles was fastest for 1, 4 and 16 regions on UVG Beauty and Jockey at 1080p. Warm-file-to-GPU latency (ms) for 64×64 regions:

RegionsIndex patchesRender touched tilesRender full frame
11.781.6822.24
47.433.5622.27
1631.0711.3822.32

Any tiled method can skip untouched tiles (submitted Table 9, now Table 10). Patch ownership also lets us read within a tile.

Across models, within one tile, N384 had the lowest median at every size. PAIRNeRV was faster than N1535 in all three passes at 16, 64 and 256 patches, but not consistently at 1 or 4. We queried central, contiguous 16×16 patches in frame 0 of each eight-frame 256×256 center crop.

Request latency (ms): median (mean) over three passes. Bold: lowest median.

Requested patchesPAIRNeRVCoANeRV N384CoANeRV N1535
11.47 (1.47)0.87 (0.89)1.46 (1.46)
41.59 (1.64)0.88 (0.87)1.50 (1.50)
161.65 (1.64)0.90 (0.92)1.91 (5.86)
642.20 (2.22)1.39 (1.41)3.23 (3.21)
2564.66 (4.87)3.28 (3.29)8.87 (14.62)

Does a larger request change the reconstruction? For the same pixels, PAIRNeRV’s largest absolute difference from full decoding was 8.94e-7 (90 configurations, three passes). Larger regions score different pixels. Mean regional YUV PSNR (dB) on UVG Beauty, HoneyBee and Jockey (best in bold):

Requested patchesPAIRNeRVCoANeRV N384CoANeRV N1535
139.3936.5139.20
440.3437.2739.71
1640.0036.8139.71
6440.3136.5839.75
25640.1334.5539.13

Each A100 pass rotated model order (one worker, FP32, TF32 off) and averaged three video medians (ten trials each). Both slow N1535 passes enter the means: 13.79 ms at 16 patches (pass 1, run third) and 26.14 ms at 256 (pass 2, run second). The timer covered warm-file reads through rendering, not each stage separately.

One patch required 159 logical bytes for PAIRNeRV vs. about 110.7 kB for CoANeRV. Submitted Fig. 6(a) allowed TF32 for PAIRNeRV. With TF32 off for both, warm resident throughput under FP16 autocast was 2.94–3.83× N384’s (revised Fig. 7a).

Fine texture and text remain difficult even with full decoding. Both Q8 models lose grass detail and score about 4 dB lower on broadcast text than on the full clip. YUV PSNR is pooled per target: three 128-frame UVG segments, then two eight-frame K400 validation clips. The last row scores text in the same Broadcast clip.

TargetPAIRNeRV (dB)CoA N1535 (dB)
Beauty38.7738.15
HoneyBee41.3240.93
Jockey42.0638.84
Grass, full clip27.0224.44
Broadcast, full clip26.6525.41
Broadcast, text region22.6021.28

The three UVG gaps average 1.41 dB, with 76% of their sum from Jockey. Revised Table 11 instead pools pixel errors across UVG7 with a different PAIRNeRV allocation (2.64 dB). These cases measure neither legibility nor failure frequency.

These timings exclude cold storage and network access.

Thank you for helping us clarify when reading fewer patch codes actually saves time. We welcome any further questions.

Dear Reviewer VhdW,

As the discussion period approaches its end, we would like to check whether any points in our response need further clarification. Please let us know if there are additional questions we could address.

Thank you for your time and careful review.

Reviewer_Njh3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PAIRNeRV, a feed-forward neural video representation framework that organizes video-specific information into patch-owned, temporally shared appearance codes and frame-specific innovation codes, enabling efficient reconstruction through local lookup without global retrieval attention.

Critical strengths
  1. The proposed framework explicitly separates clip-shared appearance information from frame-specific innovation while assigning both representations to spatial patches.
  2. PAIRNeRV achieves substantial improvements over the reported feed-forward baselines on K400, SthV2, and UCF101. Besides, the paper provides controlled experiments on appearance–innovation decomposition, patch-local readout, storage precision, and cross-frame reuse.
Critical weaknesses
  1. The main components of PAIRNeRV are closely related to existing approaches: DS-NeRV explicitly separates static and dynamic representations, CoordTok employs spatially structured features, and CoANeRV provides the underlying feed-forward token-based framework. Although combining temporal roles with patch ownership is meaningful, the paper does not sufficiently establish which aspects constitute a fundamentally new representation mechanism rather than an architectural integration of existing ideas.
  2. The principal comparison in Table 1 uses an expanded Q8 representation for PAIRNeRV, whereas the published CoANeRV baseline uses FP32 storage with fewer code values. Table 3 demonstrates that increasing code capacity through Q8 storage substantially improves reconstruction quality for both methods. Although the authors provide an expanded Q8 CoANeRV comparison, the remaining performance difference still involves changes in encoder, token organization, and decoder architecture, making it difficult to fully isolate the contribution of the proposed representation.
  3. The effectiveness of patch-owned representations for long-term temporal dependencies, motion across patch boundaries, and larger spatial resolutions remains insufficiently established.
  4. The benefits of the proposed temporal decomposition depend on the data distribution and code allocation, while its robustness across different motion characteristics remains unclear.
Questions for authors
  1. Could the authors further clarify the methodological distinctions between PAIRNeRV and existing approaches such as DS-NeRV, CoordTok, and CoANeRV?
  2. Given that Q8 storage enables a substantially larger number of code values, could the authors provide additional tightly controlled comparisons using identical storage precision, comparable encoder and decoder capacities, and matched training conditions?
  3. How does PAIRNeRV perform on substantially longer video sequences without independently partitioning them into short clips?
  4. Could the authors report the complete per-video encoding time, peak GPU memory usage, and end-to-end encoding–decoding latency compared with CoANeRV and FastNeRV?\
3 comments

Thank you for highlighting how each patch carries both shared appearance and frame-specific innovation. We agree that the overall gain cannot be credited to lookup alone: direct lookup retained comparable quality without a clip-wide search.

W1/Q1. What changes relative to prior work? Our contribution is not the static/dynamic split itself. One encoder pass assigns each patch a clip-shared code and a frame-specific code. The decoder reads that pair without searching a clip-wide token bank. DS-NeRV instead fits static and dynamic codes to each target video. Our codes have storage roles, not supervised motion labels.

CoordTok interpolates features from three planes at each coordinate, then mixes information across queried positions through decoder self-attention. PAIRNeRV indexes two codes at one patch without mixing output patches. Both methods reconstruct the same 128-frame targets, using one temporal window per tile. Entries are RGB PSNR (dB) / representation bpp:

Model / stateUVG7 (1080p)MCL16 (720p)
CoordTok FP1624.64 / 0.08320.73 / 0.083
PAIRNeRV Q829.21 / 0.13125.28 / 0.111

PAIRNeRV scored higher and used 34–58% more representation bits. Training and tiling also differ (K400/256-square vs. CoordTok’s official UCF101/128-square). Table 12 is not an equal-rate test of triplanes.

CoANeRV shares our feed-forward setting but retrieves from a clip-wide token bank for each output block. PAIRNeRV spends storage on a patch's persistent content once, keeps its frame-specific update at the same address, and can reuse the appearance projection across frames. This removes bank-wide retrieval, not all decoding cost.

W2/Q2. Which gains survive a matched comparison? With Q8 storage and matched training, direct lookup retained quality comparable to the strongest tested global reader. This isolates readout, not the full quality gap between PAIRNeRV and CoANeRV.

  1. Vary the reader at fixed state format and MLP architecture. We trained each variant separately with matched encoder, writer and MLP architectures and the same K400 input order (eight frames, seed 42, 150 epochs). K400 PSNR (dB) uses FP32 compute with TF32 off. Q8 states use 872,484 bits.
ReadoutQ8 stateFP32 state
Direct lookup of the patch pair37.0037.08
Fixed attention, separate global banks36.8637.16
Fixed attention, joint global bank35.4135.88
Same fixed joint attention, patch pair only35.6035.69
Learned attention, joint global bank33.0533.44

Restricting joint attention to the two codes of the queried patch still mixes them; direct lookup passes both codes to the MLP without attention. Restricting attention alone did not match lookup. Learned joint attention adds 90,774 parameters. Against the strongest fixed global reader, the local-minus-global Q8 difference was +0.226 ± 0.113 dB across three paired seeds (sample SD). In FP32 the mean gap was +0.074 dB and seed42 favored global. The separate Table 2 timing test gave 2.75 vs. 3.34 ms per cold clip (lookup vs. global, seed 43, batch one, reuse off).

  1. Vary temporal roles at fixed local readout. At the Table 2 eight-frame allocation, A/I minus All-I was +0.445 dB on K400, −0.152 on SthV2 and +0.819 on UCF101 across three paired seeds. All-I retains the appearance writer but discards its output (868,372 vs. 872,484 bits). This near-budget control does not show that A/I always helps. We did not compare an unstructured global writer with the A/I writer under the same learned attention decoder.

  2. Expansion helped both local systems. At equal FP32 code count, PAIRNeRV minus published N384 was −0.50/+0.06/−0.62 dB on K400 at 4/8/16 frames (Table 1). Our F8 N384 scored 29.57 vs. 32.1 published. Table 3 gives local seed-42 PSNRs at the same bit ceiling (compact FP32 to expanded Q8). Step times come from an independent A100 repeat (B2, FP16 autocast).

MethodExpanded valuesK400 PSNR (dB)Step time (ms)
PAIRNeRV109,05632.16 → 36.8928.03 → 28.09
Local CoANeRV110,52029.57 → 34.6888.45 → 133.11

Expansion gained 4.73/5.11 dB. PAIRNeRV’s fixed-width step change was within trial variation. CoANeRV’s step time rose with bank length. Against local N1535 Q8, the capacity allocation led by 2.21 dB at seed 42 and 2.80 dB over three seeds. Repeating the main allocation gave 37.01 vs. 34.09 dB (+2.92) on the same K400 metric. Model sizes are 47.65M/47.45M, not identical decoders. The N384 rerun (28.55 dB) does not calibrate the local N1535 result.

W3/Q3. Can one state cover 128 frames without splitting time? Yes. The 128-frame model encodes all 128 frames in one state per spatial tile, reaching 33.8 dB on UVG7 and 30.0 dB on MCL16. We trained a separate K400 model for each window length before submission. With the same 872,484 bits per tile, shorter windows achieved higher quality (Appendix F.3).

Dataset / PSNR (dB)32 frames64 frames96 frames128 frames
K400 validation32.130.730.329.2
UVG7, 1080p37.035.434.733.8
MCL16, 720p32.231.430.830.0

YUV PSNR uses batch means on K400 and pooled pixel error on UVG/MCL. Bits per frame fall fourfold from 32 to 128 frames. Allocation and eligible inputs also vary. On seven UVG segments, model encoding plus decoding on one A100 reached 45.22, 37.92 and 33.51 frames/s for 64, 96 and 128 frames, respectively (12, 4 and 8 tiles per call, excluding file I/O).

At nearly equal file rates (both round to 0.1045 bpp), one 128-frame Q8 window scored 29.56 dB versus 27.02 dB for two 64-frame windows on 256×256 crops from UVG Beauty, HoneyBee and Jockey (mean-frame RGB PSNR). The 64-frame model used post-training Q4, so this comparison does not isolate window length.

Spatial resolution and motion across boundaries. The models still work tile by tile in space. On full sequences using eight-frame windows, PAIRNeRV scored 42.12 dB over 3,900 native 4K UVG frames and 36.69 dB over 4,115 MCL720 frames (pooled YUV, Appendix F.2). On matched eight-frame windows, PAIRNeRV and our locally trained CoANeRV N1535 scored 40.96 vs. 38.32 dB on UVG7 and 36.75 vs. 33.75 on MCL16 (pooled YUV PSNR at nearly equal Q8 code counts, Table 11). This tests the ablation allocation, whereas timings below use the main model. The larger gaps to our compact N384 include a nearly fourfold difference in code count.

At the internal 16-pixel patch grid, boundary/interior MSE ratios on Beauty, HoneyBee and Jockey were 0.99/1.23/1.19 for PAIRNeRV and 1.01/1.12/1.00 for expanded CoANeRV. PAIRNeRV’s boundary/interior error ratio was higher on HoneyBee and Jockey, not on Beauty. At a temporal clip boundary, PAIRNeRV's frame-change error divided by its within-clip average was 0.97–1.04 across four selected sequences (one transition each). Neither test tracks a moving object across a spatial boundary or demonstrates joint encoding across tiles.

W4. When do temporal roles help? Sharing appearance helped most at low change but hurt K400 and SthV2 at high change. We compared the main eight-frame model with its innovation-only control (seed 42) on identical windows. Columns give the mean PSNR gain (dB) from sharing appearance, grouped by reference luminance change.

DatasetLow changeMedium changeHigh change
K400+1.45+0.54−0.31
SthV2+0.87+0.15−0.31
UCF101+1.90+1.07+0.12

We sampled three windows from each of 256 videos per dataset, excluding five unreadable SthV2 windows. In a separate 512-video K400 intervention, restoring innovation after zeroing it reduced MSE in both regions: in the highest-change video quartile the reductions were 0.027 vs. 0.031 for high- vs. low-change pixels. Innovation is not a motion mask. Neither this intervention nor our allocation sweep tests motion and allocation jointly.

Q4. Actual encoding time, full latency and peak memory. At four tiles per call, PAIRNeRV wrote and recovered a 128-frame 1080p segment in 13.13 s at 2.53 GiB, versus 51.79 s at 7.36 GiB for expanded CoA. Both took about 11 s to save a packet from decoded frames. Recovery explains nearly all the difference. FastNeRV finished sooner than PAIRNeRV. Cells show time to packet / total through recovery and assembly in seconds (three UVG videos, three A100 passes, FP32, TF32 off). Peak GPU allocation excludes host and reserved memory and was the same on all three segments.

Model / stateBeautyHoneyBeeJockeyPeak (GiB)
PAIRNeRV Q810.92/13.1510.89/13.1110.92/13.122.53
CoA N384 FP326.29/21.716.25/21.476.21/21.467.34
CoA N1535 Q810.55/51.8010.54/51.8010.56/51.797.36
FastNeRV FP325.26/8.645.37/8.835.31/8.710.72

FastNeRV uses our local ViT-B weights. Timings exclude source decoding, model loading, scoring and output encoding. Single-tile totals in table order (Table 4) were 38.94/48.73/79.04/37.41 s, or 46.29/56.92/87.19/45.60 s with separately timed lossless input decoding. Quality is not matched.

Thank you for pressing us to separate readout from capacity. We hope you will consider these controls when reassessing our contribution. We welcome any further questions.

Dear Reviewer Njh3,

As the discussion period approaches its end, we would appreciate knowing whether there are any remaining questions or concerns we could clarify. We welcome further discussion and would be glad to address them.

Thank you for your time and valuable feedback.

Reviewer_KucFOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

Implicit Neural Representations for videos rely on costly overfitting to encode each video. Existing methods use hypernetworks to predict per video INR weights or use a hybrid representation where they use compressed features conditioned by coordinate space. The authors here chose the latter path, where they have local and global features for a frame, conditioned using the patch coordinates. The method exploits the temporal redundancy in each GoP with general "appearance writer" features which are enhanced in the final step using local features from the "innovation writer" head. The features themselves are then quantized and stored for compression. Results show that the proposed PairNeRV out performs prior works trained on large scale video datasets like K400, SSv2 and UCF101.

Critical strengths

The paper is easy to follow and experiments are accompanied by fairly rigorous ablations.

  • The model architecture separates temporally redundant appearance features and processes them by polling across the GoP.
  • The method shows gains over the baselines under similar budgets, a marked improvement for shared encoder based INR methods. They achieve substantially quality with fewer epochs of training.
  • Decode speed is faster than CoANeRV and other baselines, but still lag behind traditional codecs.
Critical weaknesses
  • The paper does not clearly mention the time taken to encode a standard 1080p video. We can only estimate that from the current information to be around ~1fps at best. Eventhough INR based encoders are not there yet, it is important to acknowledge and present timings on real workloads.
  • The decoding speed is measured only against a single global-attention baseline, ignoring other NeRV family methods like FastNeRV and TeCoNeRV.
  • More experiments/measurements needed to check how this method scales with resolution. Current method is restricted to 256x256 crops.
Questions for authors
  • Comparisons of decoding time with all baselines is crucial.
3 comments

Thank you for highlighting the way PAIRNeRV shares appearance across frames and for your positive assessment of the ablations. We agree that decoder throughput alone does not tell readers how long a 1080p video takes to encode.

W1. How long does a 1080p encode take? On an A100, PAIRNeRV encoded 128 frames to a file in 10.91 s (11.73 frames/s) with four tiles per call, compared with 28.56 s (4.48 frames/s) with one. Reading that file and reconstructing the frames took a further 2.21 s and 10.38 s, respectively. These are averages over the first 128 decoded frames of three UVG videos: Beauty, HoneyBee and Jockey.

The table compares eight-frame models at a fixed batch of four tiles (B4). Each method uses its native tile size: 270×384 for TeCoNeRV and 256×256 otherwise. Times are in seconds per 128 frames, with B1 as the single-tile reference. File bits per pixel (bpp) and mean segment YUV PSNR are from the single-tile runs. N384 and N1535 give the token counts, while FP32 and Q8 specify 32-bit floating-point and 8-bit storage.

F8 model / stateB1 totalB4 to fileB4 recoveryB4 totalFile bppPSNR (dB)
PAIRNeRV Q838.9410.912.2113.132.11440.72
CoANeRV N384 FP3248.736.2515.3021.552.13635.74
CoANeRV N1535 Q879.0410.5541.2451.792.13739.30
FastNeRV ViT-B FP3237.415.323.418.735.69130.20
TeCoNeRV FP3217.726.893.059.941.15132.32
TeCoNeRV Q8 residual53.7717.245.1422.380.24232.29

Bold marks the best value in each column.

At B4, PAIRNeRV and expanded CoANeRV took similar time from decoded frames to a saved file (10.91 vs. 10.55 s). The difference was in recovery: 2.21 vs. 41.24 s. PAIRNeRV had the highest PSNR (40.72 dB), and residual-coded TeCoNeRV the lowest file rate (0.242 bpp).

Times exclude source-video decompression. A separate single-tile PAIRNeRV run breaks down the time to file: 5.75 s for input, 11.99 s for network encoding and 10.27 s for quantization and writing.

At fixed B4, FastNeRV had the shortest total time (8.73 s). When each method used its best tested batch, TeCoNeRV FP32 was fastest (7.54 s at B2, revised Table 4).

W2 / decoding time. With codes already in GPU memory, FastNeRV was fastest at batch 1 and TeCoNeRV at batches 2–32. PAIRNeRV decoded faster than both CoANeRV variants at every tested batch. The table reports reconstructed frames per second on an A100 using FP32 computation. B is the number of eight-frame clips per call. We cycled eight K400 clips for batches larger than eight. Bold = fastest at that batch.

8-frame model / stored stateB1B2B4B8B16B32
PAIRNeRV Q8426544914614466346804683
CoANeRV N384 FP32366364362361361381
CoANeRV N1535 Q8128127127126127129
FastNeRV ViT-B FP32763219551991199719611943
TeCoNeRV FP3260361108513079141221482715284

PAIRNeRV also outpaced FastNeRV from B2 through B32. FPS counts native output frames (270×384 for TeCoNeRV, 256×256 otherwise). Revised Fig. 7 shows these curves in panel (b) and the GINR and TransINR results in panel (c).

Precision correction. Our submission stated that TF32 was disabled, but the PAIRNeRV benchmark had allowed it. We repeated the mixed-precision test with TF32 off for both models: warm PAIRNeRV throughput was 2.94–3.83× that of CoANeRV N384. The first decode at B1 took 2.21 vs. 4.45 ms (revised Fig. 7a). For cross-frame reuse, revised Fig. 6a reports 0.90 ms per frame with reuse off and 0.83 ms with it on (FP32, geometry already cached). The submitted 1.17/1.06 ms used mixed precision and cold caches.

W3. Does training on 256×256 crops limit output resolution? We used the same 256×256-trained model for tiled HD and 4K inference. PAIRNeRV learns a shared encoder and decoder across a 10,000-video Kinetics-400 subset rather than fitting each video separately. We resize the shorter side and take the same 256×256 center crop across consecutive frames. This lets us train on videos of varied source resolutions without collecting a separate HD training set or paying the cost of full-frame attention at every training step.

For a given clip length, the same model encodes 256×256 tiles at the target video's native resolution. The tile limits the context of one encoder call, not the output frame size.

How does HD quality compare? For this dataset-wide comparison we used the first 128 frames of all seven UVG videos and 16 MCL videos, rather than the three segments timed in W1. All three models used eight-frame windows. At nearly equal Q8 code counts (109,056 vs. 110,520), PAIRNeRV scored 2.64 dB higher on UVG and 3.00 dB higher on MCL than our local expanded CoANeRV. Scores are pooled YUV PSNR (revised Table 11).

Method / stored stateUVG7, 1080p (dB)MCL16, 720p (dB)
CoANeRV N384, FP3233.5328.75
CoANeRV N1535, Q838.3233.75
PAIRNeRV, Q840.9636.75

Both CoANeRV rows use our local runs of the public architecture. Revised Table 11 also retains the separate architectural control that the submission incorrectly labeled as CoANeRV.

What does the higher resolution cost? We timed PAIRNeRV at three resolutions on the Beauty, HoneyBee and Jockey segments from W1 (128 frames each, one tile per call on an A100).

OutputNetwork encode incl. transfer (s)File recovery (s)
256×256 center crop0.410.58
720p resized4.854.73
1080p tiled11.9911.69

Peak GPU allocation was 0.812 GiB, excluding host assembly. The 11.69 s recovery is from the single-tile timing breakdown, not the Table 4 run (10.38 s). With four tiles per call the full packet pipeline took 8.81 s at 720p and 13.13 s at 1080p.

Beyond 1080p, the eight-frame PAIRNeRV model also reached 42.12 dB across all 3,900 native 4K UVG frames without fine-tuning (pre-submission evaluation in revised Appendix F.2). This is a full-sequence result for PAIRNeRV, not a matched 4K comparison.

Thank you for asking us to test real video workloads. We hope you will take the 1080p timings and broader decoder comparison into account when reassessing the paper. We welcome any further questions.

Dear Reviewer KucF,

As the discussion period approaches its end, we would like to ask whether any concerns remain unanswered in our response. We would be happy to provide further clarification and welcome any additional feedback.

Thank you for your time and thoughtful consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author summary of the discussion and revisions

We thank the reviewers for their careful reading of our paper. Their questions led us to test PAIRNeRV on real video workloads, separate the effect of patch-local readout from that of increased storage capacity, and clarify which comparisons rely on locally trained baselines.

KucF: runtime and resolution scaling. On three 128-frame 1080p segments, PAIRNeRV took an average of 10.91 s to encode decoded frames to a file and 2.21 s to recover them on an A100 at four tiles per call. Expanded CoANeRV took 10.55 s and 41.24 s, respectively. The advantage was in recovery, not encoding. FastNeRV was faster overall at this batch size. We also compared decoder throughput across batches 1–32 and evaluated the same 256×256-trained PAIRNeRV models on tiled HD and 4K video without fine-tuning.

Njh3: state organization versus capacity expansion. Our contribution is not the static/dynamic split itself. Assigning a clip-shared code and a frame-specific code to each patch lets the decoder read both directly without searching a global token bank. With matched encoder, writer, and MLP architectures, direct lookup achieved comparable quality to the strongest tested global reader (+0.23 dB on K400 over three paired seeds with 8-bit storage). In a separate eight-frame capacity comparison, expansion improved quality for both systems. PAIRNeRV’s training step remained within measurement variation (28.03 to 28.09 ms), whereas local CoANeRV’s rose from 88.45 to 133.11 ms. We also reported results for joint windows up to 128 frames and showed that sharing appearance did not consistently help on high-change clips.

VhdW: baseline calibration and selective access. We retained CoANeRV’s published scores and reported our public-code rerun separately. At equal FP32 code count, compact PAIRNeRV was close to published CoANeRV on K400 at eight frames and below it at four and sixteen. At nearly equal 8-bit code counts, PAIRNeRV’s 2.80 dB K400 gain over three seeds was relative to our locally trained expanded CoANeRV. We cannot establish that margin against an expanded model reproducing the published baseline quality. The access tests likewise distinguish fewer bytes from lower latency: a one-patch request required 159 logical bytes for PAIRNeRV versus about 110.7 kB for CoANeRV, but compact CoANeRV had lower latency at every tested request size.

As of this comment, none of the three reviewers has posted a follow-up to our responses, so we do not know whether their assessments have changed. We respectfully ask the Area Chair to weigh the additional evidence and the revised manuscript alongside the original reviews when forming the final recommendation. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.