- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
PAIRNeRV is a feed-forward neural video representation that avoids per-video fitting. A clip-wide encoder writes, for each spatial patch, an appearance code shared across frames and frame-specific innovation codes. A shared coordinate MLP reconstructs queried pixels by directly indexing the relevant codes. Q8 is used to store and evaluate the resident state, while training and reconstruction remain in floating point. The design aims to place more state within a fixed bit budget and support patch-addressable decoding. Under the authors' evaluator, Table 7 reports improvements of 1.28, 2.80, 3.03, and 3.37 dB over an expanded Q8 CoANeRV baseline on Train, K400, SthV2, and UCF101, respectively.
Critical strengths
-
The paper presents a clear and useful systems design. It combines clip-level encoding, patch-owned appearance and innovation state, and direct local lookup into a simple feed-forward interface. The write, storage, and readout paths are well specified, and the paper is unusually careful about the limits of its claims: the two code families are storage roles rather than supervised semantic factors, patch ownership is an output address rather than an encoder mask, and the speed measurements are resident-state rather than end-to-end codec results.
-
The evaluation is broad for the claimed setting. It includes matched-bit-budget baselines, cross-dataset tests, ablations, multiple seeds for the expanded comparison, high-resolution tiled experiments, and access-latency measurements. The expanded Q8 comparison and the 3.0-3.8x warm resident-state decoding result show that the organization can be practically useful when many local queries are served from an already encoded clip.
Critical weaknesses
-
The main contribution is a useful combination of existing ideas - temporal role separation, spatially local state, and coordinate-based reconstruction - rather than a clearly new primitive. In Table 2, A/I local differs from All-I local by only +0.4, +0.5, -0.1, and +0.9 dB on Train, K400, SthV2, and UCF101, and this comparison has no paired-seed statistics. The local-reader gain is also measured against one limited global reader. These results do not yet isolate which part of the design causes the much larger cross-system gap.
-
The cross-method effect size and practical use case remain uncertain. Appendix A.2 reports local CoANeRV results that are 2.03, 2.53, and 4.00 dB below the public F4/F8/F16 values, while the expanded baselines in Tables 3 and 7 use this local implementation. Most evidence concerns F<=16 short clips at 256x256; high-resolution videos are processed as independent tiles, indexed access loses its latency advantage for larger requests, and the resident-state timing excludes encoding, recovery, and I/O. The L2/PSNR evaluation also leaves visible weaknesses on fine texture and small text, and the traditional codec rate-distortion results are substantially better. These limits make the significance beyond a narrow patch-addressable setting unclear.
Questions for authors
-
Is the local CoANeRV baseline based on the public code or on a reimplementation? Please state the differences in data, learning-rate schedule, checkpoint selection, numerical precision, and evaluation. If possible, could you evaluate a public checkpoint with the same video list and evaluator, and explain how this calibration changes Tables 3 and 7?
-
Could you provide paired-seed results for A/I local versus All-I local, and compare with a stronger learned global reader or a CoANeRV-style execution control under the same short-clip protocol? This would help separate the effect of the proposed state organization from other architectural differences.
-
What video lengths and request sizes define the intended use case? Using existing outputs, could you report per-video results and a focused analysis of the difficult text and fine-texture examples, including how quality and latency change as the requested region grows?
3 comments
VhdW · 2
Q2. Paired tests of temporal roles and readout. With the same local reader, sharing appearance helped K400 but hurt SthV2 in all three runs. All-I keeps the appearance writer but discards its output. We report A/I minus All-I PSNR (dB) for eight-frame Q8 states (872.5 vs. 868.4 kbit), with SD across seeds.
| Split | Seed 42 | Seed 43 | Seed 44 | Mean ± SD |
|---|---|---|---|---|
| Train | +0.452 | +0.539 | +0.455 | +0.482 ± 0.050 |
| K400 | +0.415 | +0.503 | +0.417 | +0.445 ± 0.050 |
| SthV2 | −0.220 | −0.037 | −0.200 | −0.152 ± 0.100 |
| UCF101 | +0.784 | +0.905 | +0.769 | +0.819 ± 0.075 |
The strongest tested global reader nearly matched direct lookup. At the main allocation, we trained each variant separately using the same encoder, writer and MLP architectures and the same schedule. Against fixed attention over separate global banks, lookup gained +0.226 ± 0.113 dB on K400 (Q8, mean ± SD over three paired seeds) and +0.074 dB with FP32 storage. Learned joint attention scored 33.05 dB vs. 37.00 for lookup (Q8, seed 42). Here lookup retained comparable quality without global retrieval. This comparison does not explain the full 2.80 dB cross-system gap. In Table 2, cold decode took 2.75 vs. 3.34 ms (lookup vs. global, seed 43, reuse off).
Q3. When is selective decoding useful? We target repeated requests to already encoded 4–16-frame clips. Within PAIRNeRV, rendering only the touched tiles was fastest for 1, 4 and 16 regions on UVG Beauty and Jockey at 1080p. Warm-file-to-GPU latency (ms) for 64×64 regions:
| Regions | Index patches | Render touched tiles | Render full frame |
|---|---|---|---|
| 1 | 1.78 | 1.68 | 22.24 |
| 4 | 7.43 | 3.56 | 22.27 |
| 16 | 31.07 | 11.38 | 22.32 |
Any tiled method can skip untouched tiles (submitted Table 9, now Table 10). Patch ownership also lets us read within a tile.
Across models, within one tile, N384 had the lowest median at every size. PAIRNeRV was faster than N1535 in all three passes at 16, 64 and 256 patches, but not consistently at 1 or 4. We queried central, contiguous 16×16 patches in frame 0 of each eight-frame 256×256 center crop.
Request latency (ms): median (mean) over three passes. Bold: lowest median.
| Requested patches | PAIRNeRV | CoANeRV N384 | CoANeRV N1535 |
|---|---|---|---|
| 1 | 1.47 (1.47) | 0.87 (0.89) | 1.46 (1.46) |
| 4 | 1.59 (1.64) | 0.88 (0.87) | 1.50 (1.50) |
| 16 | 1.65 (1.64) | 0.90 (0.92) | 1.91 (5.86) |
| 64 | 2.20 (2.22) | 1.39 (1.41) | 3.23 (3.21) |
| 256 | 4.66 (4.87) | 3.28 (3.29) | 8.87 (14.62) |
Does a larger request change the reconstruction? For the same pixels, PAIRNeRV’s largest absolute difference from full decoding was 8.94e-7 (90 configurations, three passes). Larger regions score different pixels. Mean regional YUV PSNR (dB) on UVG Beauty, HoneyBee and Jockey (best in bold):
| Requested patches | PAIRNeRV | CoANeRV N384 | CoANeRV N1535 |
|---|---|---|---|
| 1 | 39.39 | 36.51 | 39.20 |
| 4 | 40.34 | 37.27 | 39.71 |
| 16 | 40.00 | 36.81 | 39.71 |
| 64 | 40.31 | 36.58 | 39.75 |
| 256 | 40.13 | 34.55 | 39.13 |
Each A100 pass rotated model order (one worker, FP32, TF32 off) and averaged three video medians (ten trials each). Both slow N1535 passes enter the means: 13.79 ms at 16 patches (pass 1, run third) and 26.14 ms at 256 (pass 2, run second). The timer covered warm-file reads through rendering, not each stage separately.
One patch required 159 logical bytes for PAIRNeRV vs. about 110.7 kB for CoANeRV. Submitted Fig. 6(a) allowed TF32 for PAIRNeRV. With TF32 off for both, warm resident throughput under FP16 autocast was 2.94–3.83× N384’s (revised Fig. 7a).
Fine texture and text remain difficult even with full decoding. Both Q8 models lose grass detail and score about 4 dB lower on broadcast text than on the full clip. YUV PSNR is pooled per target: three 128-frame UVG segments, then two eight-frame K400 validation clips. The last row scores text in the same Broadcast clip.
| Target | PAIRNeRV (dB) | CoA N1535 (dB) |
|---|---|---|
| Beauty | 38.77 | 38.15 |
| HoneyBee | 41.32 | 40.93 |
| Jockey | 42.06 | 38.84 |
| Grass, full clip | 27.02 | 24.44 |
| Broadcast, full clip | 26.65 | 25.41 |
| Broadcast, text region | 22.60 | 21.28 |
The three UVG gaps average 1.41 dB, with 76% of their sum from Jockey. Revised Table 11 instead pools pixel errors across UVG7 with a different PAIRNeRV allocation (2.64 dB). These cases measure neither legibility nor failure frequency.
These timings exclude cold storage and network access.
Thank you for helping us clarify when reading fewer patch codes actually saves time. We welcome any further questions.
Dear Reviewer VhdW,
As the discussion period approaches its end, we would like to check whether any points in our response need further clarification. Please let us know if there are additional questions we could address.
Thank you for your time and careful review.
VhdW · 1
Thank you for recognizing the value of patch-level access and for distinguishing resident decoding from end-to-end latency. Your question highlights an important distinction: using the public code is not the same as reproducing the published checkpoint.
Q1. Was the baseline reimplemented? No. We trained CoANeRV using its public model and trainer. N384 is the released 384-token architecture, and N1535 is our expansion to 1,535 tokens. We did not obtain weights verified to produce the published scores, so the published column below reports the paper’s numbers, not a run of its weights under our evaluator.
What differed? Both local N384 runs used seed 42, AdamW at 0.0001, mixed-precision training, and the final 150-epoch checkpoint, not the best validation checkpoint. We used the released K400 video lists (10,000 training and 13,732 validation videos), the first eight frames, 256×256 YUV crops and batch size two. Our earlier runs used attention temperature 0.3 and reduced the learning rate at epoch 135. The current public recipe uses 0.4 and epoch 136. We could not verify the published run’s sample order, precision flags or checkpoint-selection rule. Both local columns use FP32 storage and evaluation with TF32 off. We report YUV PSNR (dB) / SSIM using CoANeRV’s ordered-batch convention. On 32 K400 validation videos, the native evaluator and ours reconstructed identical frames from the same local N384 weights. This checks scoring on those inputs, not the published training run.
What changes in Tables 3 and 7? We retain both local comparisons. The N384 rerun did not close the K400 gap: 28.55 dB vs. 29.57 earlier and 32.1 published. Our earlier N384 runs averaged 29.49 ± 0.88 dB over three seeds. With the original seed-42 checkpoints, compact PAIRNeRV outscored local N384 by 2.59 dB but was comparable to published N384 (32.16 vs. 32.1 dB). Substituting the rerun would widen the local gap to 3.61 dB. For expanded Q8 models, Table 3 gives a 2.21 dB local gap at seed 42. Submitted Table 7 (revised Table 8) gives 2.80 dB across three seeds (36.886 ± 0.024 vs. 34.089 ± 1.468, sample SD). We retain that local comparison: the N384 rerun provides no basis for adding or subtracting a correction from N1535.
HD baseline correction. Submitted Table 10 labeled the five-encoder/one-writer-block control as native CoANeRV. Revised Table 11 retains that control and adds native CoANeRV (zero encoder blocks, six writer blocks) on the same first 128 frames of each video. The last two columns report pooled YUV PSNR (dB).
Native N384 is the earlier seed-42 run (K400 29.57), not the 28.55 rerun. With nearly equal Q8 code counts the local gaps narrow to 2.64/3.00 dB (revised Table 11). The earlier RGB check gave the same ordering (2.71/3.07 dB).
At equal FP32 code count (884.7 kbit, Table 1), compact PAIRNeRV scored above local N384 at all three lengths, but below published N384 at four and sixteen. Both local rows use seed 42. Δ = PAIRNeRV minus published CoANeRV (dB, not seed-paired).
The larger Table 1 gains (4.7/5.1/3.5 dB) compare expanded Q8 PAIRNeRV with published compact FP32 N384 at the same bit ceiling, not the same code count. At the main eight-frame allocation, three independent PAIRNeRV runs excluding the Table 1 checkpoint averaged 37.01 ± 0.06 dB under the same FP32 scoring protocol (+2.92 dB over local expanded CoANeRV).
The within-PAIRNeRV result requires no CoANeRV calibration: F8 expansion added 4.73 dB on K400 for seed42 while the measured step changed by 0.057 ms, within trial variation, and each patch remained directly addressable. The submitted F4/F16 steps showed the same qualitative pattern (CoA +18%/+78%; PAIRNeRV −1.7%/+0.9%). Only F8 was independently retimed after model loading. The small F4 decrease is not evidence that expansion accelerates PAIRNeRV.