acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Video Storage for AI Training: Breaking Cross-Frame Dependencies for Efficient Random Access

Abstract

Large-scale AI training increasingly relies on massive video datasets with random temporal sampling patterns. However, existing video storage paradigms are optimized for sequential playback and employ sophisticated inter-frame reference structures to maximize compression efficiency. When applied to the random-sampling workloads of AI training, these paradigms face substantial decoding-efficiency bottlenecks. Workarounds such as independent JPEG frames and short groups of pictures (GOPs) reduce decoding work at the cost of larger storage and transfer volumes. In this paper, we identify two sources of decode amplification: sequential reconstruction of unrelated frames and reconstruction of the reference chains required by sampled targets. We address both sources by co-designing the decoding schedule and video reference structure. Dependency-aware sparse decoding reconstructs only requested frames and their required references, skipping unrelated reconstruction without transcoding the source videos. Optional offline Dependency-Decoupled Random Access (DDRA) encoding further reduces reference reconstruction by restricting each interior frame to two bounding intra-coded anchors. Together, sparse decoding and DDRA retain inter-frame prediction while bounding reconstruction to at most three frames per requested target, independent of the intra period. Experiments on a large-scale robotics corpus show that, at comparable reconstruction quality, our method achieves the loading throughput of LeRobot v3 with less media payload.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
AI Infra, Video Compression

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YoNBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a new approach to video storage and decoding designed for the random frame-access patterns common in AI training. The authors argue that conventional video formats are inefficient for this purpose because retrieving a small number of frames may require decoding many unnecessary ones. Their approach combines dependency-aware sparse decoding, which reconstructs only the requested frames and their required references, with DDRA, a storage method that limits frame dependencies. Across three robotics datasets, the combined method improved loading performance while using substantially less storage and data transfer than JPEG-based storage and LeRobot v3.

Critical strengths
  1. The paper identifies and separates inefficient video decoding into two sources: decoding unrelated frames and decoding required reference chains. The proposed two-part design addresses each source directly, while DDRA provides a formal guarantee that reconstructing a requested frame requires at most two additional anchor frames.
  2. The evaluation covers multiple robotics datasets, access patterns, established baselines, and downstream learning tasks, with video quality aligned across the main comparisons. The results consistently show that the proposed approach improves loading efficiency and reduces storage requirements, while the ablation study confirms that both sparse decoding and DDRA contribute meaningfully to the overall gains.
Critical weaknesses
  1. The paper does not demonstrate end-to-end training speedups. It measures video-loading time rather than the total time needed to deliver tensors to the model or complete training. As the authors acknowledge, faster decoding may have a limited effect when model computation, augmentation, or another part of the pipeline is the main bottleneck.
  2. DDRA requires offline transcoding, but the paper does not fully account for this one-time computational cost or show when it is recovered through repeated training. Moreover, the main experiments use robotics videos and a controlled storage environment, so further testing is needed to confirm that the benefits extend to other video domains, hardware configurations, and real-world training systems.
Questions for authors
  1. Could you report end-to-end training time, GPU utilization, or full-tensor delivery throughput to show whether the improvements in video loading translate into meaningful training speedups?
  2. Could you quantify the offline transcoding cost of DDRA and evaluate the approach on additional video domains and system configurations to demonstrate broader applicability?
2 comments
AuthorsOpenReview ↗

Data Loading Throughput, Transcoding Costs, and Broader Evaluation

Thank you for the detailed review. We report RGB output throughput across four access patterns, offline encoding cost, and results across system configurations and additional video data.

1. RGB output throughput across access patterns (Q1)

RGB output throughput across the four access patterns is reported below in GiB/s. The setup uses 16 CPU cores, 32 workers, batch size eight, and ten requested frames per camera.

RepresentationContiguousRandomStride-2Stride-4
Original H.264 + TorchCodec1.1420.8291.0770.980
Ours native1.4011.0671.3321.221
LeRobot v31.8671.7161.8361.807
JPEG1.2661.2771.3121.310
Ours-DDRA (H.264)2.5702.3692.6412.718
Ours-DDRA (H.265)2.6882.2602.8522.727

The timed path includes OSS reads, index/reference-graph construction, RGB generation, and return of compact validation metadata from workers. Across the four patterns, DDRA H.265 achieves 1.32–1.55x LeRobot's and 1.77–2.17x JPEG's RGB output throughput.

2. Encoding CPU, storage, and bandwidth costs (Q2)

Encoding is a one-time CPU investment; smaller training copies reduce recurring storage and bandwidth requirements. This 1 TB original-H.264 example (10^12 bytes, approximately 267.21 million 720p/30-fps frames at 0.898 Mbit/s) uses Google Cloud Iowa (us-central1) prices: N2 custom CPU at USD 0.03319155/vCPU-hour and regional Standard storage at USD 0.000027397/GiB-hour (approximately USD 0.02/GiB-month). Daily storage uses 24 billable hours.

Estimated encoding CPU work for LeRobot/JPEG is 2,236.0/662.3 CPU-hours per TB; DDRA uses profiled CPU time. Encoding throughput is normalized by accumulated CPU time:

RepresentationFrames/CPU-sCPU ms/frameCPU-h / 1 TBCPU cost (USD)Storage (USD/day)
LeRobot v3 (est.)33.1930.132,236.074.221.23
JPEG (est.)112.078.92662.321.985.09
DDRA H.26438.7025.841,918.063.660.87
DDRA H.26517.2857.884,296.1142.600.51

At a common sample supply rate, LeRobot v3 is the fixed 40 Gbps anchor. Bandwidth requirements scale with the payload ratios in Table 1: LeRobot 40.00, JPEG 165.08, DDRA H.264 28.09, and DDRA H.265 16.50 Gbps. Google Cloud's USD 0.44895/hour for 50 Gbps gives a proportional reference price of USD 0.008979/(Gbps-hour). At 24 hours/day, bandwidth cost is required Gbps * 0.008979 * 24; savings are the difference between each baseline and DDRA.

Payback divides extra encoding CPU cost over each baseline by daily storage plus proportional bandwidth-cost savings:

DDRA / baselineExtra CPU (USD)Storage saved (USD/day)Bandwidth saved (USD/day, est.)Payback (days, est.)
H.264 / LeRobot0.000.372.570.00
H.264 / JPEG41.684.2229.521.24
H.265 / LeRobot68.380.725.0611.81
H.265 / JPEG120.614.5832.023.30

H.264 has USD 10.55 lower upfront CPU cost than LeRobot. H.265's estimated recovery versus LeRobot/JPEG is 11.81/3.30 days; storage-only recovery is 94.36/26.32 days.

3. System configurations and additional video data (Q2)

Appendix C covers 27 access/resource configurations. At 16 CPU cores, increasing workers from 8 to 64 reduces H.265 loading time from 358.9 to 46.7 ms/batch. At 32 workers, increasing CPU allocation from 8 to 32 cores reduces it from 132.6 to 71.1 ms/batch. Appendix D sweeps bandwidth caps over 20–800 MiB/s; at 100 MiB/s, H.265, LeRobot, and JPEG reach 10.79, 4.36, and 1.06 sampled episodes/s.

Appendix B evaluates an additional video-understanding corpus of 21,437 samples and 9,706,740 frames, with 3,800 MVBench questions per setting. Its VLM results are reproduced below; compression factor is original visual bytes divided by variant visual bytes.

RepresentationSettingCompression factorMVBench accuracy (%)
Original--1.00065.87
DDRA (H.264)CRF180.70366.13
DDRA (H.264)CRF282.06766.42
DDRA (H.264)CRF385.51164.89
DDRA (H.264)CRF4815.51063.82
DDRA (H.265)CRF180.86165.89
DDRA (H.265)CRF282.69066.03
DDRA (H.265)CRF388.08465.79
DDRA (H.265)CRF4818.59664.03
JPEGQF800.10965.97

The reviewer appreciated the authors' detailed response.

The paper improves video-loading performance over the baselines by a noticeable margin, and the design is reasonable. What I find it lacking is not sufficiently showing why the identified bottleneck matters (i.e., any end-to-end performance gains). The authors' response addressed the concerns about transcoding cost and generalizability. The end-to-end performance discussion partially addressed by the loading/throughput, but I'm still unsure whether these improvements matter in the bigger picture (the whole training pipeline). I think making this clear could improve the paper. Given that I already gave a high score for technical soundness in the initial review, the response doesn't materially change the overall evaluation, so I maintain my score.

Reviewer_q8Y3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents a novel training algorithm for video data. While the problem is timely and important, the paper suffers from an unclear and potentially incomplete solution.

Critical strengths

S1. Timely and important problem.

Critical weaknesses

Thank you for submitting to ICLR. I think you have quite valuable work, yet it suffers from an unclear and potentially incomplete solution.

W1. The solution includes many unclear points.

a. Eq. 1 is not clear. What are a and b? Are they pixels? What does “inserting” a midpoint mean? Do you mean averaging all the pixels in all the previous frames? Also, how come two “boundary” frames can form a temporal layer? I thought boundary frames are the frames that are at the “boundary” of two different layers?

b. There is a major problem in the definition of D(Q), and it complicates the whole solution, unfortunately.

“A seek-and-decode path reconstructs a coded prefix from a random-access point, potentially including pictures outside D(Q).”

First of all, what exactly is a seek-and-decode path? Second, what exactly is a prefix? I assume seek-and-decode is simply going to the target frame in constant time, and decoding frames backward. And a prefix is all the backward references to the target frame that we want to reconstruct.

Now: D(Q), based on Eq. 2, defines all the references of all the frames in the requested frames. This, as far as I understand from the background, includes all the P, I, and B frames. Then, how come a seek-and-decode path reconstructs pictures outside D(Q)?

If it is the “seek” part that causes the reconstruction of unrelated frames, we need more information on it. How do standard formats do the seeking? In an end-to-end training scenario, if we like to sample several video segments from the dataset, what do H.264/265 or Lance do? I assume there is some level of metadata management that lets you skip the unrelated frames to at least some extent. This then needs to be quantified in the experimental evaluation: yours vs. H.264/265/Lance.

And of course, all these need to be clearly spelled out.

I assume it is the seek part that you are reducing for the rest of my review (for the sparse decoding solution).

c. “Selective decoding can remove the former, but cannot shorten the encoded dependency chains.” —> What is selective decoding? Can you please cite? And, why specifically mention it here, but not other methods? Please either remove it or connect it coherently. Plus, if selective decoding can remove the unrelated reconstructions, then why spend so much time on a problem that is essentially solved?

d. Section 3; intro; why is DDRA optional? Who decides when to use it and why? This is a major problem. It makes your solution incomplete.

e. Section 3.1: “We select a segment S from a preceding random-access anchor through the last target in decoding order, extending it backward if required references are unavailable (Appendix A.4).”

i. What exactly is a segment? A continuous piece of video? How long? Is it fixed? Who decides the length?

ii. What is an anchor? You use this term many times without defining it. It is really confusing. What is a “preceding” “random-access” anchor, then? I can’t relate to either preceding or random-access…

iii. What do you mean by “last” target? How many targets are there in one, say, batch of frames that we sample for training? What is the workflow here?

iv. The appendix is not available, unfortunately. I assume you forgot to add it.

f. Are S and D(Q) the same in Section 3.1?

g. “reconstructing each frame in D(Q) once and skipping pixel reconstruction for the remaining frames” —> So, this is the sparse decoding section, where we reduce the “seek” part. How come there are unnecessary frames in D(Q)? This is against the definition of D(Q)… And, surely you need to reconstruct each frame once. And, what does “remaining ones” refer to here, I mean? It is really, really confusing… There cannot be “remaning ones” after you reconstruct all in D(Q)…

h. Line 232: decoder state alpha is undefined. Its maintenance should be reported in more detail, as right now it is not clear. What does updating picture-order count mean in this context? What does updating “reference management” mean? DPB should also be clearly shown in how it is maintained in the end-to-end algorithm… There can be some data-movement bottlenecks there…

i. Eq. 4: is this how standard storage formats store images, or is it something that you propose? I assume it is your proposal. But, this incurs additional overhead of storing extra B-frames, isn’t it? If so, this has to be clearly spelled out in the introduction so that we understand the trade-off. This is the “key” thing I was trying to refer to in my other comments about what your method contributes. It sacrifices certain storage to make things faster in a smart way. This overhead should clearly be quantified and should show why it's not a big problem (as savings from sparse coding pay off, for example).

j. What are “independently coded anchors” (section 3.2)? Each “new” B-frame that you introduce in Eq. 4? I am not sure.

k. Section 3.2; “Sparse decoding can therefore skip reconstruction of all unrequested interior frames”

So, this is Section 3.2, but it explains Section 3.1. Sparse decoding should have been explained in Section 3.1, which unfortunately was not done.

Now: how do you skip all unrequested interior frames? There are I/B/P frames, right? How do you skip them all? How do you handle P-frames? Do you mathematically reconstruct what they would be and then skip them all? But P-frames DO store the difference between the predicted frame and the original frame… So, it seems impossible to skip all the P-frames… I am not really sure about your solution then, unfortunately…

W2. Experimental evaluation should be explained better.

a. Why do you use Lance to store all the baselines? Is it a standard format? I assume H.264/5 are well-adopted in video storage isn’t it? Lance is also great, but we need technical depth in Lance to be able to interpret the results. We need the same for H.264/5, too, but they are more commonly known (as JPEG is).

b. What does Ours-H.264/5 mean? It has to be clearly described… Do you build on top of H.264/5? How come? What are the trade-offs? Shall we or shall we not build on H.264/5? It seems Ours-H264/5 is generally more successful than Ours… Why? This is a very important problem…

c. Figure 2 shows that your method has a worse speed than LeRobot. Why?

d. Table 1 does not have “random-access mode,” yet Section 4.2 refers to it as “In random-access mode”… Table 1 has an “offline” mode, which is not defined… What is an offline..?

e. How do you obtain each point in Figure 4? How your method trades off compression and success rate is not clear, unfortunately. What is the parameter you tuned in your method to obtain Figure 4?

W3. The presentation can be improved.

a. Figure 1 is missing the original storage-throughput data values. If the video is originally in H. 264 format, for example, it would be the throughput and storage that you would obtain using H.264 without any modification to the dataset.

b. The introduction nicely lays out the contribution, and it is largely fine. But it is missing the key finding/discovery that enabled you to design something novel. Contribution #2, for example, could be replaced with that key finding. What is it at the core that you discovered and that existing work missed, which enabled you to design what you designed? I will talk about this more in my other comments.

c. Alpha value in Figure 1 is not explained; “(smaller is better)” in the caption would be nice.

d. Requests containing multiple frames could be a good item for a “limitations and discussion” section. Concurrency, sharing (e.g., prefix, KV-cache, etc.).

Questions for authors

Q1. How come D(Q) includes unrequested frames? Isn't it against the definition of D(Q)? Do you refer to reducing the "seek" part with sparse decoding?

Q2. How do you reduce the seek part? What is the key point that sparse decoding exploits to be more successful than existing formats?

Q3. Why is DDRA optional? When and how to make the decision to use or not use it?

Q4. What is selective decoding?

Q5. What is an anchor?

Q6. What is Ours-H.264/5 versus Ours?

3 comments
AuthorsOpenReview ↗

Response 1/3: Sparse Decoding, DDRA, and Correctness

Thank you for the detailed review. Our approach combines dependency-aware Sparse Decoding, which removes reconstruction unrelated to a training request, with DDRA encoding, which reduces the reference reconstruction required by its targets. We clarify the method and reproduce the appendix correctness results below.

1. Requested frames, dependency closure, and seeking (Q1/Q2; W1b/f/g)

Q contains the distinct requested frames. D(Q) contains those targets and all their direct and transitive references. S is the contiguous coding-order range selected for parsing and execution. Thus, Q is a subset of D(Q), and D(Q) is a subset of S. The "remaining frames" in Section 3.1 are S - D(Q).

For example, consider coding order I_0, I_4, B_2, B_1, B_3. Suppose B_3 references B_2 and I_4, while B_2 references I_0 and I_4. For Q = {B_3}:

D(Q) = {I_0, I_4, B_2, B_3}

Sequential decoding also reconstructs B_1. Sparse Decoding skips its pixel reconstruction and retains I_0, I_4, and B_2 because they supply reference pixels for B_3. This distinguishes unrequested but necessary references from unrelated frames.

Seeking locates a preceding random-access entry, followed by decoding in coding order toward the targets. A "prefix" is the intervening coded sequence. MP4 sample offsets, timestamps, and synchronization flags support this positioning. Our method additionally parses picture-header references to select pixel reconstruction within the accessed range.

2. Sparse Decoding and DDRA anchors (Q4/Q5; W1c/j/k)

"Selective decoding" referred to our proposed dependency-aware Sparse Decoding in Section 3.1. We have removed the ambiguous phrase from the background. Sparse Decoding selects D(Q); DDRA further reduces D(Q) by redesigning the encoded reference graph.

DDRA anchors are independently intra-coded I pictures. Each interior B picture references its two bounding I anchors. In the example above, DDRA changes the closure of B_3 to {I_0, I_4, B_3}. Each target therefore requires at most three reconstructed pictures, independently of anchor spacing. All unrequested interior pictures can be skipped because their pixels supply no reference to another picture. In original streams, required P/B references remain in D(Q) and undergo normal prediction and residual reconstruction.

3. Segment selection, decoder state, and correctness (W1e/h; Q5)

The benchmark requests ten frames per camera per episode and uses eight episodes per batch. Each camera request is deduplicated, its joint closure is computed, and outputs are restored to the requested order, including duplicates.

S starts at the nearest candidate random-access point at or before the earliest target in presentation order and ends at the target with the largest coding-order position. Right-side references are coded before their dependent targets and are included. Missing references trigger retries from earlier entries, ultimately the stream start; failed graph validation selects sequential decoding. Segment length follows these endpoints.

A random-access entry initializes the decoder, while a DDRA I anchor supplies prediction. Open-GOP reference-availability checks determine whether the segment must extend backward.

The decoder state in Algorithm 1 includes POC, frame numbers, reference identities and markings, DPB entries, and output order. The header pass builds the picture-reference graph from active reference lists across slices. The patched FFmpeg state-only path derives picture identities from headers and prior state, applies H.264 reference marking or HEVC reference-picture-set updates, and retains or releases DPB entries in codec order. Each picture in D(Q) is reconstructed once; pictures in S - D(Q) retain necessary state transitions while skipping pixel reconstruction. Targets are converted to RGB.

Every active reference of a reconstructed picture belongs to the validated closure and precedes its dependent in coding order. Preserving these references and state transitions provides the same reference pixels as full decoding. Appendix A.5 validates 10,000 ten-frame requests on 100 streams, covering first/last frames, open-GOP boundaries, and duplicate or out-of-order indices. All 100,000 returned frames are pixel-identical to full sequential FFmpeg RGB24 decoding, with maximum absolute RGB error 0.

AuthorsOpenReview ↗

Response 2/3: Configurations, Baselines, and Experimental Results

1. Configurations and deployment choices (Q3/Q6; W1d/i; W2b)

The configuration names identify the encoded representation and decoder:

  • Ours native: original dataset bitstreams with Sparse Decoding.
  • Ours-DDRA (H.264): DDRA reference structure encoded with H.264, with Sparse Decoding.
  • Ours-DDRA (H.265): DDRA reference structure encoded with H.265, with Sparse Decoding.

The dataset maintainer chooses the representation before training. Native retains existing videos; DDRA applies when offline conversion is permitted. DDRA re-encodes the existing frames while preserving their count and timestamps. Restricting prediction choices can increase bitrate at a given quality and incurs offline encoding cost.

Our main setting uses intra period 32. In the existing stride-4 sweep, period 8 minimizes payload and period 32 minimizes loading time; moving from 8 to 32 reduces loading time by 14.8% and increases payload by 4.7%. Appendix A.4 specifies B31-capable x264/x265 builds, disabled B-pyramid, medium preset, and CRF16. The custom x264 build raises its B-frame limit to 32.

2. Lance and baseline comparisons (W2a/c)

Section 2.2 now explicitly describes Lance as the database wrapper and access layer in our pipeline, packaging encoded media and episode metadata into records. These payloads can contain H.264/H.265/AV1 videos or JPEG images. Section 4.1 explains that all representations use the same Lance access path on OSS, with their respective media encodings and decoders:

OSS -> Lance episode record -> encoded media -> decoder -> requested RGB frames

Ours native retains the original H.264 reference structure, whereas the evaluated LeRobot configuration uses AV1/GOP2. This explains their different access costs. Averaging equally across four patterns gives:

RepresentationMean load (ms/batch)Payload (MiB/episode)
Original H.264 + TorchCodec320.12514.62
Ours native255.814.62
LeRobot v3 (AV1/GOP2)176.029.45
Ours-DDRA (H.264)123.720.68
Ours-DDRA (H.265)121.712.15

Native provides approximately 1.25x speedup over TorchCodec on identical bitstreams. For transcoded representations, mean RGB PSNR is aligned within 0.18 dB of LeRobot (Appendix A.2). DDRA H.265 achieves approximately 1.45x LeRobot's loading throughput with 58.7% less media payload.

3. Timing and sampling definitions (W2d; W1h)

Loading timing includes OSS reads, index/reference-graph construction, state maintenance, RGB generation, and compact validation-metadata return from workers. The setup uses 16 CPU cores, 32 workers, batch size eight, and one thread per decoder (Appendix A.3).

Random is a sampling pattern: ten frames are selected without replacement from a deterministic 100-frame window, using all available frames as candidates for shorter episodes. Offline transcoding prepares the representation before loading timing begins.

4. Compression and downstream performance in Figure 4 (W2e)

Figure 4 sweeps DDRA CRF over 18, 28, 38, and 48. Each point corresponds to a model trained on the corresponding compressed corpus under a fixed budget within its model family, followed by downstream evaluation. The horizontal axis is C = B_original / B_variant, using visual-payload bytes; the vertical axis is task success or accuracy. Original is PNG for LIBERO and H.264 for the VLM corpus.

LeRobot points use AV1 CRF20/CRF50 and two bitrate-targeted settings; the insets interpolate measured points in log compression factor. These downstream sweeps are separate from the CRF16 quality-aligned loading comparison.

AuthorsOpenReview ↗

Response 3/3: Key Finding, Presentation, and Multi-frame Requests

1. Key finding and introduction revision (W3b)

The revised introduction states the key finding: designing video reference structures for training access patterns can improve random-access efficiency by reducing reference reconstruction while retaining inter-frame compression. DDRA implements this principle, and contribution 2 now presents the finding and its realization.

This connects the two components: dependency-aware Sparse Decoding reconstructs the targets and their required references; DDRA designs the encoded reference structure to reduce the references those targets require.

2. Frame indices and temporal layers in Equation 1 (W1a)

In Eq. 1, a and b are integer frame indices. For endpoints 0 and 8, the prediction levels are:

  • L_0 = {0, 8}
  • L_1 = {4}
  • L_2 = {2, 6}
  • L_3 = {1, 3, 5, 7}

Midpoint arithmetic assigns existing frames to prediction levels. Both interval endpoints belong to the base layer.

3. Original baseline and bubble metric in Figure 1 (W3a/c)

The original H.264 + TorchCodec point requested for Figure 1 is 14.62 MiB/episode and 24.99 sampled episodes/s, derived from Table 1 at batch size eight. Ours native has the same payload and reaches 31.27 episodes/s.

Bubble area encodes sigma = mean_p(p95_p) / mean_p(mean_time_p), averaging across the four access patterns after dataset macro-aggregation. Smaller bubbles indicate less relative tail inflation; the caption now states "lower is better."

4. Multiple-frame requests and sharing (W3d)

K interior targets in one DDRA interval share two anchors, giving K + 2 reconstructed pictures and amplification 1 + 2/K. The implementation merges overlapping execution ranges within a call and initializes state per call. Retaining decoded anchors and indices across calls offers a route to further reuse.

Appendix C already evaluates 27 sampling/resource configurations. At a fixed 16-core allocation, increasing workers from 8 to 64 reduces DDRA H.265 mean loading time from 358.9 to 46.7 ms/batch.

Reviewer_tcPyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies video data loading for training workloads that randomly sample sparse frames or short windows, and argues that playback-oriented video storage wastes decoding work ("decode amplification") in two ways: decoding unrelated frames after a seek, and decoding reference chains required by the requested frames. It proposes (1) dependency-aware sparse decoding, which reconstructs only the requested frames and their required references on existing bitstreams, and (2) DDRA, an optional offline re-encoding in which every interior frame depends only on its two neighboring intra-coded frames, bounding reconstruction to at most three frames per requested frame. On three robotics datasets (3,000 episodes), sparse decoding is 1.25-1.81x faster than PyAV/Decord/TorchCodec, and DDRA reaches 1.45x the loading throughput of LeRobot v3 with 58.7% less payload at aligned PSNR. Downstream VLA/WAM/VLM fine-tuning shows no significant performance drop at moderate compression.

Critical strengths
  1. Clear problem decomposition that maps directly onto the design. Separating decode amplification into unrelated reconstruction and reference reconstruction (Eq. 3) makes explicit what each component addresses, and the cumulative ablation (Table 2: Sequential -> +Sparse -> +DDRA, reporting both loading time and reconstructed frames per episode) confirms that each stage reduces the work it targets.

  2. Careful, practically relevant evaluation at the loader level. Sparse decoding works on existing videos without transcoding and is validated as pixel-exact against full decoding (A.5). The evaluation covers multiple access patterns, loader resources, bandwidth caps, and three downstream model families with quality-aligned comparisons, which makes the reported loader-level gains credible.

Critical weaknesses
  1. The main comparison changes several factors at once, so the source of the headline gain is unclear. DDRA (H.264/H.265) is compared against LeRobotv3, which uses a different codec (AV1) and a different frame structure, and against per-frame JPEG decoded with Pillow. The 1.45x / 58.7% result therefore conflates the proposed frame-dependency structure with the codec switch and the decoder implementation. The paper's own Figure6 suggests this matters: at T_I = 1 (every frame independently coded, i.e., no DDRA structure), loading time is roughly 0.105 s/batch at about 22 MiB/ep. That is already faster and smaller than LeRobot v3 in Table 1 (Stride-4: 175.8 ms, 29.45 MiB/ep) and close to DDRA at T_I = 32 (~0.10 s, ~12 MiB/ep). If so, much of the throughput gain may come from the codec rather than from DDRA, whose remaining benefit would be mainly storage. Same-codec controls at matched quality are needed to support the central claim.

  2. Practical benefit for training is not established. 1) Loading time excludes full tensor transfer, GPU-side work, and the training step, so it is unclear whether the target training jobs are data-loader-bound. 2) GPU-accelerated decoding paths commonly used in large-scale training, and an optimized JPEG decoder instead of Pillow, are not compared, although they could change the relative standing of the baselines. 3) The one-time cost of DDRA transcoding and its storage overhead relative to the original videos are not quantified, It is therefore hard to judge when DDRA, presented as "optional", is worth adopting.

Questions for authors
  1. Could you add same-codec baselines at the PSNR target of Table 1 (e.g., H.265 with every frame independently coded, H.265 with a short GOP matching LeRobot v3's structure, and standard H.265 + sparse decoding), using the same loader? This would isolate the contribution of the DDRA structure from the codec change. What PSNR does each T_I setting in Figure 6 achieve?

  2. What is the end-to-end effect on an actual training run (e.g., step time or GPU utilization for the pi0.5 or OpenWAM setups in Section 4.3) when switching from the original or LeRobot v3 data to DDRA?

  3. How do the results change with GPU-accelerated decoding for the video baselines and an optimized JPEG decoder (e.g., libjpeg-turbo, nvJPEG, or DALI) instead of Pillow?

  4. What is the wall-clock/CPU cost of transcoding the corpus to DDRA, and how many training epochs does it take to amortize? Can you give guidance on when DDRA should be used versus sparse decoding alone?

1 comment
AuthorsOpenReview ↗

Gain Attribution, Training Impact, and Deployment Costs

Thank you for recognizing the problem decomposition, correctness validation, and loader evaluation. Our main practical benefit is lower storage and bandwidth requirements while retaining efficient random access.

Q1. Attribution and the all-intra control.

The 1.45x loading throughput and 58.7% payload reduction compare the evaluated DDRA H.265 and LeRobot AV1/GOP2 configurations. They characterize the complete configurations; they do not isolate the reference structure's contribution. On unchanged original H.264, TorchCodec and Ours native average 320.125 and 255.8 ms/batch across four patterns, respectively, showing approximately 1.25x improvement from our sparse loading implementation.

All-intra's low latency is consistent with the trade-off we target. Independently coded frames remove reference reconstruction; DDRA retains temporal prediction while bounding each target's reconstruction closure to at most three pictures. Its value can therefore be lower payload at comparable access latency. Per-period PSNR is unavailable for Figure 6, so its points cannot establish a quality-matched comparison with Table 1. The requested complete same-codec control matrix remains a limitation of the reported evaluation.

Q2. Training time and resource costs.

We agree that training-time effects are small when bandwidth is ample and input processing overlaps model computation. Similar step times can nevertheless require substantially different storage and transfer resources. At comparable quality, JPEG, LeRobot, and DDRA H.265 use 121.54, 29.45, and 12.15 MiB/episode: DDRA reduces payload by 90.0% versus JPEG and 58.7% versus LeRobot.

Under the existing 100 MiB/s node-wide cap, their loading throughputs are 1.06, 4.36, and 10.79 sampled episodes/s. These measure the CPU loading path; we do not convert them into training-step speedups. The main benefit is lower storage and bandwidth requirements, with monetary savings depending on retention, access policy, and billing.

Q3. GPU decoding and faster JPEG.

GPU decoding is a useful option, with device- and codec-specific support. Migrating an existing CPU DataLoader requires adapting device buffers, tensor delivery, preprocessing, and distributed scheduling. Where compatible hardware is unavailable, additional decoding devices and resource reconfiguration may introduce substantial integration work. Where support exists, additional devices are not necessarily required. Our measurements cover the CPU path and do not establish rankings against GPU decoders. For a fixed representation and access policy, a decoder change does not reduce file size or compressed transfer bytes.

JPEG already has the fastest pure decoding among our compared methods. Its main limitation in the bandwidth-constrained setting is the roughly tenfold payload relative to DDRA H.265. At the 100 MiB/s cap, JPEG's response throughput over the full proxy window reaches 99.3 MiB/s, close to the cap; its mean loading time is 7.528 s/batch versus 0.742 for DDRA H.265. A faster JPEG backend can reduce decoding cost, but cannot remove these fixed-file storage and transfer requirements.

Q4. Conversion cost and when to use DDRA.

The existing RGB-to-MP4 microbenchmark gives 123.72/62.21 frames/s and 25.84/57.88 accumulated CPU-ms/frame for DDRA H.264/H.265. It uses three 720p clips, medium/CRF16 and GOP8, with 2,160 measured frames per codec. It excludes source reads/decoding and does not establish full-corpus conversion cost for the main GOP32 representation.

Relative to the main table's original H.264 mean payload of 14.62 MiB/episode, DDRA H.264 increases size by 41.5%, while DDRA H.265 reduces it by 16.9%. If originals are retained, the DDRA copy is additional storage. The 58.7% saving uses LeRobot as its denominator.

With little training-time improvement under ample bandwidth, we cannot give a supported fixed epoch count for training-time payback. Adoption instead depends on cumulative storage and transfer savings offsetting conversion and additional-copy costs. Sparse decoding alone suits immutable corpora or limited reuse; DDRA merits consideration for persistent, repeatedly accessed corpora where those resource savings justify conversion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.