Thank you for submitting to ICLR. I think you have quite valuable work, yet it suffers from an unclear and potentially incomplete solution.
W1. The solution includes many unclear points.
a. Eq. 1 is not clear. What are a and b? Are they pixels? What does “inserting” a midpoint mean? Do you mean averaging all the pixels in all the previous frames? Also, how come two “boundary” frames can form a temporal layer? I thought boundary frames are the frames that are at the “boundary” of two different layers?
b. There is a major problem in the definition of D(Q), and it complicates the whole solution, unfortunately.
“A seek-and-decode path reconstructs a coded prefix from a random-access point, potentially including pictures outside D(Q).”
First of all, what exactly is a seek-and-decode path? Second, what exactly is a prefix? I assume seek-and-decode is simply going to the target frame in constant time, and decoding frames backward. And a prefix is all the backward references to the target frame that we want to reconstruct.
Now: D(Q), based on Eq. 2, defines all the references of all the frames in the requested frames. This, as far as I understand from the background, includes all the P, I, and B frames. Then, how come a seek-and-decode path reconstructs pictures outside D(Q)?
If it is the “seek” part that causes the reconstruction of unrelated frames, we need more information on it. How do standard formats do the seeking? In an end-to-end training scenario, if we like to sample several video segments from the dataset, what do H.264/265 or Lance do? I assume there is some level of metadata management that lets you skip the unrelated frames to at least some extent. This then needs to be quantified in the experimental evaluation: yours vs. H.264/265/Lance.
And of course, all these need to be clearly spelled out.
I assume it is the seek part that you are reducing for the rest of my review (for the sparse decoding solution).
c. “Selective decoding can remove the former, but cannot shorten the encoded dependency chains.” —> What is selective decoding? Can you please cite? And, why specifically mention it here, but not other methods? Please either remove it or connect it coherently. Plus, if selective decoding can remove the unrelated reconstructions, then why spend so much time on a problem that is essentially solved?
d. Section 3; intro; why is DDRA optional? Who decides when to use it and why? This is a major problem. It makes your solution incomplete.
e. Section 3.1: “We select a segment S from a preceding random-access anchor through the last target in decoding order, extending it backward if required references are unavailable (Appendix A.4).”
i. What exactly is a segment? A continuous piece of video? How long? Is it fixed? Who decides the length?
ii. What is an anchor? You use this term many times without defining it. It is really confusing. What is a “preceding” “random-access” anchor, then? I can’t relate to either preceding or random-access…
iii. What do you mean by “last” target? How many targets are there in one, say, batch of frames that we sample for training? What is the workflow here?
iv. The appendix is not available, unfortunately. I assume you forgot to add it.
f. Are S and D(Q) the same in Section 3.1?
g. “reconstructing each frame in D(Q) once and skipping pixel reconstruction for the remaining frames” —> So, this is the sparse decoding section, where we reduce the “seek” part. How come there are unnecessary frames in D(Q)? This is against the definition of D(Q)… And, surely you need to reconstruct each frame once. And, what does “remaining ones” refer to here, I mean? It is really, really confusing… There cannot be “remaning ones” after you reconstruct all in D(Q)…
h. Line 232: decoder state alpha is undefined. Its maintenance should be reported in more detail, as right now it is not clear. What does updating picture-order count mean in this context? What does updating “reference management” mean? DPB should also be clearly shown in how it is maintained in the end-to-end algorithm… There can be some data-movement bottlenecks there…
i. Eq. 4: is this how standard storage formats store images, or is it something that you propose? I assume it is your proposal. But, this incurs additional overhead of storing extra B-frames, isn’t it? If so, this has to be clearly spelled out in the introduction so that we understand the trade-off. This is the “key” thing I was trying to refer to in my other comments about what your method contributes. It sacrifices certain storage to make things faster in a smart way. This overhead should clearly be quantified and should show why it's not a big problem (as savings from sparse coding pay off, for example).
j. What are “independently coded anchors” (section 3.2)? Each “new” B-frame that you introduce in Eq. 4? I am not sure.
k. Section 3.2; “Sparse decoding can therefore skip reconstruction of all unrequested interior frames”
So, this is Section 3.2, but it explains Section 3.1. Sparse decoding should have been explained in Section 3.1, which unfortunately was not done.
Now: how do you skip all unrequested interior frames? There are I/B/P frames, right? How do you skip them all? How do you handle P-frames? Do you mathematically reconstruct what they would be and then skip them all? But P-frames DO store the difference between the predicted frame and the original frame… So, it seems impossible to skip all the P-frames… I am not really sure about your solution then, unfortunately…
W2. Experimental evaluation should be explained better.
a. Why do you use Lance to store all the baselines? Is it a standard format? I assume H.264/5 are well-adopted in video storage isn’t it? Lance is also great, but we need technical depth in Lance to be able to interpret the results. We need the same for H.264/5, too, but they are more commonly known (as JPEG is).
b. What does Ours-H.264/5 mean? It has to be clearly described… Do you build on top of H.264/5? How come? What are the trade-offs? Shall we or shall we not build on H.264/5? It seems Ours-H264/5 is generally more successful than Ours… Why? This is a very important problem…
c. Figure 2 shows that your method has a worse speed than LeRobot. Why?
d. Table 1 does not have “random-access mode,” yet Section 4.2 refers to it as “In random-access mode”… Table 1 has an “offline” mode, which is not defined… What is an offline..?
e. How do you obtain each point in Figure 4? How your method trades off compression and success rate is not clear, unfortunately. What is the parameter you tuned in your method to obtain Figure 4?
W3. The presentation can be improved.
a. Figure 1 is missing the original storage-throughput data values. If the video is originally in H. 264 format, for example, it would be the throughput and storage that you would obtain using H.264 without any modification to the dataset.
b. The introduction nicely lays out the contribution, and it is largely fine. But it is missing the key finding/discovery that enabled you to design something novel. Contribution #2, for example, could be replaced with that key finding. What is it at the core that you discovered and that existing work missed, which enabled you to design what you designed? I will talk about this more in my other comments.
c. Alpha value in Figure 1 is not explained; “(smaller is better)” in the caption would be nice.
d. Requests containing multiple frames could be a good item for a “limitations and discussion” section. Concurrency, sharing (e.g., prefix, KV-cache, etc.).
Data Loading Throughput, Transcoding Costs, and Broader Evaluation
Thank you for the detailed review. We report RGB output throughput across four access patterns, offline encoding cost, and results across system configurations and additional video data.
1. RGB output throughput across access patterns (Q1)
RGB output throughput across the four access patterns is reported below in GiB/s. The setup uses 16 CPU cores, 32 workers, batch size eight, and ten requested frames per camera.
The timed path includes OSS reads, index/reference-graph construction, RGB generation, and return of compact validation metadata from workers. Across the four patterns, DDRA H.265 achieves 1.32–1.55x LeRobot's and 1.77–2.17x JPEG's RGB output throughput.
2. Encoding CPU, storage, and bandwidth costs (Q2)
Encoding is a one-time CPU investment; smaller training copies reduce recurring storage and bandwidth requirements. This 1 TB original-H.264 example (10^12 bytes, approximately 267.21 million 720p/30-fps frames at 0.898 Mbit/s) uses Google Cloud Iowa (us-central1) prices: N2 custom CPU at USD 0.03319155/vCPU-hour and regional Standard storage at USD 0.000027397/GiB-hour (approximately USD 0.02/GiB-month). Daily storage uses 24 billable hours.
Estimated encoding CPU work for LeRobot/JPEG is 2,236.0/662.3 CPU-hours per TB; DDRA uses profiled CPU time. Encoding throughput is normalized by accumulated CPU time:
At a common sample supply rate, LeRobot v3 is the fixed 40 Gbps anchor. Bandwidth requirements scale with the payload ratios in Table 1: LeRobot 40.00, JPEG 165.08, DDRA H.264 28.09, and DDRA H.265 16.50 Gbps. Google Cloud's USD 0.44895/hour for 50 Gbps gives a proportional reference price of USD 0.008979/(Gbps-hour). At 24 hours/day, bandwidth cost is required Gbps * 0.008979 * 24; savings are the difference between each baseline and DDRA.
Payback divides extra encoding CPU cost over each baseline by daily storage plus proportional bandwidth-cost savings:
H.264 has USD 10.55 lower upfront CPU cost than LeRobot. H.265's estimated recovery versus LeRobot/JPEG is 11.81/3.30 days; storage-only recovery is 94.36/26.32 days.
3. System configurations and additional video data (Q2)
Appendix C covers 27 access/resource configurations. At 16 CPU cores, increasing workers from 8 to 64 reduces H.265 loading time from 358.9 to 46.7 ms/batch. At 32 workers, increasing CPU allocation from 8 to 32 cores reduces it from 132.6 to 71.1 ms/batch. Appendix D sweeps bandwidth caps over 20–800 MiB/s; at 100 MiB/s, H.265, LeRobot, and JPEG reach 10.79, 4.36, and 1.06 sampled episodes/s.
Appendix B evaluates an additional video-understanding corpus of 21,437 samples and 9,706,740 frames, with 3,800 MVBench questions per setting. Its VLM results are reproduced below; compression factor is original visual bytes divided by variant visual bytes.