- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes an image compression framework, SPARC, for vision-language-action models. Unlike typical image compression methods that are optimized for preserving signals/pixels, this framework is designed to preserve important visual information or semantics that are required to perform robotic actions. They propose a temporal mask selector and a specialized loss function to allow adaptive bitrate allocation between camera streams and image regions, where masks are generated by leveraging hyperprior features from the current and previous camera observations. The authors conduct evaluations on both simulation benchmarks and real-world environments, where bitrate, success rate, and latency are measured and compared. The proposed method outperforms several conventional codecs (like JPEG) and neural codecs (like MS-ILLM, DCVC-RT).
Critical strengths
- The application scenario is concrete, as described in the introduction: communication-constrained environments, robotic systems that have many cameras, and multi-robot systems. This makes the work meaningful and useful.
- Successfully design a neural codec for VLA tasks. Propose the temporal mask selector for selecting important features and tilted rate loss for stabilizing training.
- Rather sufficient evaluations. The real-world experiments are highly encouraged. Results are good.
Critical weaknesses
- For VLA tasks, the topic to study should be video compression instead of image compression. These tasks can be regarded as extension of typical recognition/perception tasks (classification, detection, segmentation, etc.) in the field of image/video coding for machines (ICM/VCM). For those tasks, studying image compression is sometimes meaningful, as a single frame could provide useful information for simple applications. However, VLA tasks are complicated and highly depend on temporal context to make decisions and perform actions, which indicates a dynamic environment and long-term interactions between robots and environments. This work focuses on image compression, while the introduced temporal mask selector utilizes temporal information. In the evaluations, video codecs are also compared. These indicate obvious contradictory that should be addressed. I understand that studying image compression for a new task is always the first step and can provide solid basis for further video compression. However, if the research topic is image compression, I suggest focusing on a more static part of VLA tasks.
- The novelty and contribution of the method are limited. Selecting and preserving information that are only necessary for given tasks by masking latent features to reduce bitrate is a common approach in the field of ICM/VCM.
- The proposed method does not take text instruction inputs as priors. The temporal mask selector takes hyperprior-derived visual features, temporal embeddings, and camera embeddings as inputs. It does not directly receive the language instruction or the current action objective. Therefore, it produces the same bitrate allocation regardless of whether the robot is instructed to manipulate different objects.
- The evaluation does not include ICM/VCM codecs. The simulation experiments include several other codecs, while the real-world ones simply compares JPEG (even not JPEG2K used in simulation experiments), where more methods should be compared. The end-to-end latency evaluation contains ambiguities and potentially inconsistent assumptions: in table 3, the total encoding and decoding latency of SPARC is 124.7ms, which matches the numbers in table 4, while the results of the two tables are obtained under different bitrate settings, which are 0.0454 for table 3 and 0.15 for table 4.
Questions for authors
Please refer to weaknesses.
4 comments
W4-1 — Comparison with ICM/VCM Codecs
W4-1. The evaluation does not include ICM/VCM codecs.
As requested, we add experiments on ICM/VCM. We select SA-ICM [1] as an ICM baseline and SVC [2] as a VCM baseline, based on the availability of publicly released code and checkpoints, as well as their compatibility with latency requirements for VLA. SPARC outperforms both baselines.
ICM/VCM codec comparison
| Benchmark / Backbone | Method | bpp | Succ. (%) | bpp | Succ. (%) | bpp | Succ. (%) |
|---|---|---|---|---|---|---|---|
| RoboCasa, π0.5 | SVC | — | — | 0.056 | 27.8 | 0.085 | 39.8 |
| RoboCasa, π0.5 | SA-ICM | — | — | — | — | 0.220* | 28.0 |
| RoboCasa, π0.5 | MS-ILLM | 0.015 | 0.2 | 0.059 | 34.3 | 0.098 | 43.9 |
| RoboCasa, π0.5 | SPARC (Ours) | 0.015 | 14.4 | 0.053 | 57.2 | 0.080 | 61.3 |
| RoboCasa, π0 | SVC | — | — | 0.055 | 20.6 | 0.086 | 32.0 |
| RoboCasa, π0 | SA-ICM | — | — | — | — | 0.199* | 25.3 |
| RoboCasa, π0 | MS-ILLM | 0.014 | 4.1 | 0.058 | 19.4 | 0.096 | 24.4 |
| RoboCasa, π0 | SPARC (Ours) | 0.014 | 12.9 | 0.045 | 35.2 | 0.066 | 36.3 |
| RoboCasa, GR00T N1.6 | SVC | — | — | 0.055 | 24.8 | 0.083 | 47.3 |
| RoboCasa, GR00T N1.6 | SA-ICM | — | — | — | — | 0.163* | 50.9 |
| RoboCasa, GR00T N1.6 | MS-ILLM | 0.011 | 0.7 | 0.044 | 40.0 | 0.073 | 48.8 |
| RoboCasa, GR00T N1.6 | SPARC (Ours) | 0.011 | 10.7 | 0.042 | 46.5 | 0.070 | 51.5 |
| VLABench, π0.5 | SVC | — | — | 0.034 | 32.8 | 0.055 | 34.2 |
| VLABench, π0.5 | SA-ICM | — | — | — | — | 0.094* | 32.7 |
| VLABench, π0.5 | MS-ILLM | 0.008 | 6.5 | 0.036 | 35.9 | 0.058 | 34.6 |
| VLABench, π0.5 | SPARC (Ours) | 0.007 | 7.4 | 0.030 | 36.5 | 0.051 | 36.5 |
| LIBERO, π0.5 | SVC | — | — | 0.038 | 92.6 | 0.066 | 96.0 |
| LIBERO, π0.5 | SA-ICM | — | — | — | — | 0.136* | 97.3 |
| LIBERO, π0.5 | MS-ILLM | 0.009 | 25.6 | 0.039 | 97.0 | 0.065 | 97.2 |
| LIBERO, π0.5 | SPARC (Ours) | 0.009 | 44.7 | 0.038 | 97.1 | 0.061 | 97.2 |
*: Note for high bpp
W4-2 — Additional Latency Comparisons
W4-2. The simulation experiments include several other codecs, while the real-world ones simply compare JPEG (even not JPEG2K used in simulation experiments), where more methods should be compared.
We believe the reviewer is referring to the end-to-end latency table (Table 4), as the accuracy results already report results for MS-ILLM other than JPEG (Fig 6).
We have added MS-ILLM and JPEG-2000 to the revised latency table (also see below); the accuracies for JPEG-2000 will be added too as soon as all our tasks are done running.
Finally, we also note that we have also added an accelerated version of SPARC (opt.)—via ONNX & OpenVINO optimizations—which decreases the latency from 124.7ms to 58.0ms.
| Setting | JPEG (q95) | JPEG2000 | MS-ILLM | SPARC | SPARC (opt.) |
|---|---|---|---|---|---|
| Enc/Dec (ms) | 0.5 | 16.7 | 123.6 | 124.7 | 58.0 |
| Varying bandwidth C (N = 1) | |||||
| 250 kbps | — | — | 421.4* | 169.3 | 102.6 |
| 500 kbps | 985.3* | 393.9* | 226.5 | 145.6 | 78.9 |
| 750 kbps | 331.3 | 199.3 | 185.8 | 138.3 | 71.6 |
| 1 Mbps | 199.3 | 137.1 | 168.2 | 134.8 | 68.1 |
| 2 Mbps | 77.1 | 67.7 | 144.5 | 129.7 | 63.0 |
| Varying streams N (C = 100 Mbps) | |||||
| N = 1 | 1.7 | 17.6 | 124.0 | 124.8 | 58.1 |
| N = 128 | 2.9 | 18.0 | 124.1 | 124.8 | 58.1 |
| N = 256 | 29.0 | 19.5 | 124.2 | 124.8 | 58.1 |
| N = 512 | — | — | 124.6 | 124.8 | 58.1 |
| N = 1024 | — | — | — | 124.8 | 58.1 |
| N = 2048 | — | — | — | 124.9 | 58.2 |
| N = 3072 | — | — | — | 125.7 | 59.0 |
| Max. streams N_max | 267 | 376 | 847 | 3,390 | 3,390 |
* : Exceed the deadline |
W4-3 — Consistency of Latency Across Bitrate Settings
W4-3. The end-to-end latency evaluation contains ambiguities and potentially inconsistent assumptions: in table 3, the total encoding and decoding latency of SPARC is 124.7ms, which matches the numbers in table 4, while the results of the two tables are obtained under different bitrate settings, which are 0.0454 for table 3 and 0.15 for table 4
We clarify that the bitrate settings reported in the manuscript are correct: Table 3 reports latency at 0.0454 bpp, while Table 4 uses 0.15 bpp. The bitrate has a negligible effect on latency, with an encoding/decoding latency difference of less than 1 ms between 0.0454 bpp and 0.150 bpp.
This is because most of the latency is spent on neural-net pipelines, which is shared among all bitrates. The only rate-dependent part of the pipeline is entropy coding, accounting for less than 14% of the total encoding latency; its computational cost varies only slightly across different bitrates.
References
[1] Takahiro Shindo, Kein Yamada, Taiju Watanabe, and Hiroshi Watanabe. Image coding for machines with edge information learning using Segment Anything, ICIP, 2024.
[2] Hadi Hadizadeh and Ivan V. Bajić. Learned scalable video coding for humans and machines. EURASIP Journal on Image and Video Processing, 2024.
[3] Nam Le, Honglei Zhang, Francesco Cricri, Ramin Ghaznavi-Youvalari, Hamed Rezazadegan Tavakoli, and Esa Rahtu. Learned image coding for machines: A content-adaptive approach. In IEEE International Conference on Multimedia and Expo, 2021.
[4] Ruoyu Feng, Yunpeng Qi, Jinming Liu, Yixin Gao, Xin Li, Xin Jin, and Zhibo Chen. Diff-ICMH: Harmonizing machine and human vision in image compression with generative prior. NeurIPS 2025.
Kind Reminder
Dear Reviewer Tght,
We hope this message finds you well.
We have carefully addressed the questions and concerns you raised. In summary:
- W1: We clarified that our focus on image compression is a deliberate design choice rather than a stepping stone toward video compression since VLA observations are temporally subsampled, due to action chunking and SPARC incorporates the necessary temporal context within an image compression framework.
- W2: We clarified the novelty of SPARC in determining masks for VLA control, where the mask is conditioned on temporal context so that bit allocation adapts to the task phase and to each camera.
- W3: We respectfully clarified that language instruction conditioning is currently outside the scope of our project under the stringent on-device encoder constraints.
- W4-1: We added ICM/VCM baselines (SA-ICM and SVC) across multiple benchmarks and VLA backbones, where SPARC achieves higher or comparable success rates at substantially lower bitrates.
- W4-2: We added MS-ILLM and JPEG2000 to the latency table, introduced an accelerated version of SPARC (opt.), and added real-world JPEG2000 results (see our general comment).
- W4-3: We clarified that the encoding/decoding latency differs by less than 1 ms between 0.0454 and 0.15 bpp, as most of the computation lies in the bitrate-independent neural network pipeline.
We sincerely appreciate your constructive feedback, which has been very helpful in improving our work. As only about 36 hours remain until the end of the discussion period, we would be happy to provide any further clarification if any concerns remain unresolved. If our responses have addressed your concerns, we would greatly appreciate it if you could consider reflecting this in your evaluation.
Thank you again for your time and thoughtful evaluation of our work.
Best regards,
The Authors
Thank you for your careful review. We address your concerns as follows. Following your suggestion, we added our revised version of the manuscript. Our revision is marked with blue in the manuscript.
W1 — Image vs. Video Compression for VLA Tasks
We believe this is a misunderstanding. We did not use image compression merely as a stepping stone toward video compression—our central argument is that image compression is a better pathway for the communication of visual signals for robotics tasks, especially for manipulation tasks, because of action chunking (i.e., image inputs to VLAs are temporally discontinuous). The design of SPARC revolves around incorporating a necessary amount of temporality inside the image compression framework; thus we don’t believe there is any contradiction.
W2 — Novelty of Temporal Masking for VLA Control
The novelty of SPARC lies in how to determine masks for VLA-based robotic control. The unique characteristics of VLA, unlike downstream tasks in prior ICM/VCM literature [1,2,3,4], is that the model should make sequential predictions with subsampled frames (due to action chunking), often involving long-term interaction between robots and dynamic environments. Thus, the masking strategy should account for the temporal relevance between distant frames. Noticing this unique nature of coding for VLAs, SPARC adopts image compression and conditions the mask on temporal context, so that bit allocation adapts to the task phase and to each camera in multi-camera robots.
W3 — Language Instruction Conditioning
On the one hand, we agree that instruction conditioning could further reduce bitrate by discarding regions irrelevant to a given instruction.
On the other hand, we empirically discovered that it is very challenging to incorporate such information, especially under the stringent encoder-side on-device constraints. In fact, our early versions included language conditioning via FiLM, but it turned out that it does not add statistically significant gain. We are open to the possibility that language conditioning can indeed help, but we suspect that it may require heavier forms of encoding, which falls outside the scope of this project.
Finally, we note that SPARC indeed captures a notion of (instruction-agnostic) task-relevance, as presented in Figure 1.