acceptodds
Under review as a conference paper at ICLR 2027

SEAOTTER: Sensor Embedded Autoencoding with One-Time Transcode for Efficient Reconstruction

Abstract

In robotics, wearable, and remote-sensing systems, vast amounts of visual data are easily captured at high resolution using low-cost, low-power hardware. Yet, limited bandwidth and on-device compute resources prevent full utilization when transmitted via conventional codecs like JPEG/MPEG. Newer codecs, like AV1/AVIF, improve the rate-distortion trade-off, but demand far more resources for encoding, impractical without custom ASICs. Recent asymmetric autoencoders deliver high quality under extreme power and bandwidth constraints, but add prohibitive decoding cost and use bespoke formats that ignore decades of infrastructure built around standards like JPEG. To address these limitations, we introduce a split-inference compression framework based on a Sensor Embedded Autoencoder paired with a One-Time Transcode for Efficient Reconstruction (SEAOTTER). Because the sensor, cloud, and consumer stages face very different power and bandwidth budgets, SEAOTTER combines the compactness of a learned latent with the broad usability of a standard JPEG file. Since naive transcoding degrades performance, we propose a learnable JPEG color and quantization transform that enables increased accuracy for global, dense, and vision-language-based perception. Using SEAOTTER, we train both general-purpose and task-aware transcoding pipelines for a pre-trained, frozen encoder. At a compression ratio of 200:1 and compared to AVIF, we observe 7x faster encoding, 3.5x faster decoding, and +8% ImageNet top-1 accuracy, while retaining compatibility with JPEG infrastructure. Our code is available at URL REDACTED FOR ANONYMITY.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
image compression, JPEG proxy, representation learning, autoencoding, transcoding, split computing, cloud robotics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4YndOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes a framework for optimizing preprocessing, quantization tables, and postprocessing within the JPEG pipeline. Targeting scenarios with limited decoder-side resources, it introduces an asymmetric codec architecture for the uplink and downlink stages. Through end-to-end fine-tuning, the framework jointly optimizes rate-distortion performance and downstream task performance. The experiments demonstrate advantages in rate-distortion curves and classification accuracy after joint optimization.

Critical strengths

The learning framework is specifically designed to address the computational imbalance between edge devices and the cloud, as well as differing downstream task requirements.

Critical weaknesses
  1. The proposed techniques are already supported by well-established methods in prior work, including Differentiable JPEG [1] and JPNeO [2]. Related learned image compression technologies also have a standardization framework in JPEG AI [3] that provides practical guidance. These precedents limit the novelty of presenting such techniques as core contributions.
  2. Despite computational imbalance being a central motivation, the paper does not sufficiently explore acceleration methods on edge devices or the challenges arising from integrating them with existing optimization techniques.

[1] Differentiable JPEG: The Devil Is in the Details. WACV 2024.

[2] JPEG Processing Neural Operator for Backward-Compatible Coding. ICCV 2025.

[3] JPEG AI. https://jpeg.org/jpegai/documentation.html

Questions for authors
  1. SEAOTTER-FT exhibits a substantial drop in PSNR. How does the approach address the potential loss of decoder generality caused by adaptation to a specific task? Could the authors evaluate a broader range of tasks to characterize when this adaptation improves performance and when it degrades performance?

  2. What is the individual contribution of each learnable component? How does performance change when the cloud-side FRAPPE decoder is kept frozen rather than fine-tuned?

  3. Does the proposed approach retain its performance advantages over a learned JPEG baseline fine-tuned for the downstream task, as well as the approaches in [1] and [2]?

[1] Differentiable JPEG: The Devil Is in the Details. WACV 2024.

[2] JPEG Processing Neural Operator for Backward-Compatible Coding. ICCV 2025.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 4Ynd: The review does not describe the submitted manuscript

This review does not describe the submitted manuscript. Its summary omits the system the paper proposes and attributes to it a design it does not contain. Its first weakness cites three works, none of which performs the function it is cited for. Its second weakness names no missing experiment. Two of its three questions are answered by tables in the submitted manuscript. It assigns soundness 1 and identifies no error in any derivation, experiment, or reported number. We ask the Area Chair to weigh the review according to its correspondence with the manuscript.

Paper summary. Each sentence is inaccurate. (1) The paper proposes a three-stage system: a frozen autoencoder on the sensor, whose encoder costs 10–100 MACs per pixel; a neural decoder in the cloud; and a one-time transcode of the decoded image into a standard JPEG file. The first two stages do not appear in the summary. The sensor never runs JPEG. (2) The constraint that defines the design is the sensor-side encoder, not "limited decoder-side resources." (3) There is no "downlink" stage; the word does not occur in the manuscript. (4) Rate–distortion and task performance are not "jointly optimized." SEAOTTER-ZS sees no task, label, or downstream model, and the central result is that the transcode raises accuracy in that variant (+4.03 points of ImageNet top-1 at n=12n=12). (5) The experiments cover classification, segmentation, and SigLIP-2 zero-shot classification, with encoding and decoding throughput for every method.

Weakness 1. None of the three cited works contains a sensor-side encoder, a transcode, or a measurement of downstream accuracy for a compression system. None contains a JPEG codec whose color transform, quantization matrices, and rate proxy are learned de novo.

  • Differentiable JPEG [1] is a differentiable approximation of standard JPEG with the ITU tables. It trains no quantization table or color transform and reports no rate–distortion or rate–accuracy result. Its authors state: "the scope of this paper is to model JPEG encoding-decoding in a differentiable manner and not to model the JPEG file size." Training a codec requires that file-size estimate. The manuscript does not claim differentiable rounding as a contribution; it identifies the sandwiched codec of Guleryuz et al. as the closest prior art, with three stated differences.
  • JPNeO [2] places neural networks at both ends of JPEG. By its authors' table, its encoder costs 1,150,000 MACs per pixel, more than 10,000 times SEAOTTER's sensor-side encoder, and its 2,339,000-MAC decoder replaces the standard JPEG decoder. Its quantization matrix starts from the standard table and is trained against ∥1/Q∥1\lVert 1/Q \rVert_1, which is not an estimate of file size. It reports PSNR and SSIM only.
  • JPEG AI [3] is a separate standard with its own bitstream. A JPEG decoder cannot read a JPEG AI file. SEAOTTER's premise is that the stored artifact is an ITU T.81 file readable by existing decoders.

Weakness 2. This weakness names no acceleration method, optimization technique, or experiment whose absence affects a claim. The manuscript measures sensor-side encoding throughput for every method at every operating point: on ImageNet at n=12n=12, 178 MPx/s on one CPU, against 5.5 MPx/s for AVIF and 25.7 MPx/s for AVIF at maximum speed. The revision adds measurements on a Cortex-M55 microcontroller (Apollo510, 249 MHz), where the encoder at n=12n=12 runs at 6.18 MPx/s and AVIF at the matched rate at 0.0139 MPx/s (0.199 MPx/s at maximum speed).

Question 1. The drop in PSNR is the stated design of SEAOTTER-FT, which "trades pixel-domain reconstruction PSNR for higher downstream accuracy after the transcode." Generality is retained because the encoder is frozen: "the same transmitted latent decodes to different RGB outputs depending on the chosen snapshot." SEAOTTER-ZS is the general-purpose variant. The revision adds a cross-task evaluation of every fine-tuned variant: in 10 of 30 cross-task cells, all at n∈{6,9,12}n \in \{6, 9, 12\}, a variant tuned for another task exceeds SEAOTTER-ZS with a smaller file.

Question 2. The submitted appendix reports the frozen-decoder configurations at all five rates on all three tasks. ImageNet top-1 at n=12n=12 and 0.109 transmit bpp: FRAPPE 56.22%, FRAPPE + ITU JPEG 60.83%, and SEAOTTER-ZS 60.25%, all with the decoder frozen; SEAOTTER-FT 69.02%.

Question 3. No JPEG codec, however its tables are obtained, operates at SEAOTTER's rates. A baseline JPEG file of these images cannot be smaller than 0.18–0.25 bpp, and SEAOTTER transmits 0.010–0.142 bpp from n=3n=3 to n=12n=12. At n=15n=15, DeepN-JPEG, a quantization table selected for ImageNet accuracy, reaches 72.11% ImageNet top-1 and 33.79% ADE20K mIoU with 4:2:0 subsampling; SEAOTTER-ZS, with no task training, reaches 76.43% and 39.42%. The approach in [1] reproduces standard JPEG, which is a baseline in the manuscript. The approach in [2] cannot run on the sensor and reports no downstream result.

Thanks for the rebuttal. While some wording in my summary could be more precise, This review does not describe the submitted manuscript is not an appropriate response. Using "downlink" is not inappropriate, since the author uses "uplink" in the paper. The manuscript adopts the existing FRAPPE encoder on the sensor side and explicitly states that All SEAOTTER variants inherit the same frozen FRAPPE encoder. Therefore, I focus on the proposed learning components rather than treating the inherited sensor-side design as a new contribution. Most of the optimization is inherently on the server-side pre-processing and the quantization tables of JPEG, adapted for FRAPPE/WaLLoC. Since sensor-side encoding efficiency is inherited from FRAPPE, the major contribution of the system is therefore adapting FRAPPE/WaLLoC to the imbalanced nature of the server and app side.

The rebuttal shall focus on the improvement of the manuscript.

For Q2, the authors did not read it carefully. The only occasion that the framework finetunes FRAPPE decoder is in SEAOTTER-FT, and the ablation does not cover it (only removing the FRAPPE finetuning part in SEAOTTER-FT). This is an experiment that could strengthen the paper's technical claim of application-specific signal enhancement with specialized GS\mathcal G_S.

Since the authors emphasize the system contribution, the manuscript should provide stronger evidence of system implementation and overall optimization. It should assess compatibility with neural-network acceleration technology like quantization and software-acceleration technology like parallel execution, and explain how these techniques affect the claimed benefits. The current Wi-Fi, 5G, and BLE deployment suitability is based on assumed bandwidth thresholds, and the encoding throughput is measured only on a server-class CPU. End-to-End latency performance shall be measured. Both the proposed method and the baselines should use appropriately optimized implementations and inference frameworks, such as TensorRT where suitable for the target hardware, to establish a deployment-aligned comparison.

Supplementing the discussion of the variants is also helpful, such as when the server or the app has more or less computational capability, what changes to the system should be made.

AuthorsOpenReview ↗

Response to Reviewer 4Ynd's comment of November 11

Reviewer 4Ynd's comment of November 11 defends one word of the review's summary, restates Question 2 as a different question, and says a measurement is missing that our response of November 10 reports. It defends none of the three citations behind the review's novelty objection, and it identifies no error in any derivation, experiment, or reported number, which a soundness score of 1 requires.

The review's summary of the paper. Our response listed five inaccuracies in it. The comment contests one:

Using "downlink" is not inappropriate, since the author uses "uplink" in the paper.

We see how "downlink" follows from "uplink," and the manuscript's terminology was chosen to prevent that reading. It names three parties: sensor, cloud, and consumer. "Uplink" is the link from the sensor to the cloud. The consumer is a different device from the sensor: any machine that later reads the stored JPEG file. "Downlink" would imply that data returns to the sensor, which it never does, so the manuscript uses "consumer" throughout and does not use "downlink." The remaining inaccuracies are not matters of wording, and the comment does not address them: the summary omits the sensor encoder and the cloud decoder, places the constraint on the wrong side of the system, describes joint optimization that the zero-shot variant does not perform, and omits segmentation, zero-shot recognition, and throughput from the experiments. The comment then gives a new summary, in which the contribution is "adapting FRAPPE/WaLLoC to the imbalanced nature of the server and app side." That is the contribution the manuscript states. The ratings were assigned under the first summary.

Novelty (Weakness 1). The review states that the proposed techniques exist in Differentiable JPEG, JPNeO, and JPEG AI. Our response showed that the first trains no quantization table and declines to model file size, that the second places a network of 1,150,000 MACs per pixel at the encoder, and that the third is a separate bitstream that no JPEG decoder reads. The comment mentions none of the three, which leaves the objection without support.

Frozen decoder (Question 2).

For Q2, the authors did not read it carefully.

Question 2 reads, in full: "What is the individual contribution of each learnable component? How does performance change when the cloud-side FRAPPE decoder is kept frozen rather than fine-tuned?" Our response reported the three configurations with a frozen decoder beside the fine-tuned one, at every rate and on every task, from tables in the submitted manuscript. The comment now describes a different experiment, in which the decoder is frozen and only the JPEG components are trained on the task. Question 2 did not ask for it. The contribution of each component is nonetheless measured. The revision adds a decoder fine-tuned on the task and stored as standard ITU T.81 JPEG, with no learned JPEG component. At the 12-channel operating point and comparable file size, it reaches 47.37% SigLIP-2 zero-shot top-1 at 0.698 bpp, against 48.22% at 0.645 bpp for SEAOTTER-FT, and 27.41% ADE20K mIoU at 0.481 bpp, against 32.77% at 0.524 bpp. The decoder without fine-tuning, with standard JPEG, reaches 44.90% and 30.39% with larger files.

Edge hardware.

the encoding throughput is measured only on a server-class CPU

Our response of November 10, to which this comment replies, reports the sensor-side encoder on a Cortex-M55 microcontroller at 249 MHz: 6.18 megapixels per second (MPx/s) at the 12-channel operating point, against 0.0139 MPx/s for AVIF at the matched rate. The comment also calls the Wi-Fi, 5G, and BLE thresholds "assumed." Each is the product of a resolution, a frame rate, and a link rate from the cited measurement literature. The BLE case, 480p at 30 frames per second over 1 Mbps, requires 12 MPx/s; on the microcontroller the encoder runs at 18.4 and 11.8 MPx/s at its two lowest rates.

New requests. The comment asks for TensorRT, network quantization, parallel execution, and a discussion of variants for other compute budgets. None appears in the review, and none bears on a claim of the manuscript. TensorRT is an inference library for NVIDIA GPUs. The sensor-side encoder is a set of linear projections costing 10–100 MACs per pixel, measured on one CPU and on a microcontroller with no GPU. The baseline, AVIF, is not a neural network, so no inference framework applies to it.

Improvement of the manuscript.

The rebuttal shall focus on the improvement of the manuscript.

Our response reported three additions made in answer to this review: the microcontroller measurements, an evaluation of every fine-tuned variant on the tasks it was not tuned for, and a comparison with DeepN-JPEG, a JPEG quantization table selected for task accuracy. The comment acknowledges none of them.

Reviewer_ux63OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The SEAOTTER proposed in the paper uses frozen FRAPPE/WaLLoC encoders at the sensor side to generate latents and upload them. The cloud uses a neural decoder to reconstruct pixels. Then, through a learned JPEG color transform, per-rate quantization matrices, and a differentiable rate proxy, the reconstructed result is transcoded into standard JPEG file. The consumer side uses standard JPEG decoding, followed by an inverse transform for recovery. The goal is to balance the compression rate under extreme uplink bandwidth constraints and enable fast decoding at the consumer side. The authors validated the method’s performance on multiple datasets.

Critical strengths

The scenario of resource asymmetry across the sensor, cloud, and consumer segments is a reasonable assumption, and the authors’ discussion of the encode-once/decode-many lifecycle is also illuminating The idea of one-shot transcoding of a private latent into standard JPEG is relatively conducive to practical application. The differentiable JPEG sandwich, learned color transform, per-rate quantization matrices, and rate proxy work together and can be integrated with both the FRAPPE and WaLLoC encoders, and the appendix provides relatively complete design and experimental details. The evaluation covers three downstream tasks, multiple codecs, and five rate points, making the experiments fairly comprehensive.

Critical weaknesses
  1. Some prior work [1] has explored directly optimizing JPEG for downstream tasks, thereby leveraging JPEG’s efficient encoding and decoding characteristics. Could a comparative analysis with this work be conducted? Although adding a heavyweight intermediate stage may be reasonable in certain scenarios, it would be more practical if this stage could be omitted.

  2. The evaluation was not conducted on real edge hardware. In particular, it is unclear whether vectorized acceleration was applied to the network-based edge model and whether hardware acceleration was applied to the AVIF encoder. Comparing vectorized network inference with an encoding method that does not rely on hardware acceleration may be unfair.

[1] DeepN-JPEG: a deep neural network favorable JPEG-based image compression framework

Questions for authors

Answer the questions raised by the critical weakness.

2 comments
AuthorsOpenReview ↗

Response to Reviewer ux63 (1/2): comparison with DeepN-JPEG

Weakness 1: comparison with DeepN-JPEG

  1. Some prior work [1] has explored directly optimizing JPEG for downstream tasks, thereby leveraging JPEG's efficient encoding and decoding characteristics. Could a comparative analysis with this work be conducted? Although adding a heavyweight intermediate stage may be reasonable in certain scenarios, it would be more practical if this stage could be omitted.

[1] DeepN-JPEG: a deep neural network favorable JPEG-based image compression framework

We have updated the manuscript by adding comparisons and discussion that explain the relationship between SEAOTTER and DeepN-JPEG. Here is a summary.

DeepN-JPEG replaces JPEG's quantization table with a single table designed to preserve DNN accuracy. It measures the standard deviation of each of the 64 DCT bands over sampled training images, then maps each standard deviation to a quantization step through a piecewise-linear function. The authors set that function's nine constants by sweeping quantization steps against AlexNet accuracy on ImageNet. JPEG's color transform is unchanged, one table serves all color components, and the method has no rate parameter. The authors chose this heuristic because they considered direct optimization out of reach: redesigning the table for DNNs "is an intractable optimization problem for DeepN-JPEG because of the complexity of parameter searching, and the difficulty of a quantitative measurement suitable to DNNs."

SEAOTTER solves that optimization problem. Its JPEG codec learns the color transform, its inverse, and one quantization matrix per rate, end to end by gradient descent. The rate term of the loss is a differentiable estimate of the JPEG file size, scaled to match the size a standard JPEG encoder produces. Nothing is warm-started from JPEG: the color transform starts at the identity and the quantization matrices start at random noise. SEAOTTER-ZS trains these parameters against a rate–distortion objective and reaches parity with ITU T.81. SEAOTTER-FT trains the same parameters jointly with the decoder against the downstream task loss, which is the quantitative measure suitable to DNNs that DeepN-JPEG lacked. DeepN-JPEG cannot be adapted this way: no part of it is differentiable, so a new task or model requires repeating the sensitivity sweeps by hand, and the paper gives that procedure only for ImageNet classification.

The methods differ in what their parameters were fitted on and how they were selected:

methodfitted onselection criterion
ITU T.81nonehuman vision
DeepN-JPEGImageNetImageNet top-1
SEAOTTER-ZSLSDIRPSNR
SEAOTTER-FTthe task's datathe task's loss

Additionally, we have added a comparison to DeepN-JPEG to the appendix, which strengthens the claims presented in the original manuscript. Here is a summary of the result:

task (metric)transmit bppITU T.81 4:4:4DeepN-JPEG 4:4:4DeepN-JPEG 4:2:0SEAOTTER-ZS
ImageNet (top-1)0.34453.2545.5971.1576.43
ADE20K (mIoU)0.30420.9916.8833.1439.42
SigLIP-2 (zero-shot top-1)0.41037.9036.7355.2261.00

The rates are SEAOTTER's n=15 operating point. DeepN-JPEG cannot reach lower SEAOTTER rates, because a baseline JPEG file cannot be smaller than 0.18–0.25 bpp on these images.

In this experiment, DeepN-JPEG is the sensor-side codec with no intermediate stage, so the size of its JPEG file is the transmit rate. We built its table as the DeepN-JPEG paper specifies, from the statistics of 20,000 ImageNet training images, and evaluated it with the models and validation sets of the main results. Each row compares the methods at SEAOTTER's transmit rate at n=15.

AuthorsOpenReview ↗

Response to Reviewer ux63 (2/2): edge hardware and vectorization

Weakness 2: edge hardware and acceleration

  1. The evaluation was not conducted on real edge hardware. In particular, it is unclear whether vectorized acceleration was applied to the network-based edge model and whether hardware acceleration was applied to the AVIF encoder. Comparing vectorized network inference with an encoding method that does not rely on hardware acceleration may be unfair.

We build on FRAPPE and use its pretrained weights and architecture without modification. While demonstrating the encoding efficiency on edge hardware is outside the scope of this paper's claims, we agree that it is necessary to make the system practical for real-world deployments.

Thus, we have conducted additional experiments comparing the FRAPPE encoder to AVIF under a very extreme edge deployment scenario (Apollo510 SoC, which is a Cortex-M55 microcontroller designed for low-power wearables and edge AI). This SoC is representative of devices that operate under extreme power and bandwidth constraints. The AP510B variants of this SoC include a BLE radio, the class of link that the manuscript's 480p tier assumes at 1 Mbps. Because the design is intended for extremely power-constrained devices, it has no hardware image or video encoders, and it would not be suitable for wireless systems that require more transmit power than BLE (5G cellular, Wi-Fi, etc.).

Both encoders were run on the device (one Cortex-M55 core at 249 MHz) over the 24 Kodak images at 768×512. The FRAPPE encoder was measured in two builds: one using the Helium vector instructions of the Cortex-M55, and one compiled as scalar code with vectorization disabled. AVIF used the same libavif and libaom versions and settings as the main results. libaom has no Helium code path, so on this device AVIF runs as scalar code, and the scalar FRAPPE build is the like-for-like comparison. Each AVIF row uses the lowest quality whose rate is above FRAPPE's; for n≤9n \le 9 that is AVIF's minimum quality, whose rate (0.068 bpp) exceeds FRAPPE's.

nnFRAPPE bppFRAPPE, HeliumFRAPPE, scalarAVIFAVIF max-speed
30.00718.4 MPx/s4.81 MPx/s0.0154 MPx/s0.200 MPx/s
60.02411.8 MPx/s2.82 MPx/s0.0154 MPx/s0.200 MPx/s
90.0468.54 MPx/s1.99 MPx/s0.0154 MPx/s0.200 MPx/s
120.0786.18 MPx/s1.47 MPx/s0.0139 MPx/s0.199 MPx/s
150.2823.37 MPx/s1.01 MPx/s0.0111 MPx/s0.179 MPx/s

With neither encoder vectorized, FRAPPE is 91 to 312 times faster than AVIF and 5.6 to 24 times faster than AVIF at maximum speed. With Helium, the ratios are 303–1192 and 19–92. The encoding advantage therefore does not depend on vectorizing one method and not the other. The manuscript's BLE tier requires 12 MPx/s of encoding throughput (480p at 30 fps); on this device FRAPPE reaches 18.4 MPx/s at n=3n=3 and 11.8 MPx/s at n=6n=6, and AVIF at maximum speed reaches 0.2 MPx/s. The FRAPPE encoder uses 1.5–1.7 MB of memory, and AVIF uses 11–17 MB.

For the timings in the submitted manuscript, both encoders ran on the same high-power AMD CPU, each with the vector instructions its library enables by default, and neither used a GPU or a hardware codec. The submitted appendix describes this configuration alongside the throughput tables.

Reviewer_qbuNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SEAOTTER focuses on sensing workloads with constrained uplink resources and repeated downstream consumption. A frozen lightweight encoder transmits compact learned latents, which a cloud decoder reconstructs and transcodes into JPEG. Learned color transforms and quantization tables are optimized around this JPEG stage, optionally together with task-specific decoder fine-tuning. Experiments evaluate transmission rate, storage rate, downstream accuracy, and encoding/decoding throughput across classification, segmentation, and SigLIP-based recognition.

Critical strengths
  • Separating sensor transmission from consumer storage provides a useful way to combine lightweight encoding with inexpensive repeated decoding.
  • Concrete efficiency evidence: CPU measurements include the consumer-side inverse transform and demonstrate substantial decoding savings over the underlying neural codec.
Critical weaknesses
  • Novelty concerns. Learning neural wrappers around standard codecs has clear precedent in sandwiched compression. SEAOTTER contributes a lightweight implementation within a sensor–cloud–consumer pipeline, but the additional benefit of its learned JPEG components remains insufficiently isolated. Plain JPEG already matches or exceeds the zero-shot variant in several settings, while the strongest results jointly fine-tune the neural decoder and JPEG sandwich. A task-adapted decoder paired with ordinary JPEG would better establish the methodological advance.

  • Uncertain reuse of task-adapted artifacts. The strongest accuracy improvements accompany substantial reconstruction distortion: at ImageNet n=12, PSNR decreases from 25.08 dB for FRAPPE to 10.39 dB for SEAOTTER-FT. Target-model accuracy alone does not establish that these artifacts retain useful information for unrelated models or human viewing. This limits the evidence for using task-adapted outputs as broadly reusable image assets.

  • Underspecified task-aware training. Appendix A.7 provides an LSDIR training recipe for the learned JPEG bundle, but the manuscript does not comparably specify the task-aware fine-tuning objective, training-data split, and optimization schedule for SEAOTTER-FT. Since the headline accuracy gains depend on this stage, these details are necessary to interpret and reproduce the main results.

Questions for authors
  • How well does a task-specific SEAOTTER-FT artifact transfer to a model that was not used during optimization? A focused evaluation would help distinguish target-model adaptation from broader preservation of useful visual information.
5 comments
AuthorsOpenReview ↗

Response to Reviewer qbuN (1/4): novelty and the learned JPEG components

Novelty concerns. Learning neural wrappers around standard codecs has clear precedent in sandwiched compression. SEAOTTER contributes a lightweight implementation within a sensor–cloud–consumer pipeline, but the additional benefit of its learned JPEG components remains insufficiently isolated. Plain JPEG already matches or exceeds the zero-shot variant in several settings, while the strongest results jointly fine-tune the neural decoder and JPEG sandwich. A task-adapted decoder paired with ordinary JPEG would better establish the methodological advance.

We have updated the manuscript with additional discussion clarifying the role of the learned JPEG components and of the zero-shot variant. In brief: the claim made for the learned codec is that its color transform, quantization matrices, and rate proxy can be trained de novo by gradient descent to parity with the ITU T.81 tables, which makes the JPEG stage adaptable to a task. SEAOTTER-ZS is the demonstration of that parity, and SEAOTTER-FT is the demonstration of the adaptation.

On the precedent in sandwiched compression: the manuscript identifies the sandwiched codec of Guleryuz et al. as the closest prior art and states three differences. SEAOTTER's color transform is a single 3×33{\times}3 convolution with companding in place of a U-Net, so the consumer pays the cost of a standard JPEG decode plus 81 MACs per pixel. It is trained de novo, with no warm start from a codec. One transform pair is shared across all rates, with one quantization matrix per rate. The learned codec is also one of three stated contributions; the other two are the sensor / cloud / consumer formulation under an encode-once, decode-many lifecycle, and the finding that a one-time transcode into JPEG increases downstream accuracy over the underlying autoencoder.

On plain JPEG matching the zero-shot variant: this is the result the manuscript states for SEAOTTER-ZS, which is described as matching or exceeding the ITU T.81 tables. Its color transform starts from the identity and its quantization matrices from random noise, so it uses no information from JPEG's hand-designed tables, and it is trained with no task, label, or downstream model. Parity under those conditions is the evidence that the differentiable codec and its rate proxy work. For this reason the manuscript reports the plain-JPEG control (FRAPPE + ITU JPEG) beside SEAOTTER-ZS at every operating point on all three tasks, and evaluates the learned codec standalone against ITU T.81 on Kodak and ImageNet.

On isolating the benefit: at n=12n=12 on ImageNet, SEAOTTER-FT exceeds FRAPPE + ITU JPEG by 8.19 points of top-1 with a 13.7% smaller file, at an identical transmit rate. That comparison measures the combined effect of adapting the decoder and the JPEG parameters.

The suggested experiment, a task-adapted decoder paired with ordinary JPEG, separates these two effects and makes the contribution of the learned JPEG components directly measurable. It is a valuable addition that strengthens the paper, and it has been added to the appendix.

In this experiment, the FRAPPE decoder is fine-tuned on the task loss with the SEAOTTER-FT recipe (same training images, order, batch size, learning rate, schedule, and number of steps), with no learned color transform and no learned quantization matrix. Its output is stored as an ordinary ITU T.81 file with the standard tables, in the 4:4:4 format used by SEAOTTER's JPEG and by the FRAPPE + ITU JPEG baseline. The results are for SigLIP-2 zero-shot classification at n=12n=12 on the full ImageNet validation set. All rows share the frozen encoder, so transmit bpp is identical (0.142).

ConfigurationJPEG qualityStorage bppTop-1 (%)
FRAPPE + ITU JPEG (decoder not adapted)751.16644.90
Task-adapted decoder + ITU JPEG100.49442.56
Task-adapted decoder + ITU JPEG200.69847.37
Task-adapted decoder + ITU JPEG300.88848.74
SEAOTTER-FT (learned JPEG)n/a0.64548.22

At matched storage, SEAOTTER-FT is more accurate than the task-adapted decoder paired with ordinary JPEG: 48.22% at 0.645 bpp, against 47.37% at 0.698 bpp, which is 0.85 points higher with an 8% smaller file. Ordinary JPEG first reaches SEAOTTER-FT's accuracy at quality 30, where its file is 38% larger.

AuthorsOpenReview ↗

Response to Reviewer qbuN (2/4): task-aware fine-tuning recipe

Weakness: underspecified task-aware training

Underspecified task-aware training. Appendix A.7 provides an LSDIR training recipe for the learned JPEG bundle, but the manuscript does not comparably specify the task-aware fine-tuning objective, training-data split, and optimization schedule for SEAOTTER-FT. Since the headline accuracy gains depend on this stage, these details are necessary to interpret and reproduce the main results.

As the reviewer pointed out, we omitted part of the training recipe, which was an oversight on our part. We have updated the manuscript with this missing detail, which is also listed below:

Objective. Each SEAOTTER-FT model minimizes Ltask+λ R\mathcal{L}_{\text{task}} + \lambda \, R, where RR is the differentiable estimate of the JPEG file size in bits per pixel and λ=0.10\lambda = 0.10 for every task and rate. The FRAPPE encoder is frozen. The FRAPPE decoder, the color transform and its inverse, and the quantization matrix are trained. The JPEG parameters are initialized from the highest-rate tier (λ3=0.22\lambda_3 = 0.22) of the LSDIR-trained bundle, and the decoder from the published FRAPPE weights. The downstream model is frozen and supplies gradients only.

classificationsegmentationSigLIP-2
training imagesImageNet-1k train, 1.28MLSDIR train, 85kImageNet-1k train, first 50k
task losscross-entropy against the ground-truth label, through ConvNeXt-Tcross-entropy against the hard pseudolabel that UperNet predicts for the uncompressed image0.7 (1−cos⁡)0.7\,(1-\cos) on per-patch features plus 0.3 (1−cos⁡)0.3\,(1-\cos) on the pooled embedding, against the SigLIP-2 features of the uncompressed image
labels usedImageNet class labelsnonenone
preprocessingsquash to 3842384^2, horizontal fliprandom resize, color jitter, random 5122512^2 crop, horizontal flipnaflex resize (256 patches), horizontal flip
batch size821
base learning rate2×10−42\times10^{-4}2×10−42\times10^{-4}7×10−57\times10^{-5}
passes over the training images141
optimizer steps160k170k50k

Schedule. All three use the Adan optimizer, a raised-cosine learning-rate schedule from the base rate to 10−810^{-8}, gradient clipping at 5.0, additive uniform noise in place of rounding during training, and seed 0. The quantization matrix is trained at half the base learning rate.

No validation image is used in fine-tuning. The segmentation models are trained without ADE20K images or labels, and the SigLIP-2 models without class labels or text.

The fine-tuning code, with the defaults above, is in the submitted supplement (src/seaotter/train/recipes/). The supplement did not state the 50k-image subset used for SigLIP-2.

AuthorsOpenReview ↗

Response to Reviewer qbuN (3/4): transfer of fine-tuned variants, results

Question: transfer of SEAOTTER-FT artifacts

How well does a task-specific SEAOTTER-FT artifact transfer to a model that was not used during optimization? A focused evaluation would help distinguish target-model adaptation from broader preservation of useful visual information.

We have updated the manuscript to clarify the intended use of the two variants. SEAOTTER-FT is intended to maximize the accuracy of one application, at the cost of generality. SEAOTTER-ZS is the general-purpose variant: it is trained with no task, label, or downstream model, and the submitted manuscript shows that it preserves accuracy across global, dense, and vision-language tasks. Because every variant shares the frozen encoder, the choice between them is made in the cloud after transmission, and the same transmitted data can be decoded either way. For this reason the submitted manuscript makes no claim about how a fine-tuned variant performs with a model it was not tuned for.

The question is nonetheless scientifically interesting, and we have added an experiment to the appendix that answers it directly. Each fine-tuned variant is evaluated on the two tasks it was not tuned for, with the datasets, models, and preprocessing of the main results. A summary of the results is given below.

Each cell is task accuracy (%) with storage bpp in parentheses. All variants at a given nn share the frozen FRAPPE encoder, so transmit bpp is identical within each column. FRAPPE + ITU JPEG is the frozen FRAPPE decoder followed by a standard ITU T.81 transcode (no fine-tuning, no learned sandwich). Bold marks a cross-task cell that exceeds SEAOTTER-ZS at the same nn.

EvaluationVariantn=3n=6n=9n=12n=15
ImageNet top-1SEAOTTER-ZS8.3 (0.64)30.8 (0.88)47.8 (1.05)60.3 (1.28)76.4 (1.79)
ImageNet top-1FRAPPE + ITU JPEG7.8 (0.59)30.2 (0.77)48.1 (0.90)60.8 (1.03)77.2 (1.31)
ImageNet top-1SEAOTTER-FT (tuned for this task)17.3 (1.62)46.5 (1.22)59.7 (1.14)69.0 (0.90)77.4 (0.81)
ImageNet top-1SEAOTTER-FT (tuned for seg)6.7 (0.50)31.1 (0.47)46.9 (0.51)60.4 (0.58)72.9 (0.79)
ImageNet top-1SEAOTTER-FT (tuned for clip)5.5 (0.37)31.9 (0.46)50.4 (0.51)63.3 (0.56)75.0 (0.70)
ADE20K mIoUSEAOTTER-ZS4.8 (0.59)18.1 (0.82)25.7 (0.98)30.1 (1.18)39.4 (1.52)
ADE20K mIoUFRAPPE + ITU JPEG5.3 (0.55)18.5 (0.72)26.1 (0.84)30.4 (0.95)39.8 (1.15)
ADE20K mIoUSEAOTTER-FT (tuned for this task)6.7 (0.46)21.1 (0.44)28.2 (0.47)32.8 (0.52)38.6 (0.68)
ADE20K mIoUSEAOTTER-FT (tuned for cls)1.9 (1.52)11.4 (1.13)23.6 (1.06)31.5 (0.83)38.7 (0.69)
ADE20K mIoUSEAOTTER-FT (tuned for clip)3.1 (0.34)18.2 (0.42)26.6 (0.48)32.1 (0.52)38.6 (0.63)
SigLIP-2 zero-shot top-1SEAOTTER-ZS4.8 (0.74)17.9 (1.01)30.4 (1.20)43.3 (1.47)61.0 (2.02)
SigLIP-2 zero-shot top-1FRAPPE + ITU JPEG5.0 (0.67)18.4 (0.88)31.4 (1.02)44.9 (1.17)61.7 (1.46)
SigLIP-2 zero-shot top-1SEAOTTER-FT (tuned for this task)2.6 (0.43)20.0 (0.52)34.8 (0.59)48.2 (0.65)61.3 (0.80)
SigLIP-2 zero-shot top-1SEAOTTER-FT (tuned for cls)2.2 (1.77)15.7 (1.38)29.0 (1.30)44.1 (1.03)58.1 (0.92)
SigLIP-2 zero-shot top-1SEAOTTER-FT (tuned for seg)2.5 (0.58)16.4 (0.54)28.8 (0.59)42.5 (0.66)57.6 (0.89)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.