acceptodds
Withdrawn from ICLR 2027

Transformers from Compressed Represenatations

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Compressed file formats are the cornerstone of efficient data storage and transmission, yet their potential for representation learning remains largely under-explored. We introduce TEMPEST (TransformErs froM comPressed rEpreSenTations), a method that exploits the structured byte-stream of compressed files to design an effective tokenization and encoding strategy. By leveraging this com- pact encoding, a vanilla transformer can directly learn semantic representations from compressed data streams. Our proposal reduces the number of tokens required for semantic classification, thereby lowering both computational complexity and memory usage. Moreover, we discover unique properties of compressed byte stream for representations learning, specifically, the existence of complementary information across compression levels. Through extensive experiments across diverse datasets, coding schemes, and modalities, we show that TEMPEST achieves accuracy competitive with raw data baselines while delivering efficiency gains in memory and compute.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Compressed Formats, . Audio Classification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XQAVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The work proposes a transformer-based architecture to classify audio in MP3 format and images from a JPEG format that has been augmented to include Minimum Coded Unit (MCU) markers. A sequence of bytes representing a block (frame, MCU) is processed and reduced to a much smaller sequence length. Each block embedding corresponds to a token in a secondary transformer that performs instance classification.

A reconstruction loss is added as regularization. Streams at different bitrates representing the same input are used as data augmentation. Both aspects seem to improve classification performance.

Results show comparable performance to similar architectures that use uncompressed inputs and better performance than other approaches that use compressed data. The sequence length of the input to the classifier model is significantly reduced.

Critical strengths
  • I find the block embedding network that combines transformer layers with the channel-mixing subnetwork novel for this setting.
  • The ablation shows interesting insights regarding training and inference with multiple bitrates.
Critical weaknesses
  • The writing needs refinement. It is lacking coherence and I found some contradictions between sections. There are also several grammatical and spelling mistakes.
  • The writing also needs to improve on clarity:
    • Notation is not congruent. Constants have the same facetypes as functions and different facetypes among them. Representations have different facetypes too and the same as some constants and functions.
    • There is conflicting notation. MM is a model and also a number of layers. NN and nn are both different constants. mm seems to be different than MM but the same as jj . kk and jj seem to be the same in some contexts but different in others. jj indexes both blocks and bytes. The dimensionality Rjl’+1,N\mathbb{R}^{jl’ + 1, N} should probably be Rl’(j+1)×N\mathbb{R}^{l’(j +1) \times N} or Rl’(m+1)×N\mathbb{R}^{l’(m +1) \times N} .
    • The mathematical definition of BjB_j should specify the padding in order to be accurate. Otherwise, it might be best to drop it entirely and explain it in words.
    • Similarly for Hj\mathcal{H}_j . Either be precise or do not include a mathematical definition.
    • Table 3 should highlight the model with the highest performance.
  • I have concerns about the impact of this work. It requires that blocks are explicitly marked. Besides MP3, what other compressed formats have block markers? Changing standardized codecs to include block markers is not practically viable.
  • The number of tokens does not necessarily dictate time or space complexity. For instance, ByteFormer has lower GFLOPs than TEMPEST even though it has a higher number of tokens. I can see how a similar situation could occur in terms of space complexity. For example, the block embedding network, although smaller, still consumes all bytes initially. ByteFormer has limited attention mechanisms and strided convolutions. Since task performance is on par, space complexity is the main selling point of this work and it is not very well supported.
  • The performance of ByteFormer should be extended to the other datasets.
  • The work only focuses on classification. I think that to support some of the claims made other types of tasks should also be evaluated. The quality of this venue demands stronger support.
  • I find strange that increasing l’l’ decreases performance. Is it because of overfitting? I am not sure how that behaviour leads to the claim that “our tokenizer can effectively pack all the relevant information into a single token”. A very bold claim with very weak support.
  • λ\lambda is not defined in the main sections and an analysis of its impact on performance (besides the values 0 or 1) seems necessary given the results from the ablation study.
Questions for authors

If the authors can improve on some of the weaknesses I presented or show me that I am misunderstanding the work, I can increase my score.

3 comments
AuthorsOpenReview ↗

Official Response to XQAV Part 1

Response to Reviewer XQAV We thank the reviewer for the careful reading, highlighting the block-embedding network novelty and finding the multi-bitrate training and inference results insightful. We believe this emerging domain holds many more interesting properties to explore.

Below we address the reviewer's weaknesses and questions individually.

Grammar and Notation. We thank the reviewer for the careful reading. We are fixing the grammar and notation errors. We will upload the updated version of the PDF and we will keep updating as the discussion with the reviewers develop.

Block markers and standard codecs. We believe our presentation may have led to a small misconstruction, thus we clarify: Block markers are already part of the MP3 and JPEG standards. An MP3 file is defined as a sequence of frames, and the format defines a sync word that enables the location of beginning and end of the frames. Also, restart markers are also an integral part of the JPEG standard, and are supported by every compliant decoder.

We further emphasize that our approach does not need to change or extend the implementation of any codec. TEMPEST uses only standard codec features and flags.

Finally, our data pipeline simply parses the header and locates the byte markers (MP3 frame headers or JPEG restart markers) and feeds the bytes between consecutive markers to the block-embedding network.

We will augment our supplementary material with this clarification.

Comparison with Byteformer, computational effort We begin by noting that TEMPEST (4.68 GFLOPs, 91.24%) costs less than ByteFormer k=4 (5.49 GFLOPs, 89.19%) and is more accurate than both ByteFormer configurations (k=4 or k=8).

We agree with the review that token count alone does not determine the total computational cost, and that our tokenizer must process every byte in the first layer. Our Intra-block attention is equivalent to a non-overlapping windowed attention, as a result, tokenizers of TEMPEST and ByteFormer both scale linearly with input length. We do not claim an advantage over ByteFormer, rather we want to clarify that we are equal in this aspect.

The advantage of TEMPEST lies in what follows the tokenizer. After two layers, every block is reduced to a single token (32 tokens per second of audio vs. 1619 tokens for ByteFormer at k=4). Our network classifier runs multiple layers of full global attention over a much shorter sequence.

ByteFormer on other datasets. The ByteFormer paper reports audio results only on SC2 and the released code appears to be incomplete. We don't think that providing results from a manually patched codebase is the best reference for the performance of ByteFormer in other datasets.

TEMPEST is classification only. We agree that our evidence is limited to classification. We note, however, that classification is also the evaluation protocol of the prior work most directly comparable to ours (ByteFormer, Pérez et al. 2024, Park & Johnson 2023, Gueguen et al. 2018). We are evaluating additional experiments that can fit the window of the rebuttal and will come back to the reviewer when possible.

Applicability and impact, JPEG markers We agree that most existing JPEG files do not use a restart interval for each MCU, and the default value depends on the codec implementation. Nevertheless, existing files can be converted offline with a simple command which executes a transcoding, for example with the standard libjpeg tool:

jpegtran -restart 1B input.jpg > output.jpg

The resulting file has a restart marker after every MCU, without changing the quantized coefficients. This conversion is a one-time offline re-coding step, during inference no further encoding or transformation is required, only the parsing of the byte stream.

The main trade-off we observe is the additional storage size, as every MCU marker adds 3 to 5 bytes (2 marker bytes and between 1 to 3 additional bytes for resetting the DC component). An average TinyImageNet image at quality 80 is about 2.000 bytes, and at quality 90 it is about 2.600 bytes. With 64 MCUs the markers add about 192 to 320 bytes in both quality settings, which amounts to roughly 9.6% to 16% additional disk storage at quality 80, and 7% to 12.3% at quality 90. We do not consider this additional storage a significant increase.

AuthorsOpenReview ↗

Official Response to XQAV Part2

Applicability and impact , Block independence Upon request of another reviewer we have revisited the audio data, and learned that strict block independence is not required for effective semantic learning in TEMPEST. We have discovered that the MP3 bit reservoir was in fact active by default during the LAME/ffmpeg transcoding. After sampling 2,000 files from Speech Commands v2 we confirmed that most frames (95% of them) utilize the reservoir, meaning that inter-frame dependencies were already present in the training data used for the main results and all the ablations in the paper. The competitive results in SC2 show that TEMPEST can effectively model some level of block inter-dependencies. As the block embedding network shares no information between blocks, we hypothesize that the classification network (C), is resolving these dependencies while also building a high level representation. This finding extends the viability of our proposed method, demonstrating that our tokenizer can still operate over more complex compressed streams where blocks are not strictly independent. We will update Appendix A and revise the main text to distinguish directly locatable blocks from independently decodable units.

Applicability and impact , other formats with block markers. Other widely used formats are also organized into self-delimited blocks:

  • Audio: FLAC, Opus and Vorbis.
  • Video: H.264 .

Effect of l'>1. We agree that “our tokenizer can effectively pack all the relevant information into a single token” is a strong assumption, however, we note that the results show that a single token is enough to achieve the best performance in all the selected datasets. We will rephrase and tone down this assertion in the main paper.

We don't think increasing (l') can result in an overfit condition, l=2 adds only 289 parameters over l=1, a negligible increase in the model’s parameters count. Moreover, overfitting does not explain the observed performance variation in ESC-50 (up to 1.43) and the relative consistency for the two other datasets, where every other variation remains below 0.32.

We hypothesize that this behavior is derived from the absence of explicit positional information for the latent tokens entering the reconstruction decoder (D). The encoder’s positional encoding identifies byte positions within each block, but does not explicitly specify the order of the resulting (l') latent tokens. When (l'>1), this may pose a harder reconstruction and thereby weaken the decoder’s effectiveness as a regularizer. With (l'=1), this ordering ambiguity disappears.

Value of lambda . We clarify that lambda weights the reconstruction loss (wrec), this has been corrected in the final equation of section 3. Table 6 compares lambda=0 with lambda=0.5, the default setting in all our experiments. We are currently running additional ablations to measure the sensitivity to on each dataset.

Thank you for the response.

I think the comments made in this rebuttal are addressing some of my concerns but I do not see the discussion making its way to the document. I empathize with the time constraints the authors are facing.

Applicability and impact seem to be addressed (H.264 having markers is great) but no relevant changes were made to the document. The hypothesis around the effect of l′l' could have been discussed as well.

Due to the withdrawal, I would suggest to the authors that they fix the notation, extend the idea past classification, and strengthen the claims. I would love to see the resulting work published. Good luck!

Reviewer_YxZUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Training on stored audio and image collections requires decoding compressed files into waveforms or pixels. The authors propose TEMPEST, a Transformer that classifies audio and images from the compressed bytes themselves. The byte stream is cut at format-defined block boundaries (MP3 frames, JPEG minimum coded units), a per-block Transformer embeds each block into one token under a byte-reconstruction regularizer, and a standard Transformer classifies the resulting short sequence. The blocks are separable only because the files are encoded with non-default settings (bit reservoir disabled for MP3; a restart marker after every MCU, 4:4:4 chroma, and no header for JPEG). The claimed contributions are accuracy competitive with raw-domain baselines at one to two orders of magnitude fewer tokens than prior byte-level models, lower compute than a spectrogram Transformer on audio, and the observation that encoding at several bitrates acts as data augmentation and as test-time ensembling.

Critical strengths
  1. Learning directly from compressed media is a relevant problem for data pipelines in which storage, I/O, and decoding are costs. Using format-defined blocks as tokens is a better-motivated unit than fixed byte patches, and it reduces sequence length relative to byte-level methods by one to two orders of magnitude (32 versus 809 or 1619 tokens per second for ByteFormer on audio; 64 versus 921 tokens for Pérez et al. on CIFAR-10) while improving accuracy over them (91.24% versus 89.19% on Speech Commands v2; 83.13% versus 56.9% on CIFAR-10).

  2. The ablations on tokens per block, tokenizer depth, the reconstruction loss, and multi-bitrate training are clearly presented. The reconstruction regularizer has a large effect on ESC-50 (43.80% to 52.28%, Table 6), which is informative for others designing byte-level tokenizers.

Critical weaknesses
  1. The claim that "no decompression (not even partial) is performed at any stage" holds only for bitstreams produced with non-default encoder settings, and the paper does not report what those settings cost in file size. Blocks are independent because MP3 files are encoded with the bit reservoir disabled and JPEG files are encoded with a restart marker after every MCU, 4:4:4 chroma, and no header (Appendix A and A.1). In JPEG files as produced by default encoders, MCU boundaries fall at sub-byte positions that can only be found by Huffman decoding, DC coefficients are coded differentially across MCUs, and chroma is usually subsampled; in MP3 files with the bit reservoir enabled, a frame's main data can begin in earlier frames. Applying TEMPEST to existing collections therefore requires at least entropy decoding and re-encoding of every JPEG file (restart markers can be inserted at the Huffman level, without an inverse DCT) and a full decode and re-encode of every MP3 file, which is the decompression the method is meant to avoid. Training has the same dependence for audio: Trivial Augment is applied to the raw waveform, which is then transcoded to MP3 (Appendix A). For images, the recipes list RandAugment and random-resized cropping (Tables 9–11), which are defined on pixels, but the paper does not say where in the pipeline they are applied; if they precede encoding, as their definition implies, image training also requires raw data and an encoder in the loop, whereas Section 5 states that the approach "bypasses the need for file-format decoding or raw form storage". The format changes also increase file size: two marker bytes plus padding to a byte boundary per MCU, absolute instead of differential DC coding at every MCU, no chroma subsampling, no bit reservoir, and, for CIFAR, four times as many pixels after upscaling. Appendix A.1 notes the two marker bytes per MCU and describes the lost coupling as "a small gain in the coding efficiency", and the paper motivates its approach by "efficient data storage and transmission", but it reports no file sizes. The authors should report bits per pixel and kbit/s for their encodings against default encodings at the same quality, describe the image augmentation pipeline, and state how existing compressed datasets would be handled.

  2. The efficiency claims are not supported by the measurements in the paper. On images, TEMPEST and the ViT use the same 64 tokens, so there is no token reduction; TEMPEST adds a per-MCU tokenizer and loses 5.3–5.6 percentage points of accuracy on CIFAR-10, CIFAR-100, and Tiny ImageNet (on CIFAR-10, the error rate rises from 11.5% to 16.9%). Table 3 reports no FLOPs, and the padded MCU length ll for JPEG is not given (Table 1 lists the unpadded block length as variable), so the cost of the tokenizer on images cannot be computed. On audio, the reduction from 108 to 32 tokens per second reflects AST's overlapping patches (16×16 with stride 10, per Gong et al., 2021) rather than a property of compressed data. Non-overlapping patches give about 50 tokens per second, and grouping spectrogram frames at the MP3 frame rate gives the same token rate as TEMPEST, but no raw-domain baseline at a matched token rate is reported. The computation has also moved into the tokenizer. By Table 5, the two-layer block embedding network accounts for 4.24 of the 4.68 GFLOPs (91%). Its first layer attends over l=144l = 144 byte positions in each of 32 blocks per second and its second over 72 positions after the channel-mixing layer halves the block, about 8.3×1058.3 \times 10^5 attention entries per second of audio per head, compared with 1082=11,664108^2 = 11{,}664 for AST's whole sequence, a factor of about 71 in the other direction. Section 4.2 limits its attention comparison (1,024 versus 11,664 entries) to the classifier, but the Conclusions state without qualification that TEMPEST reduces "the size of the attention matrix by a factor of 11". The remaining FLOP difference relative to AST is largely model size: TEMPEST has 5.8M parameters at width 216, whereas the AST of Gong et al. (2021) uses a ViT-Base backbone at width 768. An AST with the same width, depth, and token rate as TEMPEST's classifier is the necessary control.

  3. The accuracy baselines are weak or uncontrolled, so "competitive with raw data baselines" holds only in a narrow sense. The AST numbers are from-scratch results copied from Gong et al. (2022), and Transformers without pretraining are known to underperform on small audio datasets. TEMPEST's 52.28% on ESC-50 is below the 64.5% reported in 2015 by the dataset's author for a two-layer CNN on log-mel spectrograms [e]. TEMPEST's hyperparameters and augmentations are tuned per dataset (Tables 9–14), whereas the baseline numbers come from other papers with different recipes. No result reports variance across training seeds, and several ablation differences (e.g., 91.24% versus 91.18% on Speech Commands v2 in Table 4) are smaller than typical seed variation.

  4. The paper overlooks or dismisses established work on inference from compressed representations after entropy decoding, which avoids pixel or waveform reconstruction. Networks operating on quantized DCT coefficients after Huffman decoding match pixel-domain accuracy (Gueguen et al., 2018, and Park & Johnson, 2023, both cited; [a]), learned token representations reduce training storage by two orders of magnitude [b], learned compression designed for compressed-domain learning has been applied to audio and images [c], and audio classification on MPEG subband data has been studied for more than two decades [d]. The paper distinguishes itself from these methods because they use "partially decoded JPEGs", but it does not compare with them on its datasets. Huffman decoding is lossless and inexpensive compared with a per-block Transformer tokenizer (4.24 GFLOPs per second of audio; the image tokenizer's cost is not reported), and it removes the bit-packing that the tokenizer must learn to undo. If a dataset is to be re-encoded anyway, as TEMPEST requires, a codec whose transform and entropy coder are designed for compressed-domain learning [c] yields latents that preserve spatial and temporal structure without per-block restart markers or a learned byte parser. A DCT-coefficient baseline on the same JPEG files is needed to show what operating before entropy decoding gains.

  5. The multi-bitrate inference result (Table 8) contradicts the paper's premise and does not support the "unique properties" claimed in the abstract. Re-encoding an input at several bitrates at test time requires the decoded waveform and an MP3 encoding pass per bitrate. The best entry combines 26 and 32 kbps (92.31% on Speech Commands v2) and is selected among six subsets reported on the test set, and adding the third bitrate lowers accuracy on every metric, contrary to the caption's statement that accuracy increases with more bitrates. The paper itself describes the procedure as test-time augmentation analogous to multi-crop evaluation, and the gains (1.07 points on Speech Commands v2, 0.6 mAP on AudioSet) are of the size test-time augmentation typically gives. No control is reported, such as averaging over several waveform augmentations at a single bitrate, so the abstract's claim of "complementary information across compression levels" as a property of compressed streams is not established.

  6. The MP3 configuration is internally inconsistent. Section 4 states a 44.1 kHz sampling rate, bitrates of 20–32 kbps, and frames averaging 144 bytes and about 32 ms; Table 1 gives about 31 ms per frame, and Table 2 gives 32 tokens per second. At 44.1 kHz, an MPEG-1 Layer III frame holds 1152 samples (26.1 ms, 38.3 frames per second) and occupies 144×bitrate/fs144 \times \text{bitrate} / f_s bytes, about 104 bytes at 32 kbps; 20 and 26 kbps are not valid Layer III constant bitrates [f]. Independently of the MPEG version, 144 bytes at 32 kbps is 36 ms of audio, not 32 ms, and at a fixed frame duration the frame size scales with bitrate, so frames cannot average 144 bytes across 20–32 kbps; 144 may be the padded length ll rather than an average. A 144-byte, 36 ms frame at 32 kbps corresponds to MPEG-1 Layer III at 32 kHz or MPEG-2 Layer III at 16 kHz. ESC-50 is recorded at 44.1 kHz, while Speech Commands and AudioSet are 16 kHz. The encoder, MPEG version, sampling rate, rate-control mode, and resulting frame lengths should be stated per dataset, since they determine the block length ll, the token rate, and the FLOP counts.

  7. The authors withhold the source code, the encoding scripts that produce the modified bitstreams, and the trained weights, and promise release only upon acceptance. There is no supplementary material and no anonymous repository link. Because the results depend on non-default encoder settings and on per-dataset tuning, the work cannot be verified as submitted. Since the code will not fit into the rebuttal text, the authors should provide it through an anonymous repository (e.g., https://anonymous.4open.science) during the rebuttal period.

  8. Several presentation errors should be corrected: Appendix A.1 contains an unresolved reference ("Section ??"); Table 9 lists the CIFAR-10 base learning rate as 5; the bibliography lists the same ByteFormer paper twice (Horton et al., 2023a and 2023b); the corporate author "Library of Congress" is cited in text as "of Congress, 2024"; Tables 12–14 list a masked-byte ratio that the text does not define; the main text refers to a "Supplementary Material" that is the appendix of the PDF; and Appendix A.1 says "we set one standard encoder flag" while the JPEG setup changes three settings.

References:

[a] Ehrlich and Davis, "Deep Residual Learning in the JPEG Transform Domain," ICCV 2019.

[b] Park et al., "SeiT: Storage-Efficient Vision Training with Tokens Using 1% of Pixel Storage," ICCV 2023.

[c] Jacobellis and Yadwadkar, "Learned Compression for Compressed Learning," DCC 2025.

[d] Tzanetakis and Cook, "Sound Analysis Using MPEG Compressed Audio," ICASSP 2000.

[e] Piczak, "Environmental Sound Classification with Convolutional Neural Networks," MLSP 2015.

[f] ISO/IEC 11172-3:1993, Coding of moving pictures and associated audio for digital storage media at up to about 1,5 Mbit/s — Part 3: Audio (cited in the submission).

Questions for authors
  1. How would TEMPEST be applied to existing JPEG and MP3 files produced with default encoder settings (no per-MCU restart markers, subsampled chroma, bit reservoir enabled) without decoding and re-encoding them? Where in the image training pipeline are RandAugment and random-resized cropping applied?

  2. What are the file sizes (bits per pixel for JPEG, kbit/s for MP3) of the modified encodings, compared with default encodings at the same quality setting on each dataset?

  3. How does TEMPEST compare with (i) an AST whose classifier has the same width, depth, and token rate as TEMPEST's and (ii) a DCT-coefficient model on the same JPEG files, in accuracy, FLOPs including the tokenizer, and wall-clock time including any decoding? What is the padded MCU length ll for JPEG on each image dataset?

  4. Which encoder, MPEG version, sampling rate, and rate-control mode were used for each audio dataset, and what are the resulting frame durations and lengths? How are 20 and 26 kbps realized?

  5. Does averaging over several waveform augmentations at a single bitrate give the same gain as the multi-bitrate inference in Table 8, and how was the subset of bitrates chosen?

  6. Will the authors provide an anonymous repository with the code, the encoding scripts, and the model configurations during the rebuttal period?

13 comments
AuthorsOpenReview ↗

Official response to YxZU Part1

Response to Reviewer YxZU

We appreciate the reviewer's extensive examination of our manuscript and the recognition of TEMPEST's contributions, in particular its improvements over existing compressed-domain methods and the ablation studies, described as "informative for others designing byte-level tokenizers."

We now address the reviewer's weaknesses (W) and questions (Q) individually.

W1/Q1–Q2. Applicability to MP3 files.

The central argument of W1 is that existing MP3 collections would need a full decode and re-encode because the bit reservoir must be disabled. After revisiting our training data, we find that this assertion does not hold.

Appendix A.1 contains an error: the bit reservoir was not disabled when we recoded the WAV files into MP3. After inspecting the code, we found that the data was encoded with LAME through FFmpeg, where the reservoir is enabled by default, and no flag to disable it was set. We checked 2,000 randomly sampled SC2 files and found that 95% of frames use the bit reservoir. All audio datasets were encoded with the same command, we confirm that the audio results in the paper were obtained on MP3 streams with the bit reservoir ENABLED.

The audio experiments in the main paper establish that TEMPEST can learn effective semantic representations even with some dependencies between the frames. Moreover, frame boundaries remain directly locatable even when the bit reservoir introduces dependencies between neighboring frames.

We will fix this statement in the appendix, explicitly distinguishing between locating compressed blocks and requiring those blocks to be independently decodable. We further note that everything else in our block formulation remains unchanged: frames are located from sync words and header fields only, and their raw byte representations are packed into the block embedding network.

W1/Q1–Q2. Applicability to JPEG files.

The review designates our JPEG data as "modified bitstreams" requiring "format changes." We clarify that TEMPEST operates on standard codecs and standard streams. Restart markers and 4:4:4 chroma sampling are features of the JPEG standard. A compliant encoder exposes them, and a compliant decoder can open files with the mentioned chroma sampling and restart markers.

In the paper, we state that we remove the header, simply because we do not need to learn from or perform inference over this preamble data. For completeness, we also clarify that JPEG provides a start-of-scan marker, which allows us to directly identify the beginning of the first MCU in the byte stream.

We agree that most JPEG files use restart markers sparsely. However, restart markers can be inserted into existing files with a single command:

jpegtran -restart 1B input.jpg > output.jpg

This performs a one-time transformation at the entropy-coding level, preserving the quantized DCT coefficients without reconstructing RGB pixels. Such a command only needs to be run once offline.

We acknowledge that this operation incurs an offline computational cost. However, it is not part of TEMPEST's inference pipeline, which processes the resulting compressed bytes directly, without any entropy decoding or transformations.

Decoding-free.

We agree with the reviewer that data preprocessing requires a specific encoding. However, we clarify that TEMPEST only scans byte markers and reads header fields, rerforming integer arithmetic over those fields. It does not perform any entropy decoding to recover coded symbols. In JPEG, TEMPEST directly scans for restart markers and assembles the MCUs on the fly. Every byte that enters the TEMPEST network is copied exactly as stored in the compressed file, padding is added as necessary. No further parsing or transformation is involved.

We are updating the manuscript to explicitly address this distinction. We will also add two algorithms describing the parsing of each format.

Dependency on 4:4:4 chroma sampling.

The reviewer also lists 4:4:4 chroma sampling as one of the settings our method depends on. We reply that this setting is not a requirement of the method, we chose it so that comparisons with the ViT baseline are controlled. We designed our experiments so that each TEMPEST token covers the exact same image region as one ViT patch.

The smallest MCU that JPEG allows is 8×8 pixels, and this size can only be obtained with 4:4:4 sampling. On CIFAR, the ViT baseline uses 4×4 patches on the 32×32 image, which gives 64 tokens. We must scale the CIFAR images to 64×64. Each 8×8 MCU covers the same region of the original image as one 4×4 ViT patch, and both models operate on 64 tokens.

We acknowledge that converting existing 4:2:0 JPEG files to the exact 4:4:4 configuration used in our experiments requires decoding and re-encoding. Similar to the restart markers, this conversion is needed once.

AuthorsOpenReview ↗

Official response to YxZU Part 2

File size. Storage efficiency is the purpose of compressed formats, but it is not a claim or side effect of TEMPEST. Our estimate is 192–320 additional bytes per Tiny ImageNet image from the per-MCU restart configuration, about 10–16% of a 2 KB image at quality 80. This quantifies the restart-related overhead only, not the difference against every default encoder configuration. For MP3, the overhead is zero.

The reviewer is correct that our data preparation has a storage cost. We emphasize that this is a storage cost, not a computational one, and it does not affect any of our central claims.

We will revisit the appendix with the updated size statistics.

In light of (i) all our MP3 data being default, reservoir-enabled LAME streams, (ii) restart intervals and 4:4:4 sampling being standard JPEG features that any compliant decoder can read, (iii) restart markers being insertable into existing JPEG files with one offline jpegtran call that leaves the DCT coefficients unchanged and reconstructs no pixels, (iv) 4:4:4 sampling being a protocol choice for a controlled ViT comparison rather than a requirement of the method, and (v) the only remaining cost being storage (10–16% for JPEG, zero for MP3), not computation, we encourage the reviewer to reconsider W1.

Alternatives in the compressed domain.

We note that the alternatives the reviewer suggests also process the stored files before learning, often more heavily than TEMPEST: DCT-domain models entropy-decode every image on every forward pass, SeiT re-encodes the dataset with a pretrained ViT-VQGAN which has about 27 times more parametes than our block embeding network. [c] requires adopting a new learned codec. A one-time, lossless, offline marker insertion is always a lighter requirement than any of these alternatives. We invite the reviewer to reassess the suggested methods in the context of their complexity.

W2/Q3. Efficiency. Our architectural contribution is not that every stage inside TEMPEST performs less computation than every other alternative. As outlined in Section 1, our contribution is that structured compressed blocks can be transformed into short sequences of learned representations suitable for standard attention-based classification. In support of this claim, we show that the reduction is substantial against previous compressed-byte methods. For example, ByteFormer operates on 809 or 1,619 tokens per second, compared with 32 for TEMPEST, and Pérez et al. use 921 tokens on CIFAR-10, compared with 64.

Tokenizer Cost. We must correct the assertion about missing computational costs. Table 2 reports end-to-end GFLOPs, explicitly including the tokenizer (4.68 for TEMPEST vs. 19.2 for AST), Table 5 breaks down the tokenizer's share separately. In fact, the reviewer's own calculation of the tokenizer cost is taken directly from the values in these two tables. TEMPEST computational cost is fully disclosed in the paper.

The reviewer correctly notes that the block embedding network accounts for a significant share of TEMPEST's computation. We complement this observation by noting that the tokenizer's attention is block-local, so its cost grows linearly with the audio duration, and all computations run in parallel across blocks. The classifier's cost is quadratic in sequence length, and this is the stage where token reduction provides a clear improvement.

Overlapping patches in AST.

The reviewer argues that TEMPEST's favorable comparison in token sequence length is attributable to an unfair comparison with AST, which uses overlapping patches. We clarify that the original AST paper explicitly evaluates the removal of patch overlap (Gong et al., 2021, Table 5). Eliminating overlap reduces AST's sequence from 1,212 to 512 patches, a reduction of approximately 58%, while also decreasing classification performance (from 34.7 to 33.6 mAP on AudioSet Balanced).

In comparison, TEMPEST already reduces the sequence from 108 tokens per second to only 32, a reduction of approximately 70%. Furthermore, the non-overlapping AST configuration in the published ablation requires 512 patches for a 10-second input, compared with approximately 290 tokens for TEMPEST over the same duration.

Even after removing the overlap, AST cannot match TEMPEST's sequence reduction, even at the expense of AST's original performance. Finally, we note that TEMPEST achieves its shorter token sequence through the structure of compressed representations, rather than by modifying the patching strategy of a spectrogram-based model.

We agree that a matched-token-rate spectrogram Transformer would provide an ideal architectural control. However, such a model does not exist and would require a separate design, tuning, and empirical evaluation. More importantly, such a comparison would not alter our central claim: compressed-byte streams can be mapped to short semantic token sequences with competitive classification performance.

AuthorsOpenReview ↗

Official response to YxZU Part3

Reduction by a factor of 11.

We agree that this sentence in the Conclusions must state that it refers to the classifier; in fact, Section 4.2 already does. We will add parameter counts to Table 2 so the model-size difference with AST is visible.

Token sequences in images. TEMPEST makes no token-reduction claim against the RGB ViT. Equal token counts (64) are a deliberate protocol for comparing against the ViT baseline, and Table 3 reports the accuracy gap openly. Our token-reduction claim is against audio streams and compressed-domain methods, which the reviewer already lists as a strength.

JPEG block length. The padded MCU length is l=50 bytes on all image datasets.

Augmentations. During training, waveform augmentations are applied before encoding, as the reviewer correctly infers. We will state this explicitly and describe the data augmentation pipeline precisely, separating raw-domain augmentation at training time from the compressed-byte operations the model uses. This does not affect our premise, which concerns what the model learns from and runs on. Inference is performed only over compressed data, and the sentence in Section 5 will be scoped to inference.

W3. Weak or uncontrolled baselines.

We reject the claim that our baselines are weak or uncontrolled.

The AST results are the exact published from-scratch numbers, reported by the AST authors under their own tuned protocol. The from-scratch AST numbers are transcribed exactly as published by the AST authors, who tuned their own recipe (Gong et al., 2022). We did not redo any of their experiments or rerun any code to obtain them. We strictly use the baseline's published numbers, which were tuned by the same authors following their own standard protocol.

The comparison is direct and fair, in fact, it could favor the from-scratch baseline, since its recipe was optimized by the same people who designed it. The from-scratch baseline was also tuned per dataset.

ESC-50 baseline comparison. A CNN on log-mel spectrograms is not the direct comparison for TEMPEST. This model belongs to a family with fundamentally different inductive biases that is less prone to overfitting. The relevant baselines for our method are transformers trained from scratch, where TEMPEST (52.28%) exceeds AST (41.90%) by 10.4 points.

Small ablation differences and seed variance. We agree that small ablation differences (e.g., 91.24 vs. 91.18) should not lead to strong conclusions. In fact, the paper does not use them for that purpose. Section 4.3 describes these variations as "minimal" and "nearly identical." We are preparing seed variance experiments for the main results and will return later with updated results.

W4. Methods that operate after entropy decoding.

We respectfully disagree. Our manuscript does not "overlook or dismiss" this line of work: Gueguen et al. and Park & Johnson are already cited and discussed in Section 2 under their own heading. We will add [a]–[d] to the related work.

We also contest that the suggested methods are direct baselines. These methods target a different research question. Our premise is that learning is feasible without entropy decoding, using the coded bytes exactly as stored. A model that first Huffman-decodes to DCT coefficients does not test that premise; it removes it. These methods are also tied to one codec's transform, whereas TEMPEST applies the very same architecture to an audio codec and an image codec.

Similarly to AST, models that use decoding might achieve higher accuracy than TEMPEST, but this does not diminish or alter the validity of our results and findings. Finally, suggesting that the datasets be re-encoded with a learned codec [c] is a proposal for a different paper.

W5/Q5. Multi-bitrate inference.

We agree on two aspects and will correct them. First, the Table 8 caption can be improved (adding the third bitrate lowers accuracy relative to {26, 32}). Second, the paper already describes this procedure as test-time augmentation "analogous to multi-crop evaluation." We will align the abstract with that wording.

The main finding, however, is multi-bitrate training (Table 7: +1.09 on SC2, +0.84 mAP on AudioSet-Bal). It needs no re-encoding at test time and has no direct counterpart in raw-domain training. As stated in the paper, the byte streams for the same signal differ in approximately 90% of their values and act as augmentation. W5 does not address this result at all, although the review lists the multi-bitrate training ablation among the strengths.

Reviewer_7LXNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TEMPEST, a framework for learning directly from compressed MP3 and JPEG data. It groups bytes into format-specific blocks, maps each block to a compact representation, and uses a transformer for classification. An auxiliary byte-reconstruction loss helps train the block representations. The approach is interesting, and the experiments demonstrate its feasibility across audio and images. However, the current evidence does not sufficiently establish its applicability to existing compressed files or how much information the learned representations retain.

Critical strengths
  1. Using the structure of compressed formats to organize bytes into meaningful local blocks is a useful direction for representation learning.
  2. The experiments demonstrate the feasibility of the approach in both audio and image classification.
  3. The ablations examine several relevant design choices, including the number of tokens per block, the depth of the block embedding network, and the auxiliary reconstruction loss.
Critical weaknesses
  1. The meaning of “decoding-free” needs a clearer definition. The method requires format-specific processing to locate and extract blocks before they enter the network. The paper should clearly distinguish this processing from partial decoding. A useful distinction is that scanning markers and reading headers and lengths constitute format parsing, whereas using Huffman or CABAC decoding to recover coded symbols constitutes partial decoding, even without reconstructing pixels or audio samples. The scope of the decoding-free claim should be stated using an explicit distinction of this kind.

  2. The Byte-to-Block modeling assumption limits general applicability. Grouping bytes into meaningful blocks is central to the method, but it depends on blocks being sufficiently small, directly locatable, and relatively independent. These properties are not generally available across compressed formats and encoding configurations. Appendix A.1 explains that JPEG uses a restart marker after every MCU and that the MP3 bit reservoir is disabled. Existing compressed files are not guaranteed to satisfy these conditions, and other formats may not provide similarly accessible small units. This raises a concern about the applicability of the proposed modeling approach itself: the current experiments do not establish how it handles streams that lack these properties, or what additional processing would be required.

  3. The amount of information retained by the representations remains unclear. All downstream evaluations are classification tasks. These results do not establish whether the embeddings retain fine details or mainly capture coarse information sufficient for classification. For JPEG, for example, the results do not show how much information is preserved beyond block-average colors or low-frequency structure. Although the method includes a reconstruction loss, it reconstructs compressed bytes, and the reported ablation only measures the resulting classification improvement. Actual reconstruction performance is not reported. Reconstruction quality would provide a useful additional measure of how much content remains recoverable from the learned representations.

  4. The visual evaluation is limited in scale and resolution. The image experiments cover MNIST, CIFAR-10/100, and TinyImageNet. These benchmarks demonstrate feasibility on low-resolution images, but provide limited evidence about performance on larger and more detailed images. Further evaluation with comparable model and training budgets would strengthen the conclusions about visual representation quality.

Questions for authors
  1. What exact operations are performed to locate and extract blocks before the network processes them? Can the authors explicitly distinguish format parsing from partial decoding and define the decoding-free claim accordingly?

  2. How would the method construct useful blocks from existing JPEG files without per-MCU restart markers, MP3 files with the bit reservoir enabled, or formats whose directly accessible units are much larger? If conversion or additional decoding is required, what are the resulting costs and limitations?

  3. How accurately does the auxiliary reconstruction network recover the input bytes? Beyond byte recovery, could the authors evaluate image reconstruction from the learned embeddings, using quantitative quality measures and visual examples? A simple reconstruction probe with frozen embeddings would help assess whether fine details remain recoverable. This is intended as a diagnostic of the representations, rather than a requirement for perfect reconstruction.

  4. Can the authors provide an evaluation on a larger or higher-resolution image benchmark, using an RGB baseline with comparable model and training budgets? If this is outside the scope of the work, please clarify the limits of the conclusions drawn from the current visual experiments.

2 comments
AuthorsOpenReview ↗

Official Response to 7LXN Part 1

Response to Reviewer 7LXN

We thank the reviewer for the constructive and careful review, and for highlighting that our main idea, using the structure of compressed formats to organize bytes into local blocks, is a useful direction for representation learning.

We now address the reviewer's weaknesses (W) and questions (Q) individually.

W1/Q1. Definition of "decoding-free". We fully agree with the reviewer and adopt the proposed distinction. TEMPEST only scans byte markers and reads header fields, performing integer arithmetic over the header fields. It does not perform any entropy decoding to recover coded symbols, every byte that enters the network is copied exactly as stored in the compressed file.

This new definition creates a clearer distinction between TEMPEST and prior work that partially decodes the stream (e.g., DCT coefficients). We are updating the manuscript to explicitly address this difference, we will also add two algorithms describing the parsing of each format.

W2/Q2. MP3 bit reservoir. This is a fantastic question!. An in-depth analysis of our audio data shows that we can relax the block independence constraint and allow for some information sharing between audio frames. This is an exciting new finding, which we think further supports the case for compressed data in semantic feature learning. We now explain in detail.

Appendix A.1 states that the bit reservoir is disabled. We found that this is not accurate. All our MP3 data was encoded with LAME through ffmpeg (-c:a libmp3lame -b:a <rate> -cbr true). The LAME documentation states that the reservoir is enabled by default. It can only be disabled explicitly (--nores in the LAME command line, -reservoir 0 in ffmpeg), and neither option was used in our transcoding command.

For further reference we provide a link to the LAME documentation in ubuntu: https://manpages.ubuntu.com/manpages/resolute/man1/lame.1.html

For a full confirmation, we verified the training data directly. In Speech Commands v2, we randomly sampled 2.000 files and found that most frames (95% of them) use the reservoir. We conclude that the blocks used by TEMPEST can have some dependencies between them and still allow for effective semantic learning. We hypothesize that the classification network CC is resolving these dependencies as it is the only module that attends to all frames at the same time.

We believe this unexpected finding further supports the case for deep networks built on compressed representations. The fact that some dependency between blocks does not hinder semantic learning could be the key observation for more general tokenizers that work over any kind of compressed format, even if the format does not strictly define independent blocks.

For completeness and full transparency we will correct this assertion in Appendix A.1 and clearly state in our main paper that some level of dependency can exist between compressed blocks.

W2/Q2. Byte-to-Block modeling on existing files. Our approach does not change or extend the implementation of any codec. It uses only standard codec features and flags. Any compliant codec implementation can provide the data streams required by TEMPEST (JPEG or MP3). Restart markers are also an integral part of the JPEG standard, and are supported by every compliant decoder.

W2/Q2. JPEG without per-MCU restart markers. We agree that most existing JPEG files do not use a restart interval for each MCU, and the default value depends on the codec implementation. Nevertheless, existing files can be converted offline with a simple command which executes a transcoding, for example with the standard libjpeg tool:

jpegtran -restart 1B input.jpg > output.jpg

The resulting file has a restart marker after every MCU, without changing the quantized coefficients. This conversion is a one-time offline re-coding step, during inference no further encoding or transformation is required, only the parsing of the byte stream.

The main trade-off we observe is the additional storage size, as every MCU marker adds 3 to 5 bytes (2 marker bytes and between 1 to 3 additional bytes for resetting the DC component). An average TinyImageNet image at quality 80 is about 2.000 bytes, and at quality 90 it is about 2.600 bytes. With 64 MCUs the markers add about 192 to 320 bytes in both quality settings, which amounts to roughly 9.6% to 16% additional disk storage at quality 80, and 7% to 12.3% at quality 90. We do not consider this additional storage a significant increase.

W2/Q2 Other formats with block markers. Other widely used formats are also organized into self-delimited blocks:

  • Audio: FLAC, Opus and Vorbis.
  • Video: H.264 (NAL units).
AuthorsOpenReview ↗

Official response to 7LXN Part 2

W3/Q3. Information retained by the representations. As stated in lines 207-210, our goal is not to generate a perfect reconstruction, rather the reconstruction is used to regularize the semantic learning. Nevertheless, we observe that the reconstruction loss decreases consistently as the network optimizes the composite loss. We are preparing additional results as a diagnostic of the representation quality. We reply in the next days with an analysis

W4/Q4. Scale and resolution of the visual evaluation. We agree with the reviewer that higher-resolution benchmarks would strengthen the visual results. However, we believe the current experiments already test the key components and contributions of this work. Higher resolution extends the classifier’s sequence but leaves the tokenizer’s task unchanged. Across CIFAR-10, CIFAR-100 and TinyImageNet, the gap to the RGB ViT stays stable between 5.3 and 5.6 points, even as the number of classes grows from 10 to 100 to 200. We see no sign that the gap widens or closes with task difficulty. To address the reviewer’s request directly, we are training TEMPEST and a ViT baseline with matched model size and training budget on ImageNet-100 dataset at 160×160, and will report the results if they complete within the rebuttal period. Regardless of that outcome, we will state the scope of our visual conclusions explicitly in the revised paper: our image experiments establish that MCU-level tokenization of JPEG streams is feasible and remains within a consistent margin of an RGB baseline on low-resolution benchmarks (up to 64×64).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.