- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The work proposes a transformer-based architecture to classify audio in MP3 format and images from a JPEG format that has been augmented to include Minimum Coded Unit (MCU) markers. A sequence of bytes representing a block (frame, MCU) is processed and reduced to a much smaller sequence length. Each block embedding corresponds to a token in a secondary transformer that performs instance classification.
A reconstruction loss is added as regularization. Streams at different bitrates representing the same input are used as data augmentation. Both aspects seem to improve classification performance.
Results show comparable performance to similar architectures that use uncompressed inputs and better performance than other approaches that use compressed data. The sequence length of the input to the classifier model is significantly reduced.
Critical strengths
- I find the block embedding network that combines transformer layers with the channel-mixing subnetwork novel for this setting.
- The ablation shows interesting insights regarding training and inference with multiple bitrates.
Critical weaknesses
- The writing needs refinement. It is lacking coherence and I found some contradictions between sections. There are also several grammatical and spelling mistakes.
- The writing also needs to improve on clarity:
- Notation is not congruent. Constants have the same facetypes as functions and different facetypes among them. Representations have different facetypes too and the same as some constants and functions.
- There is conflicting notation. is a model and also a number of layers. and are both different constants. seems to be different than but the same as . and seem to be the same in some contexts but different in others. indexes both blocks and bytes. The dimensionality should probably be or .
- The mathematical definition of should specify the padding in order to be accurate. Otherwise, it might be best to drop it entirely and explain it in words.
- Similarly for . Either be precise or do not include a mathematical definition.
- Table 3 should highlight the model with the highest performance.
- I have concerns about the impact of this work. It requires that blocks are explicitly marked. Besides MP3, what other compressed formats have block markers? Changing standardized codecs to include block markers is not practically viable.
- The number of tokens does not necessarily dictate time or space complexity. For instance, ByteFormer has lower GFLOPs than TEMPEST even though it has a higher number of tokens. I can see how a similar situation could occur in terms of space complexity. For example, the block embedding network, although smaller, still consumes all bytes initially. ByteFormer has limited attention mechanisms and strided convolutions. Since task performance is on par, space complexity is the main selling point of this work and it is not very well supported.
- The performance of ByteFormer should be extended to the other datasets.
- The work only focuses on classification. I think that to support some of the claims made other types of tasks should also be evaluated. The quality of this venue demands stronger support.
- I find strange that increasing decreases performance. Is it because of overfitting? I am not sure how that behaviour leads to the claim that “our tokenizer can effectively pack all the relevant information into a single token”. A very bold claim with very weak support.
- is not defined in the main sections and an analysis of its impact on performance (besides the values 0 or 1) seems necessary given the results from the ablation study.
Questions for authors
If the authors can improve on some of the weaknesses I presented or show me that I am misunderstanding the work, I can increase my score.
3 comments
Official Response to XQAV Part2
Applicability and impact , Block independence Upon request of another reviewer we have revisited the audio data, and learned that strict block independence is not required for effective semantic learning in TEMPEST. We have discovered that the MP3 bit reservoir was in fact active by default during the LAME/ffmpeg transcoding. After sampling 2,000 files from Speech Commands v2 we confirmed that most frames (95% of them) utilize the reservoir, meaning that inter-frame dependencies were already present in the training data used for the main results and all the ablations in the paper. The competitive results in SC2 show that TEMPEST can effectively model some level of block inter-dependencies. As the block embedding network shares no information between blocks, we hypothesize that the classification network (C), is resolving these dependencies while also building a high level representation. This finding extends the viability of our proposed method, demonstrating that our tokenizer can still operate over more complex compressed streams where blocks are not strictly independent. We will update Appendix A and revise the main text to distinguish directly locatable blocks from independently decodable units.
Applicability and impact , other formats with block markers. Other widely used formats are also organized into self-delimited blocks:
- Audio: FLAC, Opus and Vorbis.
- Video: H.264 .
Effect of l'>1. We agree that “our tokenizer can effectively pack all the relevant information into a single token” is a strong assumption, however, we note that the results show that a single token is enough to achieve the best performance in all the selected datasets. We will rephrase and tone down this assertion in the main paper.
We don't think increasing (l') can result in an overfit condition, l=2 adds only 289 parameters over l=1, a negligible increase in the model’s parameters count. Moreover, overfitting does not explain the observed performance variation in ESC-50 (up to 1.43) and the relative consistency for the two other datasets, where every other variation remains below 0.32.
We hypothesize that this behavior is derived from the absence of explicit positional information for the latent tokens entering the reconstruction decoder (D). The encoder’s positional encoding identifies byte positions within each block, but does not explicitly specify the order of the resulting (l') latent tokens. When (l'>1), this may pose a harder reconstruction and thereby weaken the decoder’s effectiveness as a regularizer. With (l'=1), this ordering ambiguity disappears.
Value of lambda . We clarify that lambda weights the reconstruction loss (wrec), this has been corrected in the final equation of section 3. Table 6 compares lambda=0 with lambda=0.5, the default setting in all our experiments. We are currently running additional ablations to measure the sensitivity to on each dataset.
Thank you for the response.
I think the comments made in this rebuttal are addressing some of my concerns but I do not see the discussion making its way to the document. I empathize with the time constraints the authors are facing.
Applicability and impact seem to be addressed (H.264 having markers is great) but no relevant changes were made to the document. The hypothesis around the effect of could have been discussed as well.
Due to the withdrawal, I would suggest to the authors that they fix the notation, extend the idea past classification, and strengthen the claims. I would love to see the resulting work published. Good luck!
Official Response to XQAV Part 1
Response to Reviewer XQAV We thank the reviewer for the careful reading, highlighting the block-embedding network novelty and finding the multi-bitrate training and inference results insightful. We believe this emerging domain holds many more interesting properties to explore.
Below we address the reviewer's weaknesses and questions individually.
Grammar and Notation. We thank the reviewer for the careful reading. We are fixing the grammar and notation errors. We will upload the updated version of the PDF and we will keep updating as the discussion with the reviewers develop.
Block markers and standard codecs. We believe our presentation may have led to a small misconstruction, thus we clarify: Block markers are already part of the MP3 and JPEG standards. An MP3 file is defined as a sequence of frames, and the format defines a sync word that enables the location of beginning and end of the frames. Also, restart markers are also an integral part of the JPEG standard, and are supported by every compliant decoder.
We further emphasize that our approach does not need to change or extend the implementation of any codec. TEMPEST uses only standard codec features and flags.
Finally, our data pipeline simply parses the header and locates the byte markers (MP3 frame headers or JPEG restart markers) and feeds the bytes between consecutive markers to the block-embedding network.
We will augment our supplementary material with this clarification.
Comparison with Byteformer, computational effort We begin by noting that TEMPEST (4.68 GFLOPs, 91.24%) costs less than ByteFormer k=4 (5.49 GFLOPs, 89.19%) and is more accurate than both ByteFormer configurations (k=4 or k=8).
We agree with the review that token count alone does not determine the total computational cost, and that our tokenizer must process every byte in the first layer. Our Intra-block attention is equivalent to a non-overlapping windowed attention, as a result, tokenizers of TEMPEST and ByteFormer both scale linearly with input length. We do not claim an advantage over ByteFormer, rather we want to clarify that we are equal in this aspect.
The advantage of TEMPEST lies in what follows the tokenizer. After two layers, every block is reduced to a single token (32 tokens per second of audio vs. 1619 tokens for ByteFormer at k=4). Our network classifier runs multiple layers of full global attention over a much shorter sequence.
ByteFormer on other datasets. The ByteFormer paper reports audio results only on SC2 and the released code appears to be incomplete. We don't think that providing results from a manually patched codebase is the best reference for the performance of ByteFormer in other datasets.
TEMPEST is classification only. We agree that our evidence is limited to classification. We note, however, that classification is also the evaluation protocol of the prior work most directly comparable to ours (ByteFormer, Pérez et al. 2024, Park & Johnson 2023, Gueguen et al. 2018). We are evaluating additional experiments that can fit the window of the rebuttal and will come back to the reviewer when possible.
Applicability and impact, JPEG markers We agree that most existing JPEG files do not use a restart interval for each MCU, and the default value depends on the codec implementation. Nevertheless, existing files can be converted offline with a simple command which executes a transcoding, for example with the standard libjpeg tool:
jpegtran -restart 1B input.jpg > output.jpgThe resulting file has a restart marker after every MCU, without changing the quantized coefficients. This conversion is a one-time offline re-coding step, during inference no further encoding or transformation is required, only the parsing of the byte stream.
The main trade-off we observe is the additional storage size, as every MCU marker adds 3 to 5 bytes (2 marker bytes and between 1 to 3 additional bytes for resetting the DC component). An average TinyImageNet image at quality 80 is about 2.000 bytes, and at quality 90 it is about 2.600 bytes. With 64 MCUs the markers add about 192 to 320 bytes in both quality settings, which amounts to roughly 9.6% to 16% additional disk storage at quality 80, and 7% to 12.3% at quality 90. We do not consider this additional storage a significant increase.