acceptodds
Under review as a conference paper at ICLR 2027

ChannelTok: Efficient Flexible-Length Vision Tokenization

Abstract

Leading flexible vision tokenizers achieve SOTA quality at an extreme cost, relying on parameter-heavy backbones and slow, multi-step generative decoders. We depart from this complex, spatial-token paradigm and introduce a simple, lightweight, and fast channel-wise flexible-length tokenizer. Our method treats each latent channel as a visual token, enabling a parameter-efficient CNN-Transformer hybrid backbone. Furthermore, employing a stochastic tail-dropping paradigm during training naturally forces channels to organize by semantic importance. This allows for flexible compression at inference by simply retaining the first channels, and naturally enables variable-length autoregressive image generation. We validate our approach through extensive experiments on ImageNet, demonstrating consistent quality across diverse token budgets. The results establish a new quality-efficiency frontier: our model achieves state-of-the-art perceptual quality (rFID 2.92) while being 8.6 faster in decoding and 2.1 smaller (159M params) than the next-best alternative. Our work establishes channel-wise tokenization as a powerful and practical paradigm for efficient visual representation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Vision tokenization, Flexible tokenization, Latent representations

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZGgbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces ChannelTok, a flexible-length visual tokenizer that treats each latent channel map, rather than each spatial location, as a visual token. It uses a lightweight VQGAN-style CNN–Transformer autoencoder, stochastic prefix masking to order channels from global structure to fine detail, and independent Binary Spherical Quantization for active channels; varying the retained prefix length kk then provides different compression levels without retraining and also supports variable-length autoregressive generation. Unlike spatial token dropping, ChannelTok preserves the full spatial grid inside every retained channel and progressively removes global feature dimensions.On ImageNet, ChannelTok achieves an rFID of 3.70 at 256 tokens while using 159M parameters and decoding 8.6× faster than FlexTok, whose rFID is 2.97. At the full 512-token budget, it reaches an rFID of 2.92 while retaining the same lightweight, single-pass decoder.

Critical strengths
  • The channel-as-token perspective is fresh and interesting. Each token represents a full H×WH\times W feature map rather than a local patch, moving beyond the standard spatial-token paradigm and enabling natural coarse-to-fine rate control.
  • Random tail dropping encourages channels to organize by semantic importance, which is similar in spirit to PCA, although the hierarchy here is learned and nonlinear. Padding inactive channels also keeps the autoencoder tensors regular, making the design friendly to parallel computation.
  • The experiments show a strong quality–efficiency trade-off. ChannelTok uses a lightweight, single-pass decoder to achieve reconstruction quality close to the best baseline while being much smaller and faster.
Critical weaknesses
  • The evaluation is limited to ImageNet at 256×256 and does not really stress-test fine-detail preservation. Results on higher-resolution images or detail-sensitive cases such as small objects, rendered text, and documents would make the claim more convincing.
  • The method does not actually choose kk based on the complexity of each image. It samples kk randomly during training and requires users to specify it at inference, so the method is flexible-length rather than truly content-adaptive;
Questions for authors
  • How should the spatial feature-map size and the number of retained channels be balanced under a fixed bitrate or compute budget? Is there a principled way to choose the best H×WH\times W and kk, rather than fixing the spatial size and tuning only the channel count?
2 comments
AuthorsOpenReview ↗

Response to Reviewer ZGgb

We're grateful for your suggestions and overall support for ChannelTok. Thank you for recognising its novelty and architectural compactness, and for appreciating our experiments. We address your concerns below.


W1. The evaluation is limited to ImageNet at 256×256 and does not really stress-test fine-detail preservation. Results on higher-resolution images or detail-sensitive cases such as small objects, rendered text, and documents would make the claim more convincing.

Thank you for the suggestion! We agree that, from a deployment standpoint, a practical tokenizer should handle varying resolutions. We trained and evaluated at 256×256 to follow the standard protocol shared by all the methods we compare against, and with academic compute, this single full run was what we could complete for this submission. Supporting resolutions beyond 256×256 is something we have been thinking about for future versions, and the channel-wise formulation lends itself naturally to it, since the spatial size of each channel map h×wh \times w and the number of channels kk are independent. Higher resolutions could therefore scale h×wh \times w while kk continues to control the coarse-to-fine budget.

To stress-test fine detail, we have added qualitative reconstructions of scene text from TextOCR and ICDAR 2013 alongside small objects from COCO in Fig. 18, as well as tiled 512×512 reconstructions in Fig. 19 (App. G). Large text becomes legible from around 96–128 tokens and small objects sharpen progressively as channels are added.


W2. The method does not actually choose kk based on the complexity of each image. It samples kk randomly during training and requires users to specify it at inference, so the method is flexible-length rather than truly content-adaptive.

The reviewer is right to point this out, and we're sorry for the misleading phrasing. We have revised "adaptive" to "flexible-length" throughout the paper. Reviewer A5CJ raised the same point, and we address it at length in our response to them, including the specific passages we revised. We kindly refer the reviewer there for more details.


Q1. How should the spatial feature-map size and the number of retained channels be balanced under a fixed bitrate or compute budget? Is there a principled way to choose the best H×WH \times W and kk, rather than fixing the spatial size and tuning only the channel count?

This is an interesting question! In our design, we fix h×wh \times w as an empirical choice and vary kk. This is similar to design choices in spatial tokenizers, which fix the channel dimension of the latent and vary the number of spatial tokens.

What makes this convenient is that, with h×wh \times w and the quantizer fixed, every token carries the same amount of information, one 16-bit BSQ index, so the bitrate budget and the token budget are tied together through kk. The bitrate is 16k16k bits, and the token budget, which determines downstream AR compute, is kk itself. Choosing kk thus sets both at once.

We chose the per-token size by balancing two effects. A larger codebook packs more information into each token, which improves reconstruction at a fixed token budget. However, it also enlarges the embedding and output layers of any downstream generator. With 2162^{16} codes, each token carries 16 bits, the same as FlexTok, while our vocabulary intentionally stays well below those of modern LLMs, such as Llama 3 with 128k and Qwen3 with 152k, so that adding our tokens to such a model remains a moderate change.

Choosing h×wh \times w and kk jointly for a given budget or image is a promising extension, but it is closer to adaptive allocation, which serves a different purpose. We wish to give the user predictable control over the budget. A sweep over h×wh \times w and codebook size under matched bitrates is something we look forward to exploring in future work.


Once again, thank you for your thoughtful review and support!

Thanks for the detailed responses. My concerns have been addressed and I would keep my original rating (3: Weak acceptance).

Reviewer_A5CJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ChannelTok treats latent channels as visual tokens and uses stochastic prefix masking to support flexible-length reconstruction and autoregressive generation with a lightweight autoencoder.

Critical strengths

(1) A simple architecture with promising model-size and decoding-efficiency advantages.

(2) Reconstruction results across multiple budgets demonstrate useful progressive refinement.

Critical weaknesses

(1) The discrete token representation is underspecified. Each channel contains a 16×16 feature map, but the quantizer uses an implicit codebook of size 2^16. The projection or grouping that maps a channel to discrete indices is not explained, leaving the actual bitrate and AR sequence length unclear.

(2) The evaluation does not fully support the headline claims. Generation FID is measured against tokenizer reconstructions rather than real images, and quantitative AR baselines are absent. Moreover, the best rFID of 2.92 uses 512 tokens; at the shared 256-token budget, ChannelTok achieves 3.70 versus FlexTok’s 2.97.

(3) The scope of adaptivity is overstated. Training lengths are randomly sampled, and inference lengths are user-specified. This demonstrates flexible-length tokenization but does not establish content-adaptive allocation. The distinction from existing nested-dropout approaches also needs clarification.

Questions for authors

(1) Where exactly are inactive channels removed during the training forward pass? Please reconcile Eq. (2), Algorithm 1, and inference.

(2) How many discrete indices and bits represent each channel, and how does LlamaGen predict them?

(3) Please report FID against real ImageNet images and compare generation quality at matched token or bit budgets.

(4) Please clarify the adaptivity claims and provide the referenced appendix and Tables 4–6, which are absent from the supplied PDF.

2 comments
AuthorsOpenReview ↗

Response to Reviewer A5CJ

We are grateful for the reviewer’s constructive feedback, which pushed us to sharpen how we frame our contributions and to state our claims more precisely. We are glad the reviewer saw value in the simplicity of our design, its efficiency gains, and its utility at decoding.


W1/Q2. The discrete token representation is underspecified. How many discrete indices and bits represent each channel, and how does LlamaGen predict them?

Each channel is represented by one discrete index, corresponding to 16 bits. We clarify that a channel token is a 4×44 \times 4 map, not 16×1616 \times 16. The encoder produces a 16×1616 \times 16 latent, and two further downsampling layers reduce it to 512×4×4512 \times 4 \times 4 before quantization. BSQ then takes the 16 values of each channel, normalises them, and keeps one sign per value, with no extra projection. On the decoder side, two extra upsampling layers bring the latent back to 16×1616 \times 16. At a budget of kk channels, an image is therefore kk indices and 16k16k bits, and the entropy-coded numbers for all methods are in our response to Reviewer drgs. We have stated this in Sec. 3 and corrected the latent resolution in Table 4.

For the AR experiments, we used a smaller ChannelTok tokenizer with a 2142^{14} codebook and up to 256 channels. We first encode each training image into its sequence of channel indices, so LlamaGen sees a vocabulary of 16,384 codes plus BOS, EOS and PAD tokens. We then train it unconditionally with standard next-token prediction. Starting from BOS, it predicts the index of the first channel, then the second, and so on, so that a budget of kk channels maps to a sequence of kk tokens. Since generation can be stopped at any point, the earliest tokens appear in every output, which is why we weight the cross-entropy towards them. At inference, we sample autoregressively from BOS until the chosen budget kk, zero-fill the remaining channels, and decode with the ChannelTok decoder, exactly as in tokenizer inference. We are glad to see how naturally the tokenizer's coarse-to-fine ordering carries over to generation, and we give further training details in App. B.1.


W2/Q3. The evaluation does not fully support the headline claims. Generation FID is measured against tokenizer reconstructions rather than real images, and quantitative AR baselines are absent. Moreover, the best rFID of 2.92 uses 512 tokens; at the shared 256-token budget, ChannelTok achieves 3.70 versus FlexTok's 2.97.

We thank the reviewer for raising this, and address each part below.

AR generation: We agree that head-to-head AR comparisons on ImageNet-1K would strengthen the paper. Training a full ImageNet-1K AR model within the rebuttal window is beyond the compute available to our academic lab, but we have started these scaled experiments and aim to include them, with standard gFID against real images at each token budget, in the camera-ready version. Our current AR experiments on ImageNet-100 are a proof of concept showing that the channel-wise ordering transfers to autoregressive generation, including generation truncated at any token budget. They are not a claim of state-of-the-art generation quality, and we agree there is scope for scaling. We kindly ask the reviewer to consider our AR experiments in this light. We also agree that our reported gFID should follow the standard protocol, rather than being measured against tokenizer reconstructions.

256-token comparison with FlexTok: We agree, and we report this comparison directly. At 256 tokens, ChannelTok and FlexTok have matched sequence lengths, and our new Huffman-coded bitrate analysis, given in our response to Reviewer drgs, shows that their compressed sizes are also matched at 4,065 and 4,084 bits per image. ChannelTok's result therefore does not benefit from a larger representation. In this rate-matched setting, FlexTok achieves better rFID, 2.97 versus 3.70, while ChannelTok attains better LPIPS, 0.169 versus 0.228, and SSIM, 0.536 versus 0.523, with the benefit of a single-step decoder that is 8.6× faster and less than half the size.

The 512-token result: We will clarify that the 512-token result is a demonstration of full-capacity scaling within a single model, not a token-matched comparison with FlexTok at 256 tokens. Our central claim is the quality–efficiency trade-off, where we show competitive reconstruction from a substantially smaller and faster tokenizer that supports any budget without retraining.

(Continued in the next comment.)

AuthorsOpenReview ↗

Continued Response to Reviewer A5CJ

W3/Q4. The scope of adaptivity is overstated. Training lengths are randomly sampled, and inference lengths are user-specified. This demonstrates flexible-length tokenization but does not establish content-adaptive allocation. The distinction from existing nested-dropout approaches also needs clarification. Please clarify the adaptivity claims and provide the referenced appendix and Tables 4–6, which are absent from the supplied PDF.

We agree that "content-adaptive" exaggerates our contribution and suggests fully automatic allocation based on visual complexity. As suggested, we have revised this wording to "flexible-length" throughout the paper, including the contributions list in Sec. 1, the method description in Sec. 3, the masking section and Algorithm 1, which we now call stochastic prefix channel masking, and the captions of Figs. 2 and 11.

With ChannelTok, our goal is to provide a quality–bitrate knob at inference. We optimise for controllability, which serves a different purpose from fully automatic token allocation. For example, our design is well suited to constrained settings where the user wants explicit control over the token budget and cannot afford larger budgets even when the image is complex. There is certainly value in content-adaptive allocation, and we take this feedback as direction for future work, where we envision training a lightweight token-budget predictor alongside the encoder.

Distinction from nested dropout: Prefix masking itself is closely related to nested dropout (Rippel et al., 2014) and Matryoshka representations (Kusupati et al., 2022), which order the units of a single feature vector. We'll make this connection explicit in the paper. Our contribution is not prefix masking in isolation, but applying nested tokenization along the channel axis of a latent. Given z∈RC×h×wz \in \mathbb{R}^{C \times h \times w}, ChannelTok treats each full slice zc∈Rh×wz_c \in \mathbb{R}^{h \times w} as one token, quantizes only the active prefix q1:k=QBSQ(zc)c=1kq_{1:k} = \\{Q_{\mathrm{BSQ}}(z_c)\\}_{c=1}^{k}, zero-fills the remaining channels, and decodes with the same single-step decoder. Unlike spatial truncation, every prefix reconstructs a complete image. Adding tokens refines the entire image rather than expanding spatial coverage, which enables efficient flexible reconstruction and variable-length AR generation.

Appendix: We apologise that the appendix was not visible in the version the reviewer received. The re-uploaded PDF now includes the full appendix, with the model architecture in Table 4, the training configuration in Table 5, and the loss and channel-masking settings in Table 6 (App. A). We have also updated the supplementary material.


Q1. Where exactly are inactive channels removed during the training forward pass? Please reconcile Eq. (2), Algorithm 1, and inference.

Sorry for the confusion here. This comes from a notational ambiguity in Algorithm 1, which Reviewer drgs also raised, and we explain it in detail in our response to them. In summary, during training, inactive channels are zeroed both before and after BSQ, so the decoder never sees them and they receive no gradient. At inference, we pick kk and apply exactly the same masking. We have fixed Algorithm 1, Eq. (2) and the Fig. 2 caption in the revision.


We would be very grateful if the reviewer would consider these changes in their assessment, and we would be happy to answer any further questions.

Reviewer_drgsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ChannelTok, a flexible-length visual tokenizer that treats individual latent channels as visual tokens. Starting from a lightweight VQGAN-style encoder-decoder, the method applies stochastic prefix masking dropping during training. Early channels are encouraged to encode coarse and semantically important information while later channels progressively refine details. The paper further applies BSQ to discretize the channel representations and demonstrates that the resulting ordering can be used for variable-length autoregressive image generation.

Critical strengths
  1. ChannelTok is substantially smaller and faster than strong flexible-tokenization baselines while maintaining competitive reconstruction quality. At 256 tokens, it is about 8.6× faster than FlexTok in decoding with less than half the parameters. This is consequential because the efficiency gain is large enough to make the method practically meaningful, not just a modest architectural improvement.
  2. A simple flexible representation that also supports variable-length generation. The learned channel ordering gives usable reconstructions across different token budgets and transfers naturally to autoregressive generation, where shorter sequences provide substantial speedups. This broad usefulness strengthens the contribution beyond reconstruction alone.
Critical weaknesses
  1. The masking implementation is unclear. Eq. 2 seems to keep inactive channels in the decoder input using stop-gradient during training, while inference sets them to zero; the authors should clarify this difference.
  2. Token counts may not be directly comparable across methods. One ChannelTok token is a full spatial channel, so reporting bitrate or bits per image would make the compression comparison more convincing.
Questions for authors
  1. Does the decoder receive the stop-gradient values of inactive channels during training, or are those channels actually set to zero before decoding? This point should be made explicit.
  2. How many bits are required for a ChannelTok representation at 32, 64, 128, and 256 tokens? This would help make the comparison with other tokenizers fairer.
  3. Can the authors report standard ImageNet generation FID in addition to FID relative to tokenizer reconstructions?
1 comment
AuthorsOpenReview ↗

Response to Reviewer drgs

Thank you for your valuable feedback. We're glad that you recognised ChannelTok's efficiency and its support for variable-length generation. We address each concern below.


W1/Q1. Does the decoder receive the stop-gradient values of inactive channels during training, or are they set to zero?

Thank you for bringing this to our attention. We agree that the notation in Algorithm 1 is ambiguous and apologise for the confusion. To answer directly, inactive channels are set to zero before decoding, in both training and inference.

The ambiguity comes from where the stop-gradient is applied. Line 5 of Algorithm 1 and Eq. (2) write it on the raw latent, sg(z)\mathrm{sg}(z), which evaluates to zz in the forward pass and therefore suggests that the decoder sees inactive channels during training. In our implementation, the stop-gradient is instead applied to the already-masked latent:

zinput=M⊙z+(1−M)⊙sg(M⊙z),q=M⊙Q(zinput).z_{\text{input}} = M \odot z + (1 - M) \odot \mathrm{sg}(M \odot z), \qquad q = M \odot Q(z_{\text{input}}).

Since the inactive entries of M⊙zM \odot z are already zero, the stop-gradient term is zero too, so in the forward pass zinputz_{\text{input}} is simply M⊙zM \odot z. Inactive channels reach the quantizer as zeros and get no gradient. We then apply the mask once more after quantization. This second step is needed because BSQ normalizes onto the unit sphere, so a zero input would otherwise come out as a nonzero code. The decoder therefore only ever sees zeros in channels k+1k+1 to CC, and we checked in training mode that this holds at both the quantizer input and the decoder input. Inference works the same way, where we build the prefix mask for the chosen kk and apply it at the same two places. That's what lets a single model serve every budget.

We'd like to add that this correction has no bearing on the method or the reported results. We have now corrected Sec. 3.3, Eq. (2), Algorithm 1, the Fig. 2 caption and Table 6 to apply the stop-gradient to the masked latent.


W2/Q2. Token counts may not be directly comparable across methods. How many bits are required for a ChannelTok representation at 32, 64, 128, and 256 tokens?

We are grateful for this suggestion. We initially kept token count as the primary axis, as all compared methods do, since downstream AR cost scales with sequence length. However, we agree that token count alone is not a perfect proxy for rate across spatial and channel-wise tokenizers, since what a token represents differs. We have therefore added bitrate to the revised paper (Table 9, App. F). As requested, the bits per image are:

Method (codebook)Bits/token32 tok.64 tok.128 tok.256 tok.
ChannelTok (2162^{16})16511102420474065
FlexTok (64,000)15.97510102020424084
KARL (4,096 + halt bit)1339479516023172
One-D-Piece (4,096)1238877715552940
ALIT (4,096)1237876714742530

Bits/token is log⁡2\log_2 of the codebook size; the other columns give entropy-coded bits per image on 10k ImageNet-val images. DOVE uses continuous tokens and is omitted.

Although a channel token spans the full spatial grid, BSQ maps each 4×44 \times 4 channel slice to a single 16-bit index, so it carries no more information than a FlexTok token, and the comparison between the two is rate-matched. Our token-matched comparison with FlexTok in Table 1 is therefore not inflated by a larger per-token payload. At this matched rate, ChannelTok still decodes 8.6× faster than FlexTok with less than half the parameters. ALIT, KARL and One-D-Piece use fewer bits per token because of their smaller codebooks, yet ChannelTok holds up favourably even at a lower bitrate. With 128 tokens, it uses about 1.2× fewer bits than ALIT at 256 tokens yet reaches a roughly 1.7× lower rFID, improves on One-D-Piece by about 1.3×, and roughly matches KARL.

We'd also like to clarify that codebook size is a design choice independent of the channel-wise formulation. For instance, the smaller tokenizer we trained for our LlamaGen experiments (App. B.1) uses a 2142^{14} codebook. For the main model, we chose 2162^{16} to stay well within the vocabulary sizes of modern LLMs, such as the 128k vocabulary of Llama 3.


Q3. Can the authors report standard ImageNet generation FID?

We agree this is the standard metric. We are scaling our LlamaGen experiments from ImageNet-100 to full ImageNet-1K, and aim to report standard gFID against real images in the camera-ready version.


Once again, thank you for your overall support for the submission!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all three reviewers for their detailed and valuable feedback. We're glad they found the channel-as-token idea novel and simple. They also highlight ChannelTok's efficiency, its compact single-step design, and its support for variable-length autoregressive generation. Their suggestions have pushed us to tighten how we frame our contributions and to revisit areas of ambiguity, and we are grateful for that.

The reviewers raised a few shared points, which we have addressed in the individual responses and in the revised paper:

  • Masking (drgs, A5CJ): The stop-gradient notation was ambiguous. Inactive channels are zeroed before decoding in both training and inference, and this correction does not change the method or any reported results. We have corrected the notation throughout the paper.
  • Bitrate (drgs, A5CJ): Each channel is a single 16-bit index, and we now report bits per image for all discrete tokenizers. ChannelTok is rate-matched with FlexTok and remains competitive against the other baselines at equal or lower bitrates.
  • Adaptivity (A5CJ, ZGgb): We agree that ChannelTok is flexible-length rather than content-adaptive, and we have revised the wording throughout the paper.
  • Fine detail (ZGgb): We have added qualitative reconstructions of scene text, small objects and tiled 512×512 images to the appendix.
  • Generation FID (drgs, A5CJ): Our existing AR experiments on ImageNet-100 are encouraging and indicate that the channel-wise ordering transfers naturally to variable-length generation. We appreciate the reviewers' suggestion to evaluate further at scale and are now training on full ImageNet-1K. As this takes longer than the rebuttal window allows on academic compute, we aim to include standard gFID against real images (for the scaled run) in the camera-ready version.

We have also updated the supplementary material, and we value reviewers' detailed feedback. We would very much welcome their thoughts on our responses and the revisions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.