acceptodds
Under review as a conference paper at ICLR 2027

Image Tokenizers Should Be Trained Between Images

Abstract

A visual tokenizer encodes an image into a smaller representation, and its decoder turns that code back into the image. However, the decoder is trained only at the codes the encoder produces from the training images. It never learns what to do with codes that lie in the space between training samples. Codes in this space lie outside of the training distribution and the decoder produces blurry images with strong artifacts there. For image generation, a second model, the generator, learns to produce images as codes of the encoder of a tokenizer. But, if generated codes land outside of what the decoder expects, the decoded image will contain imperfections. In order to make the decoder more robust to generated codes, current methods add noise to codes during decoder training. However, recent research indicates that generated codes do not simply contain noise. Instead, they lie between the codes the generator was trained on, and resemble an average of codes of training images more than one of them plus noise. Therefore, to improve quality of generated images, decoders should be trained to handle codes in the space between training samples. Motivated by these insights, we propose InterpTok, which fine-tunes MAETok by training the decoder to generate natural-looking images on mixtures of the codes the encoder produces. But no target image exists for an image between two training samples. So a discriminator judges whether each decoded mixture of codes looks as natural as the images whose codes were mixed. Additionally, a mixing-ratio loss requires the decoded image to visually keep the mixing ratio of the two codes. This prevents a collapse to a trivial solution and trains the decoder to produce a sharp and natural composition of both images. InterpTok keeps MAETok's reconstruction quality and improves interpolation quality more than threefold, the best among the thirteen tokenizers of a released comparison. Beyond interpolation, we show that our method can substantially improve generation training speed as well as the generation quality with much less computational cost. InterpTok improves the generation FID of a matched diffusion transformer from 11.61 to 8.48 without guidance, and even reaches 8.11 at an eighth of the sampling steps, where the generator on MAETok's codes worsens. Furthermore, the generator trained on our tokenizer surpasses the final results of the generator trained on MAETok at half the training steps. Our findings show that the space between image codes can and should be explicitly supervised rather than ignored.

Primary area:
generative models
Keywords:
image tokenizer, latent interpolation, latent diffusion, autoencoder, representation learning, generative models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_oBZvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses a mismatch issue in latent diffusion: a generator outputs codes that lie between training-image codes, but the tokenizer's decoder is trained only on training codes (or noisy versions of them). InterpTok fine-tunes MAETok with an interpolation objective: the decoder learns to decode mixtures of two image codes, using a DINO-feature discriminator and a mixing-ratio loss that stops it from copying one source image. Results: iFID improves from 14.42 to 4.30, the best of 13 tokenizers, and a same-recipe SiT-B generator improves from gFID 11.61 to 8.48. A decoder-only version gets 8.56.

Critical strengths
  1. The problem is real, well presented and backed by an independent benchmark study with reproduced numbers (Table 28; iFID errors under 4%).

  2. The results are strong and consistent: best iFID of 13 tokenizers with a near 2× margin over the runner-up (RAE, 7.68), and gains that hold across four feature spaces (Table 5).

  3. The ablations are useful making the reporting complete. Interpolation alone hurts the linear probe (73.0→63.5) and gFID (18.90); regularization alone doesn’t move iFID; only the combination works (Table 3). The paper reports the failures instead of hiding them. Hyperparameters, loss weights, and training cost are all given in the appendix, and code and weights are also promised.

Critical weaknesses
  1. The headline metric is close to the training objective. The interpolation loss trains the decoder on midpoints of neighbouring training codes. iFID measures decoded midpoints between a validation code and a nearest training code. The interpolation-only version, which uses no Inception features at all, still reaches iFID 5.44 (Table 3). Part of the gain may be direct optimization of what is measured.

  2. Generation is scored by FID only. In the fewer-steps analysis, samples move toward class means (within-class similarity 0.43→0.56; class accuracy 61% --> 83%, Table 23), and FID rewards that even if coverage drops. Precision and recall are reported for decoded mixtures (Table 12), not for generated samples.

  3. The cost is worth noting. The fine-tune takes about 576 GPU-hours to the 41k checkpoint and 1,152 GPU-hours for the full run. This is a tradeoff to the claimed savings.

Questions for authors
  1. Does the interpolation objective help on a second continuous 1D tokenizer (e.g., SoftVQ-VAE or l-DeTok)? If this has not been tried, should the "training principle" claim be narrowed?

  2. At matched total cost — your fine-tune plus generator training, against the baseline — does the advantage hold?

Reviewer_jvusOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The authors propose InterpTok, an image tokenizer that is trained to be more robust to predicted codes through interpolating latents. Specifically it builds on the observation that generative models that operate in latent spaces don't always predict latents in regions with high density, and that because of that other papers have added noise to the latents during tokenizer training to make them robust. The authors then propose to interpolate encoded latents of different images, and train the tokenizer with a discriminator and a bunch of other regularization losses to make the interpolated latents reconstruction a valid image. They then show that models trained on this latent space converge faster.

Critical strengths

Making decoders more robust to bad generative models is quite useful, and I do think this is a worthwhile investigation into that.

Critical weaknesses

My main concerns with the paper are around the use of only imagenet and FID for generation that makes certain parts of the somewhat large mixture of auxillary losses hard to judge. As I see it, a tokenizer that is trained for the ultimate objective of generation should show both A) good quality and B) alignment to the prompt/condition. The paper mostly investigates A, but B not sufficiently. A is easy to game as well, and the paper even uses an aux loss (Representation Fréchet Loss) that specifically targets FID. Let's assume as an example that the encoder produces random features and that the decoder is a good generative model; then it would get great gFID but poor alignment given any conditioning. I am aware that good reconstruction can rule this out, but the paper trains a decoder explicitly to be good at FID and measures gFID. Showing results only on imagenet class-to-image generation also doesn't really solve this issue, as theoretically the encoder could just output class information and the decoder samples from that. Training a text-to-image model and measuring prompt alignment could be a way to address this shortcoming. The interpolation objective of the paper could be seen as explicitly removing some of the nuance and making the decoder just more robust, but not the latents. This is also shown in the paper's experiments with MAETok (Tab 17a), where the gains between MAETok + their decoder vs InterpTok are very small and decoder fine-tuning seems to do a lot of the heavy lifting. Also it seems that with scale, the benefits vanish quite a bit.

Questions for authors
  • For some reason in many tables there are numbers missing, why? Tab 3, 16, 17. Especially the missing number in Tab 3 would show whether the DINOv3 losses drive the gFID gain.
  • I wonder about how this compares to using classifier-free guidance? CFG may do something similar, in that it predicts latents that are more "canonical". The reasons why CFG was not used or ablated seem quite weak. It would be great to show a CFG sweep for both MAETok and InterpTok.
Reviewer_7nHrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the mismatch between training an image decoder at encoded data points and using it on intermediate latent representations during generation. InterpTok adapts MAETok with supervision on interpolated codes, conditional adversarial and mixing-ratio objectives, representation alignment, and a representation Frechet loss. On ImageNet-256, it improves interpolation quality and downstream generation. Keeping the encoder and generator fixed and replacing only the decoder reduces generation FID from 11.61 to 8.56; the full tokenizer reaches 8.48 with the matched SiT-B generator.

Critical strengths
  1. The fixed-generator experiments give convincing evidence that decoder behavior beyond reconstruction matters for generation. Table 2 holds generated codes, starting noise, and sampler fixed and improves FID from 11.61 to 8.56. Appendix B.2 also improves the released SiT-XL generator from 2.46 to 2.26 using the same generated codes. These comparisons support a useful contribution without requiring generator retraining.

  2. The ablations provide meaningful support for interpolation training. At the original 174M parameter count, the frozen-encoder interpolation-discriminator variant reaches 9.71 FID, versus 11.61 for MAETok; Table 17 also reports 10.80 for reconstruction-only adaptation. The 217M joint-training variant without interpolation reaches 21.15, versus 8.48 for the full model (Table 3 and Appendix E.2). Together with evaluations in multiple feature spaces, these results make the empirical case stronger than an improvement in a single reconstruction metric.

Critical weaknesses
  1. The remaining component attribution is incomplete in the strongest fixed-generator setting. I acknowledge that the paper already includes a 217M no-interpolation ablation for joint encoder-decoder training and an interpolation-only result at the original capacity. However, the 8.56 result combines added decoder capacity with several objectives, and these controls do not fully separate those effects when the encoder and generator are fixed. In particular, Appendix E.2 includes Inception features in the Frechet loss, but does not isolate removal of that loss alone. Gains in other feature spaces mitigate metric-specialization concerns; the unresolved issue is how much of the full decoder-swap gain is due to interpolation, capacity, and distribution matching. This limits the precision of the mechanism claim rather than negating the demonstrated improvement.

  2. The efficiency claim needs a complete cost comparison. Table 25 does report tokenizer adaptation cost: 72 hours on eight A100-40GB GPUs to 41,000 steps. The remaining question is whether the saved generator training time offsets this additional cost, including feature preprocessing, at a matched target FID. Fewer generator steps do not establish a corresponding reduction in total training time, and fewer sampling steps do not include the larger decoder's cost. This matters for the practical efficiency claim, although the quality improvement remains useful.

Questions for authors
  1. In the fixed-encoder/fixed-generator setting, can you disentangle the 8.56 FID result with matched-capacity, matched-step decoder ablations that remove interpolation supervision and the Frechet loss separately? This would clarify the attribution of the strongest decoder-swap result.

  2. Including the reported tokenizer adaptation and feature preprocessing, what is the total GPU time to reach the same target generation FID as MAETok? Please distinguish generator-step savings from total training cost, and clarify whether the sampling-cost comparison includes decoding.

Comments on the paper

4 comments

Thank you for the detailed rebuttal and the many genuinely interesting new experiments. It has addressed several of my concerns. The Frechet loss is clearly not the main driver behind the gFID gains, and the reconstruction numbers rule out the illustrative extreme scenario that the encoder only keeps class information.

That said, I think that the new results mostly just confirm my concerns. With CFG, the full InterpTok shows no improvement over MAETok. Only the decoder fine-tuned on MAETok's frozen encoder improves, and it does worse than MAETok's decoder at the highest guidance scales. In t2i, InterpTok has a lower CLIP score than MAETok at every guidance scale, at guidance 4 its FID is also slightly worse, and there the decoder-only variant is the best on both. The ablations also suggest that the DINOv3 feature mixing-ratio loss is the most important component, especially on SiT-XL, while source conditioning and the pixel mixing-ratio loss barely matter for gFID. And on SiT-B, a decoder trained with noise gets most of the gFID gains while its iFID is no better than fine-tuning with MAETok's own losses, which makes me question whether better decoding between codes is really what drives the improvement.

Overall it seems that the decoder fine-tuning results are quite good, but the paper's main claims are still built on results w/o cfg and on training the full tokenizer with all of the proposed techniques, and imo the evidence doesn't support that framing. Given the thorough rebuttal, I'll raise the soundness but lean towards keeping my rating. I would suggest that the authors try to distill the main reasons of the performance gains and sharpen the claims and presentation of the paper around that.

Thank you for the follow-up and for the suggestion to distill the main reasons for the gains. We agree that most of the gain comes from the decoder. Our claim, however, is not that the encoder or the decoder matters more. It is that tokenizers should be trained between images, where generators generate, and we show that this works both for the whole tokenizer and as a fine-tuning of the decoder alone. Fine-tuned alone, the decoder improves existing generators without retraining them, on every tokenizer we tested, and on MAETok also with guidance and in text-to-image generation. This comes without the cost of the current approach of adding noise. Noise makes the decoder robust by turning it into more of a generator, which lowers reconstruction quality and can make larger generators worse. Supervising the space between codes, which standard training leaves out, makes the decoder robust while keeping it faithful to its codes. Training encoder and decoder together with our method brings further improvements, namely a more structured latent space, faster training than with our decoder alone, and good generation with far fewer sampling steps.

Fewer sampling steps and faster training. With an eighth of the sampling steps, the generator on InterpTok's codes even improves, while the generator on MAETok's codes worsens, with MAETok's decoder as with ours (Table 2). The generator on InterpTok's codes also learns faster. With 32 sampling steps, it reaches after 100,000 training steps the quality that MAETok's generator with our decoder reaches only after 200,000 (Fig. 18, page 22).

Why both parts matter. Training the encoder with our method makes the latent space more structured. A linear probe recognizes the class of more images from InterpTok's codes than from MAETok's, although the tokenizer never sees a class label (Table 3). Such a latent space is easier for a generator to learn, but it pays off only together with a decoder that decodes well between codes. Trained with only our regularization losses or only our interpolation objective, the tokenizer leads to a worse generator than MAETok, and only both together lead to a better one (Table 3).

Further tokenizers. On VA-VAE, too, training encoder and decoder together improves generation more than fine-tuning the decoder alone, at every number of sampling steps, and at 16 steps about twice as much. This holds true even when the decoder is fine-tuned three times as long (Table 28).

Guidance and text-to-image generation. With guidance, the gain comes from the decoder, and our decoder alone is the best variant. At its best guidance scale, InterpTok is on par with MAETok, and above that scale its result stays stable, while MAETok's and our decoder's worsen, so with our encoder the guidance scale needs less tuning (Table 26). In text-to-image generation, InterpTok's FID is lower than MAETok's at every guidance scale except the highest, where the difference is as small as that between two training runs of the same SiT-B on ImageNet (App. B.10). Its CLIP score is slightly lower, and the gap nearly closes at high guidance (Table 27). (1/2)

Training between codes against noise. Adding noise improves the gFID of the small SiT-B, as expected, since noise is the usual way to make a decoder more robust. However, it does only that. It makes the decoder more robust by making it more of a generator. This costs reconstruction quality, and on the much larger SiT-XL, the decoder trained with noise worsens the gFID compared with MAETok's decoder. Our decoder improves both generators, because it addresses the more specific behavior of denoising generators, which generate between the codes of training images. We train it to decode between images, where it does not simply hallucinate content but draws on the visual and semantic information of both codes. This allows improvements without sacrificing reconstruction, although we decode 3.5 mixtures for every reconstruction (Sec. 4.3), most of them near the middle between both images. We agree that the feature mixing ratio loss is the most important single component. It does not do all the work, though. On the SiT-B, our decoder keeps most of its gain without it (Table 3). Together with the pixel mixing ratio loss and the source conditioning of the discriminator, the feature loss keeps the decoder faithful to the information of both codes, visually and semantically, instead of letting it generate whatever looks natural. This supervision is possible only because both codes belong to real images, and with noise there is no information to which it could be applied. On the SiT-XL, every decoder in our ablation that decodes well between codes improves the gFID, and every one that does not worsens it, the decoder trained with noise included (App. B.11).

Training between images thus improves generation more than adding noise and keeps reconstruction quality. It works for the decoder alone, and applied to encoder and decoder together, it brings further benefits. Following your suggestion, the revised abstract now states that most of the gain comes from the decoder. We hope this answers your remaining concerns, and we would be grateful if you took it into account in your final assessment. (2/2)

Thank you for the follow-up, but I still find it hard to disentangle the effect of decoding between image codes from the one of the DINO losses. I have, however, raised the soundness score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.