- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper addresses a mismatch issue in latent diffusion: a generator outputs codes that lie between training-image codes, but the tokenizer's decoder is trained only on training codes (or noisy versions of them). InterpTok fine-tunes MAETok with an interpolation objective: the decoder learns to decode mixtures of two image codes, using a DINO-feature discriminator and a mixing-ratio loss that stops it from copying one source image. Results: iFID improves from 14.42 to 4.30, the best of 13 tokenizers, and a same-recipe SiT-B generator improves from gFID 11.61 to 8.48. A decoder-only version gets 8.56.
Critical strengths
-
The problem is real, well presented and backed by an independent benchmark study with reproduced numbers (Table 28; iFID errors under 4%).
-
The results are strong and consistent: best iFID of 13 tokenizers with a near 2× margin over the runner-up (RAE, 7.68), and gains that hold across four feature spaces (Table 5).
-
The ablations are useful making the reporting complete. Interpolation alone hurts the linear probe (73.0→63.5) and gFID (18.90); regularization alone doesn’t move iFID; only the combination works (Table 3). The paper reports the failures instead of hiding them. Hyperparameters, loss weights, and training cost are all given in the appendix, and code and weights are also promised.
Critical weaknesses
-
The headline metric is close to the training objective. The interpolation loss trains the decoder on midpoints of neighbouring training codes. iFID measures decoded midpoints between a validation code and a nearest training code. The interpolation-only version, which uses no Inception features at all, still reaches iFID 5.44 (Table 3). Part of the gain may be direct optimization of what is measured.
-
Generation is scored by FID only. In the fewer-steps analysis, samples move toward class means (within-class similarity 0.43→0.56; class accuracy 61% --> 83%, Table 23), and FID rewards that even if coverage drops. Precision and recall are reported for decoded mixtures (Table 12), not for generated samples.
-
The cost is worth noting. The fine-tune takes about 576 GPU-hours to the 41k checkpoint and 1,152 GPU-hours for the full run. This is a tradeoff to the claimed savings.
Questions for authors
-
Does the interpolation objective help on a second continuous 1D tokenizer (e.g., SoftVQ-VAE or l-DeTok)? If this has not been tried, should the "training principle" claim be narrowed?
-
At matched total cost — your fine-tune plus generator training, against the baseline — does the advantage hold?
Thank you for the detailed rebuttal and the many genuinely interesting new experiments. It has addressed several of my concerns. The Frechet loss is clearly not the main driver behind the gFID gains, and the reconstruction numbers rule out the illustrative extreme scenario that the encoder only keeps class information.
That said, I think that the new results mostly just confirm my concerns. With CFG, the full InterpTok shows no improvement over MAETok. Only the decoder fine-tuned on MAETok's frozen encoder improves, and it does worse than MAETok's decoder at the highest guidance scales. In t2i, InterpTok has a lower CLIP score than MAETok at every guidance scale, at guidance 4 its FID is also slightly worse, and there the decoder-only variant is the best on both. The ablations also suggest that the DINOv3 feature mixing-ratio loss is the most important component, especially on SiT-XL, while source conditioning and the pixel mixing-ratio loss barely matter for gFID. And on SiT-B, a decoder trained with noise gets most of the gFID gains while its iFID is no better than fine-tuning with MAETok's own losses, which makes me question whether better decoding between codes is really what drives the improvement.
Overall it seems that the decoder fine-tuning results are quite good, but the paper's main claims are still built on results w/o cfg and on training the full tokenizer with all of the proposed techniques, and imo the evidence doesn't support that framing. Given the thorough rebuttal, I'll raise the soundness but lean towards keeping my rating. I would suggest that the authors try to distill the main reasons of the performance gains and sharpen the claims and presentation of the paper around that.