- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
DynamicTok is an image tokenizer that combines variable token counts with content-dependent spatial allocation. A Sobel-gradient heatmap determines rectangular image partitions, and encoder/decoder fusion layers connect global latent tokens with their associated local regions. The method supports fixed-grid generation and a two-stage generator that first synthesizes a grayscale image to derive a partition. On ImageNet-256, the reported reconstruction rFID reaches 0.65 with the base model at an average of 64 tokens, and 0.39 at an average of 128 tokens. Fixed-grid 64-token generation achieves gFID 1.40, while two-stage generation with variable counts achieves 1.63.
Critical strengths
-
The hybrid design provides a practical way to combine spatial and token-count flexibility while retaining a conventional latent generator. The internal ablations support the importance of its components: spatially adaptive training/inference improves reconstruction rFID from 0.96 to 0.71, and adding variable counts reaches 0.65 (p. 8, Table 3); encoder/decoder fusion improves rFID from 1.36 without fusion to 0.65 (p. 14, Table 8). These controls make the architectural contribution more persuasive than benchmark comparisons alone.
-
The tokenizer is useful for both reconstruction and generation, with a credible quality/efficiency tradeoff. Dynamic training improves the matched fixed-grid generation baseline from gFID 1.66 to 1.40 with CFG (p. 9, Table 6). The two-stage variant supports multiple budgets: 48 tokens achieve gFID 1.69 at 108.4 ms versus SoftVQ-VAE's 1.78 at 143.6 ms (p. 9, Table 7), with both generation stages included in the reported costs. The paper clearly separates these results from its stronger fixed-grid one-stage result.
Critical weaknesses
-
Key experimental details are missing, limiting reproducibility and assessment of the benchmark comparisons. Section 4.1 (p. 6) and Appendix A (pp. 13–14) describe architectures and loss weights but do not specify training duration, optimizer/learning-rate schedule, or training budgets for the tokenizer and generators. Section 4.1 also omits the FID sample count/reference split and diffusion sampler/step count; the latency protocol names a batch size but no GPU model (p. 8). These omissions matter when attributing improvements over prior systems and interpreting the efficiency claims.
-
The incremental benefit of content-dependent token-count allocation is less established than that of spatial adaptivity and dynamic training. Table 3 (p. 8) changes both the training count distribution and the assignment policy when moving from 64 tokens to [32,96], without a matched random-allocation control. In Table 7 (p. 9), variable-count generation achieves almost the same gFID as fixed 64-token generation (1.63 versus 1.64) but has higher latency (217.1 versus 153.6 ms). This does not invalidate flexible budget control, but leaves the practical advantage of the automatic per-image allocation policy insufficiently demonstrated.
Questions for authors
-
Clarification: Please provide the tokenizer/generator training schedules and compute budgets, the FID evaluation sample count/reference split, and the sampler/step count and GPU used for Table 7. Were the internal uniform baselines trained and evaluated with the same schedules and sampling settings?
-
Clarification/targeted control: How much does gradient-based token-count assignment improve over randomly assigning the same count distribution while retaining gradient-based spatial partitions? For Table 7, how are variable-length batches processed, and what explains the latency increase from 153.6 to 217.1 ms despite similar average counts and FLOPs?
3 comments
Reponse to Reviewer PFgQ
W2: Targeted Control on Token Allocation
To isolate the incremental benefit of gradient-based token-count allocation, we conducted an ablation study on image reconstruction. All settings below use a variable token count of [32, 96] and retain gradient-based spatial partitions; the only difference is whether the number of tokens is assigned based on gradients or randomly. Random token-count assignment preserves the same token-count distribution as gradient-based assignment. As shown in the table, using gradient-based allocation during both training and inference yields the best performance (rFID 0.65). Furthermore, regardless of the training mode, switching from random to gradient-based allocation during inference consistently improves reconstruction quality (e.g., improving from 0.74 to 0.65, and from 0.73 to 0.70).
Regarding image generation, we hypothesize that the marginal gain in gFID is limited by the first-stage generator. In reconstruction, the token count is deterministically derived from the ground-truth image complexity. In generation, however, the first-stage model must predict the token count, introducing a prediction variance that may dilute the benefits of content-dependent allocation. We will attribute the primary contribution of dynamic training to enabling flexible spatial and computational budget control, rather than content-dependent generation quality in the revised manuscript.
| Training Mode | Inference Mode | rFID |
|---|---|---|
| Gradient | Gradient | 0.65 |
| Gradient | Random | 0.74 |
| Random | Gradient | 0.70 |
| Random | Random | 0.73 |
The analysis of variable-length latency can be found in our response to Weakness 1 above.
Q1
For detailed information regarding the training schedules and computational budgets, please refer to our response to Weakness 1 above.
Q2
For the targeted control experiment comparing gradient-based and random token allocation, please refer to our response to Weakness 2 above. Regarding the processing of variable-length batches and the detailed explanation of the latency increase, please see our response to Weakness 1 above.
[1] Dhariwal P, Nichol A. Diffusion models beat gans on image synthesis[J]. Advances in neural information processing systems, 2021, 34: 8780-8794.
I have read the rebuttal and decided to maintain my score.
Reponse to Reviewer PFgQ
W1: Experimental Details
Thank you for pointing this out. We provide the full tokenizer and generator training configurations, hardware specifications, and evaluation protocol below. All internal uniform baselines use the same training schedules and sampling settings.
Tokenizer & Generator Training Configuration
Hardware, Latency and Evaluation Protocol
Table 7 was measured on a single NVIDIA H100 80GB HBM3 GPU with batch size 256. For latency computation, we used 3 warmup runs and 5 timed runs.
Sampling uses the Euler–Maruyama SDE solver with 250 steps and CFG. For FID evaluation, we generate 50,000 images and use the ADM FID-50K reference .npz file with the official ADM evaluation suite [1].
Higher latency of variable-length batches
When handling variable token counts, we currently pad all sequences to the maximum length of 96 tokens, which explains the increased latency. The reported FLOPs are calculated based on a single model evaluation rather than the full denoising trajectory; for the [32, 96] setting, the reported 32.3 GFLOPs is the theoretical average based on valid token counts and excludes padding overhead, including in the padded row. Below, we provide a comparison of the averaged inference time (without padding overhead) between the [32, 96] setting and the fixed 64-token setting. As shown, their FLOPs and actual latencies are highly comparable. In real-world deployment, service providers can implement length-based grouping (i.e., batching sequences with the same token count) or packed attention to eliminate padding overhead and effectively reduce computation.