- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
CoFiTok decomposes a pixel-space diffusion noise prediction into an ordered sequence of additive components synthesized from continuous tokens. Bias-free synthesis ensures that a zero token produces an exact zero component. Prefix and component losses supervise cumulative predictions along a prescribed coarse-to-fine path. Experiments report large reductions in path AUC, small endpoint degradation, and mixed generation-quality results.
Critical strengths
- The factorization is mathematically coherent. Bias-free synthesis guarantees , and the pixel-space/noise-space error relation is correct. Reordering already computed components preserves the final endpoint exactly, enabling a clean test of accumulation order.
- The controlled experiments are carefully constructed. Endpoint-only models use the same architecture, the prefix and component losses are ablated separately, and the trained four-component order ranks first among all 24 permutations for both tested seeds.
Critical weaknesses
- The central result primarily demonstrates compliance with a prescribed training target rather than independently useful coarse-to-fine structure. CoFiTok directly optimizes errors against the path and is then evaluated using path AUC against that same path. Component supervision explicitly assigns ordered target increments, so the permutation experiment confirms that the model learned the imposed order. No independent semantic, spectral, perceptual, or downstream criterion establishes that this path is naturally coarse-to-fine or useful.
- No practical advantage of using prefixes is demonstrated. Endpoint reconstruction slightly worsens in the repeated studies, generation quality usually favors dense or existing diffusion baselines, and no latency, FLOP, or quality-versus-prefix-length result is reported. The experiments use very small models and short training budgets. Appendix S7 also states that the target interpolation/filter, loss reductions, token dimensions, predictor architecture, optimizer, noise schedule, and preprocessing are absent, preventing reproduction.
Questions for authors
- What is the exact definition of the target constructor , including filtering, interpolation, normalization, timestep dependence, and loss reductions?
- Can the authors validate the ordering using an independent criterion not used during training, such as frequency content, semantic structure, human judgments, or a downstream prefix-use task?
- Does using a shorter prefix produce a meaningful compute-quality trade-off? Please report actual FLOPs and latency relative to a parameter- and compute-matched dense or additive multi-head baseline.
- Do the results persist at realistic model and training scale with standard 50,000-sample generation evaluation?
5 comments
Response to Reviewer vgSH (2/2)
3. What compute–quality operating points do prefixes provide against a matched control? (W2, Q3)
CoFiTok couples an executable prefix budget to a supervised intermediate-denoising trajectory. The reference is the same-architecture endpoint-only additive multi-head model: both models have 81,808 parameters, a full-prediction core cost of 0.6231G FLOPs, and a 20K-step training budget, which fixes model capacity and full-budget arithmetic for the comparison.
The sampler uses at every step of DDIM-50. The early-exit computation evaluates the shared trunk and the heads, feedback and synthesis needed for m; its output is identical to the corresponding prefix of the full forward pass (maximum absolute difference zero). For the same CoFiTok checkpoint, m=1 reduces counted core arithmetic by 15.3% relative to m=8. The operating points are:
| Method / budget | Core FLOPs/forward (G) | Forward latency (ms) | Tiny ImageNet FID ↓ | ImageNet-64 HF FID ↓ |
|---|---|---|---|---|
| CoFiTok, m=1 | 0.5276 | 4.32 | 408.32 ± 27.59 | 401.43 ± 23.49 |
| CoFiTok, m=2 | 0.5412 | 5.28 | 411.95 ± 34.14 | 417.36 ± 16.30 |
| CoFiTok, m=4 | 0.5685 | 6.42 | 361.22 ± 30.33 | 359.31 ± 2.43 |
| CoFiTok, m=6 | 0.5958 | 6.37 | 363.43 ± 40.73 | 383.78 ± 88.17 |
| CoFiTok, m=8 | 0.6231 | 8.03 | 271.45 ± 5.86 | 273.09 ± 10.07 |
| Endpoint-only multi-head, m=8 | 0.6231 | 8.01 | 268.05 ± 30.68 | 255.10 ± 5.74 |
All generation rows use DDIM-50, identical per-index initial noise and 1,024 samples per configuration; FID is mean ± sample SD over training seeds 103/139. Compute and timing are measured on the Tiny ImageNet seed-139 checkpoints. Latencies are median single-forward times in FP32 at batch size 1 on an RTX 4090, from five synchronized runs of ten forward passes after five warm-up passes. FLOP counts cover convolutions, linear layers, attention products and fixed projections, with a multiply–add counted as two FLOPs.
The endpoint-only control supervises only the final prediction, so it offers a single full-budget operating point. CoFiTok additionally supervises the intermediate trajectory, so each m in its curve executes a defined partial prediction with measured arithmetic, latency and generation quality. Within this curve, m=8 gives the lowest mean FID on both datasets and m=1 gives the lowest core cost.
4. Do the results persist at scale under 50,000-sample generation evaluation? (W2, Q4)
At approximately 62.8M parameters and 100K training steps, CoFiTok achieves FID 112.53 versus 120.00 for its dense control, a 6.23% reduction, in a 50K-sample ImageNet-256 evaluation. Both models train on the full dataset with training seed 2027. CoFiTok and dense have 62,834,083 and 62,824,707 parameters, with full-prediction core costs of 571.024G and 570.439G FLOPs per denoiser forward. This closely matches capacity and counted forward arithmetic.
The generation comparison fixes EMA weights, full prediction budgets, DDIM-100, CFG=1.5, and identical initial noise and class labels for each sample index. Each model generates 50,000 images, evaluated against the same 50,000 validation images using the Inception-v3 features of torch-fidelity 0.4.0:
| Model | FID ↓ | IS ↑ | KID ↓ | Precision ↑ | Recall ↑ |
|---|---|---|---|---|---|
| CoFiTok | 112.53 | 9.874 ± 0.136 | 0.07521 ± 0.00189 | 0.0839 | 0.3800 |
| Dense | 120.00 | 9.298 ± 0.111 | 0.07730 ± 0.00229 | 0.0937 | 0.3661 |
IS reports SD over 10 image splits; KID reports SD over 50 subsets of 1,000 features. These describe evaluator split and subset variability within each fixed checkpoint.
CoFiTok combines lower FID and KID with higher IS and recall in this matched pair. The precision–recall pairs characterize each model's coverage–fidelity operating point; CoFiTok's lower FID accompanies higher measured feature-space coverage.
Dear Reviewer,
The authors have provided their responses to the raised concerns and revisions. Please review their responses and join the discussion before the end of the author-review discussion phase (Nov 18, AoE). Thanks.
Best, AC
Response to Reviewer vgSH (1/2)
We thank Reviewer vgSH for the constructive questions. Below we specify the target constructor and reference configuration (§1), report an independent Fourier-based ordering criterion (§2), present prefix operating points against a matched control (§3), and report a 50K-sample ImageNet-256 evaluation (§4).
1. What are the exact target constructor, loss reductions and reference protocol? (W2, Q1)
The box-pool target constructor is defined by spatial filtering, a power-law progression and its noise-space increments. Clean RGB images are normalized to [-1,1], and xt=αtx0+σtϵ, with αt=αˉt and σt=1−αˉt. For m<K, set
B_m(x_0)=\operatorname{Resize}_{H,W}\bigl(\operatorname{AvgPool}_{s_m,s_m}(x_0)\bigr),$$ and set $B_K(x_0)=x_0$. Pooling uses non-overlapping $s_m\times s_m$ windows and retains partial windows at the image border; resizing is bilinear with half-pixel-centred sampling. The constructor is $$Q(x_0,x_t,t,m)=q_m=(1-p_m)\frac{x_t}{\alpha_t}+p_mB_m(x_0), \qquad p_m=(m/K)^{1.5}.$$ Its exact endpoints are $q_0=x_t/\alpha_t$ and $q_K=x_0$. Targets retain their unclipped values in normalized image coordinates. The spatial filters and progress function are fixed across timesteps; timestep dependence enters through $x_t/\alpha_t$ and the corresponding noise-space conversion: $$e_m^*=\frac{x_t-\alpha_tq_m}{\sigma_t} =p_m\frac{x_t-\alpha_tB_m(x_0)}{\sigma_t},\qquad c_m^*=e_m^*-e_{m-1}^*,\quad e_0^*=0.$$ With $c_m=S_m(z^{(m)})$ and $E_m=\sum_{k\leq m}c_k$, the objective is $$\mathcal L=\operatorname{MSE}(E_K,\epsilon) +\frac{0.15}{K}\sum_{m=1}^K\operatorname{MSE}(E_m,e_m^*) +\frac{0.30}{K}\sum_{m=1}^K\operatorname{MSE}(c_m,c_m^*).$$ Each MSE averages over batch, channel, height and width. Both path losses average all K indices, including m=K. The reference configuration for the 64×64, K = 8 experiments on Tiny ImageNet and ImageNet-64 HF is: | Item | Reference setting | | --- | --- | | Tokens | Eight 16×64×64 token maps with 4, 4, 8, 8, 12, 12, 16 and 16 active channels | | Predictor | Small convolutional network; width 48; three trunk blocks; with token feedback | | Synthesis | Token-only, bias-free 1×1 then 3×3 convolutions; learned scalar per component | | Optimizer | AdamW; learning rate 2×10⁻⁴; weight decay 10⁻⁴; β = (0.9, 0.999); gradient clipping at 1 | | Training | Batch size 32; 20K steps; seeds 103/139 | | Diffusion | 1,000 steps; linear β schedule from 10⁻⁴ to 0.02 | | Preprocessing | RGB resize to 64×64; per-channel normalization with mean 0.5 and SD 0.5 | ### 2. Can an independent frequency criterion validate the ordering? (W1, Q2) An independent Fourier-ranking criterion gives **positive ordering gains in all 12 matched comparisons at t=50/250/500**. Its inputs are the predicted component spectra and component indices. Training uses endpoint, prefix and component MSE; the spectral ranking statistic is evaluated only on held-out predictions. For image i and component k, let $P_{i,k}(\omega)$ be channel-averaged power from an orthonormal Fourier transform. Its radial frequency centroid is $$\mu_{i,k}=\frac{\sum_{\omega}\|\omega\|_2 P_{i,k}(\omega)} {\sum_{\omega}P_{i,k}(\omega)},\qquad r_i=\operatorname{Spearman}\bigl((1,\ldots,K),(\mu_{i,1},\ldots,\mu_{i,K})\bigr).$$ Radial frequency is in cycles per pixel, and the denominator is floored at $10^{-20}$. The condition-level statistic is $\rho=N^{-1}\sum_i r_i$, directly quantifying the association between component position and frequency centroid. The matched 20K-step K = 8 models are compared on Tiny ImageNet and ImageNet-64 HF using seeds 103/139 and 1,024 class-stratified held-out images per condition, with paired images and noise. At t=500: | Dataset | Seed | CoFiTok ρ | Endpoint-only ρ | Paired gain [95% CI] | | --- | ---: | ---: | ---: | ---: | | Tiny ImageNet | 103 | 0.588 | 0.525 | +0.063 [0.059, 0.067] | | Tiny ImageNet | 139 | 0.793 | 0.500 | +0.293 [0.289, 0.296] | | ImageNet-64 HF | 103 | 0.685 | 0.621 | +0.064 [0.061, 0.068] | | ImageNet-64 HF | 139 | 0.869 | 0.524 | +0.345 [0.342, 0.347] | Gains are mean paired per-image correlation differences, CoFiTok minus endpoint-only. Each interval uses 1,000 paired image-bootstrap resamples conditional on the checkpoint pair. The same positive-gain result holds across all 12 conditions at t=50/250/500, with each paired 95% interval above zero. Across the full five-timestep grid t=50/250/500/750/950, CoFiTok's mean correlation is positive in all 20 dataset/seed/timestep conditions, ranging from 0.248 to 0.947. This provides a direct spectral characterization of the components, independent of the training target, together with paired evidence that path supervision strengthens frequency ordering.