- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Segmentation Measure Matching (SMM). The data are first normalized by marginal CDF to [0,1]^D. Each coordinate is then represented by its cardinality-K numeric path through T-layers subdivided recursively by K-elements. A transformer with a custom interleaved full/coarse labeled attention mask (KP3Attention, Figure 2, p.6) can autoregressively predict the classification distribution over the K^P joint subunits of each patch over patches and depths. Training uses cross-entropy with uniform sampling depth (Equation 4-5, page 6). The paper expresses this as the exact maximum likelihood of a finite-resolution discretization. Temperature, logit space CFG and deeper parallel refinement variants are described in Appendices A.2 and A.9.
Theoretical results appear in Appendices B–H (pp. 21–30). Proposition 1 shows that for a fixed symbol budget, the midpoint grid is minimax optimal in ℓ∞ and strictly outperforms the shared, unscaled, and decoded RQ. Theorem 2 gives the multi-scale W1 bounds for each level of condition KL. Theorem 3 restricts the factorization KL of parallel refinements of smooth positive logarithmic densities. Appendix H gives cut-through gradient difference bounds.
The experiments cover four settings. The first was a seven-toy distribution (Table 1, page 8; Table 4, page 16). The second is the D3IL robot imitation, where SMM reports the best success in the 8/9 setting (Table 2, page 8) and the highest entropy in 3/9 (Table 5, page 17). The third is pixel-space generation on a small 28×28 dataset and CIFAR-10, scored using PCA-FID (Table 3, page 9; Table 6, page 18). The fourth is the ImageNet 256 latent generation, where SMM-B achieves FID 13.17, flow matching 15.48, CFG 8.88, and SMM-XL 4.30 (Table 7-8, pp. 18-19). Ablation covers parallel refinement (Table 12-13, page 21) and reconstruction depth scaling (Figure 6-7, page 22-23).
Critical strengths
The formula is simple and coherent. Representing distributions by conditional cell mass ratios on nested K-ary partitions provides tractable exact likelihoods at finite resolution, log-space temperature and CFG, and fixed K^P head size and accuracy as K^T grows (Section 3, pp. 4-6). Toy evidence supports the exciting claim that this contributes to non-smooth support. On the Sierpinski target, SMM obtained recall 0.998/0.996 and coverage 0.906/0.909, while traffic matching was 0.909/0.920 and 0.670/0.641 (Table 1, page 8). Figure 4 (page 16) shows that at NFE=1000 the flow match is still filling the fractal hole.
The empirical scope is broad and some of the analyzes rich. A deep scaling study (Figure 6-7, pp. 22-23) shows that the per-level error rate of SMM remains around 1/K. RQ instead becomes saturated and its effective code usage drops significantly. Parallel refinement ablation (Table 12, page 21) shows that classification refinement avoids the mean collapse of MSE regression: FID 0.407 vs. 4.491 (with one AR level), with 7.7 times fewer consecutive steps. D3IL results are competitive in all nine settings. An example is Sort-4, where the optimal baseline is 64.4 vs. 56.5 (Table 2, page 8).
Critical weaknesses
Its novelty and theoretical contributions have been overstated relative to its content and related work. Digital or bit-plane extended autoregressive modeling with coarse-to-fine value accuracy, closely related to subscale/bit-depth upscaling autoregressive image models and discrete likelihood models. Section 2.2 (pages 3-4) does not position SMM against these; it only mentions Pólya trees and histograms. Proposition 1 (p.24) is the standard fact that for M points a uniform grid is ℓ∞-minimax. "Strict dominance" only applies to shared, unscaled, and decoded RQs. Theorem 3 (pp. 25-26) requires a strictly positive C^2 log density, which rules out the fractal, sparse, and disconnected goals that drive this paper (Abstract, p. 1).
The comparison of several titles is difficult to explain. Pixel-space results using PCA-FID have a value of 0.163 on CIFAR-10 (Table 3, page 9). These are not comparable to the literature and although there were 3 seeds, no seed differences were reported (A.6, p. 17). On ImageNet, flow matching obtained 15.48 without CFG but 15.269 with CFG 3.0 (Tables 7, 9), indicating insufficient baseline tuning. MAR 19.37 is a reimplementation of unclear configuration. SMM-XL's 4.30 (Table 8) was not compared with LlamaGen, despite sharing its marker recipe and backbone (A.7, page 18). The D3IL baseline was copied from Jia et al. instead of rerunning (p.7), and 6/9 tasks had lower entropy (Table 5).
Questions for authors
How to adjust flow matching baseline (sampler, NFE, CFG schedule, training length) on ImageNet? Why does CFG 3.0 barely change its FID (15.48 vs. 15.269)? How does SMM-B/L/XL compare to the released LlamaGen-B/L/XL given the same tokenizer and training budget? Are the RQ-VAE, VQ-VAE, FSQ-VAE, and MAR rows in Table 7 trained with matching backbones and epochs?
Can you report the standard (initial) FID for CIFAR-10 or other metric comparable to the literature, and the standard deviation for the three seeds? Do the pixel-space conclusions in Tables 3 and 6 hold under these metrics?
How do SMMs conceptually and empirically differ from previous autoregressive models that decompose pixel values into bit planes or coarse-to-fine value numbers? Is it possible to include such baselines, such as PixelCNN or Transformer style models, with 256-way classification per pixel under matching parameters?
For D3IL, were the 6 seeds of the SMM trained and evaluated on the launch count and initial state distribution of the baseline? The very small standard deviation (e.g., 0.2 for "Avoid") contrasts with published baselines. Can you explain why behavioral entropy drops significantly on Pushing (0.612 vs. 0.871) and Stack-1/2?
Response to Reviewer wESs
We thank the reviewer for their time and valuable feedback. Please find the responses to the raised concerns below:
1. Related work (W1 & Q3)
Thank you for raising this point. We clarify the differences between SMM and related coarse-to-fine and discrete likelihood models below:
2. Prop. 1 and comparison with RQ (W1)
We agree that the strict advantage over RQ is established for the worst case. We consider the worst case to establish a theoretical bound. Otherwise, a theoretical bound cannot be established, and the comparison has to be made empirically. The empirical results are available in Fig. 6 and Fig. 7 (Appendix A.10), where SMM's reconstruction error keeps decreasing with depth while RQ's plateaus.
3. Comparison with 256-way classification per pixel (Q3)
A flat softmax head grows exponentially with the number of channels. For example, with our ImageNet configuration (K = 4, T = 2, P = 7), a flat head requires KPT=414≈2.7×108 classes per patch, making it not practically applicable.
Thank you for the suggestion to compare with the coarse-to-fine model. We train SPN [R1] on MNIST and KMNIST and evaluate it under the same setup:
4. Thm. 3 and fractal targets (W1)
We agree that Thm. 3 does not apply to fractals. It applies to high-dimensional images, where parallel refinement is used (Tab. 12 and Tab. 13).
5. Baselines in Tab. 7 and comparison with LlamaGen (W2 & Q1)
Following LlamaGen [R6], we use the VQ-8 version (however we change latent channel size from 8 to 7). We could not use the default 8-channel latent version due to the exponential explosion of the softmax number of choices. In Tab. 7, VQ-VAE uses LlamaGen setup (7-dim VQ-8 version) and is trained with the same protocol as SMM (50 epochs).
6. D3IL baselines and behavior entropy (W2 & Q4)
We retrain BESO and DDPM-ACT, and add VQ-BeT [R5]. Success rates (%), six-seed means:
The retrained baselines are comparable to the published numbers on most tasks. SMM outperforms the retrained baselines and VQ-BeT on all nine tasks.
Regarding the lower behavior entropy on Pushing and Stack-1/2, our policy produces more predictable paths.
[R1] Menick, J., & Kalchbrenner, N. (2019). Generating high fidelity images with subscale pixel networks and multidimensional upscaling. In International Conference on Learning Representations.
[R2] Han, J., Liu, J., Jiang, Y., Yan, B., Zhang, Y., Yuan, Z., Peng, B., & Liu, X. (2025). Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 15733-15744).
[R3] Morin, F., & Bengio, Y. (2005). Hierarchical probabilistic neural network language model. In International Workshop on Artificial Intelligence and Statistics (pp. 246-252). PMLR.
[R4] Li, T., Sun, Q., Fan, L., & He, K. (2025). Fractal generative models. Transactions on Machine Learning Research.
[R5] Lee, S., Wang, Y., Etukuru, H., Kim, H. J., Shafiullah, N. M. M., & Pinto, L. (2024). Behavior generation with latent actions. In International Conference on Machine Learning (pp. 26991-27008). PMLR.
[R6] Sun, P., Jiang, Y., Chen, S., Zhang, S., Peng, B., Luo, P., & Yuan, Z. (2024). Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06525.